Fluentd repeatedly attempting to write chunks to loki after recieving a 400 error

Hey there,

We are using both fluent-bit and fluentd as a logging pipeline to ship logs to loki.

Whenever a chunk is sent to loki out of order, however, fluentd continues to retry the log, seemingly indefinitely. Loki sends a “400 Bad Request entry” error, which I would EXPECT fluentd to not retry the chunk, however this does not seem to be the case.

is there a way to have fluentd stop retrying a chunk of it gets a 400 error?

Here is our config for fluentd:

    fileConfigs:
      01_sources.conf: |
        <system>
          workers 6
          log_level info
        </system>
        <source>
          @type forward
          @label @DISPATCH
          bind 0.0.0.0
          port 24224
        </source>
      02_filters.conf: ""
      03_dispatch.conf: |
            <label @DISPATCH>
              <filter **>
                @type prometheus
                <metric>
                  name fluentd_input_status_num_records_total
                  type counter
                  desc The total number of incoming records
                  <labels>
                    tag ${tag}
                    hostname ${hostname}
                  </labels>
                </metric>
              </filter>
              <filter>
                @type record_modifier
                <record>
                  fluentd_worker "#{worker_id}"
                </record>
              </filter>
              <match **>
                @type relabel
                @label @OUTPUT
              </match>
            </label>
      04_outputs.conf: |
        <label @OUTPUT>
          <match **>
            @type loki
            extract_kubernetes_labels true
            url "http://loki:3100"
            username "#{ENV['LOKI_USERNAME']}"
            password "#{ENV['LOKI_PASSWORD']}"
            extra_labels {"parsed":"true"}
            <label>
              fluentd_worker
              container $.kubernetes.container
              namespace $.kubernetes.namespace_name
            </label>
            <label>
                env
            </label>
            <label>
              service
           </label>
           flush_interval 5s
           flush_at_shutdown true
           retry_limit 10
           <buffer time>
             timekey      180
             timekey_wait 300
             chunk_limit_size 60MB
           </buffer>
          </match>
        </label>