Fluentd repeatedly attempting to write chunks to loki after recieving a 400 error

Hey there,

We are using both fluent-bit and fluentd as a logging pipeline to ship logs to loki.

Whenever a chunk is sent to loki out of order, however, fluentd continues to retry the log, seemingly indefinitely. Loki sends a “400 Bad Request entry” error, which I would EXPECT fluentd to not retry the chunk, however this does not seem to be the case.

is there a way to have fluentd stop retrying a chunk of it gets a 400 error?

Here is our config for fluentd:

      01_sources.conf: |
          workers 6
          log_level info
          @type forward
          @label @DISPATCH
          port 24224
      02_filters.conf: ""
      03_dispatch.conf: |
            <label @DISPATCH>
              <filter **>
                @type prometheus
                  name fluentd_input_status_num_records_total
                  type counter
                  desc The total number of incoming records
                    tag ${tag}
                    hostname ${hostname}
                @type record_modifier
                  fluentd_worker "#{worker_id}"
              <match **>
                @type relabel
                @label @OUTPUT
      04_outputs.conf: |
        <label @OUTPUT>
          <match **>
            @type loki
            extract_kubernetes_labels true
            url "http://loki:3100"
            username "#{ENV['LOKI_USERNAME']}"
            password "#{ENV['LOKI_PASSWORD']}"
            extra_labels {"parsed":"true"}
              container $.kubernetes.container
              namespace $.kubernetes.namespace_name
           flush_interval 5s
           flush_at_shutdown true
           retry_limit 10
           <buffer time>
             timekey      180
             timekey_wait 300
             chunk_limit_size 60MB