Logs Status code 204

Im forwarding logs to loki with below configuration.

<match file-myapp.log> 
 @type loki
 url "http://13.xx.xx.xx:3100"
 #username "#{ENV['LOKI_USERNAME']}"
 #password "#{ENV['LOKI_PASSWORD']}"
 #extra_labels {"env":"dev"}
 flush_interval 10s
 flush_at_shutdown true
 buffer_chunk_limit 1m
 <label>
  fluentd_worker
 </label>
</match>

getting below response in logs :

  • "fluent/log.rb:306:debug: POST request was responded to with status code 204"

  • 2021-04-22 16:45:48 +0000 [warn]: #0 failed to write post to http://13.xx.xx.xx:3100/loki/api/v1/push (400 Bad Request entry with timestamp 2021-04-22 16:45:37.2231572 +0000 UTC ignored, reason: ‘entry out of order’ for stream: {},

It seems that it is caused by unexpected out of ordered timestamp in events.
If you collect logs from multiple nodes, it may better to add labels to distinct it.(I guess)