We are setting up fluentd to send logs to elasticsearch.
We set up a buffer to save data in chunks for a short time in case elasticseach hits the limit to make sure we don’t lose data .
A bad chunk is stopping fluentd from sending any data. unexpected error while checking flushed chunks. ignored. error_class=NoMethodError error="undefined method `<' for nil:NilClass"
It is one of the unrecoverable errors that is mentioned in the fluentd docs which fluentd is supposed to detect from v1.2 onwards (we are at 1.4)
On adding secondary buffer, backup directory and changing flush mode from lazy to interval - it is aborting the chunk and sending it straight to backup directory.
Is it supposed to bypass the secondary buffer that way?
I thought the expected behaviour was primary → secondary → backup