Failed to purge buffer chunk chunk_id

Hi,

In my deployment with three pods of fluentd, after an upgrade fluentd pods are in CrashLoopbackoff state and I see the below logs at the erroneous pods:

2021-08-20 13:05:50 +0000 [error]: #3 failed to purge buffer chunk chunk_id="5c9fd4dbcb0e7162a8574b70a5f18ac8" error_class=Errno::ENOENT error=#<Errno::ENOENT: No such file or directory @ unlink_internal - /data/fluentdlogs/logselk/worker3/buffer.b5c9fd4dbcb0e7162a8574b70a5f18ac8.log.meta>
  2021-08-20 13:05:50 +0000 [error]: #3 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/buffer/file_chunk.rb:161:in `unlink'
  2021-08-20 13:05:50 +0000 [error]: #3 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/buffer/file_chunk.rb:161:in `purge'
  2021-08-20 13:05:50 +0000 [error]: #3 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/buffer.rb:525:in `block in purge_chunk'
  2021-08-20 13:05:50 +0000 [error]: #3 /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
  2021-08-20 13:05:50 +0000 [error]: #3 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/buffer.rb:516:in `purge_chunk'
  2021-08-20 13:05:50 +0000 [error]: #3 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/output.rb:1018:in `commit_write'
2021-08-20 13:05:50 +0000 [info]: #3 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
  2021-08-20 13:05:50 +0000 [error]: #3 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/output.rb:1137:in `try_flush'
  2021-08-20 13:05:50 +0000 [error]: #3 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/output.rb:1439:in `flush_thread_run'
  2021-08-20 13:05:50 +0000 [error]: #3 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2021-08-20 13:05:50 +0000 [error]: #3 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2021-08-20 13:05:50 +0000 [info]: #1 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-08-20 13:05:51 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-08-20 13:05:54 +0000 [info]: #0 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.
2021-08-20 13:05:58 +0000 [warn]: #2 no patterns matched tag=nil
2021-08-20 13:05:58 +0000 [error]: #2 failed to purge buffer chunk chunk_id="5c9fd4c1abd65dc0c3e754a6ae208087" error_class=Errno::ENOENT error=#<Errno::ENOENT: No such file or directory @ unlink_internal - /data/fluentdlogs/logs/worker2/buffer.q5c9fd4c1abd65dc0c3e754a6ae208087.log.meta>
  2021-08-20 13:05:58 +0000 [error]: #2 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/buffer/file_chunk.rb:161:in `unlink'
  2021-08-20 13:05:58 +0000 [error]: #2 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/buffer/file_chunk.rb:161:in `purge'
  2021-08-20 13:05:58 +0000 [error]: #2 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/buffer.rb:525:in `block in purge_chunk'
  2021-08-20 13:05:58 +0000 [error]: #2 /opt/td-agent/embedded/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
  2021-08-20 13:05:58 +0000 [error]: #2 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/buffer.rb:516:in `purge_chunk'
  2021-08-20 13:05:58 +0000 [error]: #2 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/output.rb:1018:in `commit_write'
  2021-08-20 13:05:58 +0000 [error]: #2 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/output.rb:1137:in `try_flush'
  2021-08-20 13:05:58 +0000 [error]: #2 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/output.rb:1439:in `flush_thread_run'
  2021-08-20 13:05:58 +0000 [error]: #2 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2021-08-20 13:05:58 +0000 [error]: #2 /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.11.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2021-08-20 13:06:22 +0000 [info]: #3 disable filter chain optimization because [Fluent::Plugin::RecordTransformerFilter] uses `#filter_stream` method.

Can “not finding the chunk to delete” to lead the pod to be in Crashloogbackoff and restart of the pod?

Thank you in advance!

It looks like there is an issue in purging as Fluentd cannot find the chunks associated anymore - If you are using a buffer queue that is storage based on Kubernetes you most likely want to associate a persistent volume otherwise when upgrading / performing a change you could end up on a different machine where the buffer chunks were originally created.

Thank you for the prompt response.
So this is the reason of pod be in CrashLoopbackoff?
Is there any documentantion regarding the flow of fluentd in case of restart/upgrade?
Thank you in advance