I am using td-agent v3 on AWS and have noticed logs getting stuck when using the exec_filter plugin. If I compare the # of logs that are stored in the buffer file to the # of logs that get printed to stdout in my second match block, I am able to confirm that there are logs missing. However, if I restart the td-agent service, the missing logs appear.
I have confirmed that the python script is not filtering/excluding any records. I have also confirmed that the buffer section is working properly. If I replace exec_filter with stdout, I am able to see all of the log records get printed after the buffer flushes.
What are the possible causes?
Below is my configuration.
<system> log_level debug log_event_verbose 1 </system> <source> @type http @id input_http port 8888 <parse> @type multi_format <pattern> format json </pattern> <pattern> format none </pattern> </parse> @log_level debug </source> <match original_tag> @type exec_filter tag new_tag command python3 transform.py <format> @type json </format> <parse> @type json </parse> <buffer> @type file path /var/log/buffer flush_interval 1m chunk_limit_size 256M </buffer> @log_level debug </match> <match new_tag> @type stdout </match>