Exec_filter # of records does not match buffer file # of records

I am using td-agent v3 on AWS and have noticed logs getting stuck when using the exec_filter plugin. If I compare the # of logs that are stored in the buffer file to the # of logs that get printed to stdout in my second match block, I am able to confirm that there are logs missing. However, if I restart the td-agent service, the missing logs appear.

I have confirmed that the python script is not filtering/excluding any records. I have also confirmed that the buffer section is working properly. If I replace exec_filter with stdout, I am able to see all of the log records get printed after the buffer flushes.

What are the possible causes?

Below is my configuration.

  log_level debug
  log_event_verbose 1


  @type http
  @id input_http
  port 8888
    @type multi_format
      format json
      format none

  @log_level debug

<match original_tag>
  @type exec_filter
  tag new_tag
  command python3 transform.py

    @type json

    @type json

    @type file
    path /var/log/buffer
    flush_interval 1m
    chunk_limit_size 256M

  @log_level debug

<match new_tag>
  @type stdout

Hmm, Did you see error logs similar to Out_exec_filter EFBIG (File too large) once pipe reach 0x7fffffff ?

I don’t get any errors at all which is strange and making it more difficult to troubleshoot.

Is there a minimum size requirement within the exec_filter plugin that I’m not fulfilling?

Additional context, if I send more logs to fluentd, that also makes the missing logs appear.