Exec_filter # of records does not match buffer file # of records

I am using td-agent v3 on AWS and have noticed logs getting stuck when using the exec_filter plugin. If I compare the # of logs that are stored in the buffer file to the # of logs that get printed to stdout in my second match block, I am able to confirm that there are logs missing. However, if I restart the td-agent service, the missing logs appear.

I have confirmed that the python script is not filtering/excluding any records. I have also confirmed that the buffer section is working properly. If I replace exec_filter with stdout, I am able to see all of the log records get printed after the buffer flushes.

What are the possible causes?

Below is my configuration.

<system>
  log_level debug
  log_event_verbose 1
</system>


<source>

  @type http
  @id input_http
  port 8888
  <parse>
    @type multi_format
    <pattern>
      format json
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>

  @log_level debug
</source>


<match original_tag>
  @type exec_filter
  tag new_tag
  command python3 transform.py

  <format>
    @type json
  </format>

  <parse>
    @type json
  </parse>

  <buffer>
    @type file
    path /var/log/buffer
    flush_interval 1m
    chunk_limit_size 256M
  </buffer>

  @log_level debug
</match>

<match new_tag>
  @type stdout
</match>

Hmm, Did you see error logs similar to Out_exec_filter EFBIG (File too large) once pipe reach 0x7fffffff ?

I don’t get any errors at all which is strange and making it more difficult to troubleshoot.

Is there a minimum size requirement within the exec_filter plugin that I’m not fulfilling?

Additional context, if I send more logs to fluentd, that also makes the missing logs appear.