Out_exec_filter EFBIG (File too large) once pipe reach 0x7fffffff

We use fluentd out_exec_filter,
This issue happens after we upgrade the ruby 2.7.1p83 and fluentd 1.11.2
We can see it is creating a pipe,
fluentd (write end)-> pipe → (read end) ruby scripts
if we use the buffer type file, we could see the pipe keep increasing as per lsof, and once it reaches 0x7fffffff we are getting EFBIG (File too large).
COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ruby 46343 testuser 15w FIFO 0,9 0x7fffffff 258961662 pipe
flush_thr 46343 46348 testuser 15w FIFO 0,9 0x7fffffff 258961662 pipe{code}

If we use buffer type memory , the issue is not happening.

Below listing the config
fluentd.conf
<match test.log>
@type exec_filter
command "ruby /var/tmp/testscript.rb"
tag filtered.exec
<format>
@type json
</format>
<parse>
@type msgpack
</parse>
<buffer>
@type file
path "/var/tmp/test_buffer/test.*.buffer"
flush_interval "1s"
retry_forever true
chunk_limit_size "256m"
queue_limit_length "128"
flush_at_shutdown false
retry_wait "1s"
</buffer>
</match>
<match filtered.exec>
@type file
path /tmp/test_out.log
</match>

and testscript.rb
#!/usr/bin/env ruby
require 'json'
require 'msgpack'
begin
while line = STDIN.gets # continue to read a event from stdin
line.chomp!
json = JSON.parse(line)
json['new_field'] = "Hey from exec_filter script!"
STDOUT.print MessagePack.pack(json)
STDOUT.flush
end
rescue Interrupt # Ignore Interrupt exception because it happens during exec_filter shutdown
end

Error Trace:
2021-04-08 12:42:32 +0900 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-04-08 12:42:33.491701254 +0900 chunk="5bf6dd65044478f90a75868a53826b7c" error_class=Errno::EFBIG error="File too large - sendfile"
2021-04-08 12:42:32 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer/chunk.rb:180:in copy_stream'
2021-04-08 12:42:32 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer/chunk.rb:180:in block in write_to'
2021-04-08 12:42:32 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer/file_chunk.rb:171:in open'
2021-04-08 12:42:32 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer/chunk.rb:179:in write_to'
2021-04-08 12:42:32 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/out_exec_filter.rb:278:in write'
2021-04-08 12:42:32 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb:1133:in try_flush'
2021-04-08 12:42:32 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb:1439:in flush_thread_run'
2021-04-08 12:42:32 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb:461:in block (2 levels) in start'
2021-04-08 12:42:32 +0900 [warn]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin_helper/thread.rb:78:in block in thread_create'

https://docs.fluentd.org/output/exec_filter#num_children
https://docs.fluentd.org/output/exec_filter#child_respawn

Though It is not a direct solution, can these parameters mitigate the issue?

Hi @kenhys , Thank you for the information, i have tried with these options, but still the pipe write end is keep growing,
I tried with these options,
child_respawn -1
num_children 2

arsenth+ 101986 101979 0 10:41 pts/0 00:00:00 ruby /var/tmp/testscript.rb
arsenth+ 101989 101979 1 10:41 pts/0 00:00:00 ruby /var/tmp/testscript.rb

the lsof
$ lsof |egrep '297152897|297153844'
ruby 101979 arsenthil01 11w FIFO 0,9 0t0 297153844 pipe
ruby 101979 arsenthil01 13w FIFO 0,9 0t2015 297152897 pipe
flush_thr 101979 101981 arsenthil01 11w FIFO 0,9 0t0 297153844 pipe
flush_thr 101979 101981 arsenthil01 13w FIFO 0,9 0t2015 297152897 pipe