Fluentd + eks: skipping to send logs of short lived jobs

I’m having an issue with fluentd 1.13/1.14 where it’s not forwarding log messages of k8s Jobs to cloudwatch if the job runs just for a few seconds. For long-running jobs it works just fine.

Fluentd config:

<source>
	@type tail
	@label @container
	path /var/log/containers/*.log
	exclude_path ["/var/log/containers/cloudwatch-agent*", "/var/log/containers/fluentd*"]
	pos_file /var/log/fluentd-containers.log.pos
	tag container.*
	read_from_head true
	follow_inodes true
	<parse>
		@type multi_format
		<pattern>
			format cri
			time_format %Y-%m-%dT%H:%M:%S.%NZ
		</pattern>
		<pattern>
			format json
			time_format %Y-%m-%dT%H:%M:%S.%NZ
		</pattern>
	</parse>
</source>

It seems that fluentd is detecting the log files but it’s not forwarding it at all. It’s also a bit weird that it detects log rotation on a file that is so small (5 log lines)

Fluentd Logs:

2021-12-02 18:46:47 +0000 [info]: #0 [Aws::CloudWatchLogs::Client 200 0.020815 0 retries] put_log_events(log_events:[{timestamp:1638470804512,message:"[FILTERED]"}],log_group_name:"/convox/rrr/system",log_stream_name:"service/ingress-nginx/ingress-nginx-788976446-wwbg5",sequence_token:"49605556107201490919304478315024039166400664032086327554")  

2021-12-02 18:46:49 +0000 [info]: #0 following tail of /var/log/containers/timer-test-timer-1638470760-zdjjw_rrr-logtest3_logtest3-34477acc21d8d5f7bf8a40ebee9066f6f1184be99f1f8c10abd1452a4c885b64.log
2021-12-02 18:46:51 +0000 [info]: #0 detected rotation of /var/log/containers/timer-test-timer-1638470760-zdjjw_rrr-logtest3_logtest3-34477acc21d8d5f7bf8a40ebee9066f6f1184be99f1f8c10abd1452a4c885b64.log; waiting 5.0 seconds
2021-12-02 18:47:49 +0000 [info]: #0 following tail of /var/log/containers/timer-test-timer-1638470820-p4sgb_rrr-logtest3_logtest3-42e07d8720bad119c779b29b3a65fdde38b0116f19085f333973bc169064d537.log
2021-12-02 18:47:51 +0000 [info]: #0 detected rotation of /var/log/containers/timer-test-timer-1638470820-p4sgb_rrr-logtest3_logtest3-42e07d8720bad119c779b29b3a65fdde38b0116f19085f333973bc169064d537.log; waiting 5.0 seconds

I tried playing with read_from_head and follow_inodes values but it doesn’t work. if anything it just stops detecting log rotation on the file