Fluentd Performance degradation after upgrading to td-agent(fluentd) - 4.1.0 (1.12.1)

Hi all, my problem is that after upgrading fluentd from td-agent(fluentd) - 3.8.0 (1.11.1) to td-agent(fluentd) - 4.1.0 (1.12.1), its performance was somehow degraded. So for my base load of traffic towards elasticsearch I needed an average of 150m(cpu) to flush data in bulk api of elasticsearch and after upgrade in order to reach the same throughput I had to increase cpu limits to 1000m and the average cpu being used is 800-900m. Not to mention that I had to disable the Multi-workers feature as with that on I couldnt reach even the base load needs.

My fluentd configuration looks like this, can you please help me identify why this difference in cpu needs came after the upgrade and how can I improve it?

  workers 4

  @type http
  @log_level error
  @id input_http_ipv4
  port 9000

<filter {*security_logs*,*debug_logs*,*LI_logs*,*audit_logs*}>
  @type record_transformer
    date ${ require 'date'; DateTime.rfc3339(record["log_event_time_stamp"]).strftime('%Y-%m-%d') }

<filter {*security_logs*,*debug_logs*,*LI_logs*,*audit_logs*}>
  @type elasticsearch_genid
  hash_id_key _hash

<match {security_logs,debug_logs,LI_logs,audit_logs}>
@type copy

<buffer tag, date, facility, vnf_name>
@type file
@log_level error
path /data/fluentdlogs/logs
timekey 1d
flush_thread_count 4
chunk_limit_size 4MB
overflow_action block
flush_mode interval
flush_interval 5s
total_limit_size 5GB

@type elasticsearch
@log_level error
index_name ${tag}-${facility}-${vnf_name}-${date}
type_name logs_data
host elasticsearch
port 9200
id_key _hash # specify same key name which is specified in hash_id_key
remove_keys _hash # Elasticsearch doesn’t like keys that start with _
logstash_format false
bulk_message_request_threshold 5M
request_timeout 30s
reconnect_on_error true
reload_on_failure true
reload_connections false
user xxxxx
password xxxxx
scheme https
ssl_verify true
ssl_version TLSv1_2
#Donot change the certificate path since it is mounted by default in this path.
ca_file /etc/certs/es-root-ca.pem

Perhaps its the change in ruby version? There is a new td-agent 4.2 that I believe has a newer version of ruby, or you could try the calyptia-fluentd package which includes ruby 3.0