How to add autoincrement field to record?

I have some log files (lets say *.log) and I need to add an autoincrement field to each record.

For instance, when reading ABC.LOG and CDE.LOG, if they contains 2 records each, I need to output:

  • the parsed log record
  • the autoincrement field 1,2 for each log_file

Here below my configuration:

@type tail

path “#{ENV[‘LOG_PATH’]}/*.log”

path_key tailed_path

pos_file “#{ENV[‘FLUENTD_POS_PATH’]}/singleline.pos”

tag logs.singleline

refresh_interval 5s

read_from_head true

Parse single line

@type regexp

expression /(?<log_timestamp>\d{4}\/\d{1,2}\/\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}:\d{1,3})\s(?<message>.*)/

@type record_transformer


severity "INFO"

task "ATSB"

log_timestamp ${record.dig("log_timestamp").gsub(/.*\K:/, '.').gsub(/\s/, 'T').gsub(/\//, '-') << "Z"}

log_file ${record.dig("tailed_path").split("/").last}

tag ${tag}

<match logs.*>

@type kinesis_firehose

aws_key_id “#{ENV[‘AWS_ACCESSKEY’]}”

aws_sec_key “#{ENV[‘AWS_SECRETKEY’]}”

delivery_stream_name “#{ENV[‘AWS_KNS_LOG_NAME’]}”

region “#{ENV[‘AWS_REGION’]}”

slow_flush_log_threshold 30.0

time_key @timestamp
@type file

path "#{ENV['FLUENTD_POS_PATH']}/buffer.logs"

retry_max_interval 30

retry_forever true

flush_thread_count 8

Can anyone help me ?