Fluentd tail logs duplication in the GCP Operations Suite

I have Java SpringBoot tail logs which are shipped to the GCP Operations Suite via Fluentd. All works fine except that I’m getting the same log duplicated - as formatted and not-formatted.

Config for those logs looks like:

<source>
  @type tail
  path /var/logs/app/*
  pos_file /var/lib/google-fluentd/pos/my-app.tail.log.pos
  tag APP.tail-logs
  format multiline
  format_firstline /^\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{3}/
  format1 /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{3}) (?<severity>[^\s]+) \[(?<thread>.*)\] (?<class>[^\s]+) (?<message>.*)/
  time_format %Y-%m-%d %H:%M:%S,%N
</source>

<match my-app.**>
  @type record_reformer
  tag ${tag_suffix[1]}
</match> 

Based on Fluentular parsing regular expression is correct for what I need.

Any clue why Fluentd is sending it twice when tail log have it once?

Thanks!

PS.

What I have noticed formatted entry has different timestamp than not-formated:

timestamp: "2021-05-26T15:46:39.863Z"
timestamp: "2021-05-26T15:46:39.863605965Z"

Hmm, if you provide the example skipped logs (to reproduce the issue), someone may be able to help.

Good point!

Below is an example

2021-05-26 09:35:47,165 TRACE [http-nio-8106-exec-5] org.springframework.web.servlet.DispatcherServlet No view rendering, null ModelAndView returned.
2021-05-26 09:35:47,165 DEBUG [http-nio-8106-exec-5] org.springframework.web.servlet.FrameworkServlet Completed 200 OK, headers={masked}
2021-05-26 09:35:47,795 DEBUG [http-nio-8106-exec-1] org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter Request is to process authentication
2021-05-26 09:35:47,795 DEBUG [http-nio-8106-exec-1] org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter Request is to process authentication

Two first were not formatted when two last were:

I have updated OP with new findings.