I had a question regarding the feature of “throttling bytes per second” in in_tail plugin (PR: Add log throttling per file (revised) by cosmo0920 · Pull Request #3185 · fluent/fluentd · GitHub).
Let’s say you’re reading from a file with throttling enabled. After some time the file rotated, then there will be another tail watcher object for the same file reading the new incoming logs. The original (older) tail watcher object will be attached to a timer_execute function for detaching it from the event loop after waiting for rotate_wait seconds and ensuring that all the logs from that file have been read (EOF reached). (Link: fluentd/in_tail.rb at master · fluent/fluentd · GitHub)
My question is, what if the reading is slow and file rotation rate is so fast that new tail watchers are generated by Fluentd even before the previous tail watchers are closed (ignore the missing log rotations). Let’s say there are n tail watcher objects for a file now. So now, n-1 tw objects will be waiting to be closed (from detach_watcher_after_rotate_wait method) and the last tw object will be reading the incoming log lines.
Will this scenario cause excessive cpu usage and slow down Fluentd? Also, is this scenario even possible in for Fluentd? Please help in clarifying my understanding of in_tail plugin…