Hi FluentD community, my senior wanted to verify that FluentD can handle backpressure in a similar way as to how Logstash (log aggregator) can let Filebeat (log collector/forwarder) know that it’s busy and that it should retry after some time to write logs to Logstash.
I think the buffer section in documentation explains this to some extent but he’s saying that it applies to just regular network disconnects (which I also suspect) and is not a “true” backpressure handling solution. So, my queries to confirm are these:
Does a FluentD/bit aggregator instance let the FluentD/bit forwarding instance (on all app servers) know that “Hey I’m busy, don’t send in logs right now” which would lead to forwarding instance knowing to throttle the tries to send logs? OR If the FluentD aggregator instance is busy and is not able to accept logs, will it just stop listening on the port?
If it does or does not, how and what does the aggregator instance communicate with the forwarding instance? Is there a description of the algorithm that aggregator instances use to let the log forwarder instance know the health/load of the aggregator instance?
Is there any confirmation that the aggregator Fluentd instance sends the forwarding Fluentd/bit instance that the log message it forwarded was successfully received and recorded? How can the forwarding instance be sure that the log was received successfully by the aggregator instance?
If someone can point to the documentation stating the backpressure handling scenario between forwarding and aggregator instances, that would be great. If the above is not documented anywhere, can someone point me to the code in the GitHub repository? (I did not find it myself)