Refresh bearer token in http output plugin without restarting fluentd

I am trying to use fluentd to send some metrics to a datalake using ‘fluent-plugin-out-http’ plugin.
A short lived bearer token is configured in the output plugin while sending data to an HTTP endpoint.

authentication bearer
token "#{ENV['BEARER_TOKEN']}"

The fluentd is running in a kubernetes cluster as a POD. There is another service running in the same kubernetes cluster (say token refresh service) which refreshes the bearer token in every 30 mins and sharing it with fluentd POD. I tried two options here:

Option 1:

  • Token refresh service refreshes the token and writes the new token into a kubernetes secret
  • This kubernetes secret is mapped to a container environment variable ($BEARER_TOKEN) in fluentd POD using ‘env[].valueFrom.secretKeyRef’
    The issue with this approach is, the environment variable will not be updated until the fluentd POD is restarted. I want to avoid restarting fluentd POD in every 30 minutes.

Option 2:

  • The fluentd config file is saved in a kubernetes configmap
  • Token refresh service updates the bearer token in the config file (in configmap) when the token is refreshed
  • This configmap is volume mounted as a file in fluentd POD (Path: /etc/fluent/fluent.conf)
  • Token refresh service invokes fluentd RPC call to reload the configuration after updating the token in config (/api/config.gracefulReload)
    Fluentd documentation says “Fluentd will try to flush the entire memory buffer at once, but will not retry if the flush fails” for RPC call /api/config.gracefulReload. This may cause some data loss. I also found that the fluentd workers and plugins are getting restarted when the config is reloaded. I want to avoid the data loss and restarting workers and plugins in every 30 minutes.

Is there any option in fluentd to refresh bearer token in output plugin with out restarting fluentd, workers or plugins?