Logging non-text (A/V) data

Hi! I was wondering if fluentd was designed for logging non-textual data. Say we want to log data from audio / video sources. Do you think fluentd is designed for that kind of thing? Do you see any possible issues with using it for large volumes of data, like HD videos?

From my very limited reading of flutend, it seems like it is mostly used for (or being used for) textual logs.

FYI: Fluentd Help - Fluent Discussions category is better to post

Anyway, generally speaking, yes, Fluentd was designed to handle textual data in most cases.
But the answer depends on what you want to do with audio/video sources.
It means that If there is Fluentd’s input plugin that supports parsing/extract metadata from audio/video sources, you can handle them.

OK. Thanks for pointing it out and thank you for your reply.

We can assume that we’re extracting some metadata from A/V data and logging it.
We’d also want to log other binary blobs which have some metadata.

So, if I understand you correctly, we could do that but we’d have to write a plugin which extracts the metadata from our data source and sends across the blobs.

I guess, fluentbits would be good for this purpose because of its speed.

I think that Fluentd has enough speed of tailing logs, blob data, and so on.
But non-text (A/V) data arrangement shouldn’t shifted back and forth.
Fluentd does not ensure data arrangement.
But it ensures write-at-once if you specify require_ack_response as true on out_forward.

I guess that you should use gstreamer and writing your custom plugin in this case.