How to match complex combinations of texts

Hi, I want to make some more involved combination of matches of messages so I don’t overload the elasticsearch. I have read the manual for grep, rewrite_tag_filter and many other plugins but I just cant wrap my head around how to do it. What i want is:
Let most messages pass, but if the message comes from one specific container in kubernetes, then only some of those messages should be passed along. How should I think about this?

I know you asked this on the Fluentd category but I recommend if you are using Kubernetes that Fluent Bit may have a better getting started experience.

There are a couple concepts we need to emphasize.

  1. Tagging in Kubernetes
  2. Filtering with Fluent Bit

The first piece is that when we read from a file in Kubernetes we use that file name to determine key aspects such as Pod Name, Namespace, and Docker ID which allows us to look up additional metadata from the Kubernetes API Server. By default this filename is part of the tag that Fluent Bit uses to run through its pipeline and filter / redact / etc. For example, a tag could be kube.var.log.containers.apache-logs-annotated_default_apache-aeeccc7a9f00f6e4e066aeff0434cf80621215071f1b20a51e8340aa7c35eac6.log This gives us good information to help with your use case on performing actions based on specific apps. Documentation also inserted below

For the second piece on filtering, every event in Fluent Bit has a tag value and we can use this tag value to exclude / include certain events. For example, if we wanted to exclude any apache pods we could use the following grep filter

[FILTER]
    name grep
    match kube.var.log.containers.apache*
    exclude log .*

All the remaining events would continue to flow to their intended end destinations, here’s a visualization of that pipeline from Calyptia Cloud (Sign-up required)