How to configure DaemonSet Fluentd to read from custom log file and console

I’m trying to configure EFK stack in my local minikube setup. I have followed this tutorial Logging for Kubernetes: fluentd and ElasticSearch | MetricFire Blog.

Everything is working fine ( i can see all my console logs in kibana and elasticsearch ). But i have another requirement. I have Node.js application which is logs as files to custom path /var/log/services/dev inside the pod.

File Tree

/var/log/services/dev/# ls -l
total 36
-rw-r--r--    1 root     root         28196 Nov 27 18:09 carts-service-dev.log.2021-11-27T18.1
-rw-r--r--    1 root     root          4483 Nov 27 18:09 carts-service-dev.log.2021-11-27T18

How to configure my fluentd to read all my console logs and also read logs from the custom path configured.
My App Deployment File

apiVersion: apps/v1
kind: Deployment
metadata:
  name: carts
spec:
  replicas: 1
  selector:
    matchLabels:
      app: carts
  template:
    metadata:
      labels:
        app: carts
    spec:
      containers:
        - name: app
          image: carts-service
          resources:
            limits:
              memory: "1024Mi"
              cpu: "500m"
          ports:
            - containerPort: 4000

My Fluentd DaemonSet File

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      terminationGracePeriodSeconds: 30
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      containers:
        - name: fluentd
          image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          env:
            - name: FLUENT_ELASTICSEARCH_HOST
              value: "elasticsearch.default"
            - name: FLUENT_ELASTICSEARCH_PORT
              value: "9200"

I do know that log files written into custom path /var/log/services/dev will be deleted if pod crashes. So i have to use persistent volume to mount this path.

But i lack the experience of how to create persistent volume for that path and also link Fluentd to read from it.

Thanks in advance.

Originally Posted at: kubernetes - How to configure DaemonSet Fluentd to read from custom log file and console - Stack Overflow

Hi, this is a use case covered in the Manning book Logging In Action