Logs delay in grafana dashboard from Loki >> Fluentbit

We have installed Loki-Grafana-Fluentbit without using Helm. We have 350+ application running on Kubernetes cluster.

Some of application produce too many lines of logs in a seconds. During that time we are facing the issue delay in logs from loki to grafana. This is happen in some of application. It is starting delay from 3 min and than so on… i assume that logs are not getting in batch-size from loki to grafana.

We are using Fluent-bit official image with output Loki with this ref documentation >> Loki - Fluent Bit: Official Manual

I am sharing ConfigMap file of Loki and Fluent-bit, I am not sure where i am missing.

  1. Fluent-bit Configmap file.
data:
  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Buffer_Size         0
        Merge_Log           Off
        #Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     debug
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE out_loki.conf
  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     128MB
        Skip_Long_Lines   On
        Refresh_Interval  20
        # Control the log line length
        Buffer_Chunk_Size 4MB
        Buffer_Max_Size   8MB
        # Using the docker mode to deal with multiline messages emitted by docker
        Docker_Mode       On
        Rotate_Wait       30
  out_loki.conf: |
    [Output]
        name              loki
        match             *
        host              loki.logging.svc.cluster.local
        port              3100
        tenant_id         ""
        labels            job=fluentbit
        auto_kubernetes_labels On
        line_format       json
  parsers.conf: |
    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

  1. Loki Configmap file:
auth_enabled: false
ingester:

  chunk_idle_period: 3m
  chunk_block_size: 262144
  chunk_retain_period: 1m
  chunk_encoding: snappy
  chunk_target_size: 1536000 #0
  max_transfer_retries: 0
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
limits_config:
  ingestion_rate_mb: 15
  ingestion_burst_size_mb: 30
  max_chunks_per_query: 2000000
  max_label_name_length: 1024
  max_label_value_length: 2048
  max_streams_per_user: 10000
  max_entries_limit_per_query: 5000
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
schema_config:
  configs:
  - from: 2020-10-24
    store: boltdb-shipper
    object_store: filesystem
    schema: v11
    index:
      prefix: index_
      period: 24h
server:
  http_listen_port: 3100
  log_level: debug
  grpc_server_max_recv_msg_size: 8388608
  grpc_server_max_send_msg_size: 8388608
storage_config:
  boltdb_shipper:
    active_index_directory: /data/loki/boltdb-shipper-active
    cache_location: /data/loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /data/loki/chunks
chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: true
  retention_period: 72h
compactor:
  working_directory: /data/loki/boltdb-shipper-compactor
  shared_store: filesystem

Here is logs delay evident attached with this.