Tail drops logs in loki 3.1.*

[SmotrovaLilit]

Describe the bug

The tail loses logs when the Loki output configuration in Fluent Bit has labels. For example appName=$appName

To Reproduce
Steps to reproduce the behavior:

1. Started Loki 3.1.0
2. Started Fluent-bit 3.1.3
3. Installed logcli 2.9.8
4. Query tail: logcli query --tail '{job="logs"}'
5. In another terminal: Ingested 18 logs

Expected behavior
All 18 logs that were ingested earlier are displayed in the logcli tail output. (Step 4)

Actual behavior

Only the first 6 logs are displayed in the logcli tail output.

Environment:

• Infrastructure: Mac OS docker-compose fluent-bit, loki

Screenshots, Promtail config, or terminal output

Here is a ready-to-use Docker Compose setup with a Fluent Bit configuration to easily reproduce the issue.

Tail result:

Fluent-bit config:

service:
  flush: 1
  log_level: info
  parsers_file: parsers.conf

pipeline:
  inputs:
    - name: tail
      tag: file
      path: /var/logs/*.jsonl
      parser: json
      refresh_interval: 10

  outputs:
    - name: loki
      match: '*'
      host: loki
      port: 3100
      labels: job=logs, appName=$appName
      retry_limit: 20
      storage.total_limit_size: 1000M

Script to ingest 18 logs:

for i in {1..18}
do
  echo '{"appName":"test-app", "message": "message-'$i'"}' >> ./log.jsonl
done

Docker-compose

version: "3.8"
services:
  loki:
    image: grafana/loki:3.1.0 # tail looses logs
#    image: grafana/loki:2.9.0 # no issues with tailing
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  fluent-bit:
    image: fluent/fluent-bit:3.1.3
    depends_on:
      - loki
    volumes:
      - ./:/var/logs
      - ./fluent-bit.yml:/fluent-bit/etc/fluent-bit.yml
    command: -c /fluent-bit/etc/fluent-bit.yml