Question about http pagination

[gangtao]

Hi, I am building a pipeline to pull data from http using benthos.

here is my inputs:

input:
  broker:
    inputs:
    - generate:
        mapping: |
          root = {}
          root.now = timestamp_unix_milli()
        interval: 3s
        count: 0
        batch_size: 1
      processors:
        - http:
            url: https://server.com/api/data/views/query?timeStep=hour&from=${! this.metadata.time.not_null().catch(0) }&to=${! json("now") }
            verb: GET
            headers: {"Authorization" : "Bearer key"}
        - label: split
          unarchive:
            format: json_array
        - label: filter
          mapping: |
            root = {}
            meta time = this.timestamp
            root.raw = this

What I want to achieve use metadata time to record the last seen timestamp from the data query result, and use that timestamp as the query parameter for my next query.

The problem is that the metadata data seens cleared every time the generate run at next interval, what should I do with it?

Thanks!

[mihaitodor]

Hey @gangtao, you'll need to use a cache. There's a basic example here: #752 (comment)