Run http_client and exit

[omers]

Hello,

I'm currently utilizing Benthos for scraping CSV data using the http_client input. My aim is to use it in a batch mode daily. Therefore, I need it to exit once the download and output flow are completed. Is there a method to achieve this?

Thank you in advance for your assistance.

[mihaitodor]

@omers Hey 👋 Sure, you can either use the generate input with count: 1 combined with a http processor which executes the required query or you can wrap the http_client input in a read_until input and use check: true to exit after the first message.

[omers]

Thank you so much for the information!
Here is my config:

input:
  label: "scrape_site"
  read_until:
    input:
      http_client:
        url: "https://***************************"
        verb: GET
        timeout: 5s
        retry_period: 3s
        max_retry_backoff: 30s
        retries: 3
        backoff_on:
          - 429
        drop_empty_bodies: true
        stream:
          enabled: true
          reconnect: true
          codec: csv
          max_buffer: 1000000
        #rate_limit: "daily" # No default (optional)
    check: true
    restart_input: false
    idle_timeout: 2s

When i run benthos, it starts and then hangs:

INFO Running main config from specified file       @service=benthos benthos_version=4.26.0 path=benthos.yaml
INFO Launching a benthos instance, use CTRL+C to close  @service=benthos
DEBU Waiting for pending acks to resolve before shutting down.  @service=benthos label="" path=root.input.read_until.input
DEBU Pending acks resolved.                        @service=benthos label="" path=root.input.read_until.input

I am doing a very simple processing, and sends the output to postgresql.

Any ideas?

[omers]

OK, Managed to solve it with the following config:

input:
  label: "scrape_site"
  read_until:
    input:
      http_client:
        url: "https://************"
        verb: GET
        timeout: 5s
        retry_period: 30s
        max_retry_backoff: 30s
        retries: 3
        backoff_on:
          - 429
        drop_empty_bodies: true
        stream:
          enabled: false
          reconnect: true
          scanner:
            csv:
              parse_header_row: true
          max_buffer: 1000000
    check: true
    restart_input: false
    idle_timeout: 2s