Generate an array from source output with transform and use that array for making more http_client source requests

[nathanle]

I have an http_client source that returns a dynamic JSON object and I am using a transform to collect the items I need and store them in an array. I then need to iterate over that array and make multiple http_client source requests with each value in the array.

I apologize, but I am not clear about the proper way to reuse results from VRL on subsequent source calls.

Can I make new source requests after the transform again? I can't invoke a source request from inside the transform can I?

Is it better to use a scripting language to make the first request, get the array, and then invoke vector for each item in the array?

[pront]

I then need to iterate over that array and make multiple http_client source requests with each value in the array.

If I understand correctly, you want to make HTTP requests over the network from within a Remap transform, with is not doable for various reasons. The cleanest way here is to ensure your source outputs individual events, ready to be processed by VRL. I don't have the full context here but you could modify the API or use an ad-hoc script like you describe.

[nathanle]

Thanks @pront

Let me explain a little more. Sorry about that

Pseudo logic follows:

Source:
API_request:
http_client:
GET https://some.api.com/v1/returns_data_with_ids

Transforms:
API_transform:
VRL that captures: array_of_ids = [12345, 56789, 34545]

With this I need to do:
Source:
API_additional requests:
http_client:
for_each (array_of_ids) -> ID:
GET https://some.api.com/v1/object/${ID}

Transforms:
do some other stuff on API_additional_requests

Sink:
my_sink_not_really_important:
Sink_stuff...

[pront]

Do you own the API? You could extend it to do what you described above. Otherwise, you can introduce a proxy between Vector and some.api.com that does this.

[jszwedko]

One way to introduce this sort of feature into Vector, I think, would be an "HTTP enrichment table", but it would diverge quite a bit from existing enrichment tables in that the fetching of state would need to be dynamic. Current enrichment tables load the data into memory at start-time to avoid slow processing.

Absent that, I think a Lua transform might be able to be used here. You can execute arbitrary operations in a Lua transform (like issuing HTTP requests).

[nathanle]

Thank you both for your help on this. I think I am going to use the exec source with a python script that sets this up for Vector to consume.