Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial AP poller implementation #86

Open
wants to merge 9 commits into
base: pf/default-handler-export
Choose a base branch
from

Conversation

bryophyta
Copy link
Contributor

@bryophyta bryophyta commented Nov 12, 2024

What does this change?

Adds a new poller lambda, using the 'long polling' pattern added in #83, to get data from the AP /feed endpoint and pass it to the Newswires ingestion lambda. Getting the data needed for the ingestion lambda requires two steps:

  1. Long-polling request to the /feed endpoint, which returns immediately if there are new items since the last request (determined by the next_page value in the response from the previous request, and passed forward to the lambda via the SQS queue that invokes it) or times out after 15 seconds. (See AP docs)
  2. The /feed endpoint returns only part of the data we're interested in. To get the 'body' of the story we need to fetch and parse the NITF representation of the story, which is in an XML format.

nb. we don't yet have a clear standard for judging when our processing of a feed is complete/production-ready, so it's possible that we'll want to make some changes later (e.g. if we discover we're missing important fields).

How to test

  • Run locally, using npm run dev -w poller-lambdas and selecting the AP poller.
  • Deploy to CODE; check that it queues up the next invocation with next_page as expected.
  • Run the unit tests which check that we're extracting the expected values from some samples of the NITF rendition of a couple of stories.

How can we measure success?

Have we considered potential risks?

Images

Accessibility

@bryophyta bryophyta changed the base branch from main to infra/poller-lambdas November 12, 2024 12:18
@bryophyta bryophyta force-pushed the pf/ap-poller-business-logic branch 3 times, most recently from 38b5dda to 97fdc7b Compare November 14, 2024 12:22
@twrichards twrichards force-pushed the infra/poller-lambdas branch 5 times, most recently from 2eee7de to 976ec2b Compare December 3, 2024 00:16
@twrichards twrichards force-pushed the infra/poller-lambdas branch 7 times, most recently from 1cfec67 to 6162e55 Compare December 11, 2024 01:09
@bryophyta bryophyta force-pushed the pf/ap-poller-business-logic branch 3 times, most recently from 99e48d3 to ae6f791 Compare December 12, 2024 15:51
@bryophyta bryophyta changed the title Pf/ap poller business logic Add AP poller Dec 12, 2024
@bryophyta bryophyta force-pushed the pf/ap-poller-business-logic branch from 75fb249 to 7451c6b Compare December 12, 2024 16:39
@twrichards twrichards force-pushed the infra/poller-lambdas branch 4 times, most recently from 7260e5c to e90dcf7 Compare December 13, 2024 17:33
Base automatically changed from infra/poller-lambdas to main December 17, 2024 10:28
@bryophyta bryophyta force-pushed the pf/ap-poller-business-logic branch 2 times, most recently from 667ad13 to e93909f Compare December 18, 2024 13:51
@bryophyta bryophyta changed the title Add AP poller Initial AP poller implementation Dec 18, 2024
@bryophyta bryophyta force-pushed the pf/ap-poller-business-logic branch from a25b61c to fb0920c Compare December 18, 2024 14:43
@bryophyta bryophyta marked this pull request as ready for review December 18, 2024 14:52
@bryophyta bryophyta requested a review from a team as a code owner December 18, 2024 14:52
@bryophyta bryophyta mentioned this pull request Dec 19, 2024
4 tasks
@bryophyta bryophyta force-pushed the pf/ap-poller-business-logic branch from fb0920c to f6de1a7 Compare December 23, 2024 16:43
@bryophyta bryophyta changed the base branch from main to pf/default-handler-export December 23, 2024 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant