-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial AP poller implementation #86
Open
bryophyta
wants to merge
9
commits into
pf/default-handler-export
Choose a base branch
from
pf/ap-poller-business-logic
base: pf/default-handler-export
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bryophyta
force-pushed
the
pf/ap-poller-business-logic
branch
3 times, most recently
from
November 14, 2024 12:22
38b5dda
to
97fdc7b
Compare
twrichards
force-pushed
the
infra/poller-lambdas
branch
5 times, most recently
from
December 3, 2024 00:16
2eee7de
to
976ec2b
Compare
twrichards
force-pushed
the
infra/poller-lambdas
branch
7 times, most recently
from
December 11, 2024 01:09
1cfec67
to
6162e55
Compare
bryophyta
force-pushed
the
pf/ap-poller-business-logic
branch
3 times, most recently
from
December 12, 2024 15:51
99e48d3
to
ae6f791
Compare
bryophyta
force-pushed
the
pf/ap-poller-business-logic
branch
from
December 12, 2024 16:39
75fb249
to
7451c6b
Compare
twrichards
force-pushed
the
infra/poller-lambdas
branch
4 times, most recently
from
December 13, 2024 17:33
7260e5c
to
e90dcf7
Compare
bryophyta
force-pushed
the
pf/ap-poller-business-logic
branch
2 times, most recently
from
December 18, 2024 13:51
667ad13
to
e93909f
Compare
bryophyta
force-pushed
the
pf/ap-poller-business-logic
branch
from
December 18, 2024 14:43
a25b61c
to
fb0920c
Compare
bryophyta
force-pushed
the
pf/ap-poller-business-logic
branch
from
December 23, 2024 16:43
fb0920c
to
f6de1a7
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this change?
Adds a new poller lambda, using the 'long polling' pattern added in #83, to get data from the AP
/feed
endpoint and pass it to the Newswires ingestion lambda. Getting the data needed for the ingestion lambda requires two steps:/feed
endpoint, which returns immediately if there are new items since the last request (determined by thenext_page
value in the response from the previous request, and passed forward to the lambda via the SQS queue that invokes it) or times out after 15 seconds. (See AP docs)/feed
endpoint returns only part of the data we're interested in. To get the 'body' of the story we need to fetch and parse the NITF representation of the story, which is in an XML format.nb. we don't yet have a clear standard for judging when our processing of a feed is complete/production-ready, so it's possible that we'll want to make some changes later (e.g. if we discover we're missing important fields).
How to test
npm run dev -w poller-lambdas
and selecting the AP poller.next_page
as expected.How can we measure success?
Have we considered potential risks?
Images
Accessibility