Added workflow to index events from s3 data lake to metrics cluster #89
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #89 +/- ##
============================================
+ Coverage 83.57% 84.52% +0.94%
- Complexity 202 228 +26
============================================
Files 56 61 +5
Lines 1236 1370 +134
Branches 51 57 +6
============================================
+ Hits 1033 1158 +125
- Misses 181 189 +8
- Partials 22 23 +1 ☔ View full report in Codecov by Sentry. |
bshien
force-pushed
the
events-index-lambda
branch
2 times, most recently
from
c4ed1f5
to
ace97ba
Compare
| @Slf4j | ||
| public class GithubEventsLambda extends AbstractBaseLambda { | ||
| private final String[] eventsToIndex = {"issues.opened", | ||
| "issues.closed", |
There was a problem hiding this comment.
Please add them as enums.
| }); | ||
|
|
||
| // Create OpenSearch Metrics Lambda setup | ||
| const openSearchMetricsWorkflowStack = new OpenSearchMetricsWorkflowStack(app, 'OpenSearchMetrics-Workflow', { | ||
| opensearchDomainStack: openSearchDomainStack, vpcStack: vpcStack, lambdaPackage: Project.LAMBDA_PACKAGE |
There was a problem hiding this comment.
Please seperate this into a seperate lambda and a seperate stepfunction so that it wont mix up with existing metrics and can have its own cron and full 15 mins to execute the operation.
| } | ||
| } | ||
| openSearchUtil.createIndexIfNotExists("github-events"); | ||
| openSearchUtil.bulkIndex("github-events", finalEventData); |
There was a problem hiding this comment.
Please add as github-events-MM-YYY", this way we can clean up older data easily if required.
|
|
||
| @Override | ||
| public Void handleRequest(Void input, Context context) { | ||
| final String yesterday = LocalDate.now(ZoneOffset.UTC).minus(1, ChronoUnit.DAYS).toString(); // yyyy-MM-dd |
There was a problem hiding this comment.
Please change yesterday to something like collectionStartTime and add comments that will default to yesterday.
bshien
force-pushed
the
events-index-lambda
branch
7 times, most recently
from
bfb1524
to
d6b592c
Compare
bshien
force-pushed
the
events-index-lambda
branch
from
d6b592c
to
076b2b0
Compare
Signed-off-by: Brandon Shien <[email protected]>
bshien
force-pushed
the
events-index-lambda
branch
from
076b2b0
to
c6b32d5
Compare
bshien
changed the title
Description
Added lambda to index events from s3 data lake to metrics cluster
Issues Resolved
Part of #76
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.