Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create Orchestrator script #50

Closed
2 of 4 tasks
waseem-polus opened this issue Nov 14, 2023 · 0 comments
Closed
2 of 4 tasks

Create Orchestrator script #50

waseem-polus opened this issue Nov 14, 2023 · 0 comments
Assignees
@waseem-polus
Labels
Scrape
Milestone
Data Mining

Comments

@waseem-polus
Copy link
Collaborator

waseem-polus commented Nov 14, 2023 •
edited
Loading

As a...

maintainer of SMARE

Wants to...

understand the health of the backend modules

So that...

quickly improve performance and patch any issues as they arise

Acceptance Criteria

  • Create a new collection in MongoDB
  • Orchestrator calls each module in order
  • Orchestrator posts diagnostic data to DB (how long a job ran and performance metrics of each module)
  • Orchestrator sends run report

Resources

No response
@waseem-polus waseem-polus added this to the Data Mining (Scraping) milestone Nov 14, 2023
@waseem-polus waseem-polus self-assigned this Nov 14, 2023
@waseem-polus waseem-polus changed the title Save scraper logs in a MongoDB Collection Create Orchestrator script Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@waseem-polus waseem-polus

Labels
Scrape
Projects
None yet
Milestone
Data Mining
Development

No branches or pull requests
1 participant
@waseem-polus