feat: adds scraper script

[shiv810]

Resolves #56

• Scrapes issues based on the username passed in.
• Reads the token either as a user input or from the cli.
• Updates issues in the repo, even with same node_id exists.
• Issue Dedup and Matchmaking Results.

[github-actions]

Unused files (1)

src/handlers/issue-scraper.ts

[ubiquity-os-beta]

@sshivaditya2019, this task has been idle for a while. Please provide an update.

[shiv810]

@0x4007, This is the base scraper logic. Should I write a script for adding the issues for all the users mentioned in the auth.users.json ?

[0x4007]

Yes and please update the database with it. You can QA with some task matchmaking scoring improvements and second goal some issue dedupe improvements- right?

[shiv810]

QA:

After Issue Scraper for Issue Matching

Previously 1%

• Only admin can generate rewards non collaboratively ShivTestOrg/test-public#238 (comment)
• Automatically Handling Prerequisites ShivTestOrg/test-public#239 (comment)
• Similar Issue Editing Specification Timeout ShivTestOrg/test-public#240 (comment)

Previously 1% & 0%

• Handle HTML Input ShivTestOrg/test-public#237 (comment)
• Decimal is not properly displayed ShivTestOrg/test-public#241 (comment)

Previously 0%

• Refactor publicAccessControl ShivTestOrg/test-public#242 (comment)
• Redact switch ShivTestOrg/test-public#243 (comment)
• "Originally posted by" crediting ShivTestOrg/test-public#244 (comment)

This will not impact issue deduplication since it is restricted to issues within the same organization and repository.

[0x4007]

Is there some type of bias in the algorithm to make 75% the peak of the bell curve?

talent referrals

I looked through a few of the results and I think above 80% seems actually relevant. What are your thoughts?

It might make sense to exclude showing matches below 80%?

And then always recommend at least two contributors still.

issue deduplication

The markup seems very noisy and also based on my quick look it seems that below 80% seems kind of irrelevant. What are your thoughts on this?

For near term testing purposes I think we should leave on all the markup but I can see us needing to reduce the noisiness and hide anything below a certain threshold, like that 80% again.

[shiv810]

I looked through a few of the results and I think above 80% seems actually relevant. What are your thoughts?

It might make sense to exclude showing matches below 80%?

I think we should include matches below 80%, as this would allow for a larger pool of contributors. We can always exclude them by removing alwaysRecommend and setting the jobMatchingThreshold to 0.8.

Is there some type of bias in the algorithm to make 75% the peak of the bell curve?

The current similarity search uses a weighted sum of cosine distance (0.8) and L2 distance (0.2). Without this weighting, the results tend to cluster around 90% similarity, that is just using the cosine distance¹. With the weighted sum, they are more likely to cluster around 75%. This helps make the results more varied and accurate.

Footnotes

1. https://docs.voyageai.com/discuss/660499a8c27dbb000f201a40 ↩

[0x4007]

@sshivaditya @sshivaditya2019 can you merge now?

[shiv810]

I'll add the scraper files to knip ignore then it should be good to go.

[shiv810]

@0x4007 Could you review this ? This is good to merge otherwise.

[0x4007]

Passes the CI so it's fine