Scoring updates

[priyakasimbeg]

Addresses:

• Implement sampling function for hyperparameters and held-out workloads #540
• Scoring code needs to account for held-out workload performance #588
• Submitters need to be able to use “virtual” held-out workload sets #589
• Scoring does not take studies into account #622
• KeyError in prize qualification baseline. #623

Changes to pay attention to during review:

• PRNG Key splitting in run_workloads.py, from submission group to studies to workload trials.
• behavior of strict flag in scoring which will put strict requirements regarding number of trials and studies on a submission.

Todo:

• Add script to generate held-out workloads
• Add held-out workload sampling in script that runs workloads for submission
• Add sampling for studies and trials for in script that runs workloads for submission
• Add instructions for running workloads and scoring submissions

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[georgedahl]

I wrote some of these comments before we met yesterday, so if they contradict that discussion consider them obsolete. Posting them for completeness. Someone else should review the performance profile changes carefully in addition to another full review once you've updated this.

[priyakasimbeg]

I'm going to merge this into Dev.
Will add tests for code in performance_profile.py in a separate PR in which we can double check the correctness of the scoring calculations.