-
Notifications
You must be signed in to change notification settings - Fork 532
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add code-coverage-tools package to compare code coverage on the PR bu…
…ild. (#22452) ## Description [AB#14170](https://dev.azure.com/fluidframework/internal/_workitems/edit/14170) Add code-coverage-tools package to compare code coverage on the PR build. 1.) Before running coverage comparison, code coverage plugin identifies the baseline build for the PR. 2.) Once the baseline build is identified, we download the build artifacts corresponding to the `Code Coverage Report_<Build_Number>` artifact name for this build 3.) We then collect the code coverage stats for the PR build and then make the comparison with the baseline. 4.) If the code coverage diff (branch coverage) is more than a percentage point change, then we fail the build for the PR. We also fail the build in case the code coverage for the newly added package is less than 50%. 5.) We post the comment on the PR specifying the code coverage change if any for each package which is modified. We needed this separate module as we need specifics to do the code coverage comparison like the baseline with which we need to make the comparison and then what we need to compare and then after comparison what comment and in what format we want to report it. Sample Comment: <img width="945" alt="code coverage" src="https://github.com/user-attachments/assets/d74e612e-0da3-43b6-87d5-7278c887518d"> --------- Co-authored-by: Jatin Garg <[email protected]> Co-authored-by: Alex Villarreal <[email protected]> Co-authored-by: Tyler Butler <[email protected]>
- Loading branch information
4 people
authored
Oct 4, 2024
1 parent
842aa45
commit ad35a7c
Showing
15 changed files
with
1,285 additions
and
11 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
93 changes: 93 additions & 0 deletions
93
build-tools/packages/build-cli/docs/codeCoverageDetails.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| # Code coverage | ||
|
|
||
| ## Overview | ||
|
|
||
| This module contains all the utilities required to analyze code coverage metrics from PRs. The coverage metrics generated in the PR build are compared against a baseline CI build for packages that have been updated in the PR. If the line or branch coverage for a package has been impacted in the PR, a comment is posted to the PR showing the diff of the code coverage between baseline and PR. | ||
|
|
||
| ## Code Coverage Metrics | ||
|
|
||
| Code coverage metrics is generated when tests run in the CI pipeline. You can also generate the code coverage metrics for a package locally by running `npm run test:coverage` for the individual package or by running `npm run ci:test:mocha:coverage` from the root. | ||
|
|
||
| ## Pieces of the code coverage analysis | ||
|
|
||
| Code coverage has several steps involving different commands. This section defines those pieces and how they fit together to enable overall code coverage reporting. | ||
|
|
||
| ### Cobertura coverage files | ||
|
|
||
| Code coverage metrics is included in the cobertura-format coverage files we collect during CI builds. These files are currently published as artifacts from both our PR and internal build pipelines to ensure we can run comparisons on PRs against a baseline build. | ||
|
|
||
| ### Identifying the baseline build | ||
|
|
||
| Before running coverage comparison, a baseline build needs to be determined for the PR. This is typically based on the target branch for the PR. For example, if a pull request was targeting `main`, we would consider the baseline to be the latest successful `main` branch build. | ||
|
|
||
| ### Downloading artifacts from baseline build | ||
|
|
||
| Once the baseline build is identified, we download the build artifacts corresponding to the `Code Coverage Report_{Build_Number}` artifact name for this build. We unzip the files and extract the coverage metrics out of the code coverage artifact using the helper `getCoverageMetricsFromArtifact`. Currently, we track `lineCoverage` and `branchCoverage` as our metrics for reporting and | ||
| use both `lineCoverage` and `branchCoverage` for success criteria. The metrics is in percentages from 0..100. The final structure of the extracted metrics looks like the following: | ||
|
|
||
| ```typescript | ||
| /** | ||
| * The type for the coverage report, containing the line coverage and branch coverage(in percentage) for each package | ||
| */ | ||
| export interface CoverageMetric { | ||
| lineCoverage: number; | ||
| branchCoverage: number; | ||
| } | ||
| ``` | ||
|
|
||
| ### Generating the coverage metrics on PR build | ||
|
|
||
| As mentioned earlier, the PR build also uploads coverage metrics as artifacts that can be used to run coverage analysis against a baseline build. To help with this, we use the `getCoverageMetricsFromArtifact` helper function to extract code coverage metrics of format `CoverageMetric` that contains code coverage metrics corresponding to the PR for each package. | ||
|
|
||
| ### Comparing code coverage reports | ||
|
|
||
| Once we have the coverage metrics for the baseline and PR build, we use the `compareCodeCoverage` utility function that returns an array of coverage comparisons for the list of packages passed into it. The array returned contains objects of type `CodeCoverageComparison`. We ignore some packages for comparing code coverage such as definition packages and packages which are not inside `packages` folder in the repo. The logic is in the `compareCodeCoverage` function. | ||
|
|
||
| ```typescript | ||
| /** | ||
| * Type for the code coverage report generated by comparing the baseline and pr code coverage | ||
| */ | ||
| export interface CodeCoverageComparison { | ||
| /** | ||
| * Path of the package | ||
| */ | ||
| packagePath: string; | ||
| /** | ||
| * Line coverage in baseline build (as a percent) | ||
| */ | ||
| lineCoverageInBaseline: number; | ||
| /** | ||
| * Line coverage in pr build (as a percent) | ||
| */ | ||
| lineCoverageInPr: number; | ||
| /** | ||
| * difference between line coverage in pr build and baseline build (percentage points) | ||
| */ | ||
| lineCoverageDiff: number; | ||
| /** | ||
| * branch coverage in baseline build (as a percent) | ||
| */ | ||
| branchCoverageInBaseline: number; | ||
| /** | ||
| * branch coverage in pr build (as a percent) | ||
| */ | ||
| branchCoverageInPr: number; | ||
| /** | ||
| * difference between branch coverage in pr build and baseline build (percentage points) | ||
| */ | ||
| branchCoverageDiff: number; | ||
| /** | ||
| * Flag to indicate if the package is new | ||
| */ | ||
| isNewPackage: boolean; | ||
| } | ||
| ``` | ||
|
|
||
| ### Success Criteria | ||
|
|
||
| The code coverage PR checks will fail in the following cases: | ||
|
|
||
| - If the **line code coverage** or **branch code coverage** decreased by > 1%. | ||
| - If the code coverage(line and branch) for a newly added package is < 50%. | ||
|
|
||
| The enforcement code is in the function `isCodeCoverageCriteriaPassed`. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| `flub report` | ||
| ============= | ||
|
|
||
| Report analysis about the codebase, like code coverage and bundle size measurements. | ||
|
|
||
| * [`flub report codeCoverage`](#flub-report-codecoverage) | ||
|
|
||
| ## `flub report codeCoverage` | ||
|
|
||
| Run comparison of code coverage stats | ||
|
|
||
| ``` | ||
| USAGE | ||
| $ flub report codeCoverage --adoBuildId <value> --adoApiToken <value> --githubApiToken <value> | ||
| --adoCIBuildDefinitionIdBaseline <value> --adoCIBuildDefinitionIdPR <value> | ||
| --codeCoverageAnalysisArtifactNameBaseline <value> --codeCoverageAnalysisArtifactNamePR <value> --githubPRNumber | ||
| <value> --githubRepositoryName <value> --githubRepositoryOwner <value> --targetBranchName <value> [-v | --quiet] | ||
| FLAGS | ||
| --adoApiToken=<value> (required) Token to get auth for accessing ADO builds. | ||
| --adoBuildId=<value> (required) Azure DevOps build ID. | ||
| --adoCIBuildDefinitionIdBaseline=<value> (required) Build definition/pipeline number/id for the baseline | ||
| build. | ||
| --adoCIBuildDefinitionIdPR=<value> (required) Build definition/pipeline number/id for the PR build. | ||
| --codeCoverageAnalysisArtifactNameBaseline=<value> (required) Code coverage artifact name for the baseline build. | ||
| --codeCoverageAnalysisArtifactNamePR=<value> (required) Code coverage artifact name for the PR build. | ||
| --githubApiToken=<value> (required) Token to get auth for accessing Github PR. | ||
| --githubPRNumber=<value> (required) Github PR number. | ||
| --githubRepositoryName=<value> (required) Github repository name. | ||
| --githubRepositoryOwner=<value> (required) Github repository owner. | ||
| --targetBranchName=<value> (required) Target branch name. | ||
| LOGGING FLAGS | ||
| -v, --verbose Enable verbose logging. | ||
| --quiet Disable all logging. | ||
| DESCRIPTION | ||
| Run comparison of code coverage stats | ||
| ``` | ||
|
|
||
| _See code: [src/commands/report/codeCoverage.ts](https://github.com/microsoft/FluidFramework/blob/main/build-tools/packages/build-cli/src/commands/report/codeCoverage.ts)_ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
86 changes: 86 additions & 0 deletions
86
build-tools/packages/build-cli/src/codeCoverage/codeCoveragePr.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| /*! | ||
| * Copyright (c) Microsoft Corporation and contributors. All rights reserved. | ||
| * Licensed under the MIT License. | ||
| */ | ||
|
|
||
| import { getAzureDevopsApi } from "@fluidframework/bundle-size-tools"; | ||
| import { type IAzureDevopsBuildCoverageConstants } from "../library/azureDevops/constants.js"; | ||
| import { | ||
| type IBuildMetrics, | ||
| getBaselineBuildMetrics, | ||
| getBuildArtifactForSpecificBuild, | ||
| } from "../library/azureDevops/getBaselineBuildMetrics.js"; | ||
| import type { CommandLogger } from "../logging.js"; | ||
| import { type CodeCoverageComparison, compareCodeCoverage } from "./compareCodeCoverage.js"; | ||
| import { getCoverageMetricsFromArtifact } from "./getCoverageMetrics.js"; | ||
|
|
||
| /** | ||
| * Report of code coverage comparison. | ||
| */ | ||
| export interface CodeCoverageReport { | ||
| /** | ||
| * Comparison data for each package. | ||
| */ | ||
| comparisonData: CodeCoverageComparison[]; | ||
|
|
||
| /** | ||
| * Baseline build metrics against which the PR build metrics are compared. | ||
| */ | ||
| baselineBuildMetrics: IBuildMetrics; | ||
| } | ||
|
|
||
| /** | ||
| * API to get the code coverage report for a PR. | ||
| * @param adoToken - ADO token that will be used to download artifacts from ADO pipeline runs. | ||
| * @param codeCoverageConstantsBaseline - The code coverage constants required for fetching the baseline build artifacts. | ||
| * @param codeCoverageConstantsPR - The code coverage constants required for fetching the PR build artifacts. | ||
| * @param changedFiles - The list of files changed in the PR. | ||
| * @param logger - The logger to log messages. | ||
| */ | ||
| export async function getCodeCoverageReport( | ||
| adoToken: string, | ||
| codeCoverageConstantsBaseline: IAzureDevopsBuildCoverageConstants, | ||
| codeCoverageConstantsPR: IAzureDevopsBuildCoverageConstants, | ||
| changedFiles: string[], | ||
| logger?: CommandLogger, | ||
| ): Promise<CodeCoverageReport> { | ||
| const adoConnection = getAzureDevopsApi(adoToken, codeCoverageConstantsBaseline.orgUrl); | ||
|
|
||
| const baselineBuildInfo = await getBaselineBuildMetrics( | ||
| codeCoverageConstantsBaseline, | ||
| adoConnection, | ||
| logger, | ||
| ).catch((error) => { | ||
| logger?.errorLog(`Error getting baseline build metrics: ${error}`); | ||
| throw error; | ||
| }); | ||
|
|
||
| const adoConnectionForPR = getAzureDevopsApi(adoToken, codeCoverageConstantsPR.orgUrl); | ||
|
|
||
| const prBuildInfo = await getBuildArtifactForSpecificBuild( | ||
| codeCoverageConstantsPR, | ||
| adoConnectionForPR, | ||
| logger, | ||
| ).catch((error) => { | ||
| logger?.errorLog(`Error getting PR build metrics: ${error}`); | ||
| throw error; | ||
| }); | ||
|
|
||
| // Extract the coverage metrics for the baseline and PR builds. | ||
| const [coverageMetricsForBaseline, coverageMetricsForPr] = await Promise.all([ | ||
| getCoverageMetricsFromArtifact(baselineBuildInfo.artifactZip), | ||
| getCoverageMetricsFromArtifact(prBuildInfo.artifactZip), | ||
| ]); | ||
|
|
||
| // Compare the code coverage metrics for the baseline and PR builds. | ||
| const comparisonData = compareCodeCoverage( | ||
| coverageMetricsForBaseline, | ||
| coverageMetricsForPr, | ||
| changedFiles, | ||
| ); | ||
|
|
||
| return { | ||
| comparisonData, | ||
| baselineBuildMetrics: baselineBuildInfo, | ||
| }; | ||
| } |
Oops, something went wrong.