Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DOP-4209 #958

Merged
merged 195 commits into from
Jan 22, 2024
Merged
Show file tree
Hide file tree
Changes from 178 commits
Commits
Show all changes
195 commits
Select commit Hold shift + click to select a range
f4b5188
experiments, added nextGenParse call to jobHandler
mmeigs Nov 14, 2023
8f941ba
stage ecs - no makefiles
mmeigs Nov 14, 2023
706af67
alter snooty version
mmeigs Nov 14, 2023
1a63501
next gen html
mmeigs Nov 15, 2023
0be0b24
command key
mmeigs Nov 15, 2023
6c43e45
parse
mmeigs Nov 15, 2023
e733122
catch
mmeigs Nov 15, 2023
2442b6f
logger
mmeigs Nov 15, 2023
901a229
remove error
mmeigs Nov 15, 2023
9aa2fd2
more logging
mmeigs Nov 15, 2023
8bd6ade
cwd
mmeigs Nov 15, 2023
c11687b
repoDir correct
mmeigs Nov 15, 2023
68eed83
quotations
mmeigs Nov 15, 2023
e9d9b36
successful parse? odd address
mmeigs Nov 15, 2023
dc4c2a1
oas page build
mmeigs Nov 16, 2023
1873ce9
xlarge
mmeigs Nov 16, 2023
478ebf3
persistence module
mmeigs Nov 17, 2023
ab4f652
quote
mmeigs Nov 17, 2023
71751a4
no parse
mmeigs Nov 17, 2023
38c0f7d
try catch parse
mmeigs Nov 17, 2023
e313e5f
type error
mmeigs Nov 17, 2023
b97835c
no parse, use persistence and build
mmeigs Nov 20, 2023
97cbc8b
log errors
mmeigs Nov 20, 2023
2832b0e
localApp working
mmeigs Nov 20, 2023
1d45d72
remove /dist
mmeigs Nov 20, 2023
11bac2c
add build deps & nextgenhtml
mmeigs Nov 22, 2023
7aee7af
add logging to build deps
mmeigs Nov 22, 2023
afadedc
log html
mmeigs Nov 22, 2023
e320a67
remove build deps for envs
mmeigs Nov 22, 2023
178e680
env vars
mmeigs Nov 22, 2023
cf18066
build deps before executeBuild
mmeigs Nov 22, 2023
3bc4550
staging monorepo jobs
mmeigs Nov 27, 2023
781b7ab
comment
mmeigs Nov 27, 2023
bae5f35
localApp type cleanup
mmeigs Nov 27, 2023
53d68da
stage
mmeigs Nov 27, 2023
62a9747
deploy
mmeigs Nov 27, 2023
733211c
clean fs
mmeigs Nov 27, 2023
2f3bebd
status
mmeigs Nov 27, 2023
6b4dcbe
no event body
mmeigs Nov 27, 2023
56b11c7
throw new error
mmeigs Nov 27, 2023
9c97ae1
commented out all unnecessary steps
mmeigs Nov 28, 2023
4cb8bbb
explicitly call build steps
mmeigs Nov 28, 2023
f11e4b5
remove steps that should not be used
mmeigs Nov 28, 2023
c3ae28e
include job inqueue for error
mmeigs Nov 28, 2023
e352d0b
Merge branch 'master' into DOP-4127-no-makefiles
mmeigs Nov 28, 2023
b576633
log status
mmeigs Nov 28, 2023
c099d3b
Empty-Commit
mmeigs Nov 29, 2023
a2e9a64
comment
mmeigs Nov 29, 2023
02aba79
used process.env to find URL and BUCKET
mmeigs Nov 29, 2023
7a10135
save localApp
mmeigs Nov 29, 2023
9e36e45
uncomment prepNextGenBuild
mmeigs Nov 29, 2023
504cb84
remove throw error
mmeigs Nov 29, 2023
0c8de69
not build on preprd
mmeigs Nov 29, 2023
62adea1
Merge branch 'master' into DOP-4127-no-makefiles
mmeigs Nov 29, 2023
86dbf92
buildCommands array
mmeigs Nov 29, 2023
857a1db
log publish
mmeigs Nov 29, 2023
d87b56e
use normal deploy
mmeigs Nov 29, 2023
cb8080e
deploy
mmeigs Nov 29, 2023
01463ff
add cp commands
mmeigs Nov 29, 2023
0a2c7b5
cd snooty
mmeigs Nov 29, 2023
c0717cf
log outputs
mmeigs Nov 29, 2023
02b61af
ref repoDir rather than cwd
mmeigs Nov 29, 2023
a3cbc72
correct reposDir vs repoDir
mmeigs Nov 29, 2023
7d0ba08
add slash
mmeigs Nov 29, 2023
375a97d
add repos to prodFileName
mmeigs Nov 29, 2023
482f841
log build deps
mmeigs Nov 30, 2023
9f82e55
change prodFIleName to cwd/snooty
mmeigs Nov 30, 2023
93fbc40
run parse
mmeigs Nov 30, 2023
68137c4
remove deploy
mmeigs Nov 30, 2023
be0d0a0
merged latest master
mmeigs Nov 30, 2023
c386506
override build in stagingJobHandler
mmeigs Nov 30, 2023
6534d77
override build, set up debugger
mmeigs Nov 30, 2023
d86040f
vscode launch
mmeigs Nov 30, 2023
fe4ddca
remove dist
mmeigs Dec 1, 2023
8d7182f
remove dist
mmeigs Dec 1, 2023
312ba00
use project
mmeigs Dec 4, 2023
3c7156d
readd build deps
mmeigs Dec 4, 2023
22d75fd
add cp and cd commands to nextGenHtml
mmeigs Dec 4, 2023
6ae841f
cp correct paths
mmeigs Dec 4, 2023
21207c7
remove first cp and cd
mmeigs Dec 4, 2023
e494c7f
remove cp
mmeigs Dec 4, 2023
8f48f51
ref snooty filepath correctly
mmeigs Dec 4, 2023
b87d85c
dockerfile snooty branch mm-log
mmeigs Dec 4, 2023
f1b6e57
remove first cp
mmeigs Dec 4, 2023
f48ac30
cd ..
mmeigs Dec 5, 2023
f24ef53
commands
mmeigs Dec 5, 2023
76a0677
stringify, logs
mmeigs Dec 5, 2023
901792c
use chdir
mmeigs Dec 5, 2023
5254b44
cp second
mmeigs Dec 5, 2023
09cc84a
log in clicommand
mmeigs Dec 5, 2023
94312e8
pass logger to mut publish
mmeigs Dec 5, 2023
f502182
dotcomstg -> stg
mmeigs Dec 5, 2023
52590c4
checkout and pull branch
mmeigs Dec 5, 2023
243ac89
add pull repo and change clone
mmeigs Dec 5, 2023
32496d4
commented out incorrect code
mmeigs Dec 5, 2023
b770001
local run works for both cloud-docs and monorepo/cloud-docs
mmeigs Dec 7, 2023
3667b5d
log event and boyd
mmeigs Dec 7, 2023
2650e8f
log out trigger build
mmeigs Dec 7, 2023
0108bc4
log repo name and feat flag
mmeigs Dec 7, 2023
150b11f
feature flag
mmeigs Dec 7, 2023
39d84b8
ssmprefix
mmeigs Dec 7, 2023
a249c26
env
mmeigs Dec 7, 2023
178a527
dist
mmeigs Dec 7, 2023
a6f3261
remove feature flag for feature branch build
mmeigs Dec 7, 2023
f0b535c
log why no paths
mmeigs Dec 7, 2023
045180b
change slash of path
mmeigs Dec 7, 2023
1d69643
clean
mmeigs Dec 7, 2023
7713c19
organize code
mmeigs Dec 8, 2023
c6c5c69
clean
mmeigs Dec 8, 2023
f796c4a
redoc
mmeigs Dec 8, 2023
ffbc331
Co-authored-by: Brandon Ly <[email protected]>
mmeigs Dec 8, 2023
39837b4
fixes from merge
mmeigs Dec 8, 2023
c3313a3
comment out builddeps, use redoc rc
mmeigs Dec 11, 2023
40e5c26
duplicate clone
mmeigs Dec 11, 2023
c25b533
clean, get bucket and url
mmeigs Dec 11, 2023
248df20
log env vars
mmeigs Dec 11, 2023
d23da3c
pass logger to getEnvVar
mmeigs Dec 11, 2023
c0c4c57
takeover preprd
mmeigs Dec 11, 2023
2adf502
add to v1?
mmeigs Dec 11, 2023
4c8d4c3
log which build
mmeigs Dec 11, 2023
64cf314
force directory
mmeigs Dec 11, 2023
07967fb
add directory to debug command and local build
mmeigs Dec 12, 2023
2a8386b
remove from preprd
mmeigs Dec 12, 2023
f5dbead
clean up
mmeigs Dec 12, 2023
07caee9
keep conditionals for buildCommands in normal build
mmeigs Dec 12, 2023
e45fce2
further cleaning
mmeigs Dec 12, 2023
5c558da
remove logs
mmeigs Dec 12, 2023
4ac7b01
number of logs
mmeigs Dec 12, 2023
b57f0ac
curl into repoDir/targetDir
mmeigs Dec 12, 2023
493edaa
try new flow of logging
mmeigs Dec 13, 2023
52b2f15
remove comments
mmeigs Dec 13, 2023
492d7c5
allow output and error text to be returned from nextGenStage
mmeigs Dec 13, 2023
7f3299d
clean logs
mmeigs Dec 14, 2023
b8dec8c
revert targetDir for downloadBuildDependencies
mmeigs Dec 14, 2023
bfe3cbd
clean, wrapWithBenchmark
mmeigs Dec 14, 2023
93d2642
Merge branch 'master' into DOP-4127-no-makefiles
mmeigs Dec 14, 2023
038337f
PR feedback
mmeigs Dec 15, 2023
211d1c2
[DOP-4127]: Update dockerfile.local to have redoc installed properly
branberry Dec 18, 2023
841e6f4
[DOP-4127]: Use new SQS queue URL
branberry Dec 18, 2023
033d58a
[DOP-4127]: Install redoc bundle
branberry Dec 18, 2023
1eaf867
[DOP-4127]: Revert how redoc is installed
branberry Dec 18, 2023
c4052bb
PR feedback, second round
mmeigs Dec 19, 2023
64e6af4
Merge branch 'master' into DOP-4127-no-makefiles
mmeigs Dec 19, 2023
d1367a7
replace useWithBenchmarks with isNextGen
mmeigs Dec 19, 2023
e16a789
source patchId from getBuildAndGetDependencies
mmeigs Dec 20, 2023
ffe258c
Merge branch 'master' into DOP-4127-no-makefiles
mmeigs Dec 20, 2023
58b7c14
[DOP-4204]: Update README for local Autobuilder (#954)
branberry Jan 2, 2024
f7d968f
added logging and error throwing in wrapWithBenchmark
mmeigs Jan 3, 2024
8e5e756
conditionally write patchId and commitHash env vars
mmeigs Jan 3, 2024
0f27590
seeing if we have build dependencies
mayaraman19 Jan 3, 2024
cdaa1d8
adding await
mayaraman19 Jan 3, 2024
341e1d3
log
mayaraman19 Jan 3, 2024
41f10a7
strringify
mayaraman19 Jan 3, 2024
9ef2582
adding monorepo handling
mayaraman19 Jan 3, 2024
57bbafd
snooty toml
mayaraman19 Jan 3, 2024
f3b7748
logging
mayaraman19 Jan 3, 2024
04a687a
Adding slash
mayaraman19 Jan 3, 2024
2a99aaa
adding timeout to curl
mayaraman19 Jan 3, 2024
845a0b7
adding logging
mayaraman19 Jan 8, 2024
1c59684
fix merge conflicts
mayaraman19 Jan 8, 2024
178cb7e
finally
mayaraman19 Jan 8, 2024
3c5ab4d
bruh
mayaraman19 Jan 8, 2024
a9daaa8
L
mayaraman19 Jan 8, 2024
cb63290
oh well
mayaraman19 Jan 8, 2024
7b7901d
adding repodir
mayaraman19 Jan 9, 2024
e444059
adding monorepo support
mayaraman19 Jan 9, 2024
510172d
log
mayaraman19 Jan 9, 2024
248257e
cwd
mayaraman19 Jan 9, 2024
c901a06
adding to both curl and mkdir
mayaraman19 Jan 9, 2024
419ced8
adding logging
mayaraman19 Jan 10, 2024
6218371
testing with executeclicommand not throwing
mayaraman19 Jan 11, 2024
8181640
Merge branch 'master' into DOP-4209
mayaraman19 Jan 11, 2024
5ee7c73
avoiding merge conflict
mayaraman19 Jan 11, 2024
749077c
Merge branch 'DOP-4209' of https://github.com/mongodb/docs-worker-poo…
mayaraman19 Jan 11, 2024
313af9c
cleaning up
mayaraman19 Jan 11, 2024
52f9ed7
restructuring
mayaraman19 Jan 11, 2024
6e7627d
[DOP-4269]: Refactor error handling
branberry Jan 11, 2024
97ae5f4
[DOP-4269]: Add text to error
branberry Jan 12, 2024
068c8ee
consolidating
mayaraman19 Jan 16, 2024
6af15c4
Merge branch 'master' into DOP-4209
mayaraman19 Jan 16, 2024
afd3064
fixing merge stuff in readme
mayaraman19 Jan 16, 2024
b7e6bab
Merge branch 'DOP-4209' of https://github.com/mongodb/docs-worker-poo…
mayaraman19 Jan 16, 2024
802340e
merge
mayaraman19 Jan 16, 2024
882dff2
async in dep helper
mayaraman19 Jan 16, 2024
4ef5cf8
error message
mayaraman19 Jan 16, 2024
6c05afa
nvm removing error msg
mayaraman19 Jan 16, 2024
f06608f
trying
mayaraman19 Jan 16, 2024
ce59317
seems like the best way for await idk
mayaraman19 Jan 16, 2024
a0d9a4b
one more attempt
mayaraman19 Jan 16, 2024
da37185
Nvm
mayaraman19 Jan 16, 2024
20c2bc6
not working
mayaraman19 Jan 16, 2024
57c3fda
Merge branch 'master' into DOP-4209
branberry Jan 19, 2024
577fba4
trying this out
mayaraman19 Jan 22, 2024
fc56fc7
nvm
mayaraman19 Jan 22, 2024
bb41a5f
Merge branch 'master' into DOP-4209
mayaraman19 Jan 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Dockerfile.enhanced
Original file line number Diff line number Diff line change
Expand Up @@ -117,4 +117,4 @@ ENV OAS_MODULE_PATH=${WORK_DIRECTORY}/modules/oas-page-builder/index.js

RUN mkdir repos && chmod 755 repos
EXPOSE 3000
CMD ["node", "enhanced/enhancedApp.js"]
CMD ["node", "--enable-source-maps", "enhanced/enhancedApp.js"]
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,12 +52,17 @@ To view all of the options for the command, you can run:

Here is an example of running the local debugger for `cloud-docs`:

<<<<<<< HEAD
`npm run debug -- -o 10gen -n 'cloud-docs'`
=======
`npm run debug -- -o 10gen -n cloud-docs`

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Make sure the merge finishes and gets rid of these characters

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the diff: The quotes are not necessary, but they are also not an issue. Either way is fine, but I would lean toward all the examples showing the same way for each input

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noting here that this entire file has multiple unfinished merge flags to take care of!

Here is an example of running the local debugger for `docs-monorepo/docs-landing` on branch `groot`:

`npm run debug -- -o 10gen -n docs-monorepo -d docs-landing -b groot`

> > > > > > > d5194e973afbcf295f5bd32ca5f2b5dc24dcf9b2

By default, the environment that is used for the local Autobuilder is `stg`.

### Debugger Behavior
Expand All @@ -72,16 +77,28 @@ When the command is run, there are several steps that occur before the Autobuild
4. The data from step 2 is then added as a record in the `pool_test.queue`.
5. The container is then run, and waits for the user to connect to it via the VSCode debugger.

<<<<<<< HEAD
Once the container starts successfully, you should see the following message:

# `Container started. Please attach to the debugger to run the Autobuilder.`

Once the container starts successfully, you should see something like the following message:

`Debugger listening on ws://0.0.0.0:9229/....`

> > > > > > > d5194e973afbcf295f5bd32ca5f2b5dc24dcf9b2

To connect, click on the debug tab on the left side of your VSCode editor. Make sure the dropdown to the right of the green play button is set to the `Docker: Attach to Autobuilder` configuration. Press the green play button, and you will attach to the container.

### Troubleshooting

<<<<<<< HEAD
The most frequent cause of build failures will be related to expired AWS credentials, or not having Docker running. Also, if you haven't ran `npm ci` in a while, you will need to do so as a new dependency was added to run the command.
=======
The most frequent cause of build failures will be related to expired AWS credentials, or not having Docker running. Also, if you haven't run `npm ci` in a while, you will need to do so as a new dependency was added to run the command.

> > > > > > > d5194e973afbcf295f5bd32ca5f2b5dc24dcf9b2

Occasionally, errors may occur inexplicably, and the error messages may seem unrelated to any change made. Oftentimes, running the following commands can resolve these sporadic issues:

```sh
Expand All @@ -95,8 +112,13 @@ Also, another potential error could be due to the Dockerfile.local not being upd

By default, the container will break at the first line of code, which will be in a file called `bind.js`. Press the fast-forward button to continue the execution. You are also able to add other breakpoints to stop the application. Once the application is complete, press `CTRL + C` for the terminal to exit out of the connection to the container.

<<<<<<< HEAD
If you receive `CredentialsProviderError: Could not load credentials from any providers`, make sure that the `~/.aws/credentials` file has the `[default]` profile defined.
=======
If you receive `CredentialsProviderError: Could not load credentials from any providers`, make sure that there is no env `AWS_PROFILE` defined as a different profile anywhere (such as in the global `~/.zshrc` file). Otherwise, ensure that `AWS_PROFILE` matches the same profile defined in `~/.aws/credentials`.

> > > > > > > d5194e973afbcf295f5bd32ca5f2b5dc24dcf9b2

## Run Tests

```
Expand Down
1 change: 1 addition & 0 deletions api/controllers/v2/github.ts
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@ export const TriggerBuild = async (event: APIGatewayEvent): Promise<APIGatewayPr

/* Create and insert Job for each monorepo project that has changes */
for (const path of monorepoPaths) {
consoleLogger.info(body.repository.full_name, `Create Job for Monorepo directory: /${path}`);
// TODO: Deal with nested monorepo projects
/* For now, we will ignore nested monorepo projects until necessary */
if (path.split('/').length > 1) continue;
Expand Down
2 changes: 1 addition & 1 deletion cdk-infra/lib/constructs/api/webhook-env-construct.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export class WebhookEnvConstruct extends Construct {
const ssmPrefix = getSsmPathPrefix();
const env = getEnv();

const featureFlagMonorepoPath = StringParameter.valueFromLookup(this, `${ssmPrefix}/flag/monorepo_path`);
const featureFlagMonorepoPath = StringParameter.valueFromLookup(this, `${ssmPrefix}/DOP-4127/flag/monorepo_path`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noting that this needs to be changed back before merge

const dbName = StringParameter.valueFromLookup(this, `${ssmPrefix}/atlas/dbname`);
const snootyDbName = StringParameter.valueFromLookup(this, `${ssmPrefix}/atlas/collections/snooty`);
const repoBranchesCollection = StringParameter.valueFromLookup(this, `${ssmPrefix}/atlas/collections/repo`);
Expand Down
2 changes: 1 addition & 1 deletion cdk-infra/lib/constructs/worker/worker-env-construct.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ export class WorkerEnvConstruct extends Construct {
// doing this for the time being, but I think we don't need to necessarily retrieve this from ssm for feature branches, nor would we want to in that case
const previewBuildEnabled = StringParameter.valueFromLookup(this, `${ssmPrefix}/flag/preview_build/enabled`);
const featureFlagUpdatePages = StringParameter.valueFromLookup(this, `${ssmPrefix}/flag/update_pages`);
const featureFlagMonorepoPath = StringParameter.valueFromLookup(this, `${ssmPrefix}/flag/monorepo_path`);
const featureFlagMonorepoPath = StringParameter.valueFromLookup(this, `${ssmPrefix}/DOP-4127/flag/monorepo_path`);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to change back before merge


const entitlementCollection = StringParameter.valueFromLookup(
this,
Expand Down
22 changes: 15 additions & 7 deletions src/commands/src/helpers/dependency-helpers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,16 +43,25 @@ async function createEnvProdFile({
}
}

export async function downloadBuildDependencies(buildDependencies: BuildDependencies, repoName: string) {
export async function downloadBuildDependencies(
buildDependencies: BuildDependencies,
repoName: string,
directory?: string
) {
const commands: string[] = [];
await Promise.all(
buildDependencies.map(async (dependencyInfo) => {
const repoDir = getRepoDir(repoName);
const repoDir = directory ? getRepoDir(repoName, directory) : getRepoDir(repoName);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to do this conditional! The directory argument can be undefined. You can simple state:

const repoDir = getRepoDir(repoName, directory);

const targetDir = dependencyInfo.targetDir ?? repoDir;
let options = {};
if (targetDir != repoDir) {
options = { cwd: repoDir };
}
try {
await executeCliCommand({
command: 'mkdir',
args: ['-p', targetDir],
options: options,
});
} catch (error) {
console.error(
Expand All @@ -63,17 +72,19 @@ export async function downloadBuildDependencies(buildDependencies: BuildDependen
}
commands.push(`mkdir -p ${targetDir}`);
await Promise.all(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: I did not use allSettled here because it seemed to not exist with this version of Javascript?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What version of node are you using?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

v18.17.1!

dependencyInfo.dependencies.map((dep) => {
dependencyInfo.dependencies.map(async (dep) => {
try {
executeCliCommand({
command: 'curl',
args: ['-SfL', dep.url, '-o', `${targetDir}/${dep.filename}`],
args: ['--max-time', '10', '-SfL', dep.url, '-o', `${targetDir}/${dep.filename}`],
options: options,
});
} catch (error) {
console.error(
`ERROR! Could not curl ${dep.url} into ${targetDir}/${dep.filename}. Dependency information: `,
dependencyInfo
);
return commands;
}
commands.push(`curl -SfL ${dep.url} -o ${targetDir}/${dep.filename}`);
})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, interesting thing here. I could be wrong, but I believe that because we're returning a static value (even though it's a Promise because of the async keyword) it can return before the executeCliCommand function finishes.

I think this can be slightly changed to do what you're trying to do and cleaner too. If you take away the async keyword and add return keyword to the executeCliCommand invocation that will ensure that the Promise.all waits for each Promise to finish. Then move the commands.push line up into the try block before the return. Then you can completely delete the unused line return commands. Doing this will ensure that the Node thread will not continue on until all of these commands have either finished or errored.

I have an example below.

Like this:

await Promise.all(
     dependencyInfo.dependencies.map((dep) => {
          try {
            commands.push(`curl -SfL ${dep.url} -o ${targetDir}/${dep.filename}`);
            return executeCliCommand({
              command: 'curl',
               args: ['--max-time', '10', '-SfL', dep.url, '-o', `${targetDir}/${dep.filename}`],
              options: options,
            });
          } catch (error) {
            console.error(
              `ERROR! Could not curl ${dep.url} into ${targetDir}/${dep.filename}. Dependency information: `,
              dependencyInfo
            );
          }
       })
)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are all great points, @mmeigs. The one small nit with the example though is that we won't hit this catch block, I think. This is because we return the promise, but we don't await it, not giving the promise the opportunity to settle before the catch block can be entered.

I think we can add the async back and do return await executeCliCommand... so that it will wait for the promise to settle before returning.

This also has me thinking as well, do we want to do an allSettled here maybe as well? That way, we can know all of the dependencies that are failing for whatever reason.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Testing this out with some incorrect curl links but this approach still doesn't seem to catch the error 😞

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I wonder if doing return await executeCliCommand(...) we try:

 const cliResult = await executeCliCommand(...);

return cliResult;

in the try/catch? I wonder if return is sort of overwriting the await in a sense

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this didn't work either 💔 i dropped a new ticket for this issue though!

Expand All @@ -87,12 +98,9 @@ export async function prepareBuildAndGetDependencies(
repoName: string,
projectName: string,
baseUrl: string,
buildDependencies: BuildDependencies,
directory?: string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're not getting dependencies in this function anymore, can we rename this function? prepareBuild could be fine by itself!

) {
const repoDir = getRepoDir(repoName, directory);
await downloadBuildDependencies(buildDependencies, repoName);
console.log('Downloaded Build dependencies');

// doing these in parallel
const commandPromises = [
Expand Down
26 changes: 19 additions & 7 deletions src/commands/src/helpers/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,11 @@ const EPIPE_SYSCALL = 'write';

export class ExecuteCommandError extends Error {
data: unknown;
constructor(message: string, data: unknown) {
exitCode: number | null;
constructor(message: string, exitCode: number | null, data?: unknown) {
super(message);
this.data = data;
this.exitCode = exitCode;
}
}

Expand Down Expand Up @@ -80,7 +82,7 @@ export async function executeAndPipeCommands(
return;
}

reject(new ExecuteCommandError('The first command stdin (cmdTo) failed', err));
reject(new ExecuteCommandError('The first command stdin (cmdTo) failed', err.errno, err));
hasRejected = true;
});

Expand All @@ -89,7 +91,7 @@ export async function executeAndPipeCommands(
});

cmdFrom.on('error', (err) => {
reject(new ExecuteCommandError('The first command (cmdTo) failed', err));
reject(new ExecuteCommandError('The first command (cmdTo) failed', 1, err));
hasRejected = true;
});

Expand All @@ -105,7 +107,7 @@ export async function executeAndPipeCommands(
});

cmdTo.on('error', (err) => {
reject(new ExecuteCommandError('The second command failed', err));
reject(new ExecuteCommandError('The second command failed', 1, err));
});

cmdTo.on('exit', (exitCode) => {
Expand All @@ -127,7 +129,13 @@ export async function executeAndPipeCommands(
console.error('error', errorText.join(''));
}

reject(new ExecuteCommandError('The command failed', { exitCode, outputText, errorText }));
reject(
new ExecuteCommandError('The command failed', exitCode, {
exitCode,
outputText: outputText.join(''),
errorText: errorText.join(''),
})
);
return;
}

Expand Down Expand Up @@ -177,7 +185,7 @@ export async function executeCliCommand({

executedCommand.on('error', (err) => {
console.log(`ERROR in executeCliCommand.\nCommand: ${command} ${args.join(' ')}\nError: ${err}`);
reject(new ExecuteCommandError('The command failed', err));
reject(new ExecuteCommandError('The command failed', 1, err));
});

executedCommand.on('close', (exitCode) => {
Expand All @@ -204,7 +212,11 @@ export async function executeCliCommand({
Options provided: ${JSON.stringify(options, null, 4)}\n
Stdout: ${outputText.join('')} \n
Error: ${errorText.join('')}`,
exitCode
exitCode,
{
outputText: outputText.join(''),
errorText: errorText.join(''),
}
)
);
return;
Expand Down
5 changes: 4 additions & 1 deletion src/commands/src/shared/next-gen-parse.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import path from 'path';
import { Job } from '../../../entities/job';
import { getDirectory } from '../../../job/jobHandler';
import { CliCommandResponse, executeCliCommand } from '../helpers';
import { CliCommandResponse, ExecuteCommandError, executeCliCommand } from '../helpers';

const RSTSPEC_FLAG = '--rstspec=https://raw.githubusercontent.com/mongodb/snooty-parser/latest/snooty/rstspec.toml';
interface NextGenParseParams {
Expand Down Expand Up @@ -37,6 +37,9 @@ export async function nextGenParse({ job, patchId, isProd }: NextGenParseParams)
});
return result;
} catch (error) {
if (error instanceof ExecuteCommandError && error.exitCode !== 1) {
return error.data as CliCommandResponse;
}
throw new Error(`next-gen-parse failed. \n ${error}`);
}
}
21 changes: 12 additions & 9 deletions src/job/jobHandler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -214,16 +214,21 @@ export abstract class JobHandler {

@throwIfJobInterupted()
private async getBuildDependencies() {
const buildDependencies = await this._repoBranchesRepo.getBuildDependencies(this.currJob.payload.repoName);
const repoName = this.currJob.payload.repoName;
const directory = this.currJob.payload.repoName === MONOREPO_NAME ? this.currJob.payload.directory : undefined;
const buildDependencies = await this._repoBranchesRepo.getBuildDependencies(repoName, directory);
if (!buildDependencies) return [];
await this._logger.save(this._currJob._id, 'Identified Build dependencies');
return buildDependencies;
}

@throwIfJobInterupted()
private async getAndBuildDependencies() {
private async getAndDownloadBuildDependencies() {
const repoName = this.currJob.payload.repoName;
const directory = this.currJob.payload.repoName === MONOREPO_NAME ? this.currJob.payload.directory : undefined;
const buildDependencies = await this.getBuildDependencies();
const commands = await downloadBuildDependencies(buildDependencies, this.currJob.payload.repoName);
this._logger.save(this._currJob._id, commands.join('\n'));
const commands = await downloadBuildDependencies(buildDependencies, this.currJob.payload.repoName, directory);
await this._logger.save(this._currJob._id, `${commands.join('\n')}`);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can combine these again. The reason to separate is gone now, and also it would combine the redundancy of accessing the repoName and directory twice in quick succession.


@throwIfJobInterupted()
Expand Down Expand Up @@ -550,7 +555,7 @@ export abstract class JobHandler {
this._logger.save(this._currJob._id, 'Checked Commit');
await this.pullRepo();
this._logger.save(this._currJob._id, 'Pulled Repo');
await this.getAndBuildDependencies();
await this.getAndDownloadBuildDependencies();
this._logger.save(this._currJob._id, 'Downloaded Build dependencies');
this.prepBuildCommands();
this._logger.save(this._currJob._id, 'Prepared Build commands');
Expand Down Expand Up @@ -581,8 +586,8 @@ export abstract class JobHandler {
await this.setEnvironmentVariables();
this.logger.save(job._id, 'Prepared Environment variables');

const buildDependencies = await this.getBuildDependencies();
this._logger.save(this._currJob._id, 'Identified Build dependencies');
await this.getAndDownloadBuildDependencies();
this._logger.save(this._currJob._id, 'Downloaded Build dependencies');

const docset = await this._docsetsRepo.getRepo(this._currJob.payload.repoName, this._currJob.payload.directory);
let env = this._config.get<string>('env');
Expand All @@ -595,12 +600,10 @@ export abstract class JobHandler {
job.payload.repoName,
job.payload.project,
baseUrl,
buildDependencies,
job.payload.directory
);
// Set patchId on payload for use in nextGenStage
this._currJob.payload.patchId = patchId;
this._logger.save(this._currJob._id, 'Downloaded Build dependencies');

let buildStepOutput: CliCommandResponse;

Expand Down
3 changes: 2 additions & 1 deletion src/repositories/repoBranchesRepository.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@ export class RepoBranchesRepository extends BaseRepository {
super(config, logger, 'RepoBranchesRepository', db.collection(config.get('repoBranchesCollection')));
}

async getBuildDependencies(repoName: string): Promise<BuildDependencies> {
async getBuildDependencies(repoName: string, directoryName?: string): Promise<BuildDependencies> {
const query = { repoName: repoName };
if (directoryName) query['directories.snooty_toml'] = `/${directoryName}`;
const repo = await this.findOne(
query,
`Mongo Timeout Error: Timedout while retrieving build dependencies for ${repoName}`
Expand Down
Loading