Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add more info for ndjson errors #157

Merged
merged 8 commits into from
Sep 5, 2024
Prev Previous commit
Next Next commit
reworked skipping of Parameters resource to handle starting with empt…
…y lines. added comments to explain some work
hossenlopp committed Aug 28, 2024
commit 3cd652d4ab5bde146ed1a5ca311a2084effa49e2
13 changes: 11 additions & 2 deletions src/server/ndjsonWorker.js
Original file line number Diff line number Diff line change
@@ -55,6 +55,9 @@ ndjsonWorker.process(async job => {

const ndjsonLines = ndjsonResources.split(/\n/);

// keep track of when we hit the first non-empty line
let hitNonEmpty = false;
lmd59 marked this conversation as resolved.
Show resolved Hide resolved

const insertions = ndjsonLines.map(async (resourceStr, index) => {
resourceStr = resourceStr.trim();

@@ -65,16 +68,22 @@ ndjsonWorker.process(async job => {

// attempt to parse the line
try {
// capture the value of if we already hit a non empty line incase we can parse this resource
const wasNotEmptyHit = hitNonEmpty;
// set this to true now that we have reached a non empty line
hitNonEmpty = true;

const data = JSON.parse(resourceStr);

// check if first line is Parameters header and skip it
if (index === 0 && data.resourceType === 'Parameters') {
// check if first non empty line is a Parameters header and skip it
if (!wasNotEmptyHit && data.resourceType === 'Parameters') {
lmd59 marked this conversation as resolved.
Show resolved Hide resolved
return null;
}

checkSupportedResource(data.resourceType);
return updateResource(data.id, data, data.resourceType);
} catch (e) {
// Rethrow the error with info on the line number. This fails the async promise and will be collected later.
throw new Error(`Failed to process entry at row ${index + 1}: ${e.issue?.[0]?.details?.text ?? e.message}`);
}
});
Loading