Pagination: Limit, Offset & downloadCSV

[bh2smith]

This PR makes quite a few changes to the routing while adding limit and offset as "GetParameters".

Unit Tests are introduced validating that limits are respected where possible.

[msf]

doesn't this mean that you're blowing up on memory usage because you created leftData and rightData with full copies and a lot more overhead of ts object per line?

[msf]

I've asked an LLM about this (because I'm not good at ts) and this is his answer:

import { createReadStream, createWriteStream } from 'fs';

export async function concatCSVs(leftPath: string, rightPath: string) {

  const output = createWriteStream('out.csv');

  const writeStream = (fileStream: NodeJS.ReadableStream) => {
    fileStream.pipe(output, { end: false });
  }

  const leftStream = createReadStream(leftPath);
  writeStream(leftStream);

  const rightStream = createReadStream(rightPath);
  rightStream.once('data', () => {
    // pause stream after first chunk which contains header
    rightStream.pause(); 
  });

  // resume stream after header
  rightStream.once('end', () => {
    rightStream.resume();
  });

  writeStream(rightStream);

  await finishStreams([leftStream, rightStream]);

  output.end();
}

const finishStreams = (streams: NodeJS.ReadableStream[]) => {
  return Promise.all(
    streams.map(stream => {
      return new Promise((resolve) => stream.on('end', resolve));
    })
  );
}

a bit overkill, I wonder if we can avoid the streams...

[bh2smith]

So I looked into this a bit and the code that was shared here is for concatenating two csv files (for which one can create streams). However, we have strings and the code proposed with this information gave basically the same thing as is here:

function concatCSVStrings(firstCSV: string, secondCSV: string): string {
    // Split the first and second CSV strings into arrays of lines
    const firstLines = firstCSV.split('\n');
    const secondLines = secondCSV.split('\n');

    // Remove the header from the second CSV (assuming the first line is the header)
    secondLines.shift();

    // Concatenate the first CSV lines with the modified second CSV lines
    const allLines = firstLines.concat(secondLines);

    // Join all lines back into a single CSV string
    const outputCSV = allLines.join('\n');

    return outputCSV;
}

// Example usage
const firstCSV = "header1,header2\nrow1_col1,row1_col2\nrow2_col1,row2_col2";
const secondCSV = "header1,header2\nrow3_col1,row3_col2\nrow4_col1,row4_col2";

const concatenatedCSV = concatCSVStrings(firstCSV, secondCSV);

console.log(concatenatedCSV);

The memory I expect to be consumed is bounded above by the total result set size. One improvement we could make here is to write the file one page at a time and continue appending to it. Although I am not so sure that downloadCSV (and write to file) is the entire purpose of this method.

[msf]

there should be a way of doing:

const data = discardLines(right.data, 1) // consume header line, return the remaining
const total = stringConcat(left.data, right.data)

I just wonder how much this will amplify the memory consumption..
maybe this is a nitpick?