feat: adds streaming query

[nathanielc]

No description provided.

[nathanielc]

Here is an example using the new SDK streamQuery function.

import { CeramicClient } from '@ceramic-sdk/http-client'
import type { ModelDefinition } from '@ceramic-sdk/model-protocol'
import { ModelClient } from '@ceramic-sdk/model-client'
import { ModelInstanceClient } from '@ceramic-sdk/model-instance-client'
import { getAuthenticatedDID } from '@didtools/key-did'
import { ClientOptions, createFlightSqlClient } from '@ceramic-sdk/flight-sql-client';
import { tableFromIPC } from 'apache-arrow';



async function main() {
  await Promise.all([read(), write()])
}

async function write() {
  const client = new CeramicClient({ url: "http://localhost:5101" })

  // Create an authenticated DID
  const authenticatedDID = await getAuthenticatedDID(new Uint8Array(32))

  // Create a model client
  const modelClient = new ModelClient({
    ceramic: client,
    did: authenticatedDID,
  })

  const model: ModelDefinition = {
    version: '2.0',
    name: 'Post',
    description: 'test model',
    accountRelation: { type: 'list' },
    interface: false,
    implements: [],
    schema: {
      type: 'object',
      properties: {
        title: { type: 'string', maxLength: 12 },
        body: { type: 'string', maxLength: 1024 },
        nonce: { type: 'number' },
      },
      required: ['userName'],
      additionalProperties: false,
    },
  };
  const modelStream = await modelClient.postDefinition(model);
  // Create a model instance client
  const modelInstanceClient = new ModelInstanceClient({
    ceramic: client,
    did: authenticatedDID,
  })
  while (true) {
    // creating a MID using a `list` model
    await modelInstanceClient.postSignedInit({
      model: modelStream,
      content: { title: 'This is a new post', body: 'This is the body for a new post', nonce: Math.random() },
      shouldIndex: true,
    })
    await sleep(1000)
  }
}

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

async function read() {
  const options: ClientOptions = {
    username: undefined,
    password: undefined,
    tls: false,
    host: '127.0.0.1',
    port: 5102,
    headers: [],
  };
  const client = await createFlightSqlClient(options);
  const stream = await client.streamQuery(`
    SELECT
      "index",
      cid_string(event_cid) as event_cid,
      data::varchar as data
    FROM conclusion_events_stream
    WHERE
    "index" > 43
  `);
  while (true) {
    const batch = await stream.next();
    if (batch == null) {
      break;
    }
    const table = tableFromIPC(batch);
    for (let i = 0; i < table.numRows; i++) {
      const row = table.get(i);
      console.log(row);
    }
  }
  console.log('read finished');
}

main()

[dav1do]

Other than fixing/understanding the package.json changes and CI, this looks good to me.

[nathanielc]

@dav1do This is ready for another review. Summary of recent changes.

1. Tests for feed queries
2. Support for prepared feed queries with tests
3. Fixed logging from flight-sql-client rust code
4. Changed tests to not sleep and instead use flight-sql-client to wait till the specific event has been aggregated
5. Added support for binary parameters my multibase encoding them.
6. Update tests to be less interdependent. Tests still fail if run in parallel but they no longer make assertions about actions previous tests made
7. Changed default image to one with debug symbols

[nathanielc]

Explicitly pass the target so that we can use external tooling that does the same.

[nathanielc]

All of these are removed as eslint is not configured as a dependency and the workspace is configured to use biome which lints the whole repo so individual package linting is not needed.

[dav1do]

Small comments/questions but this looks great.

[dav1do]

does this panic if you call it twice? wondering if we should ignore the result of tracing::subscriber::set_global_default or if this is fine

[nathanielc]

Setting it a second time will fail but we ignore the failure, its a returned result not a panic. Therefore we should be fine.

[dav1do]

better 👍

[dav1do]

it looks like all these tests use the same options. you could define this somewhere shared (index.ts in this case) and import them but it's also kind of nice to keep config close to tests