Web Crypto API

[ajayvignesh01]

Can we use the web crypto api across the board instead of node crypto? This could make this library fully compatible not only with traditional node environments and browsers, but also relatively newer runtimes geared towards edge functions such as deno, cloudflare workers, etc that don't support all of the node apis such as crypto.

So instead of using something like the following to generate the signature:

import { createHmac } from 'crypto' // +427.266kB
const signature = createHmac('sha256', api_secret).update(param_str).digest('hex')

We would use something like this:

const encoder = new TextEncoder()
const signature = await crypto.subtle
  .importKey('raw', encoder.encode(api_secret), { name: 'HMAC', hash: 'SHA-256' }, false, ['sign'])
  .then(async (key) => await crypto.subtle.sign('HMAC', key, encoder.encode(param_str)))
  .then((buffer) => Array.from(new Uint8Array(buffer)).map((byte) => byte.toString(16).padStart(2, '0')).join(''))

Currently, we are not able to use this package on cloudflare workers. On Deno, we're able to use 3.8.2 (nothing above) thanks to it's partial node support, but even that displays logs saying that something the library uses will be deprecated in a later release of the deno runtime (which I think is related to node:crypto).

The cost implications of being limited to node are also note-worthy. Our calculations estimate ~$150 to run a function 1M times for 10 seconds. On deno and cloudflare workers, our cost is $2 and $1.30 respectively for the similar scenario while also being notably faster due to a low-weight runtime + almost no cold starts.

Due to this, we created a custom implementation similar to this library but with the fetch api, web crypto and 0 dependencies. Our bundle size is ~3kB (no websocket support). Fetch api could also be looked into to replace axios, but I have a feeling people are likely still running node versions earlier than 18, when fetch became stable. Considering, crypto.subtle became stable in node 16, might be worth switching over.

[tiagosiebler]

Absolutely! My biggest concern here is broad compatibility, but since crypto.subtle became stable in node 16, hopefully that already is compatible for almost everyone. They can always use an older version of the SDK if they're not ready to upgrade yet. I'll do some testing here to evaluate it works as needed. Thank you for putting this together.

Regarding axios, given the ability to simply inject an inherited axios config into the REST clients, it's possible/likely a number of axios features are being used by some of the userbase. At the very least, proxy usage is definitely one common feature (CI/CD also uses it in all my repos, thanks to IP blocks to US IPs where action runners are typically hosted). Would be curious to better understand your motivations in replacing axios with fetch eventually.

[ajayvignesh01]

Awesome! I happened to stumble upon this earlier issue from 2 years ago suggesting to use crypto.subtle instead of crypto-browserify for browser environments. Maybe you could use some of that testing for this as it's almost the exact same thing. Would just need to test in node and edge environments in that case.

#90

As for fetch vs axios, that's a good point, didn't think of that. I don't think the native fetch api has support for proxies like axios. We're US based as well, but just use our backend eu-central servers to interact with ByBit's API. My motivation for using fetch is mainly because Cloudflare Workers don't support axios, which uses a lot of node specific apis that cloudflare doesn't support. I wonder if there's a viable solution to proxy using native fetch. Will get on this. Are there any axios specific features people use a lot?

Also, curious, what recommendation do you have for setting up a proxy server? I'm thinking of maybe using some setup involving it to fetch currently open user trades from the client-side. Having a serverless function fire up for this every time gets pricey pretty quick.

[tiagosiebler]

Definitely useful, thank you for sharing this.

Regarding axios specific features, I'm primarily aware of:

• proxies
• custom http user agents
• custom headers

Not much else comes to mind. Curious about how I could support cloudflare workers the way that you need them without a breaking change. An under-the-hood switch for using fetch in place of axios perhaps, but then there's still the import from axios (which ideally has to happen on the top level, not within a function call somewhere).

Regarding setting up proxies, depends on your needs. If you can delegate your API calls to the browser (especially if you're querying account data, not submitting orders), I would try to do so as that is the cheapest and most scalable solution. With read-only API keys, most exchanges won't be strict about IP whitelisting. For everything else, you would either need to host your own (perhaps a cheap way to deploy tiny proxy-hosting docker containers and then a load balancing mechanism that allocates requests to specific IPs/containers), or use proxy services hosted by 3rd parties.

Luminati (now brightdata) was a popular proxy provider, but they're now against crypto trading via their proxies (one of their BDs pointed me towards their terms & conditions). It may work but if they catch wind, you may face service interruption: https://brightdata.com/luminati

There are other services out there for relatively cheap data-center proxies, but I haven't tried them. Would also be curious to hear what experience others had with different providers.

Btw, if you're on telegram I highly recommend joining my engineering community: https://t.me/nodetraders

I'm sure others in the group also have recommendations and it would be an interesting discussion.

[tiagosiebler]

Currently evaluating signing using web crypto vs node crypto. Functionally both solutions look consistent. I don't see any reason why the web crypto api wouldn't work here. I do have performance concerns though.

Before even looking at hex or base64 encoding, just running a sha256 sign on a piece of text using a secret is 15x faster using node crypto. It's consistently avg 0.004ms for the current node sign on my laptop (across 10k test runs per "method"), vs 0.061ms with the web crypto api:

  const RUNS = 10000;

  const tNode1 = performance.now();
  for (let i = 0; i < RUNS; i++) {
    const hmac = createHmac('sha256', secret).update(testMsg);
  }
  const tNode2 = performance.now();
  const diffNode = tNode2 - tNode1;
  const msPerRunNode = diffNode / RUNS;

  const tWeb1 = performance.now();
  for (let i = 0; i < RUNS; i++) {
    const encoder = new TextEncoder();
    const key = await globalThis.crypto.subtle.importKey(
      'raw',
      encoder.encode(secret),
      { name: 'HMAC', hash: { name: 'SHA-256' } },
      false,
      ['sign'],
    );

    const signature = await globalThis.crypto.subtle.sign(
      'HMAC',
      key,
      encoder.encode(testMsg),
    );
  }
  const tWeb2 = performance.now();
  const diffWeb = tWeb2 - tWeb1;
  const msPerRunWeb = diffWeb / RUNS;

  console.log(`diffs: `, {
    diffNode,
    msPerRunNode,
    diffWeb,
    msPerRunWeb,
  });

Seems I can make it 3x faster (but still 6.5x slower than node) by preparing the "key" once and caching it, instead of preparing the key each time a sign is about to happen:

  const tWeb21 = performance.now();

  const encoder = new TextEncoder();
  const key = await globalThis.crypto.subtle.importKey(
    'raw',
    encoder.encode(secret),
    { name: 'HMAC', hash: { name: 'SHA-256' } },
    false,
    ['sign'],
  );

  for (let i = 0; i < RUNS; i++) {
    const signature = await globalThis.crypto.subtle.sign(
      'HMAC',
      key,
      encoder.encode(testMsg),
    );
  }
  const tWeb22 = performance.now();
  const diffWeb2 = tWeb22 - tWeb21;
  const msPerRunWeb2 = diffWeb2 / RUNS;

  console.log(`diffs: `, {
    diffWeb2,
    msPerRunWeb2,
  });

But this would only deal with part of the difference, and it wouldn't give any improvement for a single-fire cold start API call.

It's relatively tiny either way, but the difference could be significant for some that are using my libraries over others for performance reasons (I've had comments in the past). It's not a deal breaker although I would prefer a solution that caters to bother types of systems.

One idea that immediately comes to mind is using the universal web crypto api by default (since that carries no import burden) but allowing a custom signMessage (with a copy paste example that uses the much faster node version). I could add an optional function into the RestClientOptions for example, which anyone could include when instancing any of my clients.

This would eliminate the problematic import from node's crypto module, which causes problems here and in the browser, while still offering a solution to those with more advanced & latency needs. Feels a bit dirty but might be a decent compromise... If you (or anyone else reading this) have other ideas, please let me know.

[tiagosiebler]

Could look something like this to inject a custom sign method (ignore the differences to the bybit sdk, this is a different not-yet-released sdk for another exchange):

Maybe a decent compromise, as long as it's well documented in the next major release.

[ajayvignesh01]

• Web Crypto API #321 (comment)
  Good points. Think it would be a headache to replace axios in that case. I primarily just need to get the user's current open trades when they login. Either through webhook, or one time fetch. Proxy might be the cheapest, but I'm also considering setting up a cheap server and running deno oak (similar to express) on it. Very light footprint compared to express, and especially on arm64. Will join the telegram!

#321 (comment)
That's a good idea to prepare the key before-hand. I haven't really tested the performance implcations of the approaches, but it seems like a small enough number to not matter too much?

#321 (comment)
This is great, perfect solution honestly. 100% for the documentation! Is that for Blofin by any chance?

And as for support Cloudflare specifically, there are also a couple of other factors to consider that might make it not worth the effort to think of a solution for axios vs fetch. Main one being that there's not explicit way to make the cloudflare worker trigger from a specific region instead of closest to the use. So this would be hit with geo blocks. There's a hacky work around, but it's hidden in a forum somewhere and I doubt the masses are going find it, or use it.

Vercel uses a similar runtime to cloudflare for their edge compute network, but you can specify which region to run it in. But I think people could just go about just making a normal fetch request tbh. Could just provide code snippets in the documentation on how to generate signature and headers to make it easier.

For example:

async function genSignature({
    payload = '',
    timestamp = `${Date.now()}`,
    recv_window = '5000'
  }: Signature = {}) {
    const param_str = timestamp + this.key + recv_window + payload
    const encoder = new TextEncoder()
    const signature = await crypto.subtle
      .importKey('raw', encoder.encode(this.secret), { name: 'HMAC', hash: 'SHA-256' }, false, [
        'sign'
      ])
      .then(async (key) => await crypto.subtle.sign('HMAC', key, encoder.encode(param_str)))
      .then((buffer) =>
        Array.from(new Uint8Array(buffer))
          .map((byte) => byte.toString(16).padStart(2, '0'))
          .join('')
      )

    // import { createHmac } from 'node:crypto' // +427.266kB
    // const signature = createHmac('sha256', this.secret).update(param_str).digest('hex')

    return { signature, recv_window, timestamp }
  }

function genHeaders({ signature, timestamp = `${Date.now()}`, recv_window = '5000' }: Header) {
    return {
      'X-BAPI-SIGN-TYPE': '2',
      'X-BAPI-SIGN': signature,
      'X-BAPI-API-KEY': this.key,
      'X-BAPI-TIMESTAMP': timestamp,
      'X-BAPI-RECV-WINDOW': recv_window,
      'Content-Type': 'application/json'
    }
  }