Add support to retrieve the observable events from the contracts in a given Clarinet project

[sabbyanandan]

As a user, I want the ability to discover all observable chainhook events in a given project so I can look at the list of events and create the chainhooks easily as opposed to manually defining each predicate by reading the docs, making errors in the process, and wasting so much time.

The rest of additional context is at https://github.com/hirosystems/platform/issues/546, and I am skipping it for brevity here.

Previous context:
We want to use clarity-events in other places, so that requires making this component a JavaScript-readied WASM library.

[lgalabru]

We probably want to see a bunch of our libraries being exported to wasm.
@hugocaillard what's your take on introducing a component clarinet-sdk, that would essentially be some sort of umbrella component, along with exposing a few wasm entrypoints.
This SDK could be published both on crates.io and npm.
The platform would consume the wasm version available on npm (for the clarity-events, the deployment plans, and maybe other future features like formatter, etc).
I guess the Vs code extension could potentially also become a consumer?

[hugocaillard]

Totally makes sense.
Non exhaustive list of WASM components that could be expose

• deployment plan helpers (for the platform)
• clarity-events
• get contract interface (cf [clarity-repl] Expose method to construct JSON abi from contract source #913)
• lsp bridge

With this approach, the clarinet-sdk would be the only one requiring wasm-bindgen

Although I'm not sure of the need to publish anything on crates.io? What's the use case?

I guess the Vs code extension could potentially also become a consumer?

Yes

[andresgalante]

This work will be a dependency for https://github.com/hirosystems/platform/issues/546

[hugocaillard]

Sounds like we can tackle this in Q3 (September) with the progress made on the JS clarinet-sdk 🙌

[hugocaillard]

Hey @sabbyanandan
I wanted to point out that the clarinet-sdk is mainly used to be used within Node.js, but not in the browser.
So as it is, it could be used on the back-end but not on the front-end. We could do so, but it would also mean for the platform to provide a way for the SDK to "read" the files (contracts and Clarinet.toml). Within Node, it relies on Node FS, but in the browser, it could rely on a virtual file system (provided by the platform).
What was the plan with this package?

@lgalabru also mentioned to possibility to get the events from the devnet package (I think it already returns the ABI, it could also have the events)

[lgalabru]

The ABI is unfortunately not part of the packaged plans, but this is something we can address, and would make sense to me, so I'm going re-purpose this issue.
@csgui do you think you could help us with that?

[hugocaillard]

Also, I'm still interesting in adding it in the JS SDK, but just wanted to warn that it won't be available in the browser (at least not out of the box)

[csgui]

@csgui do you think you could help us with that?

Sure thing! 👍

[smcclellan]

Idea: New command clarinet deployments package - @beguene to review technical implementation spec that @csgui writes up here.

[csgui]

After working on an initial implementation we have the following proposal to the specification:

Leveraging the use of the Clarinet contracts command, we can add a sub-command abi. The output can be a JSON format with both event and function types. Thus, the JSON output can be handled by the client, i.e.: filtering by type value.

Any feedback on that? @sabbyanandan @beguene @hugocaillard @lgalabru @MicaiahReid

(print "hello!")

(define-public (say-hello) 
    (ok u1))

;; ---------------------------
 
(define-private (transfer-to-recipient! (recipient principal) (amount uint))
  (if (is-eq (mod amount u10) u0)
      (stx-transfer? amount tx-sender recipient)
      (err u400)))

$ clarinet contracts abi contract-file.clar

[
  {
    "trigger": "say-hello",
    "name": "print",
    "data_type": {
      "value": "hello!",
      "type": "string"
    },
    "type": "event"
  },
  {
    "trigger": "transfer-to-recipient!",
    "name": "transfer_stx_event",
    "type": "event"
  },
  {
    "name": "transfer-to-recipient!",
    "access": "private",
    "args": [
      {
        "type": "principal",
        "name": "recipient"
      },
      {
        "type": "uint",
        "name": "amount"
      }
    ],
    "outputs": [
      {
        "type": "response",
        "value": {
          "ok": "bool",
          "err": "string"
        }
      }
    ],
    "type": "function"
  },
  {
    "name": "say-hello",
    "access": "public",
    "args": [],
    "outputs": [
      {
        "type": "string"
      }
    ],
    "type": "function"
  }
]

[hugocaillard]

Some thoughts about the data structure.

What about splitting the ABI and events like so:

{ events: [], functions: [] }

Or even having two commands

clarinet contract interfaces <contract-name>
clarinet contract events <contract-name>

(<contract-name> could be optional and return the data for all contract when omitted)

I think it also depends on the platform needs, @sabbyanandan

• Will the platform always need events and interfaces at the same time or is it 2 use cases?
• In clarinet, the contract interface exposes the functions, as well as the variables, maps, FT, NFT, is this data needed on the platform? Anyway, it might be useful to have it in the future

@csgui

[andresgalante]

@beguene @sabbyanandan would you mind reviewing the json output and adding feedback on this issue, please?

[sabbyanandan]

Hey @sabbyanandan
I wanted to point out that the clarinet-sdk is mainly used to be used within Node.js, but not in the browser.
So as it is, it could be used on the back-end but not on the front-end. We could do so, but it would also mean for the platform to provide a way for the SDK to "read" the files (contracts and Clarinet.toml). Within Node, it relies on Node FS, but in the browser, it could rely on a virtual file system (provided by the platform).
What was the plan with this package?

@hugocaillard: My apologies for the delayed response. The DevEx and the use-case requirements for why clarity-events is useful (in the Platform) is explained at https://github.com/hirosystems/platform/issues/546 — it doesn't have to be a WASM module; as far as we have a lib, we can use that in the Platform backend to hopefully extract the event-streams for a given set of contracts in the project. And that's all we need.

[sabbyanandan]

Any feedback on that? @sabbyanandan @beguene @hugocaillard @lgalabru @MicaiahReid

@csgui: If I understand the data structure correctly, it appears we have a few different ways to get the "possible" event-streams for a given contract.
(1) using the clarity-events package/lib directly in code
(2) using the clarinet contracts abi or clarinet contracts events commands in Clarinet
(3) using the RESTful API backing the Clarinet commands in (2)

Did I get this right?

Regardless of these options (it looks like we can get the data multiple ways, anyway), the goal for the Platform is to discover the possible event streams automatically and offer an improved DevEx for developers to select and create Chainhooks rapidly instead of manually drafting each Chainhook. I hope this clarifies the ask.

For further context into Chainhook improvements in the Platform, see here.

[beguene]

@csgui I think the output format you described makes sense, it's perfectly fine to have everything in a plain array. We can filter in our app if required.
Agree with @sabbyanandan: as such we don't necessarily need a wasm as we have a node backend and a pod for each user with clarinet installed. Though having a clarity-events nodejs package will be easier and more performant for the platform.

[hugocaillard]

@beguene

Though having a clarity-events nodejs package will be easier and more performant for the platform.

So it looks like the first idea of having it on the clarinet-sdk still make sense.

Would something like that work for you:

import { initVM } from "@hirosystems/clarinet-sdk";

const vm = await initVM();

const contractInterfaces = vm.getContractsInterfaces()
// contractInterfaces interfaces is a map of all contract interfaces

const counterInterface = contractInterfaces.get(`${deployerAddr}.counter`);
console.log(counterInterface?.functions) // array of the functions

// not implemented yet / tbc
const events = vm.getContractEvent(`${deployerAddr}.counter`);

The clarinet-sdk is basically Clarinet wrapped in WASM. Also right now the package only work in Node.js (not in the browser), but that's something we could work on

[beguene]

@hugocaillard Yes this should work.
Is initVM compute or memory intensive ?
Looks like it's expecting a Clarinet.toml. Does this mean we also need all the contracts files in a well structured project like this ?
If that's the case, we can't really do that in our backend but we can do it on the user's pod where the project folder and Clarinet.toml lie.

[csgui]

Does this mean we also need all the contracts files in a well structured...

Would be better to follow a pattern, like contract files under a contracts folder. And the contracts folder in the same place where the Manifest file (Clarinet.toml) lives.

[csgui]

@hugocaillard Thinking about the platform requirements seems that makes more sense to implement this on the clarinet-sdk.

[hugocaillard]

Long story short, the manifest file (Clarinet.toml) allows to get the exact path of each contracts along with the setting (clarity_version, epoch) that could change the result of the ABI.
Also, because contracts call other contract (from the same project or from requirements from mainnet), it's necessary to load the whole session to know the ABI of a contract.

So the Clarinet.toml is a source of truth and a good way to make that our tools behave the same way.

Is initVM compute or memory intensive? - @beguene

It is slightly, it also depends on the size of the project. But it basically what happens every time the Clarity VSCode extension starts.

Looks like it's expecting a Clarinet.toml.

Yes

Does this mean we also need all the contracts files in a well structured project like this? - @beguene

Not necessarily, because the contracts path are specified in the manifest (Clarinet.toml), allowing for flexibility. (sbtc is using multiple directories)

[contracts.hello-world-v1]
path = "contract-v1-directory/hello-world.clar" # dir 1
clarity_version = 1
epoch = 2.0

[contracts.hello-world-v2]
path = "contract-v2-directory/hello-world.clar" # dir 2
clarity_version = 2
epoch = 2.4

[contracts.hello-world-tests]
path = "some-other-directory/hello-world.clar" # dir 3
clarity_version = 2
epoch = 2.4

Would be better to follow a pattern, like contract files under a contracts folder. - @csgui

I'm not sure if we want to add this constraint to Clarity developers on the platform, having the Clarinet.toml as a source of truth to list contract and get their paths and settings (epoch / clarity_version / maybe more in the future) is quite useful.

---

Ideas for the future

• The sdk could actually be usable in the browser, it does need some work but definitely doable if we want to prioritize that
• For flexibility, we could allows to boot an empty VM. The VM could also return the manifest content (to know contracts paths and settings). And one could manually deploy arbitrary contract (with vm.deployContract()). But it would end up doing basically stuff that the clarinet-sdk is currently doing.

[hugocaillard]

@csgui @beguene

Thinking about the platform requirements seems that makes more sense to implement this on the clarinet-sdk.

Yes it does 👍

So the ABI is already available if you want to start implementing that @beguene

const contractInterfaces = vm.getContractsInterfaces()

I can also prioritize implement contract events, it shouldn't be too long. What's your timeline @beguene @sabbyanandan?

(I could implement it this week it could make it the v1 or v1.1)

[beguene]

Got it @hugocaillard ! Sounds good to me. We anyway rely a lot on the Clarinet.toml for deployment plan and devnet so we can continue like this.
Regarding the folder, I meant we must have a file system to use const vm = await initVM();.
Hence we won't do it in the platform backend but on the user vscode pod where all the user's projects and contracts are persisted (just like we do now for deployment plan and devnet) cc @BLuEScioN

[hugocaillard]

@beguene Ok I see.

In the vscode pod, do you have access to the actual FS (ie: node:fs will work out of the box) or is it the VSCode virtual file system?

[beguene]

@hugocaillard We do have the actual filesystem so everything should work fine. We access it via a custom nodejs server and indeed we use node:fs to interact with the user's clarity project files.

[hugocaillard]

@beguene ok perfect, thanks.
So two questions on my end:

• what would be ideal deadline for your to have access to the event with the sdk?
• could you work on POC just to make sure that the sdk and initVM work in the pods?

Thx

[sabbyanandan]

@hugocaillard: Nakamoto/sBTC is the immediate priority, and access to events via the SDK can take a back seat until you free up. There is no rush; the Platform can pick it up whenever it is ready (even in a milestone capacity versus a full-blown release).

[lgalabru]

We anyway rely a lot on the Clarinet.toml for deployment plan and devnet so we can continue like this.

@beguene this makes me think that the Platform team could be missing a feature that we shipped in v1.8: packaged deployment plans.
Thanks to this new command, you should be able to dump the entire project (including contracts, wallets) in one json file that you can use for executing the deployment plan - which should probably help you delete a lot of code on your end.

[beguene]

Nice! This is good to know. Yes, we missed it because we started working on deployment plan before this was available, but we use the devnet package command for our devnet feature, which is indeed cleaner. We will refactor this when we touch the deployment plan again.
Those features in the clarinet are precious and make it easy for us to integrate into the platform.

[hugocaillard]

Will try to do it in October in sdk v1.1.
Lower the the priority to medium

[sabbyanandan]

@smcclellan: Once 2.2 ships, could we timebox to review the scope and estimate this effort?

There are additional signals from devs using Chainhooks, but they also create things manually, which we can entirely automate once we automatically provide visibility into observable events.

[smcclellan]

@sabbyanandan Yes, this is on my radar. Thanks for flagging.

[smcclellan]

@hugocaillard to clarify the approach, possibly opening a new ticket to track the new scope.

[hugocaillard]

@sabbyanandan Considering all of the noise that happened on this issue, could we open a new one instead?

Ideally, I'd like the new issue to focus on the "what" / "why" instead of the "how".
(eg, instead of "Include ABI + and Clarity events analysis in packaged deployment plans", we could have "platform need access to ABI and events analysis for ...")

[sabbyanandan]

@hugocaillard: I would repurpose this issue; on that note, I updated the title and description. I would appreciate it if we could scope this first and then decide on the timeline feasibility. This can be impactful for the Platform users.

[hugocaillard]

Thanks @sabbyanandan
Have you considered opening a new issue as I suggested in my previous comment?
I'll provide some details later today

[smcclellan]

Closing in favor of #1410 to simplify the thread. /cc @sabbyanandan