Distribution RFC (Milestone 1) #107
Conversation
rfcs/0002-distribution-gh.md
Outdated
| "source": { | ||
| "hash": "98801634b0e1396c933188eef88178952f412a8c", | ||
| } |
There was a problem hiding this comment.
Could you please describe the use of this stanza for me?
There was a problem hiding this comment.
Sure thing. I'll add clarification in the RFC, but basically the source link can reference any field in the sources stanza in order to show the provenance of a definition.
| * Add link and checksum to package definition archive to DB Schema | ||
| * Add link and checksum to package definition archive to Github metadata format |
There was a problem hiding this comment.
I am not sure if I understand this fully. Do you mean the git tag references in our current DB?
There was a problem hiding this comment.
This references updating the DB/schmea to be able to store and fetch the new packages stanza. Not sure of the changes required to do this.
rfcs/0002-distribution-gh.md
Outdated
| "checksum": { | ||
| "type": "sha256", | ||
| "value": "9283dh9283u...hd092ujdf2ued" | ||
| }, |
There was a problem hiding this comment.
we will probably need the information of whether a version of connector is of type PreBuiltDockerImage or ManagedDockerBuild.
|
|
||
| The database backing the API provides all of the APIs state management capabilities outside of package archive storage (as described in "Storage"). | ||
|
|
||
| The initial implementation of the Database will be an extension of the exiting hub registry Postgres instance. |
There was a problem hiding this comment.
correct! For the beta release, we have created an entirely new schema in the existing postgres DB that could be used now and in the future according to the distribution spec.
rfcs/0002-distribution-gh.md
Outdated
| "definition": { | ||
| "packagingDefinition": { | ||
| "type": "PrebuiltDockerImage" | ||
| "dockerImage": "hasura/ndc-clickhouse:v0.1.0" | ||
| }, | ||
| "supportedEnvironmentVariables": [ | ||
| { | ||
| "name": "username", | ||
| "description": "user name to access click house DB" | ||
| } | ||
| ], | ||
| "cliPlugin": { | ||
| // fill in the blanks | ||
| }, | ||
| ... | ||
| } |
There was a problem hiding this comment.
since this information is being already submitted via the package definition archive, should we omit it to avoid duplication? Right now, the two pieces of information that gets stored in the hub registry DB is packagingDefinition.type and packagingDefinition.dockerImage
There was a problem hiding this comment.
I thought it might be worth including in the hub since there's no way to browse or search for that information inside a tarball. This way it can be indexed and displayed without having to download the distribution first. IMO it's worth the extra storage.
There was a problem hiding this comment.
Maybe we store only those two pieces of information?
There was a problem hiding this comment.
I'd store all of it in a JSONB or something personally since you never know what information someone will want. What's the motivation for storing less? Space usage shouldn't be much of an issue.
There was a problem hiding this comment.
ah cool, sorry! Didn't notice your above comment. (GitHub failed in delivering that notification to my browser 😄)
There was a problem hiding this comment.
@imperfect-fourth If we store this definition in our DB as jsonb, will it help skip the tar unpacking step for PreBuildDockerImage type of connectors?
There was a problem hiding this comment.
if we store this definition in our DB as jsonb, will it help skip the tar unpacking step for PreBuildDockerImage type of connectors?
we are not doing this currently either since we have type and image columns in the connector table
rfcs/0002-distribution-gh.md
Outdated
| // Imported directly from the package definition archive - See: https://github.com/hasura/ndc-hub/pull/89 | ||
| "definition": { | ||
| "packagingDefinition": { | ||
| "type": "PrebuiltDockerImage" | ||
| "dockerImage": "hasura/ndc-clickhouse:v0.1.0" | ||
| }, | ||
| "supportedEnvironmentVariables": [ | ||
| { | ||
| "name": "username", | ||
| "description": "user name to access click house DB" | ||
| } | ||
| ], | ||
| "cliPlugin": { | ||
| // fill in the blanks | ||
| }, | ||
| ... |
There was a problem hiding this comment.
let us remove this as we will be directly uploading from the package.tgz to db if needed
There was a problem hiding this comment.
Feel free to merge after addressing https://github.com/hasura/ndc-hub/pull/107/files#r1520856482
RFC for distribution aspects of connectors.
Related specifications:
While the precursor specifications outline the structure and mechanisms of packaging, this RFC outlines the initial distribution of package definitions.
See the Rendered Version of the RFC here.