[WIP] Generic provider template #1140
Conversation
| - name: Download schema-embed.json | ||
| uses: #{{ .Config.ActionVersions.DownloadArtifact }}# | ||
| with: | ||
| # Use a pattern to avoid failing if the artifact doesn't exist | ||
| pattern: schema-embed.* | ||
| # Avoid creating directories for each artifact | ||
| merge-multiple: true | ||
| path: provider/cmd/pulumi-resource-#{{ .Config.Provider }}#/schema-embed.json |
There was a problem hiding this comment.
Not all providers use schema-embed right now. It's only realy a performance optimisation which could be implemented internally within the provider rather than needing CI to have insight of.
We mostly still have the original schema commited (though this has been taken out Azure Native due to the file being too big for GitHub, and LFS causing a load of other problems) but I think it would be more correct to treat the schema like a build artifact as you're doing here, though it would be nicer to not assume the Go provider directory structure if possible within the CI job. I think this could be abstracted by the makefile if needed.
- Prerequisites writes all artifacts to the bin directory including the embeddable schema if it needs it.
- Restore the bin directory in this subsequent job.
- Within the provider_dist target, copy the embeddable schema into the provider directory before building.
| merge-multiple: true | ||
| path: provider/cmd/pulumi-resource-#{{ .Config.Provider }}#/schema-embed.json | ||
| - name: Build & package provider | ||
| run: make provider_dist-${{ matrix.platform.os }}-${{ matrix.platform.arch }} |
There was a problem hiding this comment.
Perhaps we should use the file target here to match the path we're going to upload rather than having the assumption of the path this phony target will write to:
make dist/pulumi-resource-#{{ .Config.Provider }}#-v${{ inputs.version }}-${{ matrix.platform.os }}-${{ matrix.platform.arch }}.tar.gz
| - name: Install plugins | ||
| run: make install_plugins |
There was a problem hiding this comment.
This is a bit of an odd step. I know for TF providers it's related to installing other bridged TF providers which the bridge will use for examples generation, but for a generic template we might want something a little more generic such as make prepare or something that allways runs at the start of the job before restoring any targets or doing make --touch ... etc.
There was a problem hiding this comment.
Yeah, I was also wondering about this because the CLI should automatically download these plugins anyway. Related - these plugins eat a ton of space! pulumi/pulumi#17718
There was a problem hiding this comment.
I think for bridged providers, if they're opted into using the converter plugin, then it explicitly disables auto-download because we need to have the provider version pinned, so failing fast is good. For generic providers, perhaps we can just let the provider figure out what it needs and how it will install them internally within the makefile.
| # otherwise use the current SHA for any other type of build. | ||
| sha: ${{ github.event.pull_request.head.sha || github.sha }} | ||
|
|
||
| # TODO: Extract into shared action. |
| @@ -0,0 +1,48 @@ | |||
| # WARNING: This file is autogenerated - changes will be overwritten if not made via https://github.com/pulumi/ci-mgmt | |||
|
|
|||
| name: license_check | |||
There was a problem hiding this comment.
I think this should already be included from the provider template .. unless there's something subtle we're tweaking here I didn't spot?
There was a problem hiding this comment.
Ah yeah, I duplicated it here because of #1133
| case "generic": | ||
| return []string{"provider", "pulumi-provider", "generic"}, nil |
There was a problem hiding this comment.
When we use this template for the native boilerplate, we'll probably add external-generic which will exclude the pulumi-provider sub-template.
There was a problem hiding this comment.
Or maybe "third-party"?
There was a problem hiding this comment.
Just above is the external-bridged-provider template which is like bridged-provider but with bits removed for third parties. Whatever name we use should aim to keep these consistent.
This is a very WIP fork of our bridged workflows which is intended to be used by native, component, or other miscellaneous providers not currently managed by ci-mgmt.
There are two primary design principles which deviate from the bridged template:
maketargets define the entire contract between a provider andci-mgmt. We do not assume anything about the provider, e.g. it could be implemented in a language other than Go.ci-mgmtis responsible for exposing the appropriate hooks (Allowmaketarget customization #1131).For example, if the provider needs some additional tooling then it is responsible for ensuring the tool is installed. Historically these sorts of customizations have been handled as one-offs in
ci-mgmt, but we want to push this down to themaketarget or into the test itself. Not only does this makeci-mgmteasier to maintain and refactor, but it also forces us to write tests that are more easily reproducible in local environments. The SSH key handling between the docker and docker-build providers is illustrative.I've been testing this in EKS and haven't gotten tests passing reliably yet. I was planning on consolidating the test into a shared action once I got that far, but also feel free to do that sooner than later!
EKS was already sharding tests so I've opted to keep that behavior here -- a provider just needs to implement a
test_shardtarget which takes aTESTSandTAGenv var. I've tweaked it somewhat so the sharding is totally arbitrary, which in practice just means we need to install all the SDKs instead of one at a time.Other differences:
TODO: