[Feature] Publish plugins SNAPSHOT artifacts (along with SNASPHOT distribution) #5096

reta · 2024-10-11T19:23:00Z

Is your feature request related to a problem? Please describe

At the moment, we do not publish plugins SNAPSHOT artifacts but min distribution only (see please https://build.ci.opensearch.org/job/publish-opensearch-min-snapshots/). However, it becomes increasingly demanded to have plugins SNAPSHOT artifacts published as well.

Describe the solution you'd like

Publish plugins SNAPSHOT artifacts under S3 bucket https://artifacts.opensearch.org/snapshots/plugins/ so OpenSearch tooling (opensearch-plugin, etc) could be used to install them for SNAPSHOT distributions.

Describe alternatives you've considered

Publish ZIPs to Apache Maven but that would change completely the distribution model for releases and snapshots.

Additional context

See please opensearch-project/OpenSearch#16219

The text was updated successfully, but these errors were encountered:

reta · 2024-10-11T19:23:08Z

CC @prudhvigodithi

prudhvigodithi · 2024-10-11T20:06:14Z

We have been pushing the created core-plugins already to https://ci.opensearch.org but not in https://artifacts.opensearch.org/snapshots/plugins/ format.

Example:
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/core-plugins/analysis-icu-3.0.0.zip

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/rpm/builds/opensearch/core-plugins/analysis-icu-3.0.0.zip

https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/deb/builds/opensearch/core-plugins/analysis-icu-3.0.0.zip

and we do the same for arm64

@peterzhuamazon @cwperks @reta can we just the leverage this ? :)

Thank you

reta · 2024-10-11T20:28:16Z

@peterzhuamazon @cwperks @reta can we just the leverage this ? :)

Thanks @prudhvigodithi . So this is somewhat inconsistent: all these 3 URLs point to the same plugin (at least they should) because the distributions of the plugins do not depend on the OpenSearch packaging. We could leverage that, indeed, but looks more like a workaround, not a solution.

peterzhuamazon · 2024-10-11T20:49:50Z

At the moment we just let every distribution build the native plugins zip, but when release we only release x64/tar version.

peterzhuamazon · 2024-10-11T20:50:11Z

So that we do not need to change the workflow too much.

dblock · 2024-10-23T15:59:04Z

Will this address my problem in opensearch-project/opensearch-api-specification#610?

I am using the 3.0 docker build and need to install one of the optional plugins.

Normal docker reference:

FROM opensearchstaging/opensearch:2.18.0@sha256:f311da085b968663fadf041bafc78b24dc21556f0726e625caf3560f7323a020

# attempt to install the plugin here
RUN ./bin/opensearch-plugin install https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.18.0/10380/linux/x64/tar/builds/opensearch/core-plugins/analysis-phonenumber-2.18.0.zip

The above example is problematic because (it doesn't work), but I also need to manually match the version to the docker image SHA somehow.

reta · 2024-10-23T16:03:20Z

Will this address my problem in opensearch-project/opensearch-api-specification#610?

It will @dblock

peterzhuamazon · 2024-10-24T16:17:16Z

After some talk on the triage meeting I fully understand what @reta is talking about.

Similar to:
https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/<>.tar.gz

Do a:
https://artifacts.opensearch.org/snapshots/plugins/opensearch/3.0.0-SNAPSHOT/<>.zip

We can start this with opensearch first, as I just checked we havent done any core snapshot for OSD core on artifacts.opensearch.org for a long time.

Thanks.

Divyaasm · 2024-10-28T22:31:08Z

Acceptance criteria:
Publish native plugins along side the core snapshot ->
https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/<>.tar.gz

peterzhuamazon · 2024-10-28T22:31:44Z

We also need to add the checksum files alongside the native plugin zips.
Probably need to use the same folder structure for snapshot as the release ones.

peterzhuamazon · 2024-10-28T23:30:29Z

Hi @reta if we put the snapshot native plugins zips on s3, I assume you can directly run opensearch-plugin install <>? Or do we need to make changes in core to make it happen.

reta · 2024-10-28T23:33:23Z

Hi @reta if we put the snapshot native plugins zips on s3, I assume you can directly run opensearch-plugin install <>? Or do we need to make changes in core to make it happen.

Thanks @peterzhuamazon, we only need to know the location, if it is different from the released versions fe

peterzhuamazon · 2024-10-30T15:24:51Z

Take a look in the next two days, assuming we should be able to add it as part of min snapshot workflows.

peterzhuamazon · 2024-10-30T19:32:57Z

After offline sync, we will use this folder structure:

https://artifacts.opensearch.org/snapshots/plugins/analysis-icu/2.10.0-SNAPSHOT/analysis-icu-2.10.0-SNAPSHOT.zip

peterzhuamazon · 2024-10-31T20:08:07Z

PRs:

Add core plugins snapshots publishing alongside min snapshots opensearch-build-libraries#556
Upload core plugins snapshots / gradle check updates in lib 7.3.2 #5165

peterzhuamazon · 2024-11-06T03:17:03Z

Hi @reta confirms 2.19.0-SNAPSHOT is up with core plugins
https://build.ci.opensearch.org/job/publish-opensearch-min-snapshots/1947/console

Ex: https://artifacts.opensearch.org/snapshots/plugins/analysis-icu/2.19.0-SNAPSHOT/analysis-icu-2.19.0-SNAPSHOT.zip

3.0.0-SNAPSHOT still in progress here: https://build.ci.opensearch.org/job/publish-opensearch-min-snapshots/1948/console
Should follow the same url pattern as above.

Thanks.

reta · 2024-11-06T14:19:39Z

Thanks @peterzhuamazon , we have one more issue to clarify, for official plugins we also sign them and publish signature, it seems like we don't do this for snapshots?

/opensearch-2.19.0-SNAPSHOT$ ./bin/opensearch-plugin install transport-reactor-netty4                                                                                                                                                                                                     [15/178]
-> Installing transport-reactor-netty4                                                                                                                                     
-> Downloading transport-reactor-netty4 from opensearch                                                                                                                    
[=================================================] 100%                                                                                                                   
-> Failed installing transport-reactor-netty4                                                                                                                              
-> Rolling back transport-reactor-netty4                                                                                                                                   
-> Rolled back transport-reactor-netty4                                                                                                                                    
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://artifacts.opensearch.org/snapshots/plugins/transport-reactor-netty4/2.19.0-SNAPSHOT/transport-reactor-netty4-2.19.0-SNAPSHOT.zip.sig
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2013)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1614)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:223)
        at java.base/java.net.URL.openStream(URL.java:1325)                                                                                                                
        at org.opensearch.plugins.InstallPluginCommand.urlOpenStream(InstallPluginCommand.java:490)
        at org.opensearch.plugins.InstallPluginCommand.verifySignature(InstallPluginCommand.java:630)
        at org.opensearch.plugins.InstallPluginCommand.downloadAndValidate(InstallPluginCommand.java:608)
        at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:316)
        at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:271)
        at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:248)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

peterzhuamazon · 2024-11-06T18:01:54Z

Thanks @peterzhuamazon , we have one more issue to clarify, for official plugins we also sign them and publish signature, it seems like we don't do this for snapshots?

/opensearch-2.19.0-SNAPSHOT$ ./bin/opensearch-plugin install transport-reactor-netty4                                                                                                                                                                                                     [15/178]
-> Installing transport-reactor-netty4                                                                                                                                     
-> Downloading transport-reactor-netty4 from opensearch                                                                                                                    
[=================================================] 100%                                                                                                                   
-> Failed installing transport-reactor-netty4                                                                                                                              
-> Rolling back transport-reactor-netty4                                                                                                                                   
-> Rolled back transport-reactor-netty4                                                                                                                                    
Exception in thread "main" java.io.IOException: Server returned HTTP response code: 403 for URL: https://artifacts.opensearch.org/snapshots/plugins/transport-reactor-netty4/2.19.0-SNAPSHOT/transport-reactor-netty4-2.19.0-SNAPSHOT.zip.sig
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:2013)
        at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1614)
        at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:223)
        at java.base/java.net.URL.openStream(URL.java:1325)                                                                                                                
        at org.opensearch.plugins.InstallPluginCommand.urlOpenStream(InstallPluginCommand.java:490)
        at org.opensearch.plugins.InstallPluginCommand.verifySignature(InstallPluginCommand.java:630)
        at org.opensearch.plugins.InstallPluginCommand.downloadAndValidate(InstallPluginCommand.java:608)
        at org.opensearch.plugins.InstallPluginCommand.download(InstallPluginCommand.java:316)
        at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:271)
        at org.opensearch.plugins.InstallPluginCommand.execute(InstallPluginCommand.java:248)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
        at org.opensearch.cli.Command.main(Command.java:101)
        at org.opensearch.plugins.PluginCli.main(PluginCli.java:60)

Yes we dont have signings for snapshots, on both plugins zips and min artifacts.
Do we need them in order to install?

reta · 2024-11-06T18:07:30Z

Do we need them in order to install?

Could we do that? I think it will be helpful, or there are any drawbacks?

peterzhuamazon · 2024-11-06T19:00:35Z

Do we need them in order to install?

Could we do that? I think it will be helpful, or there are any drawbacks?

The existing min pipeline is not designed to have signing in mind, let me take a look on it today.

peterzhuamazon · 2024-11-06T19:20:11Z

Related OS PR that is pending due to this:

Support installing plugin SNAPSHOTs with SNASPHOT distribution OpenSearch#16581

peterzhuamazon · 2024-11-06T21:58:29Z

More PRs:

Upload signature alongside snapshots artifacts and checksums opensearch-build-libraries#561

reta added enhancement New Enhancement untriaged Issues that have not yet been triaged labels Oct 11, 2024

reta mentioned this issue Oct 22, 2024

Add pluginZip publication for plugins in the plugins folder opensearch-project/OpenSearch#16219

Closed

3 tasks

dblock mentioned this issue Oct 23, 2024

Add spec for opensearch phone plugin. opensearch-project/opensearch-api-specification#610

Closed

peterzhuamazon removed the untriaged Issues that have not yet been triaged label Oct 24, 2024

peterzhuamazon self-assigned this Oct 30, 2024

peterzhuamazon added jenkins Jenkins related issue Build Libraries & Interfaces labels Oct 30, 2024

peterzhuamazon mentioned this issue Oct 31, 2024

Add core plugins snapshots publishing alongside min snapshots opensearch-project/opensearch-build-libraries#556

Merged

peterzhuamazon mentioned this issue Nov 5, 2024

Upload core plugins snapshots / gradle check updates in lib 7.3.2 #5165

Merged

prudhvigodithi closed this as completed in #5165 Nov 6, 2024

reta mentioned this issue Nov 6, 2024

Support installing plugin SNAPSHOTs with SNASPHOT distribution opensearch-project/OpenSearch#16581

Open

3 tasks

peterzhuamazon reopened this Nov 6, 2024

github-actions bot added the untriaged Issues that have not yet been triaged label Nov 6, 2024

peterzhuamazon removed the untriaged Issues that have not yet been triaged label Nov 6, 2024

peterzhuamazon mentioned this issue Nov 6, 2024

Upload signature alongside snapshots artifacts and checksums opensearch-project/opensearch-build-libraries#561

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Publish plugins SNAPSHOT artifacts (along with SNASPHOT distribution) #5096

[Feature] Publish plugins SNAPSHOT artifacts (along with SNASPHOT distribution) #5096

reta commented Oct 11, 2024

reta commented Oct 11, 2024

prudhvigodithi commented Oct 11, 2024 •

edited

Loading

reta commented Oct 11, 2024 •

edited

Loading

peterzhuamazon commented Oct 11, 2024

peterzhuamazon commented Oct 11, 2024

dblock commented Oct 23, 2024

reta commented Oct 23, 2024

peterzhuamazon commented Oct 24, 2024

Divyaasm commented Oct 28, 2024

peterzhuamazon commented Oct 28, 2024 •

edited

Loading

peterzhuamazon commented Oct 28, 2024 •

edited

Loading

reta commented Oct 28, 2024

peterzhuamazon commented Oct 30, 2024

peterzhuamazon commented Oct 30, 2024 •

edited

Loading

peterzhuamazon commented Oct 31, 2024 •

edited

Loading

peterzhuamazon commented Nov 6, 2024 •

edited

Loading

reta commented Nov 6, 2024

peterzhuamazon commented Nov 6, 2024

reta commented Nov 6, 2024

peterzhuamazon commented Nov 6, 2024

peterzhuamazon commented Nov 6, 2024

peterzhuamazon commented Nov 6, 2024

[Feature] Publish plugins SNAPSHOT artifacts (along with SNASPHOT distribution) #5096

[Feature] Publish plugins SNAPSHOT artifacts (along with SNASPHOT distribution) #5096

Comments

reta commented Oct 11, 2024

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you've considered

Additional context

reta commented Oct 11, 2024

prudhvigodithi commented Oct 11, 2024 • edited Loading

reta commented Oct 11, 2024 • edited Loading

peterzhuamazon commented Oct 11, 2024

peterzhuamazon commented Oct 11, 2024

dblock commented Oct 23, 2024

reta commented Oct 23, 2024

peterzhuamazon commented Oct 24, 2024

Divyaasm commented Oct 28, 2024

peterzhuamazon commented Oct 28, 2024 • edited Loading

peterzhuamazon commented Oct 28, 2024 • edited Loading

reta commented Oct 28, 2024

peterzhuamazon commented Oct 30, 2024

peterzhuamazon commented Oct 30, 2024 • edited Loading

peterzhuamazon commented Oct 31, 2024 • edited Loading

peterzhuamazon commented Nov 6, 2024 • edited Loading

reta commented Nov 6, 2024

peterzhuamazon commented Nov 6, 2024

reta commented Nov 6, 2024

peterzhuamazon commented Nov 6, 2024

peterzhuamazon commented Nov 6, 2024

peterzhuamazon commented Nov 6, 2024

prudhvigodithi commented Oct 11, 2024 •

edited

Loading

reta commented Oct 11, 2024 •

edited

Loading

peterzhuamazon commented Oct 28, 2024 •

edited

Loading

peterzhuamazon commented Oct 28, 2024 •

edited

Loading

peterzhuamazon commented Oct 30, 2024 •

edited

Loading

peterzhuamazon commented Oct 31, 2024 •

edited

Loading

peterzhuamazon commented Nov 6, 2024 •

edited

Loading