publish snapshot not accepting -component (publishing to s3)

[troy2914]

Detailed Description

started with
aptly mirror create -architectures=amd64 noble-main http://atl.mirrors.clouvider.net/ubuntu/ noble main restricted universe multiverse
aptly mirror create -architectures=amd64 noble-security http://atl.mirrors.clouvider.net/ubuntu/ noble-security main restricted universe multiverse
aptly mirror create -architectures=amd64 noble-updates-main http://atl.mirrors.clouvider.net/ubuntu/ noble-updates main restricted universe multiverse
then the mirror update for those 3
after that i created a snapshot
aptly snapshot create noble-main-24.04.1 from mirror noble-main
aptly snapshot create noble-updates-date +%Y%m%d from mirror noble-updates
aptly snapshot create noble-security-date +%Y%m%d from mirror noble-security
then a snapshot merge
aptly snapshot merge -latest noble-main-final-date +%Y%m%d noble-main-24.04.1 noble-updates-date +%Y%m%d noble-security-date +%Y%m%d
but when trying
aptly publish snapshot -component="main,universe,restricted,multiverse" -label "Ubuntu" -origin "Ubuntu" -distribution=noble noble-main-final-date +%Y%m%d s3:endpoint:prefix
or
aptly publish snapshot -component=",,," -label "Ubuntu" -origin "Ubuntu" -distribution=noble noble-main-final-date +%Y%m%d s3:endpoint:prefix
it's coming out with a
ERROR: unable to parse command
removing the -component arg and it works.

Your Environment

This was on Ubuntu 22.04 host running aptly 1.5.0 (tested both installed locally, and from apt)

[neolynx]

I think the error comes from not having enough arguments. the number of [[:]] needs to match the number of components provided.

cmd/publish_snapshot.go:

   20         if len(args) < len(components) || len(args) > len(components)+1 {
   21                 cmd.Usage()
   22                 return commander.ErrCommandError
   23         }

so adding multiple s3:endpoint:prefix with different prefix should work.

I wonder what is the intention behind that... Would this work for you ?

[troy2914]

adding multiple s3:endpoint:prefexies, lead me to... actually i needed to list the snapshot for each component. eg: noble-main-final-20241003 noble-main-final-20241003 noble-main-final-20241003 noble-main-final-20241003, which leads me to believe I should have snapshot for each component? Though mirror create let me create it listing multiple components...
letting this publish snapshot to finish, to see what i wind up with.

[troy2914]

as expected instead of 188 GB the new one was 751G. so sounds like a have to create separate snapshots of each component, yuck.

[neolynx]

this is weird indeed. I think via the API if is possible to publish multiple components to one prefix...

@cfiehe what do you think ?

[cfiehe]

Yes, you can publish multiple repositories/snapshots under the same prefix, but they must cover different distributions.

@troy2914
I think you want to do multi-component publishing and want to preserve the package to component mapping. In that case, you have to start with the definition of 4 separate mirrors for each distribution, one for each component:

aptly mirror create -architectures=amd64 noble-main http://atl.mirrors.clouvider.net/ubuntu/ noble main
aptly mirror create -architectures=amd64 noble-restricted http://atl.mirrors.clouvider.net/ubuntu/ noble restricted
aptly mirror create -architectures=amd64 noble-universe http://atl.mirrors.clouvider.net/ubuntu/ noble universe
aptly mirror create -architectures=amd64 noble-multiverse http://atl.mirrors.clouvider.net/ubuntu/ noble multiverse

aptly mirror create -architectures=amd64 noble-security-main http://atl.mirrors.clouvider.net/ubuntu/ noble-security main
...

aptly mirror create -architectures=amd64 noble-updates-main http://atl.mirrors.clouvider.net/ubuntu/ noble-updates main
...

Use the same pattern with different mirror names for the distributions noble-security and noble-updates. You will end up with 12 different mirrors. It is important to understand, that mirrors and snapshots are just a set of packages. When you define a mirror with multiple components, all packages are put in the same set and you loose the package to component mapping. In the next step you must create a snapshot for each mirror and you will end up with 12 different snapshots:

aptly snapshot create noble-main-20241018 from mirror noble-main
...
aptly snapshot create noble-security-main-20241018 from mirror noble-security-main
...
aptly snapshot create noble-updates-main-20241018 from mirror noble-updates-main
...

Components come back into play, when you create a published repository. Each component must have a single source which must be a local repository or (recommended) a snapshot. The number of given components must be equal to the number of given sources. In your case, you must define 3 published repositories:

aptly publish snapshot -component="main,restricted,universe,multiverse" -distribution=noble noble-main-20241018 noble-restricted-20241018 noble-universe-20241018 noble-multiverse-20241018 s3:endpoint:prefix
aptly publish snapshot -component="main,restricted,universe,multiverse" -distribution=noble-security noble-security-main-20241018 noble-security-restricted-20241018 noble-security-universe-20241018 noble-security-multiverse-20241018 s3:endpoint:prefix
aptly publish snapshot -component="main,restricted,universe,multiverse" -distribution=noble-updates noble-updates-main-20241018 noble-updates-restricted-20241018 noble-updates-universe-20241018 noble-updates-multiverse-20241018 s3:endpoint:prefix

I do not recommend to use the syntax like -component=,,. Currently, I am not sure, what is returned when you leave it to Aptly to pick out a component from a mirror that comprises multiple components. It is less error-prone, when you always name the components explicitly in the publish command and ensure that you give a single package source for each component.

I hope, that it helps to solve your issue.

[troy2914]

Thanks, might be nice to have a new tutorial on the website showing preserving components.