Upload to S3 broken

[nzjrs]

I get the following

ERROR: unable to publish: unable to process packages: error caching paths under prefix: error listing under prefix in S3: eu-central-1:XXX/: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.

Any idea what is going on here?

[aureq]

Hi @nzjrs

aptly should use the AS "SigV4" to authenticate against the S3 service.
While this is not mandatory in most S3 regions, eu-central-1 (Frankfurt) requires such mechanism.

The authentication process is described here: http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html

To me, it looks like aptly should use SigV4 instead of the old mechanism so it is compatible in all S3 regions. Though, I haven't check aptly's code for sure....

[aureq]

So, probably not an aptly specific issue, but to a related package.

In https://github.com/mitchellh/goamz/blob/master/s3/sign.go#L111 you can read HMAC+SHA1.
However, The official doc for SigV4 uses SHA-256 http://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html

This is also tracked in a separate issue mitchellh/goamz#166 for the exact same reason: Frankfurt.

[aureq]

And it looks like there's already a PR for this as well: mitchellh/goamz#230

[nzjrs]

The last commit on goamz is in March, is it still actively developed?

On 9 December 2015 at 05:05, Aurélien Requiem [email protected]
wrote:

And it looks like there's already a PR for this as well:
mitchellh/goamz#230 mitchellh/goamz#230

—
Reply to this email directly or view it on GitHub
https://github.com/smira/aptly/issues/326#issuecomment-163098334.

[smira]

If goamz gets updated, I can pull new version and release new nightly build for that.

[aureq]

@smira To me, goamz seems a bit dead'ish. No commits in 9 months with a nice pull request like highlighted is not a good sign. Especially considering that SigV4 has been around for a while and it's more secure.

The good news though is that AWS has release an SDK for Go: https://github.com/aws/aws-sdk-go

While I understand that will be a lot of work to migrate from one SDK to another, I think it may be beneficial in the long term.

Just sharing my personal thoughts on this...

[ColinHebert]

@smira as @aureq said, goamz has been dropped in favour of aws-sdk-go. Even the projects that triggered the creation of the goamz library moved away from it.

hashicorp/packer#2034
hashicorp/terraform#1195

I would strongly recommend to drop goamz.

[smira]

Yeah, I agree we should rewrite in AWS Go SDK. I haven't look close, but the tricky part could be retries and stull like that.

[iiro]

+1

[nwheeler]

+1 ... retries are currently broken. I've tested from .deb and source, and even tried to troubleshoot the problem. I was not very successful, but the retry logic leads to the request body being 0 on retry, which throws an error:

ERROR: unable to publish: unable to process packages: error uploading /usr/local/mirror/pool/f5/f8/xul-ext-noscript_2.6.9.3-1_all.deb to S3: us-east-1:mybucket/: Put https:/mybucket/production/pool/main/m/mozilla-noscript/xul-ext-noscript_2.6.9.3-1_all.deb: http: Request.ContentLength=762948 with Body length 0

Not opening a new ticket because I'm pretty sure this would be resolved with going away from goamz. Thanks!

(Mirroring to s3 is obviously currently unusable as it's not reliable at all, presently)

EDIT:
Ah, mitchellh/goamz#243

[smira]

@nwheeler good catch! if goamz doesn't come, probably switching to AWS Go SDK is a way to fix/workaround such problems going forward

[bitglue]

#344 replaces goamz with the official SDK. Haven't tested this particular issue, though.

[smira]

Should be fixed as #344 is merged