s3proxy on other K8S distro (kubeadm)

[rune1979]

I have been looking around and tested out different solutions for secure storage at (insecure) external s3-like storage providers. Which seems to be difficult find solutions for.

Constellations s3proxy seems to be very much what I am looking for. BUT, is it in anyway possible to run this in another setting?

• k8s, deployed with Kubeadm
• Running on Wireguard

If possible, is "KeyService" and "s3proxy" enough to make this work or what would it require?

Constellations looks very interesting and would be something I have to test out at some point in the future. But, for now the whole "package" is not an option.

Thanks - Rune

[daniel-weisse]

Hi there

While this does sound like an interesting project, it is not officially supported.
As you have already figured out, the s3proxy depends on Constellation's KeyService to manage encryption/decryption keys.

You can try deploying the KeyService as a standalone application into your cluster, but there are no guarantees it will work.
We also do not publish any Helm charts for the KeyService, since they are included as part of the CLI and Terraform provider.
If you want to take a look nonetheless, the files can be found here.

Theoretically, you could also roll your own implementation of a KeyService.
The only requirement is that it implements this grpc interface.

Hope this helps!

[rune1979]

Thanks a lot @daniel-weisse I'll try to dig a bit deeper into this and post a followup if I succeed in this.

[rune1979]

It seems to be doable. I don't use Amazon S3, so I think that's where things starts be a bit more difficult. I can see buckets at my storage provider and even create a new bucket, view content of exisiting buckets. This seem to be the case for both bucket storage providers that I have tested. But, some conflicts starts to happen when trying to create a file or directory.

Filestash:
BucketRegionError: incorrect region, the bucket is not in 'us-east-1' (seems like a default AWS region) region at endpoint '[my].[region].[provider].com' status code: 301, request id: , host id: (both seems to be empty) I have seen some other errors too.

s3proxy log:
{"time":"2024-10-09T15:50:09.710904966Z","level":"DEBUG","source":{"function":"github.com/edgelesssys/constellation/v2/s3proxy/internal/router.Router.Serve.handlePutObject.func2","file":"s3proxy/internal/router/handler.go","line":44},"msg":"intercepting","path":"/test1/test","method":"PUT","host":"[region].[my_host].com"} {"time":"2024-10-09T15:50:09.83786996Z","level":"ERROR","source":{"function":"github.com/edgelesssys/constellation/v2/s3proxy/internal/router.object.put","file":"s3proxy/internal/router/object.go","line":150},"msg":"PutObject sending request to S3","error":"operation error S3: PutObject, https response error StatusCode: 301, RequestID: SOMEIDXXXX, HostID: BASE64KEYXXXXXX, api error PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."}

Just thought I would let you know, in case others are having the same issues (are not on AWS).

[rune1979]

I suspect it could be the bucket URL formatting. Like the two providers I am currently using have the follwing setup:
First one: [bucket].[UID].[region].[domain].com
Second one: [region].[domain].com/[bucket]

[msanft]

Hey @rune1979! Did you supply the region flag? It defaults to eu-west-1, which I assume could be something the object store gets confused about when it's specified but the region doesn't exist.

[rune1979]

Yes that was missing in the previous example, some fields still seems to be missing or get an error.
GUI Filestash:
BucketRegionError: incorrect region, the bucket is not in 'eu' region at endpoint '[region].[domain].com' status code: 301, request id: , host id:

Logs Filestash:
2024/10/11 22:53:49 HTTP 200 GET 9.6ms /login
2024/10/11 22:53:50 HTTP 200 GET 0.9ms /custom.css
2024/10/11 22:53:50 HTTP 401 GET 2.2ms /api/session
2024/10/11 22:53:50 HTTP 200 GET 3.5ms /api/config
2024/10/11 22:53:51 HTTP 304 GET 11.5ms /api/backend
2024/10/11 22:54:32 HTTP 400 POST 415.2ms /api/session
2024/10/11 22:55:29 HTTP 200 POST 231.6ms /api/session
2024/10/11 22:55:30 HTTP 200 GET 71.0ms /api/session
2024/10/11 22:55:32 HTTP 304 GET 1747.7ms /api/files/ls?path=%2F
2024/10/11 22:55:37 HTTP 304 GET 155.5ms /api/files/ls?path=%2Ftest1%2F
2024/10/11 22:55:42 HTTP 200 GET 25.6ms /assets/icons/file.svg
2024/10/11 22:55:45 HTTP 200 GET 0.3ms /assets/icons/edit.svg
2024/10/11 22:55:45 HTTP 500 POST 217.1ms /api/files/touch?path=%2Ftest1%2Fhh

Log s3proxy:
{"time":"2024-10-11T22:55:45.687180442Z","level":"DEBUG","source":{"function":"github.com/edgelesssys/constellation/v2/s3proxy/internal/router.Router.Serve.handlePutObject.func2","file":"s3proxy/internal/router/handler.go","line":44},"msg":"intercepting","path":"/test1/hh","method":"PUT","host":"[region].[buckethost].com"}
{"time":"2024-10-11T22:55:45.893188257Z","level":"ERROR","source":{"function":"github.com/edgelesssys/constellation/v2/s3proxy/internal/router.object.put","file":"s3proxy/internal/router/object.go","line":150},"msg":"PutObject sending request to S3","error":"operation error S3: PutObject, https response error StatusCode: 301, RequestID: xxxxxxxxxxxxxx, HostID: xxxxxxxxxxx, api error PermanentRedirect: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint."}

[msanft]

This reads to me as if you would set --region eu, which the bucket you're querying isn't available in within your Filestash? Can you somehow list the available buckets along with their regions in Filestash?

[rune1979]

I think I figured it out. AWS s3 is fine. But, as soon as you are starting to use other providers a lot of individual setting often starts to appear like bucket path etc. (looking into rclone remote s3, one would have to choose a specific provider: https://rclone.org/s3/)

So, it would probably be a mess to try and make something generic. S3proxy is just following the default standard provided by AWS and it is probably best, to keep it simple. (The problem is a lot of the secondary bucket providers, kind of have their own conventions probably based on their storage layer).