Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Non standard S3 endpoint download url issue #21

Open
JoeeGrigg opened this issue Mar 23, 2021 · 5 comments
Open

Non standard S3 endpoint download url issue #21

JoeeGrigg opened this issue Mar 23, 2021 · 5 comments

Comments

@JoeeGrigg
Copy link

Hi,

I could be wrong but I think there is a problem with generating a url for an uploaded file when using a custom s3 storage endpoint. I am running Minio locally on my machine and uploading works fine. Unfortunately, when I try to request a download url via UploadedFile#url I get a url that doesn't point to my local Minio server.

I am using it within a Lucky project. Below is my configuration and usage.

# config/shrine.cr

client = Awscr::S3::Client.new(
  "minio",
  "minioadmin",
  "minioadmin",
  endpoint: "http://localhost:9000"
)

Shrine.configure do |config|
  config.storages["store"] = Shrine::Storage::S3.new(bucket: "day", client: client)
end
Shrine::UploadedFile.new("7609952bf7b7973b03595c315d728cd9.jpg", "store").url

# => https://s3-minio.amazonaws.com/day/7609952bf7b7973b03595c315d728cd9.jpg?X-Amz-Expires=86400&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minioadmin%2F20210323%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20210323T083243Z&X-Amz-SignedHeaders=host&X-Amz-Signature=5bafad2cb415d929198ba3b0f9ede9fd78943cef30b81e3711ec16b1baeb6ff3

As you can see from the above, I have my endpoint configured to http://localhost:9000 but when I request a url it uses https://s3-minio.amazonaws.com.

Am I doing something wrong or is this a bug? Any help would be appreciated!

@da1nerd
Copy link

da1nerd commented Mar 28, 2021

I had a similar issue in a project of mine. The solution was to patch Shrine's S3 storage class.
I was going to open a PR with a fix, but I sort of forgot... 🙄

Here's the patch I applied in my local code to get Shrine working with Digital Ocean Spaces. I still have a lot on my plate at the moment, so if someone else wants to turn this into a PR that would be great 😄

# TRICKY: we patch this because it's not providing the correct download url
class Shrine
  module Storage
    class S3 < Storage::Base
      def url(id : String, **options) : String
        endpoint : String?
        if ep = client.@endpoint
          endpoint = ep.gsub("https://", "")
        end
        presigned_options = Awscr::S3::Presigned::Url::Options.new(
          aws_access_key: client.@aws_access_key,
          aws_secret_key: client.@aws_secret_key,
          region: client.@region,
          object: "/#{object_key(id)}",
          bucket: bucket,
          host_name: endpoint
        )

        url = Awscr::S3::Presigned::Url.new(presigned_options)
        url.for(:get)
      end
    end
  end
end

@JoeeGrigg
Copy link
Author

Thanks that will help massively!

I will add it into my project and if I get chance turn it into a PR 😀

@JoeeGrigg
Copy link
Author

I tried adding this into my project but unfortunately it still didn't work for me. The problem seemed to be partly that the awscr-s3 shard only supports https endpoints. While I can understand this, it makes it a lot harder to run a local Minio server in development. I will keep working on a solution and potential end up submitting a couple of pull requests...

@nikobojs
Copy link

nikobojs commented Nov 2, 2021

What do you think of this diff: nikobojs@c5a3677
It will not solve the forced https but at least use the right hostname. The signature stuff works and the file downloads on the generated link as long as i remove the 's' from https. I can't run specs right now, so not sure if its good enough for a PR?

@wout
Copy link
Contributor

wout commented Sep 2, 2023

The problem seemed to be partly that the awscr-s3 shard only supports https endpoints.

In that case, you're probably better off using Ngrok or something for your local minio server, so you run it over https.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants