Skip to content

Setting up an S3 storage

Jump to bottom
L. Le Meur edited this page Dec 28, 2021 · 5 revisions

S3 storage in the LCP server is managed via the official AWS SDK stored on Github. This document is a quick "how-to", extracted from the AWS SDK documentation.

Create a bucket.

Make files publicly readable

source: https://docs.aws.amazon.com/AmazonS3/latest/dev/WebsiteAccessPermissionsReqd.html

  • Select the bucket.
  • Go to Permissions.
  • Add this bucket strategy:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "PublicReadGetObject",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::name-of-your-bucket/*"
        }
    ]
}

warning: change name-of-your-bucket for the name of your bucket.

Get your credentials

Source: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/setting-up.html

If you don’t have your credentials yet, you can create them by using the AWS Management Console. We recommend that you use IAM access keys instead of AWS root account access keys. IAM lets you securely control access to AWS services and resources in your AWS account.

  • Open the IAM console.
  • create a user "lcp-server" + programmatic access
  • create a group "s3-client" + strategy "AmazonS3FullAccess"
  • add the user to the group
  • no optional key
  • download the csv -> access key + secret access key
  • store it securely.

ex. https://s3.console.aws.amazon.com/s3/buckets/name-of-your-bucket?region=eu-west-3&tab=objects

Set environment variables

Source: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

The AWS Region needs to be provided in the AWS config file (.aws/config) or as the AWS_REGION environment variable.

The AWS SDK for Go requires credentials (an access key id and a secret access key) to sign requests to AWS. These are used to sign programmatic requests that you make to AWS. You can specify your credentials either in the AWS credentials file (.aws/credentials) or as the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

ex. on MacOS and Linux

$ export AWS_REGION=YOUR_REGION
$ export AWS_ACCESS_KEY_ID=YOUR_AKID
$ export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY

warning: make sure these environment variables are set permanently.

Test

Go to https://github.com/aws/aws-sdk-go/

Copy/paste the sample in s3test.go in a gospace/src/s3test folder (replace "gospace" by your own go space folder name). This cli uses bucket and object key as parameters.

Copy an epub file (e.g. moby-dick.epub) into this test folder.

$ cd /Users/laurentlemeur/work/gospace/src/s3test
$ go run s3test.go -b name-of-your-bucket -k moby-dick.epub -d 10m < moby-dick.epub

-> is now fetchable via something like https://edrlab-lcp-storage.s3.eu-west-3.amazonaws.com/moby-dick.epub (the exact URL is found in the AWS console, object screen).

Configure properly the LCP server

Read carefully the LCP Server configuration page.

A few S3 parameters are used in the LCP Server codebase but not described in the LCP Server configuration page, because they don't seem to be used in practice.

This is the case for the token parameter, described in AWS Temporary Security Credentials in IAM, potentially used if credentials are defined in the configuration file, by a call to NewsStaticCredentials.

This is also the case for the endpoint parameter, described at the bottom of the AWS Go SDK.

And the same for the disable_ssl and path_style parameters, for which some explanations can be found here and here.

Clone this wiki locally