Unable to access file from S3 bucket file url.

[Aazad23]

while running titiler with a url of file from S3 bucket, I m getting response code 500. How to reslove it?

[vincentsarago]

do you have credential to read the S3 file?
is TiTiler deployed somewhere? did you white listed the S3 bucket to have access to your file? https://github.com/developmentseed/titiler/blob/master/deployment/aws/cdk/config.py#L32-L36

[Aazad23]

yes , as suggsted by you I have made changes in config.py
buckets: List = ['*']
and got following logs

D:\IFFCO_Tokio\titiler>docker-compose up --build
Creating network "titiler_default" with the default driver
Building titiler
[+] Building 1.3s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 32B                                                                                0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 34B                                                                                   0.0s
 => [internal] load metadata for docker.io/tiangolo/uvicorn-gunicorn:python3.8                                     1.1s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 7.41kB                                                                                0.0s
 => [1/4] FROM docker.io/tiangolo/uvicorn-gunicorn:python3.8@sha256:179848e06aa0e45dc15e3db531c8c9519f038ff08a5e9  0.0s
 => CACHED [2/4] COPY src/titiler/ /tmp/titiler/                                                                   0.0s
 => CACHED [3/4] RUN pip install /tmp/titiler/core /tmp/titiler/mosaic /tmp/titiler/application --no-cache-dir     0.0s
 => CACHED [4/4] RUN rm -rf /tmp/titiler                                                                           0.0s
 => exporting to image                                                                                             0.0s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:b20e430d890c80932ba73a3e5a342bf796ab6ed421003527ddafc604506596c1                       0.0s
 => => naming to docker.io/library/titiler_titiler                                                                 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Creating titiler_titiler_1 ... done
Attaching to titiler_titiler_1
titiler_1  | Checking for script in /app/prestart.sh
titiler_1  | Running script /app/prestart.sh
titiler_1  | Running inside /app/prestart.sh, you could add migrations to this file, e.g.:
titiler_1  |
titiler_1  | #! /usr/bin/env bash
titiler_1  |
titiler_1  | # Let the DB start
titiler_1  | sleep 10;
titiler_1  | # Run migrations
titiler_1  | alembic upgrade head
titiler_1  |
titiler_1  | [2021-09-22 09:25:36 +0000] [1] [INFO] Starting gunicorn 20.1.0
titiler_1  | [2021-09-22 09:25:36 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
titiler_1  | [2021-09-22 09:25:36 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
titiler_1  | [2021-09-22 09:25:36 +0000] [9] [INFO] Booting worker with pid: 9
titiler_1  | [2021-09-22 09:25:36 +0000] [17] [INFO] Booting worker with pid: 17
titiler_1  | [2021-09-22 09:25:36 +0000] [18] [INFO] Booting worker with pid: 18
titiler_1  | [2021-09-22 09:25:36 +0000] [19] [INFO] Booting worker with pid: 19
titiler_1  | [2021-09-22 09:25:36 +0000] [41] [INFO] Booting worker with pid: 41
titiler_1  | [2021-09-22 09:25:36 +0000] [42] [INFO] Booting worker with pid: 42
titiler_1  | [2021-09-22 09:25:36 +0000] [50] [INFO] Booting worker with pid: 50
titiler_1  | [2021-09-22 09:25:36 +0000] [58] [INFO] Booting worker with pid: 58
titiler_1  | [2021-09-22 09:25:40 +0000] [17] [INFO] Started server process [17]
titiler_1  | [2021-09-22 09:25:40 +0000] [17] [INFO] Waiting for application startup.
titiler_1  | [2021-09-22 09:25:40 +0000] [17] [INFO] Application startup complete.
titiler_1  | [2021-09-22 09:25:40 +0000] [9] [INFO] Started server process [9]
titiler_1  | [2021-09-22 09:25:40 +0000] [9] [INFO] Waiting for application startup.
titiler_1  | [2021-09-22 09:25:40 +0000] [9] [INFO] Application startup complete.
titiler_1  | [2021-09-22 09:25:40 +0000] [18] [INFO] Started server process [18]
titiler_1  | [2021-09-22 09:25:40 +0000] [18] [INFO] Waiting for application startup.
titiler_1  | [2021-09-22 09:25:40 +0000] [18] [INFO] Application startup complete.
titiler_1  | [2021-09-22 09:25:40 +0000] [42] [INFO] Started server process [42]
titiler_1  | [2021-09-22 09:25:40 +0000] [42] [INFO] Waiting for application startup.
titiler_1  | [2021-09-22 09:25:40 +0000] [42] [INFO] Application startup complete.
titiler_1  | [2021-09-22 09:25:40 +0000] [41] [INFO] Started server process [41]
titiler_1  | [2021-09-22 09:25:40 +0000] [41] [INFO] Waiting for application startup.
titiler_1  | [2021-09-22 09:25:40 +0000] [41] [INFO] Application startup complete.
titiler_1  | [2021-09-22 09:25:40 +0000] [50] [INFO] Started server process [50]
titiler_1  | [2021-09-22 09:25:40 +0000] [50] [INFO] Waiting for application startup.
titiler_1  | [2021-09-22 09:25:40 +0000] [50] [INFO] Application startup complete.
titiler_1  | [2021-09-22 09:25:40 +0000] [19] [INFO] Started server process [19]
titiler_1  | [2021-09-22 09:25:40 +0000] [19] [INFO] Waiting for application startup.
titiler_1  | [2021-09-22 09:25:40 +0000] [19] [INFO] Application startup complete.
titiler_1  | [2021-09-22 09:25:40 +0000] [58] [INFO] Started server process [58]
titiler_1  | [2021-09-22 09:25:40 +0000] [58] [INFO] Waiting for application startup.
titiler_1  | [2021-09-22 09:25:40 +0000] [58] [INFO] Application startup complete.
titiler_1  | HTTP response code: 403
titiler_1  | NoneType: None
titiler_1  | {"loglevel": "info", "workers": 8, "bind": "0.0.0.0:8000", "graceful_timeout": 120, "timeout": 120, "keepalive": 5, "errorlog": "-", "accesslog": "-", "workers_per_core": 1.0, "use_max_workers": null, "host": "0.0.0.0", "port": "8000"}
titiler_1  | 192.168.16.1:50758 - "GET /cog/metadata?url=https://mod13q1-cog.s3.ap-south-1.amazonaws.com/2021225/erdas_spnad83.tif&max_size=256 HTTP/1.1" 500

[vincentsarago]

@Aazad23 the file either doesn't exist or isn't in public access

rio cogeo info https://mod13q1-cog.s3.ap-south-1.amazonaws.com/2021225/erdas_spnad83.tif
...
rasterio.errors.RasterioIOError: HTTP response code: 403

gdalinfo /vsicurl/https://mod13q1-cog.s3.ap-south-1.amazonaws.com/2021225/erdas_spnad83.tif
ERROR 11: HTTP response code: 403
gdalinfo failed - unable to open '/vsicurl/https://mod13q1-cog.s3.ap-south-1.amazonaws.com/2021225/erdas_spnad83.tif'.

Note:

• the buckets: List = ['*'] part is only if you have titiler deployed on S3
• it's not an S3 url so we don't need credentials, files using https scheme should be in public read or then use the proper S3 url s3://mod13q1-cog/2021225/erdas_spnad83.tif

[Aazad23]

we have not made S3 bucket as public. in addition we are using AWS credentials to access the objects in the S3 . how is it possible to fetch the raster available in private bucket using credentials and visualized in cod viewer

[vincentsarago]

@Aazad23 I'm not sure to follow 💯 of that.

Maybe we are missing the region information in the docker-compose file, could you try to set AWS_REGION='ap-south-1' in https://github.com/developmentseed/titiler/blob/master/docker-compose.yml#L12-L13 and make sure to use s3://mod13q1-cog/2021225/erdas_spnad83.tif url please

[Aazad23]

thank you now we are able to access it through s3 url