From 3dd0a02a7b08b8c2b3c29c556cd7fd6e0ce3e91b Mon Sep 17 00:00:00 2001 From: Fabio Buso Date: Tue, 12 Nov 2024 21:56:59 -0800 Subject: [PATCH] [FSTORE-1604] Add option to avoid setting s3a global options --- docs/user_guides/fs/storage_connector/creation/s3.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/user_guides/fs/storage_connector/creation/s3.md b/docs/user_guides/fs/storage_connector/creation/s3.md index 59003a14f..cf09dbb4a 100644 --- a/docs/user_guides/fs/storage_connector/creation/s3.md +++ b/docs/user_guides/fs/storage_connector/creation/s3.md @@ -71,8 +71,12 @@ If you have SSE-KMS enabled for your bucket, you can find the key ARN in the "Pr ### Step 5: Add Spark Options (optional) Here you can specify any additional spark options that you wish to add to the spark context at runtime. Multiple options can be added as key - value pairs. -!!! tip - To connect to a S3 compatiable storage other than AWS S3, you can add the option with key as `fs.s3a.endpoint` and the endpoint you want to use as value. The storage connector will then be able to read from your specified S3 compatible storage. +To connect to a S3 compatiable storage other than AWS S3, you can add the option with key as `fs.s3a.endpoint` and the endpoint you want to use as value. The storage connector will then be able to read from your specified S3 compatible storage. + +!!! warning "Spark Configuration" + When using the storage connector within a Spark application, the credentials are set at application level. This allows users to access multiple buckets with the same storage connector within the same application (assuming the credentials allow it). + You can disable this behaviour by setting the option `fs.s3a.global-conf` to `False`. If the `global-conf` option is disabled, the credentials are set on a per-bucket basis and users will be able to use the credentials to access data only from the bucket specified in the storage connector configuration. + ## Next Steps Move on to the [usage guide for storage connectors](../usage.md) to see how you can use your newly created S3 connector. \ No newline at end of file