Upload neverworld data to OSN?

[rabernat]

I just wrote a rough data guide as the README in this repo: https://github.com/ocean-eddy-cpt/cpt-data

It would be great to get some of the neverworld data into OSN. I'm using this issue to track that task. Let me know how I can help.

xref https://github.com/orgs/ocean-eddy-cpt/teams/members/discussions/35

[neerajabhamidipati]

Thanks! I'm having some trouble with open_mfdataset on xarray so I'm getting help from Raphael Dussin at GFDL. I'll keep you posted on any developments.

[rabernat]

Great. @raphaeldussin is an expert on this stuff! But also feel free to post your issue here. Even if you manage to resolve it privately, it can be good to have a public record of the problem and how it was resolved, for others to learn from.

[neerajabhamidipati]

Hi @rabernat, I've had a couple of issues so far:

• I found that I have to run aws configure to create the $HOME/.aws directory (before editing the config file).

• Step 2 with s3 --profile osn cp /local/path s3://Pangeo/ocean-eddy-cpt/<dataset name> needs to be replaced with aws s3 --recursive --profile osn cp /local/path s3://Pangeo/ocean-eddy-cpt/<dataset name>. Otherwise, I get an error saying either that s3 command doesn't exist (without aws) or that upload failed: ... [Errno 21] Is a directory: ... (without --recursive).

One issue which I haven't managed to resolve is that when I run the last command above, it says that An error occurred (InvalidAccessKeyId) when calling the CreateMultipartUpload operation: The AWS Access Key Id you provided does not exist in our records. I've copy-pasted the access key ID from your email so I'm not sure what's going on.

[rabernat]

Thanks for trying to move forward with this!

aws s3 --recursive --profile osn cp /local/path s3://Pangeo/ocean-eddy-cpt/<dataset name>

Yes, leaving out aws was a typo, sorry for that.

The AWS Access Key Id you provided does not exist in our records

Let's try a list command. Can you do

$ aws s3 --profile osn ls s3://Pangeo

You should get

                           PRE cm26_control/
                           PRE cm26_control_temp.zarray/

[neerajabhamidipati]

$ aws s3 --profile osn ls s3://Pangeo gives me an error:
An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.

[rabernat]

Ok, the credentials must be wrong. I'll double check and send them to you again.

[neerajabhamidipati]

I still get the same error after copy-pasting the lines from your email An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.

[rabernat]

Let's switch to slack to iterate faster.

[rabernat]

xref https://github.com/NOAA-GFDL/MOM6/issues/1203