Can I use .aws/config file to provide AWS S3 credentials?

I have been able to successfully access files on an AWS bucket via the syntax

readtable('s3://bucketname/filename.csv')

when I store my AWS credentials in the file .aws/credentials.

However, when I try to use .aws/config instead, I get the error message

===========================================================================================

Location 's3://bucketname/filename.csv' requires credentials. You must setup

an AWS credentials file, or set the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

===========================================================================================

The .aws/config file method works for both command line and Python access, so I am confident I have the file itself correct.

Answer 1

[Kojiro Saito]

readtable checks AWS Credentials either

• environment variable "AWS_SHARED_CREDENTIALS_FILE"
• ".aws/credentials" file
• environment variable AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

in matlab.io.internal.vfs.validators.validateCloudEnvVariables function.

.aws/credentials is hard coded in \toolbox\shared\virtualfileio\+matlab\+io\+internal\+vfs\+validators\hasS3CredentialsFile.m, so if you want to change the path, currently you need to edit the code

from

credentialsFilePath = fullfile(basePath, '.aws', 'credentials');

to

credentialsFilePath = fullfile(basePath, '.aws', 'config');

But, as AWS document says, "The AWS CLI stores sensitive credential information that you specify with aws configure in a local file named credentials, in a folder named .aws in your home directory. The less sensitive configuration options that you specify with aws configure are stored in a local file named config".

The credentials are sensitive and I would store them in credentials file as by default.

[the cyclist]

Thanks for the answer!

Making that change redefining the credentialFilePath from credentials to config was not sufficient to fix the problem. It seems to find the credentials, but then I get the error

=================================================================================

Error using readtable (line 498)

Unable to access: 's3://bucketname/filename.csv'

Permission denied.

=================================================================================

So, it's not quite equivalent.

Also, I guess that since AWS allows the config method, I would expect the MATLAB implementation would allow it as well, and without needing to change a read-only file that will have to be changed with every new version.

[Kojiro Saito]

I was able to access to my S3 bucket with .aws\config file after copying & pasting aws_access_key_id and aws_secret_access_key and modifying hasS3CredentialsFile.m.

Permission denied means your IAM role is not allowed to access S3 object. Could you confirm you have proper policy?

"Action": "s3:GetObject"

"Resource": "arn:aws:s3:::bucketname/*"

is necessary for reading S3 files from MATLAB.

[the cyclist]

I can't see the policy list. (I don't sufficient privileges in my org.)

However, the fact that I can use Python or the command line (via config file) suggests to me that this is not the issue. Right?

This is not particularly high-priority to me at this point, as I can in fact just use a credentials file. (It just means an additional manual step that I was hoping to avoid.) But, if you have other ideas I may try them later, for the sake of someone finding this solution.

[Kojiro Saito]

The necessary policy for reading from and writing to S3 depends on softwares. Some policies might not needed in other tools but as my investigation, MATLAB needs GetObject, ListBucket and GetBucketLocation policies.

How about putting AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY written in .aws\config to MATLAB with setenv then, doing readtable?

setenv('AWS_ACCESS_KEY_ID', 'xxx')
setenv('AWS_SECRET_ACCESS_KEY', 'xxx')
%setenv('AWS_REGION', 'xxx') % Optional Ex) us-east-1
readtable('s3://bucketname/filename.csv')

If this does not work, there are some more policy needed in IAM policy for MATLAB to read from S3.