Script to access Amazon S3 files

81 views (last 30 days)
André Bibaud
André Bibaud on 25 Sep 2017
Commented: Dmitry Kaplan on 8 Feb 2022
I setup a cluster on AWS EC2 to run my scripts faster. Right now my script load input variables locally (on my C: drive) and write simulation results also locally. I setup an S3 bucket with a sub-directory for my input variables and a sub-directory for the results. How do I change the load and write function to point to the S3 bucket instead of my c: drive?
Thanks

Answers (3)

Jhon Wine
Jhon Wine on 24 Jan 2018
I found datastore to be very easy to use. Run the code below in each worker.
%Set S3 Credentials
setenv('AWS_ACCESS_KEY_ID', 'ID');
setenv('AWS_SECRET_ACCESS_KEY','Key');
setenv('AWS_REGION', 'us-west-2');
%Load Data
fp= 's3://mybucket/data/file.data';
ds=fileDatastore(spectralFilePath,'ReadFcn',@AWSRead);
data=ds.read;
Also define this function
function data= AWSRead(fileName)
fid = fopen(fileName);
data= fread(fid,inf,'short');
fclose(fid);
end
Make sure the EC2 cloud is on the same site as S3 to avoid data transfer bottlenecks

David Fink
David Fink on 28 Sep 2017
Since S3 uses a RESTful architecture, use 'webread' and 'webwrite' to access the S3 bucket.
  1 Comment
Jhon Wine
Jhon Wine on 17 Nov 2018
Is there an example, how to use webread and webwrite with S3? There is no good tutorial..

Sign in to comment.


Harish Sundaresh
Harish Sundaresh on 26 Jun 2021
can MATLAB read .json files from aws s3? I get these errors
%% webread
webread(fname)
Error using webread (line 119)
The 's3' protocol specified in URL, 'url', is not supported. Specify the URL with the protocol 'http://' or 'https://'.
%% jsondecode
Error using jsondecode
JSON syntax error: expected value but found end of text.
  1 Comment
Dmitry Kaplan
Dmitry Kaplan on 8 Feb 2022
I could not get webread to work. I am sure there is a way, but I could not figure it out.
Instead, I use something like this
fds = fileDatastore('s3://blah/blah/filename.json','FileExtensions',{'.json'},'readfcn',@fileread);
file_data = read(fds);
This defines the read function for the datastore to be fileread(). Invoking the read() actually runs fileread() and passes to it the filename -- because the fds.ReadMode is 'file'.

Sign in to comment.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by