fileDatastore request to AWS S3 limited to 1000 files

I have a bucket on an AWS S3 server which I can reach without any problems.

The issue is: The answer from the server is limited to 1000 files. How can I increase this number? My target is to get just the list of filenames.

I also tried different calls of fileDatastore.

imds = fileDatastore(['s3://',<bucket>,','/'],'FileExtensions','.csv','ReadFcn',@load,'IncludeSubfolders',true)

[Sameer Gummuluru]

Hey MaKe,

Could you find a solution for this? I have the same issue.

[MaKe]

Sadly no.

In the end I found a workaround via PowerShell and the AWS Tools. Not pretty, but it works:

https://blog.ardourdigital.co.uk/b/blog/posts/copying-all-files-from-an-aws-s3-bucket-using-powershell

[Sameer Gummuluru]

I was able to figure out a temporary solution to this, today. On older versions of MATLAB, I was able to use filedatastore to access more than 1000 files from S3. (I tried on MATLAB R2019a)

You can give it a try if working on older MATLAB versions is fine by you.

[Jhon Wine]

I have the same issue. When using Matlab R2020b I cannot get more than 1,000 files.

However while using Matlab R2019b, fileDataStore seems to return all the files I need.

Using AWS CLI seems to work fine, but is not as elegent of a solution.

Answer 1

[Div Tiwari]

Hi all, this issue was resolved in MATLAB R2021a and R2020b Update 5: https://www.mathworks.com/support/bugreports/details/2407120.

If you're using R2020b, you can install updates by clicking on the notification icon towards the top-right of the MATLAB window.

[Jhon Wine]

I just installed R2021a, problem persists.

In fact I have two versions of Matlab on the same computer. 2019a shows 2000 files, 2021a shows 1000 files...