Upload files to Matlab cluster

Hi Everyone!

Im using matlab cluster (AWS based). my code uses local folders and openes large amount of data from local muchine (~3 GB, of 10K files). All files are on Matlab Drive (synced), but for some reason i cant get the cluster to read them - it doesnt recognize their path.

The option of uploading them at the cluster properties is not relevant (taeks forever).

Any ideas?

attached is a scheme of my code

c = parcluster;
parpool(c)
my_folder = ...
    'D:\MATLAB_Drive\new_folder\test_files\';
    
parfor i = 1: table_length
    %tries to access files ffrom my_folder - cant do it...

Answer 1

[Walter Roberson]

Im using matlab cluster (AWS based)

What operating system is the AWS running? My understanding is that AWS clusters are mostly Linux based

my_folder = ...
    'D:\MATLAB_Drive\new_folder\test_files\';

A Windows file name would not work on a linux system.

It would be safer to use the matlabdrive() function together with fullfile():

https://www.mathworks.com/help/matlab/ref/matlabdrive.html

my_folder = fullfile(matlabdrive, 'new_folder', 'test_files');

[paul_dak]

Hey walter, thank you.

I tried using this method, but for some reason the cluster still doesnt recognize the folder. i guess since code runs online on some server, even the fullfile() method is not enought fir it to recognuze files location on drive :(

[paul_dak]

I justfound the problem i had.

part of my file path was cappitalized, and linux is case sensitieve, therefore could not identify the path.

It works now, after fixing that issue, and using the suggested functions above.

TNX!

[Walter Roberson]

The situation is slightly more complicated.

The issue is not exactly Linux being case sensitive but Windows not being case sensitive.

Linux, MacOS, and Windows: all three of them delegate case sensitivity to the file system being used. All three of them support mixing case-sensitive filesystems and case-insensitive file systems. NTFS filesystems (Windows) can be set to be case-sensitive (but it isn't common.) Mac HFS+ and APFS filesystems can be set to be case-sensitive (might be a bit more common.) Linux ext4 filesystems can be built to be case-insensitive (probably not very common yet.)

[Raymond Norris]

This is a good example that we often talk to folks about when running elsewhere. How best to write your code so that, within reason, you can rehost your code on another machine with no changes.

For example, how would you expand on the above example? Do you ensure that my_folder exists? Should it? If it doesn't, do you throw an error immediately (because you were expecting files in it) or do you create it because you'll need to write files to it? The sooner you address this, the easier it will be to debug issues when they do arise.

[paul_dak]

yeah, it is very important, i will pay more attention next time.