Best way to process large data from Dropbox
6 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
I have a very large permanent Dropbox link which has 6 folders in it. Each folder has the following file organization (I specified the organization for Folder 1 only where - represents a folder and *represents a file and 3 lines with dots means continue same pattern, but all of the main folders and subfolders are organized the same).
- Folder 1
- 12390r3398
- 20253023432
* A2308432.edf
* calibration.xlsx
* EventList.xlsx
* Stage.csv
- 20253023789
* A2308449.edf
* calibration.xlsx
* EventList.xlsx
* Stage.csv
.
.
.
- 202530243808
* A23086903.edf
* calibration.xlsx
* EventList.xlsx
* Stage.csv
- 12390r4490
- 20258900023
* A23489001.edf
* calibration.xlsx
* EventList.xlsx
* Stage.csv
.
.
.
- 20258978290
* A23489876.edf
* calibration.xlsx
* EventList.xlsx
* Stage.csv
.
.
.
- Folder 2
.
.
.
- Folder 3
.
.
.
- Folder 4
.
.
.
- Folder 5
.
.
.
-Folder 6
.
.
.
Now, I wrote a function that I will call myfunction. The inputs of myfunction are a .edf file and a .xlsx file. The output of myfunction is a 6 column array, and the number of rows depends on the input files.
I want to apply myfunction to all of the file pairs in each of the smallest subfolders in this Dropbox link. For example, I want to have arr1 = myfunction('A2308432.edf', 'EventList.xlsx') for the file pair located in subfolder '20253023432' above and then I want arr2 = myfunction('A2308449.edf', 'EventList.xlsx') for the file pair located in subfolder '20253023789' and so on up until the last smallest subfolder in Folder 3 (please notice that even though each subfolder contains an 'EventList.xlsx' file, that each of these Excel files are actually different despite having the same name). I cannot directly download this dropbox link, or even any of the 3/6 main folders, or even a certain subfolder of a main folder as they are too large (the entire Dropbox has about 100 GB of data).
Does anyone know how I can do this in MATLAB by calling myfunction on all of these file pairs directly from the Dropbox link, or if I can iteratively download file pairs from Dropbox and then run my function on the file pair and then delete the file pair before moving on to the next one? If you can provide code to help, I would very much appreciate as I do not have a systems background. Thanks.
4 Kommentare
Walter Roberson
am 27 Mär. 2023
I mean writing code in MATLAB that would do the downloading.
Do you have "Dropbox Plus" ? That would allow you to install dropbox desktop to get a mountpoint such as ~/Dropbox that can be used to refer to files stored on Dropbox.
If you do not have the "Plus" subscription then Dropbox Desktop would want to copy all of the files to your local system.
Antworten (1)
Walter Roberson
am 27 Mär. 2023
Verschoben: Walter Roberson
am 27 Mär. 2023
I will use MacOS / Linux syntax for referring to files for this purpose
DBdir = "~/Dropbox/Fab403/Infrared/Tests";
EL = "EventList.xlsx";
Tdir = tempname() + ".cache";
mkdir(Tdir);
edfinfo = dir(DBdir, '**/*.edf');
edfnames = fullfile({edfinfo.folder}, {edfinfo.name});
numedf = numel(edfnames);
for K = 1 : numedf
thisedf = edfnames{K};
[thisfolder, thisbase, thisext] = fileparts(thisedf);
thisEL = fullfile(thisfolder, EL);
copyfile({thisedf, thisEL}, TDir);
Tedfname = fullfile(TDir, [thisbase thisext]);
TELname = fullfile(TDir, EL);
try
myfunction(Tedfname, TELname);
end
try delete(Tedfname); end
try delete(TELname); end
end
3 Kommentare
Walter Roberson
am 30 Mär. 2023
Sorry, I do not have Dropbox Plus to test this with (it will not work regular Dropbox), and I am not willing to pay for Dropbox Plus.
Siehe auch
Kategorien
Mehr zu File Operations finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!