Guide program - calculation of multiple data sets

Hello all,

I have a GUIDE program in MATLAB where you can load folders containing datasets (many files with strain gauge data) and perform a variety of functions on these datasets. Currently when you select multiple folders in the GUI and click run, a for loop executes which will in turn execute functions which perform calculations (cycle counting, mean stress etc.) on the first dataset (it extracts data from each file within the folder and then plots the total result at the end), then after it has finished going through the first data set, it will loop back to the next folder and perform the same calculations.

I desire to modify the code so that if I select more than one folder the calculations will be performed on the datasets from both folders and provide a 'total' output rather than two separate outputs for two datasets. I appreciate this may be difficult to answer but any guidance or help would be greatly appreciated.

Let me know if there is any other information I can provide that might help.

Cheers

****UPDATE 15/09/2017

Thanks for the help per isakson and Walter.

I've decided that the best way to fix this issue is to change the logic of the code rather than trying to fix a poorly coded program. There was plotting inside for loops etc.

So i'm going to save the data produced by each loop in .mat files (one for each dataset). I will then concatenated these .mat files outside the loop. Then, from outside the loop, I'll plot this concatenated data.

Thanks

[per isakson]

Now you have

loop over all selected folders
   loop over all files in one folder
       perform the same calculations
   endloop
endloop

and you want

loop over all files in all selected folders
    perform the same calculations
endloop

Is that a proper summary of your question?

"first data set" does that comprise the data of all files in the first folder?

[HoboJoe]

Hello!

yes that's pretty much it.

yes when I refer to the first dataset that's all the files in the first folder.

Answer 1

[per isakson]

Without details it is difficult to be specific.

compile a "list", sad, of the files of the selected folders

sad = dir( fullfile( selected_folders{1}, 'pattern' ) );
for jj = 2 : length( selected_folders )
    d = dir( fullfile( selected_folders{jj}, 'pattern' ) );
    sad(end+1:end+length(d),1) = d;
end

loop over all files

for jj = 1 : length( sad )
    ffs = fullfile( sad(jj).folder, sad(jj).name );
    data = read_the_file( ffs );
    perform the same calculations
end

The field folder was added to the output of dir recently. If you have an older Matlab release you need to add it to the first loop.

[d.folder] = deal(selected_folders{jj});

and before the first loop

[sad.folder] = deal(selected_folders{1});

In response to comment

I assume your folder structure is similar to

h:\m\cssm
+---dataset_1_folder
¦   +---subfolder_1
¦   +---subfolder_2
¦   +---subfolder_3
+---dataset_2_folder
¦   +---subfolder_1
¦   +---subfolder_2
¦   +---subfolder_3
+---dataset_3_folder
¦   +---subfolder_1
¦   +---subfolder_2
¦   +---subfolder_3

Compile a master file list on Matlab R2016a. (R2017a has a more powerful function dir. There is a function, rdir, in the File Exchange, which also searches subdirectories.)

data_folders = {'h:\m\cssm\dataset_1_folder' ...
            ,   'h:\m\cssm\dataset_2_folder' ...
            ,   'h:\m\cssm\dataset_3_folder' };
% or generate data_folders automatically

all_data_files = struct( 'name',{}, 'date',{}, 'bytes',{}   ...
                       , 'isdir',{}, 'datenum',{}, 'folder',{} );        
for folder = data_folders
    sub_folders = dir( fullfile( folder{1}, 'subfolder_*' ) );
    sub_folders(not([sub_folders.isdir])) = [];
    for subfolder = reshape( sub_folders, 1,[] )
        d = dir( fullfile( folder{1}, subfolder.name, '*.csv' ) );
        [d.folder] = deal( fullfile( folder{1}, subfolder.name ) );
        all_data_files = cat( 2, all_data_files, reshape( d, 1,[] ) );
    end
end

Inspect

>> all_data_files(13)
ans = 
       name: '13.csv'
       date: '13-sep-2017 01:17:43'
      bytes: 18
      isdir: 0
    datenum: 7.3695e+05
     folder: 'h:\m\cssm\dataset_3_folder\subfolder_1'

And perform the calculations

for file = all_data_files
    ffs = fullfile( file.folder, file.name );
    % import data from file with existing code
    % data = read_the_file( ffs );
    % perform the same calculations
    disp( ffs )
end

outputs

...
h:\m\cssm\dataset_2_folder\subfolder_2\27.csv
h:\m\cssm\dataset_2_folder\subfolder_3\33.csv
h:\m\cssm\dataset_2_folder\subfolder_3\37.csv
h:\m\cssm\dataset_3_folder\subfolder_1\13.csv
...

[HoboJoe]

Thanks for the response. I'll try to work this into the code.

What do you mean by

data = read_the_file( ffs )

I'm not sure what 'read_the_file' here refers to.

also what does

'pattern'

refer to?

Cheers

P.S i should have already made it clearer initially, but each folder or dataset contains about 20 sub folders with each subfolder containing a number of .csv files numbered 0.csv, 1.csv 2.csv etc. Currently, depending on the desired data, functions will look up the sub folder within the dataset (folder) selected. Most of the time, there is overlap in the file names. i.e. there will be a file called 45.csv in dataset_1_folder/subfolder_5 and also in dataset_2_folder/subfolder_5.

I'm not sure whether this creates any issue when constructing the 'list' and executing the loop you posted.

[Walter Roberson]

read_the_file = @xlsread

[per isakson]

• "data = read_the_file( ffs )" &nbsp was my way of saying: import the data as you see fit.
• "pattern" See glob patterns specify sets of filenames with wildcard characters, e.g. "*.csv".

I added an elaborated version of my code to the answer

• "overlap in the file names" should not cause problems, since absolute paths are used throughout, thus fullfile everywhere.
• for folder = data_folders requires that data_folders is a row, thus, reshape

Answer 2

[Walter Roberson]

With R2016b or later you can use

datadir = 'Name/of/top/level/folder';
dinfo = dir( fullfile(datadir, '**', '*.csv') );
fullnames = fullfile( {dinfo.folder}, {dinfo.name} );

numfiles = length(fullnames);
results = cell(numfiles, 1);

for K = 1 : numfiles
   thisfile = filenames{K};
   thisdata = xlsread(thisfile);
   this_result = perform_the_calculation(thisdata);
   results{K} = this_result;
end

For older MATLAB versions, there are several recursive directory File Exchange contributions.