Code has to load too many structs being loaded. Better way ??

I am working on code that loads cnt files in EEGLAB from all the subjects that have been studied. There are about 80 some odd subjects and they have 8 cnt files each that load as structures. This is roughly 640 files. I am noticing that this code slows down the computer quite appreciably when it crosses about the 65 or the 70th person.I am trying to figure out if there is a better way of doing it. This is only a part of a larger data processing function that i am working. Once these cnt files are loaded, i will run processing code on each one of these files. That is a lot of processing. The code is as follows:

 function cnt =imported_cnt()
tic
cnt_home = uigetdir()
cnt_folders = strcat(cnt_home, '\sub*')
folder_want = dir(cnt_folders)
%The folders will now start with the right syntax always
counter = 1;
for y= 1: size(folder_want)
    fold_name = (strcat(cnt_home,'\', folder_want(y).name, '\01-cnt\'))    %generates the address of the folder in the computer.
    files_cnt = dir(strcat(fold_name, '*.cnt'))   
    for z = 1: size(files_cnt,1)
        try
        % read file
        % if it works it does more code.        
        cnt_loaded = strcat(fold_name, files_cnt(z).name)
        strCorrupt = strfind(cnt_loaded, 'corrupt')
        if isempty(strCorrupt)
            cnt(counter) =pop_loadcnt(cnt_loaded)% , 'dataformat', 'int16')
            counter = counter + 1;
        end  
        catch
    % File can't be read - it's corrupt.  Alert the user.
            errorMessage = sprintf('Error in function %s() at line %d.\n\nError Message:\n%s', ...
      ME.stack(1).name, ME.stack(1).line, ME.message);
            fprintf(1, '%s\n', errorMessage);
            uiwait(warndlg(errorMessage));
            continue; 
    end
end % of for loop
end
toc

I would appreciate any help that i can get.

Answer 1

[Walter Roberson]

You did not mention how large the data is, or how much memory you have. It could be that you are using more memory than you have available RAM and that your system is swapping to disk to hold the remaining files.

Generally speaking:

if you have restricted memory, load only as much as you need to process independently. If the cnt files for each subject are to be handled independently that might mean loading only one cnt file at a time; if the cnt files are used together that might mean loading one subject at a time.

However, in some cases it is more efficient to load a group of data at one time. In that case, load only as much as can "easily" be processed.

If you start getting into situations where you need all of the data for all of the subjects simultaneously, then you start needing different techniques, such as using mapreduce()