How to efficiently extract 2-D data from multiple text files and save it as a single 3-D variable?

I have 456 comma-delimited text files. Each file represents one month of climate data, containing 600 rows and 900 columns.

I want to extract the data from each text file and store it in one three-dimensional variable (600x900x456).

The names of the text files are standardized. For example, the first file is called 'data198201.txt' (which means refers to January 1982), the second file is called 'data198002' (which refers to February 1982), and so forth all the way to December 2019.

I have a successful way of extracting the data for individual 12-month periods, for example, Jan-1982 to Dec-1982:

% Vector of months
mon = [{'01'};{'02'};{'03'};{'04'};{'05'};{'06'};{'07'};{'08'};{'09'};{'10'};{'11'};{'12'}];

% Extract the monthly data for 1980
data1982 = zeros(600,900,12); % Preallocate memory
for b = 1:12
	filename = ['data1982',mon{b},'.txt']; 
	data = table2array(readtable(filename));
	data1982(:,:,b) = data;
end

But for efficiency I want to create a loop that goes through all 456 months at once. Here is what I tried:

% Create a variable of years as cells
a = 1982:2019;
y = [];
for n = 1:length(a)
	ycell = {num2str(a(n))}; % save year as a cell
	y = [y ycell]; % final output: [{'1982'},{'1983'},{'1984'},...,{'2019'}]
end

% Extract and store all the data as one variable
alldata = zeros(600,900,456); % pre-allocate memory
for n = 1:length(y)
for b = 1:456
for c = 1:12
	filename = ['data',y{n},mon{c},'.txt'];
	data = table2array(readtable(filename));
	alldata(:,:,b) = data;
end
end
end

I had the loop run for ~20 minutes before I came to the conclusion that I either did something wrong or that there must be a more efficient way to do this. I would greatly appreciate any corrections or tips that I can try to implement.

Answer 1

[Voss]

Using sprintf to construct your file names will make that easier. Also, you only need one additional loop (loop over years) to wrap around the loop over months you already had.

a = 1982:2019;
alldata = zeros(600,900,numel(a)*12); % pre-allocate memory
for n = 1:numel(a)
    for c = 1:12
        filename = sprintf('data%4d%02d.txt',a(n),c);
        % does readmatrix work for these files?
        % instead of reading to a table and immediately converting to an array
        data = readmatrix(filename);
%         data = table2array(readtable(filename));
        alldata(:,:,12*(n-1)+c) = data;
    end
end

[Austin M. Weber]

Hi @_,

Thank you for your help. I just tested it out and the readmatrix() function was a success. I am not entirely sure I understand how the sprintf() function works, but it does seem to do the job.

Your soluion for wrapping around the loop is simple and clear. Thanks again!

[Voss]

Here's an explanation of the format specification (first argument) in sprintf:

%  sprintf('data%4d%02d.txt',a(n),c)
%  ^^^^^^^                  sprintf: (print) (f)ormatted '(s)tring' (actually a character vector in MATLAB)
%           ^^^^            data   : the literal character vector 'data'
%               ^^^         %4d    : a formatted (d)ecimal number, (4) characters long
%                  ^^^^     %02d   : a formatted (d)ecimal number, (2) characters long,
%                                    preceded by (0)s if necessary, e.g., 9 becomes '09'
%                      ^^^^ .txt   : the literal character vector '.txt'

Then the remaining arguments to sprintf are the numbers used in the formatted parts of the first argument

sprintf('data%4d%02d.txt',1982,6) % returns 'data198206.txt'

ans = 'data198206.txt'

[Austin M. Weber]

WOW! That is a really useful function. Thanks for explaining!

[Voss]

You're welcome!