Importing multiple .txt files containting single colums of numerical data

I'm trying to import multiple .txt files, up to 100 at a time. The files are named in a fashion like 'test_1.txt, test_2.txt'... etc. These .txt files simply contain a 1024 long column of integers. I'm hoping that I can get some code such that I can import all of these files at once, and obtain a variable for each imported file, which will have a suitable name, such as 'test_1' 'test_2' etc. The purpose of this is to perform analysis on each of these sets of data, so first, would it be better to import them into one matrix, with each column or row representing a separate test file, or would my suggestion of separate variables be better suited?

I found the following code somewhere online, and it almost gives me what I want.

datafiles = dir('*.txt');

filename = cell(length(datafiles),1);
data = cell(length(datafiles),1);

for k = 1:length(datafiles)
    filename{k} = datafiles(k).name;
    data{k} = importdata(filename{k});
end

The result is a 100x1 cell, where each element within the cell is a 1024x1 double; I am not sure if this is a good format for me to do further analysis. How can I alter the code to get the best result I am looking for?

Many thanks.

Answer 1

[Davide Ferraro]

Hi Ross,

it depends on the type of analyses you need to do. If you are not expert working directly with cell array using command such as CELLFUN you may like to transform this variable into a standard matrix. By using the command CELL2MAT:

data_matrix = cell2mat(data);

you should be able to get a single matrix. In order to get the matrix with each variable in a column a transposition may be needed in your case:

data_matrix = cell2mat(data');

At this point you can use statistical operator directly on the matrix. In example:

mean_values = mean(data_matrix);

will compute the mean of each individual column. This makes your code faster, shorter and simpler. I would avoid getting 100 different variables, you will get crazy in automatically process all of them with complex loops and EVAL instructions.

[Ross]

Hi Davide, thanks for the quick reply.

The line of code with the transpose worked perfect; I knew my code wasn't far away from what I needed.

The analysis I need to perform is primarily Fourier transforms, so would this matrix form be most suitable, as you suggest?

If I need to split it into separate variables, this should be easy from the single matrix.

[Davide Ferraro]

If you are going to use FFT this format is ok. From FFT documentation:

"If the input X is a matrix, Y = fft(X) returns the Fourier transform of each column of the matrix."

[Ross]

Sounds perfect. The result of my code plus the line you suggested means my data is in rows, not columns, but that's not an issue. Thanks again for the help.

EDIT: Scrap that, it does produce the data in columns. I'm a bit rusty with MATLAB.

[Ross]

I've come across a problem. As mentioned before, my files are named "TRIAL1__1.txt, TRIAL1__2.txt' up to 'TRIAL1__100.txt'. Using the previously mentioned code, it imports 'TRIAL1__1.txt' then 'TRIAL1__10.txt' then 'TRIAL1__100.txt' instead of the correct order. I could manually sort this with some code, but not all of my data sets are only 100 files, so is there a simple way to fix this?

Answer 2

[Azzi Abdelmalek]

for k=1:100
data{k}=dlmread(sprintf('test_%d.txt',k))
end

[Ross]

Hi Azzi, thanks for the quick reply.

This seems like a very nice neat code, but I'm getting:

Error using dlmread (line 139)
Mismatch between file and format string.
Trouble reading number from file (row 1025u, field 1u) ==> \n

Error in testimport (line 19)
    data{k}=dlmread(sprintf('TRIAL1__%d.txt',k))

when I try to run. (Note my files are named TRIAL1__#.txt (double underscore) where # is 1:100. Any idea what would give this error? The code seems good to me.

[Azzi Abdelmalek]

Check if your files are in the same folder then your matlab working folder. type

dir TRIAL*.txt

[Ross]

They are in the correct folder. I get a list of the 100 files. Still no luck. This solution looks like it would solve the problem I having on the other answer on this page, where the order of the filenames generated by

dir('*.txt')

is incorrect. I get 1, 10, 100, 11, 12 etc.

[Ross]

Sorry for the multiple posts, but I got it working. Used the code you suggested but importdata instead of dlmread. Thanks for the pointers!