reading multiple text file from two different folders using nested for loop
3 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
i am using the following lines to read multiple text files and do some comparison between them. Baiscally the first loop will read text files of certain format (we have 30 files of this type). the second loop will read text files of another format (we have 33000 of them) and we do some comparison between them.
I have my code working and it is fine but it is extremely slow because of the nested loop, and its important to mention that i dont really need these loop for anything else, only for reading the files from folders.
So is there a more effecient way of doing this.
for n = 1 :nfiles2
fprintf('.......... File - %d of %d\n',n, nfiles2)
fullFileName2 = fullnames2{n};
t2 = readtable(fullFileName2); %t2 stores the ionosonde profiles
for m = 1 :10
fprintf('.......... insideFile - %d of %d\n',m, 10)
fullFileName1 = fullnames1{m};
t1 = readtable(fullFileName1); %t1 stores the ISR profiles
0 Kommentare
Antworten (1)
Konrad
am 31 Mär. 2022
Hi Salma
It depends on what exactly you want to do with the data in the text files, but reading the file line by line seems to be considerably faster compared to readtable()
tic;
for k = 1:100
t = readtable('test.csv');
end
toc
tic;
for k = 1:100
fid = fopen('test.csv');
clear tl;
tl{1} = fgetl(fid);
while ischar(tl{end})
tl{end+1} = fgetl(fid);
end
fclose(fid);
end
toc
Of course readtable() converts numbers to double while fgetl() returns char and if you have to do type conversion the speed gain might disappear. But I think its worth trying.
Best, Konrad
2 Kommentare
Konrad
am 31 Mär. 2022
Bearbeitet: Konrad
am 31 Mär. 2022
Yes, thats what I ment with type-conversion ;)
I assume your data is comma delimited.
get the header in the first row:
fid = fopen('test.csv');
tline = fgetl(fid);
header = strsplit(tline,','); % <- the variable names as cellstring
If the data is all numeric (with variable names in the first row) you can do the following:
tic;
data = csvread('test.csv',1,0);
toc
Else: you can read the text data, store it in a cellstring and convert the numeric part (you will have to know which columns you need). This takes much longer due to the type conversion (and the growing array, which could be avoided by preallocating), but could be made faster if you only need a small fraction of the data for your comparison. But I don't know what you want to compare, so here's how you get everything from the test.csv:
tic;
dataCellstr = cell(0);
selectedCols = 1:3;
tline = fgetl(fid);
while ischar(tline)
cols = strsplit(tline,',');
dataCellstr(end+1,:) = cols(selectedCols);
tline = fgetl(fid);
end
data2 = str2double(dataCellstr);
fclose(fid);
toc
Siehe auch
Kategorien
Mehr zu Data Import and Export finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!