How to import large text file data in to workspace?

Hi, I am having a text file with following format

First Line is a string contains device parameters

rest is 18000x12 Lines format is double

I am using uigetfile to access my file then fscanf to read data. But this text file should reside in the same workscpace where the reading script is placed.

Is there any way to read above file without placing .txt file into current directory. The command importdata is not useful in my case only.

Answer 1

[Walter Roberson]

[filename, pathname] = uigetfile(.....)
fullname = fullfile(pathname, filename);

ncol = 12;
fmt = repmat('%f', 1, ncol);

fid = fopen(fullname, 'rt');
data_cell = textscan(fid, fmt, 'HeaderLines, 1, 'CollectOutput', 1);
fclose(fid);

data = data_cell{1};

[Muhammad Abdurrehman]

Working perfectly ! Dear thanks a lot for such a quick reply.

[LeChat]

I have a similar issue (so I am not making a new post from scratch), except that I have two sets of arrays to load within the given (HUGE) text file:

IGOR

WAVES/O/R/S it	ntot	nsi	nsi1	viol	Ek	Ep	fydown	fxdown	fyup	fxup	height	wallx	ncontglis	ramil	ttheta	flowrate	gravity	sigyy	Pdiss
BEGIN
0.000000e+00	2285	1177	49	4.291105e-04	2.199055e-01	6.468194e-01	1.043197e+00	1.770681e-01	1.046892e+00	1.253301e-01	4.724858e+01	0.000000e+00	4	5.071849e-01	0.000000e+00	1.849395e-01	0.000000e+00	1.044623e+02	5.545544e-02
2.182179e-01	2192	1162	62	4.089641e-04	1.057425e-01	5.829052e-01	1.935262e+00	4.054038e-01	1.316429e+00	5.103208e-01	4.725055e+01	4.364358e-03	120	5.071849e-01	0.000000e+00	1.926225e-01	0.000000e+00	9.435535e+01	1.474838e-01
4.364358e-01	2146	1161	72	4.276555e-04	1.387203e-01	5.057031e-01	1.752111e+00	4.153276e-01	1.295198e+00	5.141023e-01	4.725333e+01	8.728716e-03	118	5.071849e-01	0.000000e+00	2.206827e-01	0.000000e+00	8.305721e+01	1.052721e-01
...
...
...
6.546537e-01	2101	1159	77	4.018822e-04	1.758128e-01	4.401088e-01	1.287929e+00	2.003077e-01	1.204818e+00	3.809746e-01	4.725580e+01	1.309307e-02	157	5.071849e-01	0.000000e+00	2.442545e-01	0.000000e+00	7.423966e+01	1.112252e-01
8.728716e-01	2038	1149	82	4.071467e-04	1.716760e-01	4.085304e-01	8.694503e-01	1.602622e-01	1.232470e+00	4.363594e-01	4.725785e+01	1.745743e-02	174	5.071849e-01	0.000000e+00	2.959853e-01	0.000000e+00	6.873186e+01	6.264896e-02
2.182179e+03	1702	1053	176	2.414867e-02	1.142044e-01	1.988853e+01	1.037340e+00	2.408913e-01	1.040168e+00	2.314350e-01	4.806088e+01	1.864359e+01	89	5.071834e-01	0.000000e+00	2.751452e-01	0.000000e+00	7.295313e+01	4.865551e-03
END

WAVES/O/R/S rr	xx	yy	vx	vy	ome
BEGIN
5.294830e-01	-4.487053e+00	4.813038e+00	-2.987710e-04	3.890736e-04	2.466829e-05
4.238057e-01	-3.958683e+00	4.266787e+01	1.540010e-02	-3.687625e-05	-7.041431e-03
5.676467e-01	1.034062e+01	1.001319e+01	-2.464676e-04	6.216398e-04	-2.657809e-04
4.838913e-01	-6.190074e+00	3.665010e+01	1.417410e-02	1.503664e-04	4.720945e-05
5.109182e-01	-6.538011e+00	3.758183e+01	9.950496e-03	-1.148410e-03	8.513154e-03
...
...
...
5.134217e-01	1.014562e+01	1.784603e+01	3.745296e-05	9.864389e-04	1.788069e-04
4.405835e-01	1.125609e+01	1.043561e+01	-5.660598e-07	1.738286e-04	-2.165469e-04
4.304009e-01	1.204259e+01	2.494275e+01	1.842662e-03	8.021805e-04	5.170441e-04
4.785989e-01	-1.188209e+01	9.460645e+00	2.490334e-04	7.324075e-04	-5.055283e-04
5.524777e-01	-1.605586e+00	2.877578e+01	-2.453181e-03	4.989078e-04	-1.470961e-02
5.005685e-01	-5.499865e-01	2.135791e+01	3.699902e-03	3.077965e-03	-3.379855e-03
5.381660e-01	1.208255e+01	1.447031e+01	7.099516e-04	4.243457e-04	-4.595301e-04
END

I am using the following code, using fgetl, textscan and cell2mat to load my file:

% %% load timeout.igr text file
%
%% load the first part of the file: it	ncontot	nsi	nsi1	viol	Ek	Ep	fydown	fxdown	fyup	fxup	height	wallx	ncontglis	ramil	ttheta	flowrate	gravity	sigyy	Pdiss
tic
fid = fopen('myfile.igr');
tline = fgetl(fid);
tline = fgetl(fid);
tline = fgetl(fid);
tline = fgetl(fid);
tline = fgetl(fid);
%
datatime=[];
colonLocation=[];
while ischar(tline) & isempty(colonLocation)
    colonLocation = strfind(tline, 'END');
    if isempty(colonLocation)
        newline=cell2mat(textscan(tline,'%f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64 %f64',20,'Delimiter',{'\t'}));
        datatime = [datatime; newline];
    end
    tline = fgetl(fid);
end
toc
%% load the second part of the file: rr	xx	yy	vx	vy	ome
tline = fgetl(fid);
tline = fgetl(fid);
tline = fgetl(fid);
%
datafinal=[];
colonLocation=[];
while ischar(tline) & isempty(colonLocation)
    colonLocation = strfind(tline, 'END');
    if isempty(colonLocation)
        newline=cell2mat(textscan(tline,'%f64 %f64 %f64 %f64 %f64 %f64',6,'Delimiter',{'\t'}));
        datafinal = [datafinal; newline];
    end
    tline = fgetl(fid);
end
fclose(fid);
toc

When I generate a text file that is ~2Mb, this code runs in ~7-8sec. The issue comes when I run a longer simulation and my output text file is then >100Mb. My current Matlab code takes ages to run (I have actually never waited until it completes the task).

Is there something I could to to make it run faster? What (if possible automatic/script) other way should I try?

Thank you for your help and support.

[Walter Roberson]

@LeChat

        datatime = [datatime; newline];

That is slow. Each time you append data to the end of an array, MATLAB has to allocate space for the new larger array, then copy the old values into the array, copy in the new value, then release the old space. The first time that would involve copying 0 existing rows; the second time it would involve copying 1 existing row; the third time, 2 existing rows, the 4th time, 3 existing rows, and so on. After N times that would be 0+1+2+3+...(N-1) existing rows copied, which would be (N-1)*N/2 rows copied from the beginning. It is an N^2 copying process.

Instead, if you can establish a maximum size, pre-allocate the memory, and assign into the array,

        maxents = 50000;
        
        datatime = zeros(maxents, 6);
        
        ...
        datatime(counter,:) = newline;

Note: you are asking for the entire format of six %f64 items to be processed 6 times, so you are trying to read in 6 rows of 6 columns at a time. Are you sure that the data has a multiple of 6 rows in each block?

[LeChat]

Dear Walter, thank you very much for your answer.

Would you know how I can get the maximum row number to load in order to pre-allocated the memory and assign into the array? I am thinking of something similar to "wc -l filename" in shell, but here in Matlab, with -if possible- the possibility to get the respective size of each of the two blocks of data...

I actually don't remember why I was processing the data by sets of 6 rows... I know the number of columns in each block (the first block has 20 columns, the second one has 6 columns), but the number of rows is a priori unknown.

Thanks you very much for your help and interest.