how to extract data from the fixed-width-field format using fscanf or textscan
13 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
Dmitrii Semikin
am 16 Jun. 2019
Kommentiert: Dmitrii Semikin
am 18 Jun. 2019
I am trying to parase large file, containing among others numerical data fixed in fixed-width-fields format without whitespaces (or any other separators) between the fields. Consider the example:
1223244
55 6 788
1122 3 4
In this text each line contains four numbers each - two digits wide. Expected result after parsing this flie is:
1 22 33 44
55 6 7 88
11 22 3 4
The prerformance is important for me, as the files I read may be hunreds of MBs large. From my experiments I discovered, that fgetl() is drammatically slower, than fscanf (I did not check the performance, but I expect, that textscan should also be significantly faster.
The problem is that I cannot find the way to parse data like this with fscanf or textscan. Could someone tell, if it is possible at all? If not, is there any other way to parse such a text file with good performance?
P.S. format of actual string is, of course, somewhat more complicated. Example string is:
1101332.18685714711829.064533733772535.874264373485 0 0
this string should be parsed into the numbers with the following widths: 8, 16, 16, 16, 8, 8, which in this particular case would result in the following numbers:
110
1332.18685714711
829.064533733772
535.874264373485
0
0
P.S.2: Differnence between the performance of fgetl and fscanf: On my laptop for the file, which contains 35597 lines the following code with fscanf complete in 0.253614 seconds
function [data] = test_fscanf_nodes_only_01()
file_name = 'myfile.txt';
file_id = fopen (file_name, 'rt');
cleanup_obj = onCleanup(@() fclose(file_id));
data = fscanf(file_id, '%8d%16f%16f%16f%8f%8f', [6, Inf]);
end
while the following code with fgetl needs 6.343209 seconds to complete, even though it does much less
function [data] = test_fscanf_nodes_only_02()
file_name = 'myfile.txt';
file_id = fopen (file_name, 'rt');
cleanup_obj = onCleanup(@() fclose(file_id));
lines_count = 0;
while ~feof(file_id)
current_line = fgetl(file_id);
lines_count = lines_count + 1;
end
data = 1;
fprintf('Lines count: %d', lines_count);
end
The main problem with the first snippet is that it returns wrong result (because for fscanf the width of the field counts from the first digit it finds and not from the current position of the file pointer (which means, that the leading whitespaces are not counted as field width).
EDIT: The dimension in fscanf is changed from [5, Inf] to [6, Inf] according to Jan's comment.
8 Kommentare
Walter Roberson
am 17 Jun. 2019
textscan() is not the same as fscanf(): with textscan it is possible to change the Whitespace and Delimiter properties in a way that does not skip leading whitespace. because of the way that the numeric fields parse numbers, a leading whitespace in a numeric field in such an arrangement would be counted as an invalid character, so this turns out to only be useful for fixed-width character fields.
Akzeptierte Antwort
Walter Roberson
am 17 Jun. 2019
fileread into a character vector. regexprep to insert a space before every field. textscan the character vector.
4 Kommentare
Walter Roberson
am 17 Jun. 2019
newS = regexprep(S, '^(.{8})(.{16})(.{16})(.{16})(.{8})(.{8})', '$1 $2 $3 $4 $5 $6', 'lineanchors');
Weitere Antworten (1)
Stephen23
am 17 Jun. 2019
A simple solution based on regeprep and sscanf. Before the file-reading loop:
vec = [8,16,16,16,8,8];
rgx = sprintf('(.{%d})',vec);
rpl = sprintf('$%d,',1:numel(vec));
In the file-reading loop:
str = fileread('temp2.txt');
str = regexprep(str,rgx,rpl,'dotexceptnewline');
mat = sscanf(str,'%f,',[numel(vec),Inf]).'
Giving (the test file is attached):
mat =
110.00000 1332.18686 829.06453 535.87426 0.00000 0.00000
220.00000 2332.28686 829.06453 535.87426 0.00000 0.00000
330.00000 3332.38686 829.06453 535.87426 0.00000 0.00000
440.00000 4332.48686 829.06453 535.87426 0.00000 0.00000
550.00000 5332.58686 829.06453 535.87426 0.00000 0.00000
Siehe auch
Kategorien
Mehr zu String Parsing finden Sie in Help Center und File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!