Why does textscan read only 95% of my data file?

My dat file has 423,000+ ascii lines that look like:

2017-08-30 12:34:56 7.89

When I use textscan, each of the 7 cells in the returned variable only have 413315 values. If I do it line by line with fgetl, I get all 423,000+ values. Textscan takes a few seconds. fgetl takes several minutes.

DATAX=textscan(fid,'%d-%d-%d %d:%d:%d %f');

Next thing to try is to sed replace '-', ':', and ' ' with \t then try again. Unfortunately I have almost 400 files like this. An opportunity to improve my shell scripting...

Any help is greatly appreciated.

[Janice Nelson]

Here's what worked:

        filecontent = fileread(TheFileName);
        tokens = regexp(filecontent, ...
          '(?<date>[-0-9]+\s+[0-9:.]+)\s+(?<value>-?[0-9.]+e?+)*', ...
          'names');
        %dates = datetime({tokens.date});
        dates = datenum({tokens.date});
        values = str2double({tokens.value});

It gets both f & e formatted values. datenum seems slower than datetime but semilogy doesn't like datetime numbers.

[Janice Nelson]

Thanks to all for your help!

Answer 1

[Janice Nelson]

Here's what worked:

filecontent = fileread(TheFileName);
tokens = regexp(filecontent, ...
  '(?<date>[-0-9]+\s+[0-9:.]+)\s+(?<value>-?[0-9.]+e?+)*', ...
  'names');
%dates = datetime({tokens.date});
dates = datenum({tokens.date});
values = str2double({tokens.value});

It gets both f & e formatted values. I used datenum though it seems slower than datetime so semilogy can function. Plot() works with datetime number types, but semilogy doesn't.

[Cedric]

Next time accept the answer of the people who helped you getting to some solution.

Depending how fast you need the approach to be, in my experience it is often faster to find/update discrepancies in a text buffer using a very short and efficient regular expression, and then to parse it using SSCANF, TEXTSCANF, DATENUM, etc.

If your content is really something like

 2017-08-30 12:34:56 7.89
 2017-08-30 12:34:56 7.89
 ..

why not just splitting on white spaces using STRPLIT or REGEXP with a \s+ pattern, reshaping, concatenating columns 1 and 2, and converting with DATENUM and STR2DOUBLE?

[Walter Roberson]

semilogy is the same as plot() followed by set() 'yscale', 'log' on the axis

Answer 2

[Walter Roberson]

filecontent = fileread(TheFileName);

tokens = regexp(filecontent, '^(?<date>[-0-9]+\s+[0-9:.]+)\s+(?<value>-?[0-9.]+)', 'names');

dates = datetime({tokens.date});
values = str2double({tokens.value});

This code has been designed to permit negative numeric values, but it does assume that if there is a negative sign then the numeric values are immediately afterwards with no space. Also, this code is not designed to recognize exponential format.

[Janice Nelson]

Almost! I got one each dates and values. filecontent is 11E6 chars. I'm not regexp savvy. The lines are separated by \n (ascii 10)

    Name             Size                     Bytes  Class       Attributes

    TheFileName      1x11                        22  char                  
    ans              1x50                       100  char                  
    blvac            1x3537126              7074252  char                  
    dates            1x1                        121  datetime              
    filecontent      1x11759953            23519906  char                  
    id               1x423632               3389056  double                
    tokens           1x1                        402  struct                
    values           1x1                          8  double                

>> tokens

tokens = 

       date: '2017-08-30 19:47:39'
      value: '2.4639'

In bad news, some of the files contain exponentials 4.23e-11 for example.

[Janice Nelson]

Getting closer! I got rid of the ^ at the beginning of the regexp and now I'm getting 423631 instead of 423632. I can do without the last one but it would be nice to figure this out. Also values with an E in them...

[dpb]

Does not

fmt ='%{yyyy-MM-dd}D %{HH:m:ss}D %f');
data=textscan(fid,fmt %f');

work? (If the dates are poorly formatted as 13: 4:29 it will fail as Walter notes but one would hope that isn't so...)

If are, use

fmt='%q %q %f');
data=textscan(fid,fmt,'collectoutput',1);
dates=datetime(strjoin(data{:,1});

[Walter Roberson]

When you use a %D for times within a day but without the date, then the result is taken relative to the date on which it was scanned. You cannot then just add that to the date portion. You have to do things like:

DatePortion + (TimePortion - shiftdate(TimePortion, 'start', 'day'))

[dpb]

That's what strjoin is for--concatenate the date/time strings into one for datetime to parse as a whole.

I think it's a major wart in implementation of '%D' that it doesn't handle the cases directly; I'm hoping that's because it's still the new kid on the block and just isn't yet ripe but like wine will improve with aging...

Excepting owing to the cell structure it won't work as written as that will concatenate all elements in order of all dates followed by all times in one long string...it's a pain to deal with no matter what you do.

Answer 3

[dpb]

There'll be a formatting discrepancy in the file at the offending line that causes textscan to fail. fgetl otoh reads the line as character string without formatting it so content is totally immaterial.

Since what you have is a date/time field, I'd suggest using the '%D' format string and return the data as datetime class instead of a string of variables. See the format field description for details on the format.

You might attach the last portion of the file with the offending line so folks can see what might be the actual issue -- only the section in the neighborhood of the place where the code fails is pertinent.

[Walter Roberson]

The %D format specifier is a bit tricky because spaces cannot be present in the date, unless you have set 'whitespace' to exclude space. However if you set 'whitespace' to exclude space then it is not going to be able to recognize the spacing between the time and the value.

Answer 4

[Jeremy Hughes]

textscan can read time-of-day as datetime,

having the format

d = textscan(fid,'%D%D%f','Delimiter',' ','ReturnOnError',false);

might work. The you'd have to do something like this to post-process:

[date,time,n] = d{:};
date = date + timeofday(time);

textscan will simply stop reading when it encounters an error. To see what data is messing up the read, setting ReturnOnError=false will issue an error instead of just stopping. This should give an indication of the problem.

[dpb]

It would be most interesting if OP would return and show us the offending record in the file to see just what broke what...

Are there plans to fix some of the observed warts with '%D' in the future with the embedded blank issue, etc., ... ?? It's a strong step forward but there are still "issues" that it doesn't handle well as well as the rest of the datetime class.