Help using textscan on a .dat file

Hi guys,

I'm having some trouble trying to automaticallly load .dat files in my matlab code.

I'm pretty sure I know what the problem is but here's the error message anyway:

 Error using load
 Number of columns on line 201
 of ASCII file G:\Thesis
 DATA\New
 Test\sunday_test_4.dat must be
 the same as previous lines.

Basically my .dat file is pretty standard, it has a time column followed by 16 channel of transducer depth values.

The problem is the final line of the file has a date line which has less columns than the rest and so the process fails here.

An example of the final lines of text within the file is shown below.

17:01:07.458  10.28  10.40  10.50  10.44  10.46  10.40  10.32  10.20  10.23  10.40  10.68  10.66  10.80  10.80  10.74  10.83
17:01:15.459  10.26  10.40  10.49  10.50  10.47  10.43  10.32  10.23  10.29  10.38  10.57  10.65  10.77  10.81  10.71  10.86 
17:01:23.459  10.28  10.44  10.43  10.46  10.49  10.47  10.32  10.28  10.34  10.38  10.56  10.62  10.83  10.81  10.71  10.77 
17:01:31.459  10.34  10.40  10.49  10.52  10.49  10.50  10.38  10.26  10.29  10.44  10.56  10.65  10.80  10.81  10.72  10.75 
Data end time 07/12/2015 17:03:55

And here is my code so far:

 [filename2, directory_name2] = uigetfile('*.dat', 'Load the test data');
 fullname2 = fullfile(directory_name2, filename2);
 C3=load(fullname2);
 C1 =textscan(C3,'%s %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32');

Simply put, I need the code to either ignore the final line within the .dat file or to store it somehow (the latter would be ideal).

However the number of lines within the .dat file changes depending on how long I leave the transducers running for, therefore I cant simply make the program ignore the 201'st line because it may be different the next time.

However it IS always the last line, so if there's a way of doing it, it shouldn't be that difficult. I'm just not sure how.

Any help would be greatly appreciated, Many Thanks, Nath

[Star Strider]

If your .dat file (or one of them) is not too large, attaching it (use the ‘paperclip’ icon and do both the ‘Choose file’ and ‘Attach file’ steps) would allow us to write definitive code to read it.

[Nathaniel Howard]

Hi Star Strider, Many thanks for you help. I have since attached my dat file, it is compressed because for some reason Mathworks does support .dat files as attachments?!!

Hope this helps and you assistance is very much appreciated. Cheers, Nathan

[Star Strider]

For some reason, I can’t unzip it. Your original file doesn’t look so large that there would be a problem uploading it. If the .dat extension is the problem, open it in ‘notepad’ and save a copy of it as a .txt file, then upload that version.

[Nathaniel Howard]

Hi again Star Strider, Again, many thanks for your help. Ive since attached a .txt file version of the data file. Yeah for some reason Mathworks doesn't support .dat...strange. Cheers again. Nathan

Answer 1

[Walter Roberson]

[filename2, directory_name2] = uigetfile('*.dat', 'Load the test data');
fullname2 = fullfile(directory_name2, filename2);
fid = fopen(fullname2, 'rt');
C1 = textscan(fid,'%s %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32 %f32', 'CommentStyle', 'Data'); 
fclose(fid);

That is, lines that begin with 'Data' are to be treated as comments and discarded.

[Nathaniel Howard]

Hi Walter, That's perfect for my needs. Thank you very much. I've been banging my head over this for ages, so it's much appreciated!

[dpb]

I thought "either ignore the final line within the .dat file or to store it somehow (the latter would be ideal)." Hence reading the full file and then processing the N-1 lines; the last line is still around from which to parse the end date/time.

[Nathaniel Howard]

Hi dpb, Sorry for the late reply. I'm not very good with parsing and the date of acquisition isnt really essential to me. So I went with Walter's code as it did what I needed.

Many thanks however for your assistance. It's greatly appreciated.

[dpb]

Don't be intimidated--it's really the same solution as Walter's other than reading the file into memory first. Once it's there, parsing is the same as his on the input file except from memory instead.

Answer 2

[dpb]

Either just use fgetl on each line and test for the end line characteristic string, parsing each line and building the arrray or read the full file as a character array and find the number of lines then use textscan on the array in memory.

dat=textread('nat.dat','%s','delimiter','\n','whitespace','');  % read to cell array
dat=char(dat(1:length(dat)-1));  % convert excepting last line to character array

Parse the dat array for the numeric data and then read the last line specifically.

[Nathaniel Howard]

Hi dpb, Thank you very much for your response. However I'm afraid I'm not experinced enough with Matlab to do anything with the character variable:

Also the code does not appear to omit the last line:

Here is my code thus far, would you be able to tell me how I can now convert the char file into it's useful fields, Namely time and each channels recording?

 [filename2, directory_name2] = uigetfile('*.dat', 'Load the test    data');
 fullname2 = fullfile(directory_name2, filename2);

 dat=textread(fullname2,'%s','delimiter','\n','whitespace','');  % read to cell array
 dat=char(dat(1:length(dat)-1));  % convert excepting last line to character array

Many thanks , Nathan

[dpb]

Same formatting as Walter used(*) excepting from the char() array instead of the file. Use the first N-1 lines as noted in the conversion but if do want to get the last date/time from the last line don't overwrite the original variable as the above does.

(*) Or, convert directly to mo,da,yr,h,min,etc., with '%2d:%2d:%2d.%3f' instead of keeping the string; either is suitable for datenum()