Binary file input help?

Hi Im trying to import some old Eddy Covariance data in SLT file format into matlab. I know how the data should look but cannot get anything that looks correct using fread. thanks lot moe

fid = fopen('030031130.SLT','r','n');
x = fread(fid,[36419 7],'*uint16'); % 36419 in known in this file but not in them all

the data should look like this, with time stamp first

00:01.30   212  -369   -12 16760  2352  1407
00:01.35   206  -407     4 16761  2352  1407
00:01.40   219  -411   -24 16755  2362  1408
00:01.45   224  -407   -19 16757  2362  1408
00:01.49   214  -418    -8 16758  2354  1412
00:01.54   229  -403   -18 16758  2361  1412

but comes out looking like this

12  1405  2351  16761  65506  79
25  230  1375  2341  16765  65328
23443  65263  135  1372  2338  16768
0  65505  65181  170  1391  2344
21260  16760  65491  65276  329  1405
12336  2362  16755  32  65366  276

"2.2.3 Raw data files (.SLT) format

These are binary files made up of records of 16 bit integers the format is shown below:

Column
1
2
3
4
5
6
Variable
un  
vn
wn
Cs
input 1
input 2
Units
m s-1
m s-1
m s-1
1/ 50 m s-1
mV
mV

The number of fields (or columns) varies with the number of analogue inputs to the Solent anemometer: four fields if there are no analogue inputs up to nine fields if there are five analogue inputs. The first record of the file is a header the first byte of which contains the record size in bytes (=2 * number of fields). "

[dpb]

Format the question please using the ()Code button. Or place two blanks in front of each line to indicate code follows. NB that must have blank line between paragraphs for formatting to take effect but no blank lines within a code section. Oh that TMW would fix this to make code instead of text the default view--90% of these problems would go away.

[dpb]

Well, it's better but still not clear on the actual format...

If I read the description it says the first int16 is the record length in bytes -- that looks from your data to be 12 which would indicate a record length of 6 fields. Is that correct for this case?

[moe]

HI Yes sorry about the formatting

6 fields plus the time stamp

this may be beyond my level

thanks moe

[dpb]

We need better definition of the timestamp formulation -- is it encoded somehow or three bytes? Are the colons you've shown embedded or are those just for display and implied, not actually in the data file?

Is there a link to a vendor's description, perhaps? My search wasn't fruitful.

I don't think it'll be any problem to read once get a precise definition of how it was written.

[moe]

HI All the info I have is already above but it comes from here edisol link The sample out put above is from a 1991 DOS program and Im trying to contact the publishers for more info. The data comes out of the DOS program as above and thats all I know.

00:01.30   212  -369   -12 16760  2352  1407

Ive now attached a zip with an SLT inside if it helps.

Answer 1

[dpb]

Ah....that does help. The description at the link doesn't say there's any timestamp inside the file; just the header record of number of fields (bytes/record) and then the data. The timestamp in the sample program output must have been written by it from some other knowledge of the data-acq rate.

That's the disconnect I had from what you posted vis a vis the file description. I don't suppose they give the source for the program or a sample source routine to read a file, do they? Anyway, whether do or not, the following should solve the problem--

fid = fopen('030031130.SLT','r');
n = fread(fid,1,'*uint16');        % read the header/# bytes/record
x=fread(fid,[n/2,inf],'*uint16')'; % now the data in that number of words/record
fid=fclose(fid);

See if that doesn't solve the problem...

NB: the detail of reading the number of words/record then the transpose to arrange since Matlab storage is column-major...

Answer 2

[moe]

thanks a lot dpb its starting to work but most of the numbers are slightly or totally wrong. Could matlab not be parsing the data correctly. I have tried many of the precision values without success to see if it was that.

Column 6 should be number one, which is not an issue but maybe a clue as I can move it. The first 5 numbers look like they are part of the header .

25  23443  0  21260  12336  124
-245  -30  16760  2358  1407  116
-243  -59  16767  2358  1407  105
-240  -74  16760  2347  1400  132
-246  -81  16752  2347  1400  106
-230  -76  16757  2355  1411  123

If they are working they should look this this

    212  -369   -12 16760  2352  1407
    206  -407     4 16761  2352  1407
    219  -411   -24 16755  2362  1408
    224  -407   -19 16757  2362  1408
    214  -418    -8 16758  2354  1412
    229  -403   -18 16758  2361  1412
    etc

[dpb]

Well, looking at the datafile definition again, it slipped by me that the number of records record is a byte not a word. Anyway, for the first part, try

n = fread(fid,1,'*int8');
x=fread(fid,[n/2,inf],'*int16')';

If your example data are correct then you want signed int not unsigned uint

Unfortunately, the way the description is written it's very ambiguous what the content of the header record is...it doesn't say anything specific about the length of it; only that the first byte of it is the count of channels in the rest.

Do you have any sample code to read it or can you find any--C, Fortran, even BASIC would be good enough to see what the actual structure is.

Have you looked at the file w/ a binary viewer to see if there's any discernible text in the header? Can you create a file w/ a known number of records so you can count bytes and see if can discern the header that way?

If all else fails, send me a copy and I'll see if can decipher it -- dp bozarth run together at the domain swko punctuation net, not com.

Just an unambiguous file description from the vendor would be nice... :(

OBTW...I poked around the downloads from the site and unfortunately, afaict there's nothing available but executables--no source code so that doesn't help much. I'd guess since looks like last updates were ~2004 time frame likely not much help going to be forthcoming from there...but who know, maybe they'll surprise.

ADDENDUM:

Do you know the first data values are definitely 212 and -369 for the file you're reading or are those just examples? If you know that you can look at the binary content of the file and determine which bytes correlate and then get the header length from there. Then you can try to decipher what the header bytes are if more than just one byte or word to see if can get a generic length.

If there is an actual header, then it would seem you'll have to read the first byte/word, extract the number of files, read the rest of the first record and then, finally, the data.

All for the lack of a nail...

ADDENDUM 2:

OK, I've read and reread and reread the header description...try

n = fread(fid,1,'*int8');
frewind(fid);
h=fread(fid,[n/2,1],'*int16');     % read a header record
x=fread(fid,[n/2,inf],'*int16')';  % _now_ read the data...

Double check the value of n assuming a byte; if it's not right then try int16 and see which way they stored it.

Answer 3

[moe]

HI dpb I can work with what Ive got now realy hanks a lot for your help

[dpb]

So what was the final answer???? (Curious minds and all that... :) )