How do I read in a binary file that has a very unique data structure?

I have a binary file of which I know the structure (i.e. A x uint16, B x 12-bit unsigned, C x uint16 etc.). I suspect I have to use "memmapfile" in some capacity to map and then read the file into a usable form but I haven't had any success after some time trying. If anyone has any ideas how one can "Read this file where bits 1 to A are uint16, A+1 to B are 12-bit unsigned, etc." that'd be a huge help. Many thanks!

[Scott]

Thanks for the replies. However, I'm having issues even with step 0: is there any way to even just load it in as an array or string of 1s and 0s?

[Walter Roberson]

Safest would be to fread uint8 and convert that to a bit array.

[Scott]

Walter, Just saw your comment now as I was going to post an update. That's exactly what I did. Used fopen and fread which read in all data as unit8. Converted that to binary, and used my outline to convert the desired number of bits into the appropriate int. Thanks for everyone's replies and support. Much appreciation!

Answer 1

[dpb]

There's no facility in memmapfile to handle arbitrary bit-lengths; you'll have to map the 12-byte type to an array of 3-x-uint8 and then combine those results.

You also need to know endianness of the underlying data as well to interpret correctly; that should be provided by the application documentation that created the file.

Answer 2

[Walter Roberson]

You need to know whether the "12 bit unsigned" is 12 bits out of every 16 with the other 4 ignored, or if the 12 bits are packed together, 2 of them for every 3 8-bit bytes? If they are packed together, then you need to have defined what happens if you end in the middle of a byte because B is not a multiple of 2.

Answer 3

[Teja Muppirala]

Would it be possible to use FREAD with the precision specified?

For example:

%% Just making some test data... (16bit,12bit,16bit // 16bit,12bit,16bit // ...)

    digitsOfPi = [3 1 4 1 5 9 2 6 5 3 5 9]
    fileID = fopen('pi.dat','w');
    for n = 1:numel(digitsOfPi)
        switch mod(n,3)
            case 1
                fwrite(fileID,digitsOfPi(n),'ubit16');
            case 2
                fwrite(fileID,digitsOfPi(n),'ubit12');
            case 0
                fwrite(fileID,digitsOfPi(n),'ubit16');
        end
    end
    fclose(fileID);

%% Now read the file back in:

    fileID = fopen('pi.dat','r');
    n = 0;
    while ~feof(fileID)
        n = n+1;
        switch mod(n,3)
            case 1
                data = fread(fileID,1,'ubit16')
            case 2
                data = fread(fileID,1,'ubit12')
            case 0
                data = fread(fileID,1,'ubit16')
        end
    end
    fclose(fileID);

This gives me back

data =
   3
data =
   1
data =
   4
data =
   1
data =
   5
data =
   9
data =
   2
data =
   6
data =
   5
data =
   3
data =
   5
data =
   9
data =
   []

[Walter Roberson]

Note that each fread will start at a byte boundary and that when the count is exhausted that any partial byte unread will be discarded. It is thus not suitable for advancing through a file as if the file were a bit array.

[dpb]

Interesting! I wasn't aware there was such a thing as 'uint12' that Matlab would recognize, even. Of course, as you note the above "works" for the OP as it writes 16-bits to store 12...

>> fid=fopen('test.dat','w');
>> fwrite(fid,3,'ubit12');
>> fid=fclose(fid);

Open file in a binary mode viewer...

C:\ML_R2012b\work> ty /x test.dat

0000 0000 03 00 ♥.

shows two bytes were written so by symmetry fread read the same two bytes when used 'uint12' there as well. The demonstration shows the underlying C i/o library is consistent but doesn't, as you note, do anything for the OP if the file is actually a packed bitstream as would seem to be, probably from some lab instrument.