How to convert AnsiChar (char8_t) into HEX?

Hello. I save raw sensor data to an SD card. When I open such a file, for example, in the HxD program, I get a preview of the data in HEX, Int8, UInt8 and other formats. Can I do the same directly in MATLAB (R2015a)? How can I convert AnsiChar (char8_t) data into other data types? For example, I would like to convert the data string "ę˙°ţiţ{ý" into the EA FF B0 FE 69 FE 7B FD form.

[John D'Errico]

@Radoslaw Puchalski - please don't use flags to make a comment. Flags are there for moderation purposes.

Answer 1

[Walter Roberson]

It isn't completely clear what you want

datastring = "ę˙°ţiţ{ý"

datastring = "ę˙°ţiţ{ý"

fprintf('%02x ', typecast(swapbytes(uint16(char(datastring))),'uint8'))

01 19 02 d9 00 b0 01 63 00 69 01 63 00 7b 00 fd 

The above is in "big endian" order, which is not the internal order for any currently supported architecture. Big-endian as in the 01 19 is to be interpreted as 0x01 * 256 + 0x19 rather than as 0x01 + 0x19 * 256

dec2hex(datastring{1},4)

ans = 8×4 char array
    '0119'
    '02D9'
    '00B0'
    '0163'
    '0069'
    '0163'
    '007B'
    '00FD'

But maybe you want utf-8 bytes?

fprintf('%02x ', unicode2native(datastring, 'utf8'))

c4 99 cb 99 c2 b0 c5 a3 69 c5 a3 7b c3 bd 

[Radoslaw Puchalski]

Thanks, but from this datastring I need values: EA FF B0 FE 69 FE 7B FD. HxD editor shows this is AnsiChar (char8_t) data when it is formatting as Windows ANSI. Every other editor shows values like yours. I'm not sure where is the problem. Maybe char8_t and char are different types?

[Walter Roberson]

hex='EA FF B0 FE 69 FE 7B FD';
bytes = uint8(sscanf(hex, '%02x', [1 inf]))

bytes = 1×8

234 255 176 254 105 254 123 253

bytepairs = typecast(bytes, 'uint16')

bytepairs = 1×4

65514 65200 65129 64891

char(bytepairs)

ans = '￪ﺰ﹩ﵻ'

char(swapbytes(bytepairs))

ans = '냾槾篽'

native2unicode(bytes, 'utf8')

ans = '����i�{�'

native2unicode(bytes, 'utf16-le')

ans = '￪ﺰ﹩ﵻ'

native2unicode(bytes, 'utf16-be')

ans = '냾槾篽'

for ch = "windows" + (1250:1258)
  disp(ch); native2unicode(bytes, ch)
end

windows1250

ans = 'ę˙°ţiţ{ý'

windows1251

ans = 'кя°юiю{э'

windows1252

ans = 'êÿ°þiþ{ý'

windows1253

ans = 'κ�°ώiώ{ύ'

windows1254

ans = 'êÿ°şiş{ı'

windows1255

ans = 'ך�°‏i‏{‎'

windows1256

ans = 'êے°‏i‏{‎'

windows1257

ans = 'ź˙°žiž{ż'

windows1258

ans = 'êÿ°₫i₫{ư'

So that leads us to:

datastring = "ę˙°ţiţ{ý"

datastring = "ę˙°ţiţ{ý"

fprintf('%02X ', unicode2native(datastring, 'windows1250'))

EA FF B0 FE 69 FE 7B FD 

[Radoslaw Puchalski]

Wow, nice job. Thank you very much. I'll check this and let you know, but it looks very well.

[Radoslaw Puchalski]

Your solution helped me a lot. I am currently loading data from a csv file this way:

filename = 'E:\file.csv';
delimiter = '';
formatSpec = '%s%[^\n\r]';
fileID = fopen(filename,'r');
dataArray = textscan(fileID, formatSpec, 'Delimiter', delimiter, 'ReturnOnError', false);
fclose(fileID);

Raw_from_csv = dataArray{:, 1};
Raw_uni = [Raw_from_csv{:}];
Uint8_from_raw = unicode2native(Raw_uni, 'windows1250');
Int16_from_raw = typecast(Uint8_from_raw, 'int16');

This works well, however there is a problem. I have 32768 data in file.csv, but Raw_uni only contains 32746, so 22 are missing.

Raw_from_csv, on the other hand, has a dimension of 23x1 cell. I don't know why there are so many lines. I would expect more like 1x1 cell. That is, these 22 missing characters divide the file.csv into 23 lines in MATLAB.

There seems to be a problem where I have two NULL characters next to each other.

Do you have any ideas on how to solve this problem?

Maybe I should change delimeter or formatSpec?

Answer 2

[Radoslaw Puchalski]

I checked with another file and have a similar problem. Maybe someone could check the example file and see what can be done with these "virtual" newline characters?

The attached file has 32768 characters corresponding to 16384 values of type int16, while my code generates 16373 values.

According to me, this is caused by double NULL characters, which are treated as newline characters and omitted from the conversion.

How can I deal with this problem?

I am attaching my code again:

filename = 'E:\file.csv';
delimiter = '';
formatSpec = '%s%[^\n\r]';
fileID = fopen(filename,'r');
dataArray = textscan(fileID, formatSpec, 'Delimiter', delimiter, 'ReturnOnError', false);
fclose(fileID);
Raw_from_csv = dataArray{:, 1};
Raw_uni = [Raw_from_csv{:}];
Uint8_from_raw = unicode2native(Raw_uni, 'windows1250');
Int16_from_raw = typecast(Uint8_from_raw, 'int16');

I have also attached a slice of the plot of the Int16_from_raw variable. It should be a pure sine wave, but strange (very large) values also appear.

[Walter Roberson]

Don't use textscan() for this purpose. Do not use line oriented operations. Use binary files with fread()

[Radoslaw Puchalski]

Wow, however, it is very simple. Here is the complete (and working) code:

filename = 'E:\file.csv';
fileID = fopen(filename,'r');
Data_int16 = int16(fread(fileID, 'int16'));
fclose(fileID);

Thank you so much!

[Walter Roberson]

You can improve slightly:

filename = 'E:\file.csv';
fileID = fopen(filename,'r');
Data_int16 = fread(fileID, '*int16');
fclose(fileID);

[Radoslaw Puchalski]

Thanks a lot