Why does unicode2native returns different values on different Matlab versions?

I use 2 Matlab versions 2020b and 2022b. With eachversion, the function unicode2native returns differently for 1 character.

unicode2native('�')

On 2020b, it returns 26.

While in 2022b, it returns an array of 3 uint8.

unicode2native('�')

ans = 1×3

239 191 189

Why does this happen?

Answer 1

[Rik]

I couldn't find this in the release notes, but apparently the default encoding changed between these versions (the documentation claims it choses the "user default encoding").

You can reproduce the results when specifying the target encoding:

unicode2native(char(65533),'ISO-8859-1') % picked on R2020b

ans = uint8 26

unicode2native(char(65533),'UTF-8') % picked on R2022b

ans = 1×3

239 191 189

In general I think people expect UTF-8 when converting char to uint8, so the new default makes sense.

[Ky Tran]

Thanks for the answer. This is very helpful! I just found out that we can also check for the default character set on each version by

feature('DefaultCharacterSet') %Ran on 2022b

ans = 'UTF-8'

while in 2020b it was

'windows-1252'

Anyway, you were right. They should have mentioned this in the release note. Haha.

[Steven Lord]

We did mention the encoding change in the Release Notes, though that Release Note entry doesn't specifically mention the unicode2native function. See this entry in the release R2022a Release Notes.

[Rik]

@Steven Lord Thank you for confirming the section in the RN is indeed related to this. I did read that part, but I didn't fully understand it, since it looked to me as if this would have been the expected behavior since R2020a ("UTF-8 was adopted as MATLAB's default character encoding to ensure that all Unicode code points can be correctly represented in files and byte streams." - RN 2020a).

But in retrospect it makes sense that there is some change in behavior. Why else mention it in the release notes? (and on close inspection, there is a qualification for Windows: "On the Windows platform, if the Use Unicode UTF-8 for worldwide language support option is enabled in the Windows Region settings dialog box, then MATLAB uses UTF-8 as its system encoding.")

Anyway, the mystery seems to be solved.