Working with unicode paths

The following is a followup to:

http://www.mathworks.com/matlabcentral/answers/85508-directory-listing-of-extended-ascii-in-windows

This question however is a bit more specific. I have a file which was created using a program on Windows. I can browse to the file in Windows Explorer (Win 7). I am however unable to:

1. Open the file in Matlab (using fopen)
2. If I create a directory with the same name, I am unable to cd to the directory. cd(directory)

I have uploaded the file to a public folder on my dropbox account. https://www.dropbox.com/sh/d2mghr9xyb426lz/ZEM4DH8XTp

The files are: v. Békésy - 1957.txt v. Békésy - 1957.zip

 I am currently unable to provide instructions as to how one would create such a file in Matlab (hence providing them for download). For handling naming, I have also included the file in a zip, so that even if the zip is renamed on download, the file inside should maintain the same name. Incidentally, it was by exporting the zip to a folder with the same name that created the folder which I cannot cd to with Matlab.

Thus, the question is how do I get around issues #1 and #2 (without renaming them using manually using a windows interface). I am assuming this might mean using a custom library (mex and/or Java code).

The ideal solution is to provide a generic class of code that actually works for path/file manipulation instead of needing manual interference any time this problem is encountered.

Thanks, Jim

[per isakson]

With R2013a 64bit on Windows 7, I downloaded your file from Dropbox and run

    filespec = fullfile( 'C:\Users\poi\Downloads', 'v. Békésy - 1957.txt' );
    str = fileread( filespec )

which returned

    str =
    test

???

.

More:

• I download interactively with Firefox without being asked about text encoding.
• I copy&pasted the file name with Windows Explorer, e.g. right-click, rename, copy.
• Notepad++ says that both the pasted file name string and the file content, "test", are ANSI-encoded
• Matlab double says that "é" of the file name is character 233
• Here (Sweden) "é" is an "extended ascii" character(?)

It just works automagically!

Some years ago I received some text files, which I had to rename, interactively. The names of these contained large plus-signs and other "graphical" characters. I recall that I was able to rename some files with movefile and a wildcard(?).

[Jim Hokanson]

On mine I get:

Error using fileread (line 27)
Could not open file C:\v. B�k�sy - 1957.txt. No
such file or directory.

Could you comment on the output of the following?

a = get(0, 'Language')

import java.nio.charset.Charset
b = Charset.defaultCharset()

c = feature('DefaultCharacterSet')

Automagically working is nice. I'd like to have an automatic method instead of needing to manually manipulate files.

[per isakson]

Now, I'm on a different computer (same installation: R2013a 64bit on Windows 7). I read your file without problems here too. "Standard Swedish" installation, I guess. And:

    a = get(0, 'Language')
    import java.nio.charset.Charset
    b = Charset.defaultCharset()
    c = feature('DefaultCharacterSet')

returns

    a =
    sv_se
    b =
    windows-1252
    c =
    windows-1252

[Jim Hokanson]

EDIT: I am now having trouble NOT getting this to work. The interface through dropbox might be causing a problem and changing the character type. This should be kept in mind given the original response below.

Thanks Per, setting my language to Swedish works, although is obviously not ideal.

Specifically I can read the file if I do: set(0,'Language','sv_se')

Matlab documentation is very vague (from what I can tell) as to what these options actually do. I would have guessed that option only changed the way that figures were rendered, but apparently not.

The question still remains open then as to how to read the file on a more generic basis.

Answer 1

[Jim Hokanson]

As others have alluded to, the problem seems to be with Matlab touching the character data. I still don't have a solution for changing the directory, since I don't know of a way of doing this without using Matlab strings.

Here's how to read a file (in this case to bytes) using Java which bypasses the unicode problem.

dir_obj    = java.io.File(DIR_ROOT);
dir_files  = dir_obj.listFiles;
file_bytes = typecast(org.apache.commons.io.FileUtils.readFileToByteArray(dir_files(end)),'uint8');

NOTE: There are other methods of extracting bytes given a file but the method alluded to above exists on my system and seemed the most straightforward.

Found from: http://stackoverflow.com/questions/6058003/elegant-way-to-read-file-into-byte-array-in-java

At this point native2unicode() or char() would be fine if you wanted the content as a string.

It seems like the problem is most likely tied to combining characters, which is one way of adding something like an accent to a "normal" letter.

http://en.wikipedia.org/wiki/Combining_character

I believe that the file on disk which has caused the problem actually consists of a combined character which adds an accent to an e, thus the 101 769, which is the letter e followed by a combining acute accent:

http://www.fileformat.info/info/unicode/char/301/index.htm

Answer 2

[Malcolm Lidierth]

Jim

MATLAB/Java need to talk to an OS and a file system beneath so this is likely to vary across FAT12/16/32, NTFS etc as well as OS or MATLAB/Java.

From @Pers comments: the windows-1252 charset is proprietary, not unicode, and to convert a Java String to the originating byte[] requires the CharSet.

So, telling the difference between "'v. Békésy" on this screen to the byte[] that it was created from requires information that the string does not contain and, AFAIK, neither does the directory entry of any file system.

On my Mac:

>> java.nio.charset.Charset.availableCharsets.size()

ans =

     166

The answer then is that there is no answer beyond "don't use special characters in file names" as suggested by Jan on your first post. But, on the assumption that nobody is likely to have used anything but 8 bit encoding:

>> java.lang.String('v. Békésy').getBytes()

ans =

118
 46
 32
 66
-23
107
-23
115
121

But compare that with the MATLAB char array:

>> char(java.lang.String('v. Békésy').getBytes())

ans =

v .

B

k

s y

and with:

>> uint8(java.lang.String('v. Békésy').getBytes())

ans =

118
 46
 32
 66
  0
107
  0
115
121

For this problem, MATLAB's uint arithmetic rules may not be the most useful.

>> java.lang.String(java.lang.String('v. Békésy').getBytes(),java.nio.charset.Charset.defaultCharset())

ans =

v. Békésy

>> java.nio.charset.Charset.defaultCharset()

ans =

ISO-8859-1

but:

>> java.lang.String(java.lang.String('v. Békésy').getBytes(), 'US-ASCII')

ans =

v. Bks

Regards ML

[Walter Roberson]

Try

typecast(java.lang.String('v. Békésy').getBytes()), 'uint8')

[Malcolm Lidierth]

@Walter My edits overtook your post- but the point that an integer array, signed or otherwise, may be better to work with remains.

[Jim Hokanson]

Malcom,

I am surprised that TMW just says it can't handle anything beyond 7-bit ascii on a Mac (link to this on previous question). What bothers me is that Windows can handle this file just fine (via explorer), but Matlab can't. Perhaps what you are indicating is that part of the problem is with Java (Java 6 maybe?). At some point when I have some time I'll look at writing a mex solution to the problem.

Thanks, Jim

[Malcolm Lidierth]

@Jim Not a Java issue. On my Mac e.g. with both a Mac HD and a FAT32 drive

>> java.io.File('v. Békésy')

ans =

v. Békésy

>> ans.exists()

ans =

     1

Its a MATLAB issue.

P.S. SciLab works fine while R gives the name as v. Be\314\201ke\314\201sy but works happily with that.

[Jim Hokanson]

@Malcom,

Thanks for the clarification.

Jim

Answer 3

[per isakson]

Googling taught me

• NTFS stores file names in Unicode.
• Not all zip-tools are Unicode-aware.
• The name of files transferred in zip-files between systems with different default character sets may be "corrupted".

Links

• Character Sets Used in File Names
• Zipping up Unicode file names
• Error while zipping files with unicode characters in names with Win7's