Why would csvread read all data into a single column
21 views (last 30 days)
dpb on 12 May 2017
Edited: dpb on 12 May 2017
Expanding upon the above comments, I did a test that looked like--
N=1E6; % the long row length
csvwrite('test.csv',randi(127,2,N)) % write a 2-row file of same N (@)
csvwrite('test.csv',randi(127,2,N)) % write a 2-row file of same N
Result was for N=62500 which seems to prove conclusively there's an internal limit in textscan; probably some sort of buffer limit one would guess when the format string isn't provided.
I didn't try to refine the result to between 62500 125000 where it breaks, but that definitely seems to be the cause of the issue.
I tried the venerable textread, it never completed the 1E6 case before I gave up, so that's not a workaround.
While it would be butt-ugly as a solution, tried the explicit format and
Out of memory. Type HELP MEMORY for your options.
Looks like a support request to TMW to see they can find a workaround or put it onto the enhancement list to resolve. Certainly seems as though Matlab should be able to read any file in whatever form it is in on disk as long as it can actually fit in memory without gyrations by the user.
(@) Just to be sure, I did scan the long record file by reading as stream character file and confirmed csvwrite wrote the linefeeds where should have so that, in fact, the file was actually two records on disk.
I hate it when get fixated on something... :(
But, I did a couple of additional tests and confirmed there's a hard limit apparently buried inside the textscan code at 100000--
Fails beginning at 100,000 elements in length; 99,999 is ok, you're just not supposed to have a file with records any longer than that, it seems.