CSV file reading and applying loop to plot graphs

Hello

How can I do it for multiple csv file name of "WT_201119.csv" ,"WT_201120.csv" contain 52 main header with column. All headers are constant in csv files. Above csv read data correctly using readtable command but only for few files. where I have to do changes? similar to above one csv file Because I got error- Where I have to do changes always so that it can detect the total number of header and rows which we wan b "Error using matlab.io.internal.shared.TextInputs/set.DataLines (line 101) 'DataLines' must be specified as a positive integer or a 2-column matrix of positive integers."y

clc
clear all
d=dir('WT_201119.csv');  % "*" will detect the sequence of your file

opts = detectImportOptions(d(1).name,"Delimiter",","); % already know it's csv, avoid detecting it
T = {};
for i=1:numel(d)
    fid = fopen(d(i).name,'r');
    nhdr = 8 + cell2mat(textscan(fid,'Total Channels:%f',1,'HeaderLines',5,'Delimiter',','));
    fid = fclose(fid);
    if nhdr == 8   % skip no data in file cases if that occurs (dpb conjecture)
        warning(['No data.  Skipped file: ' d(i).name])  
        continue
    end        
   
    opts.DataLines = [nhdr+2 inf];
    opts.VariableNamesLine = nhdr+1;
   
    T{i} = readtable(d(i).name,opts)
end

See more (1)

[Mathieu NOE]

hello

could you share as well a working csv file like "WT_201120.csv" I presume

tx

[Mathieu NOE]

hello

seems to me the two files are very similar (both have 37 header lines) , and your provided code fails the same with both files ..

indeed , what was the purpose of this line ?

 nhdr = 8 + cell2mat(textscan(fid,'Total Channels:%f',1,'HeaderLines',5,'Delimiter',','));

you said the code is operating ok on several csv files (not the two attached) - so there must be an evident difference between the two that I am not seeing right now .

What am I missing ?

[Arshey Dhangekar]

Hello,

Both WT csv are same with 52 columns with headers. But it retrieved raw data some of the WT csv file so I don't want to retrieve it

[Mathieu NOE]

hello

sorry I don't understand the meaning of your second sentence... what do you want to retrieve or not ? and finally what was the issue in first place because now I am bit lost...

[Arshey Dhangekar]

"Error using matlab.io.internal.shared.TextInputs/set.DataLines (line 101) 'DataLines' must be specified as a positive integer or a 2-column matrix of positive integers."

getting error below the line

  opts.DataLines = [nhdr+2 inf];

Answer 1

[Jeremy Hughes]

Without an example file, it's impossible to say why the code doesn't work as expected. I'm just working backwards from the error message and what I know about the returns of textscan.

I see the example uploaded in a comment for a different answer, but that file works as expected with the code you included.

[Arshey Dhangekar]

Hello,

I attach file. readtable command works here but just want to know how your suggested code will work?

[LO]

I am sorry Arshey, your files are even different in size and contain errors, how are you supposed to read from them in a loop and plot data ?

[Jeremy Hughes]

This file is an entirely different format. The code above was specifically for the other format file. You should experiement with the options and try to understand why the code works for the first file, and adapt it to work for the second set with different formats.

[LO]

I agree, that's what I meant.

I think before writing any code about this, Arshey should have clear in mind where are the data files coming from, how are they formatted and standardize the format as much as possible.

If you scroll though the CSV files there are multiple columns containing several errors and the "tabulation" of the data is also incoherent (besides the different file format).

[Arshey Dhangekar]

Hello Jeremy,

I have two csv file WT and both have fixed have 52 header. I tried to use below command to read however header is as Var1,Var2..to Var52 instead of Store no,Date and Time. How can I fix the header problem with importing all the datas from those files?

table = readtable('WT_201120.csv'); 
table1 = readtable('WT_.csv'); 

Answer 2

[LO]

first import one single CSV file and select coumns and rows of what you want usually to import.

Generate the code by clicking the 'generate code' button, in the import menu (right side of the screen, up)

Then use this code and put it where IMPORT CODE is written here below

[logfname, pathname] = uigetfile('*.csv','Pick a log file'); %select a csv file 
cd(pathname); % move to that folder

file_list = dir ('*.csv') ; %  create a list of csv files based on folder content
conversion = struct2cell(file_list);
conversion1 = transpose(conversion);
name_list = (conversion1(:,1));

% these two limits give you the option to select only a certain file range
% but you could actually remove them and simply set the k in the for loop
% from 1 to Nr of files
first= 1; % set first file to analyze
last= 10; % set last file to analyze

for k= first:last
    
    filename = [pathname name_list{k}]; 
        delimiter = ',';
        startRow = 5;
        endRow = 11831; % you can remove this or set it to a level that would fit all your data logs
        formatSpec = '%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%s%[^\n\r]';
        fileID = fopen(filename,'r','n','UTF-8');
        fseek(fileID, 2, 'bof');
        dataArray = textscan(fileID, formatSpec, endRow-startRow+1, 'Delimiter', delimiter, 'TextType', 'string', 'EmptyValue', NaN, 'HeaderLines', startRow-1, 'ReturnOnError', false, 'EndOfLine', '\r\n');
        fclose(fileID);
        data_table = table(dataArray{1:end-1}, 'VariableNames', {'Name','x1','x2','x3','x4','x5','x6','x7','x8','x9','x10','x11','x12','x13','x14','x15','x16'});
        data_table = data_table(22:end,:); % this crops away the header part of your file (somehow the file format is not properly detected by matlab and there is some problem with formatting
        clearvars filename delimiter startRow endRow formatSpec fileID dataArray ans;
        
        size_t = size(data_table);
        plot_size = sqrt(size_t(2));
        
        if plot_size > floor(plot_size)
            plot_size=plot_size+1;
        end
        
        for i = 1:size_t(2)
            
            subplot(round(plot_size),round(plot_size)-1,i)
            % pre allocate var space
            time_vector = zeros(numel(data_table(:,i)),1);
            var_vector = zeros(numel(data_table(:,i)),1);
            
               for j = 1:numel(data_table(:,i))
                time = table2array(data_table(j,1));
                time(time == ' ') = [];
                time_vector(j) = str2double(time);
                var = table2array(data_table(j,i));
                var=strrep(var, ' ', '');
                var_vector(j) = str2double(var);
               end
            plot(time_vector,var_vector)
            title('title here')
            xlabel('label')
            ylabel('label')
        
        end
  

end

[Arshey Dhangekar]

Hello,

I can't always import and read the data for 100 csv files and everytime select number of columns and rows

[LO]

no you don't have to. Just do it once, create the code, then paste the code where I wrote "import code".

This will basically import the data in a loop, going through all your CSV files (assuming it is formatted in the same way)

[Arshey Dhangekar]

Could you just whole code so that I can understand how to plot?

[LO]

well it depends on which variables you want to plot, from the CVS file and also which kind of plot. could you share your file again, I can't see it

[Arshey Dhangekar]

Hello,

I uploaded file. I ´want to time vs every M variable in csv using loop

[LO]

I have updated the script, it works. However there is an issue with the text formatting of your data: due to the presence of whitespaces between characters the variables are converted into NaNs so you cannot see the data. I have tried to eliminate them using the function (var is one of the values in your data array)

strip(var,char(133)) or the other char codes you can see below but it does not remove them.

char(133) Next line

char(160) Nonbreaking space

char(8199) Figure space

char(8239) Narrow no-break space

the strip function could replace the line "var=strrep(var, ' ', '');", which is meant to have analogous results (although strrep does not affect special whitespace characters I suppose).

The problem may be in the formatting of the text and how MATLAB reads it. Do you have by chance the older file you posted ?

Apart from this, it should work.

Answer 3

[Jeremy Hughes]

You should look at what nhdr is when you see the error. It's likely an empty value which makes nhdr+2 empty, and [nhdr+2 inf] just [inf].