Searching through files for missing data

Hi,

I have a set of 8000 files in the format of

YYYYMMDDHHMMSS

year/month/day/hour/minute/second

The files should increase by 5 minutes each time and I need to write a function that would check the file names are named in a logical way. And if any files are missing it can identify this for me.

The function strcmp and join have been reccomended to me.

Does anyone know how to do this?

Answer 1

[KSSV]

You may follow something like this:

files = dir('*txt') ;    % give your extension
% Create a datetime vector for the files present with the names mentioned 
[P,N,E] = cellfun(@fileparts,f,'UniformOutput',0) ; 
t = datetime(N,'InputFormat','yyyyMMddHHmmSS') ; % this is datetime for the files present 
%% Create 5 mins possible datetime vector 
file1 = files(1).name ;
[path, name1, extension] = fileparts(file1) ;
t0 = datetime(name1,'InputFormat','yyyyMMddHHmmSS') ; 
file2 = files(end).name ;
[path, name2, extension] = fileparts(file2) ;
t1  = datetime(name2,'InputFormat','yyyyMMddHHmmSS') ; 
% Make datetime arrray 
t0 = t0:minutes(5):t1 ;   % this is used for comparison 
% Get the indices which are present 
idx = ismember(t0,t1) ; 
% Dates which  donot exist 
t0(~idx)

[Adam Danz]

What's 'f' in the 2nd line of code?

It looks like the 3rd line of code assumes that the entire file name is a datetime string which isn't the case. For example,

N = 'T_PAAH72_C_EIDB_20210204140000'; % From OP's description
t = datetime(N,'InputFormat','yyyyMMddHHmmSS')

Error using datetime (line 647)
Unable to convert 'T_PAAH72_C_EIDB_20210204140000' to datetime using the format 'yyyyMMddHHmmSS'. 

[drb17135]

Hi, yes I had an undefined variable, f

[Jan]

@drb17135: Replace "f" by "{files.name}"

I assume the 2st line is:

dir('*.txt')
%     ^ insert a dot

[drb17135]

Index exceeds array bounds.

Error in searching (line 7)

file1 = files(1).name ;

clc
clear all
close all
 
files = dir('*.C:\Users\drb17135\Documents\August_Radar') ;    % give your extension
 
% Create a datetime vector for the files present with the names mentioned 
[P,N,E] = cellfun(@fileparts,{files.name},'UniformOutput',0) ; 
t = datetime(N,'InputFormat','yyyyMMddHHmmSS') ; % this is datetime for the files present 
%% Create 5 mins possible datetime vector 
file1 = files(1).name ;
[path, name1, extension] = fileparts(file1) ;
t0 = datetime(name1,'InputFormat','yyyyMMddHHmmSS') ; 
file2 = files(end).name ;
[path, name2, extension] = fileparts(file2) ;
t1  = datetime(name2,'InputFormat','yyyyMMddHHmmSS') ; 
% Make datetime arrray 
t0 = t0:minutes(5):t1 ;   % this is used for comparison 
% Get the indices which are present 
idx = ismember(t0,t1) ; 
% Dates which  donot exist 
t0(~idx)

[Jan]

% Replace
files = dir('*.C:\Users\drb17135\Documents\August_Radar')
%            ^^
% by
files = dir('C:\Users\drb17135\Documents\August_Radar\*.*')
%                                                     ^^^ 

[drb17135]

Error using datetime (line 635)

Unable to convert 'T_PAAH72_C_EIDB_20200901000000' to datetime using the format 'yyyyMMddHHmmSS'.

Error in searchfiles (line 16)

t1 = datetime(name2,'InputFormat','yyyyMMddHHmmSS') ;

I think because the whole line is not in date time format this may be failing. Would you suggest I try parsing the line to leave me with the - 20200901000000 part?

[Adam Danz]

That's because this solution missing step 2 in my modifed list of 4 clear steps as I mentioned in a comment under this answer as well.

Answer 2

[Adam Danz]

>Does anyone know how to do this?

Lots of people know how to do this and we're here to help but few people will devote a portion of their day to do it for you.

Let's start by figuring out where you're stuck. There are just a few basic steps in your process and you can find lots of information in this forum, on the web, and in the documentation for each step.

1. Get a list of files. See dir()
2. Read in the file. There are lots of ways to read files depending on the filetype and content (review).
3. Are your time stamps in datetime format? If not convert them to datetime.
4. If all you want to do is check whether a file is missing, you just need to store the following 3 data points for each file as 2 separate variables. This will be done in your loop: The first and last datetime value can be stored in an nx2 matrix for n files and the filename stored as an nx1 string array.
5. Once all files are read and the 3 data points are stored for each file, you can sort the datetime values in case the files are read out of order and then compare the first datetime of file n with the last datetime from file n-1. If that difference is more than 5 minutes, you know you're missing a file and you can use the filename array to help identify which file is missing.

If you get stuck on any step leave a comment below and show us where you're at with the code and what the problem is.

[Stephen23]

I suspect that the difference is not exactly five minutes... most likely some tolerance will be required in the comparison.

[drb17135]

Hi, thank you for the help

The file names are written as: 'T_PAAH72_C_EIDB_YYYYMMDDHHmmSS.hdf'

Where YYYY is the year, MM is the month, DD is the day, HH is the hours, mm is the minutes and SS is the seconds.

For example, if the file was from today at 2pm, the file would be named:

T_PAAH72_C_EIDB_20210204140000.hdf

Each file progresses by 5 minutes. So the next file would be:

T_PAAH72_C_EIDB_20210204140500.hdf

then

T_PAAH72_C_EIDB_20210204141000.hdf

and so on,

This continues for a whole month worth of time.

Does this still work with what you have suggested?

[Adam Danz]

Oh, I see. So, the datetime values are stored in the filename. That's even easier. There's no need to read in the files.

New list of tasks:

1. Get a list of files. See dir()
2. Loop through each file name and isolate the datetime portion of the file name. This can be done several ways: by parsing the final segment between the last underscore and the file extension, or by using regular expressions.
3. Convert the isolated time stamp string to a datetime value using t = datetime(DateStrings,'InputFormat',infmt) and store those datetime values in an nx1 array.
4. After all file names are analyzed, differentiate the datetime vector and determine if any durations are more than 5 minutes (also consider Stephen's advice about tolerance).

I suspect step 2 will be most difficult for beginners. Give it a shot and circle back if you get stuck.

[drb17135]

Hi, sorry I am fairly new to Matlab,

I was just wondering what you mean by differentiating the datetime vector? What information willl that give me?

Thank you

[Adam Danz]

Example:

The first line below produces a vector of datetime values (ignore how that's done for now).

The second line uses diff() to differentiate the datetime vector via (n+1)-n and converts the difference to minuts using minutes().

dt = datetime('now','Format','HH:mm') + cumsum(minutes(randi(5,1,10)))

dt = 1×10 datetime array
   17:01   17:03   17:06   17:11   17:16   17:18   17:19   17:21   17:23   17:26

minutes(diff(dt))

ans = 1×9

2 3 5 5 2 1 2 2 3

[KSSV]

@drb17135 did you try the code which I have given?

[drb17135]

Hi Adam

So this loops through the files and differentiates the date time vector, is this correct? I was wondering if there was a way to easilt identify which file is missing?

 
clc
clear all
close all

files = dir('C:\Users\drb17135\Documents\August_Radar\*.*')
 
tmp=dir;
 
for idx = [1:8000] % looking through 8000 files
   
  fileID = tmp(idx).name;
  
  myfile = split(fileID,{'_','.'});  %seperates the file name
  
  file = myfile(5:5);   %isolates the file in terms of 'yyyyMMddHHmmSS'
  
  datetime(file,'InputFormat','yyyyMMddHHmmSS'); %gives name in format of  '10-Sep-2020 04:00:00' - for example
  
 x = minutes(diff(datetime)); %differentiate vector
  
end

[drb17135]

I added this at the end. How do i make it add 5 minutes per each file?

t2 = datetime('01-Sep-2020 00:05:00','Format','yyyyMMddHHmmSS') + cumsum(minutes(5))
  
 dt = between(t,t2)

  

[Adam Danz]

Looks like you're making progress.

1. In this line, you could add the file extension if it's the same for all files. That should list all of the files you need, assuming they are all in the same folder.

files = dir('C:\Users\drb17135\Documents\August_Radar\*.*') 
files = dir('C:\Users\drb17135\Documents\August_Radar\*.hdf') % change to this 

2. This is where things go wrong. "tmp" should be the "files" variable above. You don't need this line. Replace tmp with "files".

tmp=dir;  

3. Instead of these 3 lines,

files = dir('C:\Users\drb17135\Documents\August_Radar\*.*') 
file = myfile(5:5);   %isolates the file in terms of 'yyyyMMddHHmmSS'
datetime(file,'InputFormat','yyyyMMddHHmmSS'); %gives name in format of  '10-Sep-2020 04:00:00' - for example

Use these two, based on example: 'T_PAAH72_C_EIDB_19991231134501.hdf' (YYYYMMDDHHmmSS)

% >>files(idx).name
% ans =
%     'T_PAAH72_C_EIDB_19991231134501.hdf'

[~, timestamp] = regexp(files(idx).name, '([0-9]*).hdf','match','once','tokens');

% Which returns
% timestamp =
%    {'19991231134501'}

timestampDT = datetime(timestamp{1},'InputFormat','yyyyMMddHHmmss')

% Which returns
% timestampDT = 
%   datetime
%    31-Dec-1999 13:45:01

4. Instead of assuming you have 8000 files, use the actual number of files identified to define the loop.

for idx = [1:8000]      % Not this

for idx = 1:numel(files) % use this, where "files" is defined in my step #1 above.

5. Lastly, you need to store the datetime stamps in the loop so the loop should be structured like this (using variable names above).

timestampDT = nat(numel(files),1);  % preallocate the loop variable
for idx = 1:numel(files)
    
    % < PUT YOUR OTHER STUFF HERE >
    
    % store all datetimes from the file names
    timestampDT(idx) = datetime(timestamp{1},'InputFormat','yyyyMMddHHmmss');
end

Then you can differentiat

dt = diff(timestampDT)