listening to the creation/deletion of a file with specific name.

Hi,

I'm wondering if it is possible somehow to listen to the creation/deletion of a file with an specific name. I know in Windows one has the option of controling .NET events, but I need a similar functionality also in Linux and Mac... I know I can create timers that scan continuously the file system, but this solution is rather inelegant and seems to compromise the performance of my code, so I'm interested in a solution more oriented to actual listeners... Any ideas?

Thanks in advance, Daniel

[Jan]

Using a tiny C-mex function I can check the existence of a (not existing) file more then 40'000 times per second: all you need is an _open(FileName, _O_RDWR). The timer would cause some overhead also, but if 1 sec reaction time is sufficient, I do not see, why this should be inelegant or a performance problem.

Please explain, what should happen in case of a file creation and which reaction time is required.

[Arabarra]

Hi Jan, thanks for your answer.

My application controls a database, which can contain thousands of files. Testing for all of them means that the system will very busy just with this operation... In general my users work in a integrated environment where they operate visualization of 3D data. This task is already rather slow in matlab, so that I cannot really afford an additional performance penalty from continuous (and unnecessary) checking the disk. Also, my users will not always have an SSD disk, I have to consider that many of them will have very slow disk access. Reaction time is not a problem per se, one second is ok.

[Steven Lord]

Be careful with how (and when) you try to process the files that the "watcher" detected.

Suppose your local supermarket is out of your favorite brand of breakfast cereal. The manager tells you that they're getting more soon because the truck carrying the shipment is already on the way. So you camp out watching the supermarket's loading dock with binoculars (you REALLY like this cereal :) and as soon as the truck arrives you rush into the store to the cereal aisle. But the cereal doesn't teleport from the truck to the shelves. You still need to wait for the truck to be unloaded before the cereal is available on the shelf.

In this analogy, the file is the truck carrying the cereal and you plucking a box off the shelf is the operation you want to perform on the file. Just because the truck is at the supermarket (the file is present on the disk) doesn't mean it's available for you to use quite yet.

[Arabarra]

haha, good analogy. And good point. Well, actually in my application I'm on the safe side. I'm lucky enough that the action to trigger when a truck is in the supermarket is just to broadcast an announcement notifying customers that the truck is there, and the cereal will come in a short time.

[Jan]

 My application controls a database, which can contain thousands of files.

If this is a real data base application, it should be able to trigger events on the file creation. A data base can e.g. lock the file status during the access from the outside. Using the file system and a polling loop is very indirect and prone to timing problems, which can be extremely hard to predict and to debug. Therefore Steven's analogy hits the point: Instead of looking for the truck, let the staff of the supermarket inform you, that your resources are available.

[Guillaume]

Certainly, if you're using a proper database engine, it should have triggers that could be captured more easily in matlab than watching the file system.

Answer 1

[Guillaume]

Yes, with .Net it's very easy using System.IO.FileSystemWatcher. See this recent question where I show how to use it. Of course, it is indeed windows only.

For other platforms, Java has an equivalent class, the WatchService. It seems to be more complicated to use that the .Net equivalent, and I'm not very well versed in java, so I couldn't figure out all the details. Using the example from there, this is how it would start:

fs = java.nio.file.FileSystems.getDefault;
watcher = fs.newWatchService;
javapath = fs.getPath('C:\somewhere\somefolder', 'somefilename');

At this point you're supposed to be able to register the watchservice but I get a no method with matching signature error that I haven't got the time now to dig into to solve

javapath.register(watcher, java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY)  %does not work

But hopefully, this should get you started.

edit: fat fingers

[Arabarra]

Hi Guilllaume,

thank you very much! Sounds close to what I need... Not very versed in Java myself, but I'm trying to look at this.

According to methodsview, the arguments of the register method are

java.nio.file.WatchService and

java.nio.file.WatchEvent$Kind[] Not certain what is going wrong, as the two entries in your code do seem to match the signature.. hm, I'll have to dig deeper into the java interfacing system of Matlab...

[Arabarra]

Well, apparently one needs to use the []. I tried

    eventkinds = [...
              java.nio.file.StandardWatchEventKinds.ENTRY_CREATE,  ...
              java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY,  ...
              java.nio.file.StandardWatchEventKinds.ENTRY_DELETE,  ...
              ];

as found in https://svn.oss.deltares.nl/repos/openearthtools/trunk/matlab/applications/+wps/+runner/watch_directory.m

... still does not work as intended, but looks like the correct way...

[Jan]

The examples I've found for the nio watchService contained an infinite loop to check the changes. I assume, this is not more efficient than running a timer in Matlab.

[Guillaume]

Really can't figure the correct arguments for that register method.

I must say that compared to the .Net FileSysemWatcher, the java way is a lot more complicated and a lot less flexible. From posts on StackOverflow it looks like the java watcher can't watch for individual files, only whole directories. You then have to see which file triggered it. And as Jan pointed out, there's no asynchronous way of getting a notification. You either have to poll (with poll) or block (with take). Still, if you can get it to work it should be more light weight than polling with _open(FileName, _O_RDWR) as it should rely on the OS built-in mechanism (when available. on some OS it may revert to polling).

[Arabarra]

Hi Jan and Guillaume,

yes, I've come to the same conclusion. The java WatchService does not seem to be something conceptually different to Matlab timers, unfortunately.

I wonder how come that an elegant solution like System.IO.FileSystemWatcher is not available for Linux and Mac.

[Guillaume]

It's probably not what you're after but you could always install mono, the open source version of .Net. System.IO.FileSystemWatcher is fully implemented in mono (with the caveat that on those OSes that don't have the required machinery it resort to polling the filesystem)

[Jan]

How many files or folders do you want to control? What is the required reaction time? You could create a multi-threaded C-Mex function, which performs the checks in an own thread, while the main thread return to Matlab already. This would be a dull polling loop also, but at least separated in an own thread.

[Arabarra]

the number of files to control can be around 10000+, but launching a separate thread can be a compromise solution. Not certain on how to get started, though. Any hint is welcome. Installing mono is less feasible. By now, my users just download my software, untar it and that's it. Making them install further software it's something that I want to avoid.

[Jan]

I do not think that any function can control 10'000+ files efficiently. Do you mean that a folder is watched, which can contain this amount of files?

Instead of waiting passively for a change, it might be much easier, if the tool, which modifies the files mentions the activity actively.

[Arabarra]

hm, not, its more like 100-300 folders are watched, and each folder has 30-100 files. But all of them have a common ancestor folder... Is it possible to simply watch for any modifications in any subfolder below a given one?

About the active/passive approach, the problem in this case is that there isn't a single tool modifying the files. One required feature of the application is that users can modify those files externally (even from different machines), and the application notices it. This works well in the practice using NET functionalities in Windows, probably because file modification is a rare event when using the application. Say, once every ten minutes

[Guillaume]

Well, I've finally figured out a way to use the java methods. However, since it's got no kind of event mechanism I'm not sure how useful it would be. You probably still would have to use some timer to poll the watcher but at least on some OSes it wouldn't poll the filesystem, relying instead on the built-in notification mechanism of the OS to tell the watcher that something happened:

fs = java.nio.file.FileSystems.getDefault;  %get a valid FileSystem object
javapath = fs.getPath(somefolder, "");  %folder to watch.
watcher = fs.newWatchService;  %get a watch service
eventkind = javaArray('java.nio.file.WatchEvent$Kind', 1); %create array with one element. Couldn't find another way to get the right kind of argument for register
eventkind(1) = java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY;  %to watch for modification
key = javapath.register(watcher, eventkind);

After that you can poll (non-blocking) or take (blocking) the watcher. With poll, if a modification occurs, you'll get key back, otherwise empty

key2 = watcher.poll();
if ~isempty(key2)   
    %key2 will be the same object as one of the key registered to the watcher. 
    %In this example, key2 == key
    eventlist = key2.pollEvents;  %get list of event that triggered the key
    for event = eventlist.toArray  %convert to array as I don't know how to extract elements from a javalist
       kind = event.kind;  %should be java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY
       filename = event.context;
       %...
    end
    key2.reset;  %otherwise you won't catch more events. 
end

Phew! Somebody more versed in java may find a less roundabout way of doing this.

[Arabarra]

Thanks for looking into that... truth is, I don't quite understand the advantage of the watcher object: if one really needs to time it to actively search for file modification events, then, isn't it exactly like _open(FileName, _O_RDWR)?

[Guillaume]

Oh yes, I agree there's not many advantages. It's an overcomplicated beast. I can only see two minor advantages:

• you don't have any external dependencies and don't need to compile a mex file
• on most system it relies on the filesystem hooks (filesystemwatcher on windows, inotify on linux/mac) to get notifications. This does not require polling of the disk. (You have to poll the watcher, but the watcher doesn't touch the disk, it gets notified by the OS whenever something happens). Whether or not that makes a difference, I don't know.

You're probably better off going the mex route. If you do, instead of polling with _openfile you could use the inotify machinery. Not something I can help with though. Windows only here.

[Guillaume]

Actually, considering that you want to detect modifications to thousands of file, the java way may actually be the way to go. I wouldn't think that thousands of simultaneous fopen would be very good for performance. Whereas you would only need one watcher registered to all the folders to watch.

[Arabarra]

oh, I see... but then, if it doesn't access the disk continuously looking for remotely locate files, it should actually be useful. I'll try to go for it, then. I'm not certain on how to actually use it, though. I don't get what the 'poll' method does: after it is started, the watcher will change the value of key2 every time a modification happens in the watched file, correct? And I still need to use a timer to periodically extract the events in key2?

In any case thanks for all the suggestions, this does get me started!

[Guillaume]

I don't really understand why the whole thing is designed the way it is, it seems overly complicated.

You can register any number of paths to a watcher. For each path you get a key. When something happens in the corresponding path, the state of the key changes and you can retrieve the cause(s) for the change by using the pollEvents method of the key. You could actually not bother with the poll method of the watcher and just regularly pollEvents all your keys but with many keys that would be a bit inneficient.

Instead, you can regularly poll the watcher. It will return you whichever key has changed state, if any. Note that if several keys have changed state, you'll have to issue several poll. I imagine you could design something like this (untested, expects bugs and typos, using OOP as it makes life easier):

classdef fswatcher < handle
    properties (Access = private)
       fs;
       watcher;
       timer;
    end
    methods (Acess = public)
        function this = fswatcher()
            this.fs = java.nio.file.FileSystems.getDefault;  %get a valid FileSystem object
            this.watcher = this.fs.newWatchService;  %get a watch service
            this.timer = timer('ExecutionMode', 'fixedRate', 'Period', 1, 'TimerFcn', @(~, ~) this.poll);         
            this.timer.start;     
        end
        function watchfolder(this, folder)
            javapath = this.fs.getPath(folder, "");
            eventkind = javaArray('java.nio.file.WatchEvent$Kind', 1); %create array with one element. Couldn't find another way to get the right kind of argument for register
            eventkind(1) = java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY;  %to watch for modification
            javapath.register(this.watcher, eventkind);
        end
     methods (Access = private)
         function poll(this)
             while true  %keep polling until all keys (if any) have been retrieved
                 key = this.watcher.poll;
                 if isempty(key)
                      break;  %nothing happened
                 end
                 eventlist = key.pollEvents;  %get list of event that triggered the key
                 for event = eventlist.toArray  %convert to array as I don't know how to extract elements from a javalist
                      kind = event.kind;  %should be java.nio.file.StandardWatchEventKinds.ENTRY_MODIFY
                      filename = event.context;
                      fprintf('%s was modified\n', filename);
                 end
                 key.reset;
             end
         end
     end
end

And usage example:

   >> watcher = fswatcher;
   >> watcher.watchfolder(pwd);
   >> watcher.watchfolder(fullfile(pwd, 'subfolder');
   >> watcher.watchfolder(fullfile(pwd, 'subfolder2');
   >> %modifying file 'test.txt' in local folder
   test.txt was modified
   >> %modifying file 'test2.txt' in subfolder
   test2.txt was modified

[Arabarra]

thanks for all the usage examples... I agree that this is not as easy as one would like, but well... hopefully Mathworks will offer something similar to the NET system in all platforms at some point...

[Guillaume]

It's bit more complicated than that. Matlab can interface with a number of external runtimes. On Windows, you've got Microsoft .Net and COM because they're built into windows. Microsoft does not provide a .Net runtime for other platforms, hence why it's not supported. There is an open-source effort to provide a .Net runtime: mono.

On all platforms, you've also got Java because the java runtime is cross-platform.

Recently, mathworks added support for python, which is also a cross-platform runtime. In fact, that's another option for you, I've just done a search and there's a watchdog python library.

You can also use perl which is another cross-platform runtime.

In all cases, the runtime is not developed by Mathworks, so they have no control over what is available. I have no inside knowledge over what Mathworks is planning, but I don't see them ever developing a filesystem watcher directly into matlab.

[Arabarra]

I see... well, that's too bad.

In any case, even if they don't control over what is available, they could offer a clean wrapper on top of whatever engine would be actually running below. I guess could end figuring out how to use python watchdog, or java watchers or whatever, but I'm certain that Mathworks engineers would make a much better job at it :-)

Answer 2

[Jose Luis Susa Rincon]

I found an easy way to check for changes, not the fanciest but it works. the idea is to compare two strings with the dates of when the file changed, if the date changed voila!

      function GetFileTime()
          DateNumberOld='17-Aug-2018 16:45:51' %this is just initial time to start checking for changes 
          DateNumberNew=DateNumberOld   %the new and old are the same initially
          while(DateNumberNew==DateNumberOld) % if new is different than old means that the file changed
              fileName='SOMEPATH/SOMEFILE.txt' %file to check
              listing = dir(fileName);
              % check we got a single entry corresponding to the file
              assert(numel(listing) == 1, 'No such file: %s', fileName);
              modTime = listing.datenum; %listing is a structure, you can get datenum or any other
              DateNumberNew = listing.date %I take date and compare it with the previous date
          end
          disp('it changed!!!')
      end

[Walter Roberson]

I recommend comparing the numeric listing.datenum field instead of the text listing.date field. The text .date field reflects local time, so it is after timezone adjustments, where the numeric datenum is before timezone adjustments. You don't want your code ignoring changes for an hour during the shift back to Standard Time.