Parallel Computing with Command Line Scripts

I am using Matlab as a wrapper code to run a windows exe. Currently, I am writting input files for the exe with Matlab. Then writing a .bat file to submit all of the files to the exe. Then Matlab reads the output files. Then Matlab processes the data. Currently I am writing one .bat file and so each input file is submitted serially to the exe. However, I would like to make use of my multiple cores to speed up the process. How can I run these files in parallel? I was thinking I could create the same number of .bat files as number of cores and evenly divide up the input files across all of the .bat files and thus cores. Is there a better way to submit work in command line for parallel computing? Also, how do I determine how many cores I have? Is this the same as the number of workers Matlab displays when performing parallel loops?

Current steps:

1. Matlab writes input files
2. Matlab writes .bat file
3. .bat file submits input files to exe and moves output files to correct folder
4. Matlab reads output files
5. Matlab plots data

Potential steps:

1. Matlab writes m-input files using parfor
2. Matlab writes n-.bat files (n = number of cores) with m/n input files per .bat file (how do I determine n?)
3. Matlab submits n-.bat files using parfor to Windows command line
4. Matlab reads in and processes data

[Walter Roberson]

What is the relative time requirements of preparing input files compared to time to execute the .exe ?

Is it correct that the .exe executes in the command window, so that you invoke it with the executable name and some options on the command line (such as file names to process), and it does not return until it is finished? As opposed to the .exe opening a separate GUI to execute, since .exe that open a GUI usually exit their original process almost immediately after opening the GUI, before the work has been completed. Not returning until it is finished is easier to deal with for automated purposes.

[Christopher Saltonstall]

Each input file takes about 1-2 seconds to write. The exe then takes about 2 minutes per input file.

Yes, the exe executes in the command window. No GUI.

[Walter Roberson]

Outline:

1. find/create an output directory.
2. begin loop
3. prepare an input file.
4. create a bat that instructs running the .exe on the input file, writing to an output file in the output directory. In the step after the .exe is run (so after it is finished processing) have the bat create file [output_file_name '.ok'] or similar. That is, the .exe will probably start creating the output file early, but the output file will not be ready for post processing until the exe finishes executing, and monitoring for the end of the exe is a nuisance. So have the .bat create a file that will not even exist until the output file is completely ready
5. system() the .bat using '&' at the end of the command line to cause the bat to be run asynchronously. The system() will not wait until the .bat finishes executing if you use & on the command line.
6. Add the .ok file name to a list of files you are expecting to see later
7. dir() the output directory for .ok files . Each of them represents a completed run. For each of them, run the post-processing phase on the corresponding output file and delete the .ok file (or rename it to something else if you want to keep track of which ones have already been processed.) Remove the .ok file name from the list of files you are expecting to see later
8. end loop
9. after all of the inputs have been prepared, then loop until the list of files you expect to see later is empty. The loop would be the same as the above "dir() the output directory for .ok files" step. At any given time that all of the existing .ok files have been handled but there are still files left in the list of what you expect to see later, then pause() for a time to give time for .exe to finish processing in the background. You might want to put in a failsafe execute of the loop in case some output file did not get created (e.g., a problem with the .exe)