Bidirectional interface to MATLAB (allowing callbacks)

I would like to implement a bidirectional programming interface to MATLAB, to make MATLAB accessible from another programming language. I'd like to be able to:

1. Evaluate MATLAB code from within language X
2. Call back to language X from the MATLAB code being evaluated

I'm looking for ways to implement this. Please see the detailed description below.

Point 1. can be solved using the MATLAB Engine interface, but that doesn't allow for callbacks (point 2.).

An alternative solution would be using the MEX interface: language X is running in a separate process and supports a bidirectional communication protocol. I could write a MEX function which will open the communication channel and makes everything possible that one can do with the MATLAB Engine. This MEX function would never return (until MATLAB is closed), it would listen for incoming messages on the communication channel and would use mexPutVariable, mexGetVariable, mexEvalString to evaluate MATLAB code when necessary. It would work just like the MATLAB Engine. All is good so far. But what about callbacks?

First I thought that I could implement a second MEX function which would allow calling back to language X. But for this, I would need access to the data structures related to the communication protocol, which means I would need to be able to share memory with the first MEX function that initiated the connection.

Question 1 Is it possible to share memory between MEX functions? From what I could google up, the answer seems to be no. One possibility that might work would be this: create a C struct which has function pointers to all the functions I need to use. Get a pointer to the struct and store it as an integer in a MATLAB variable. Then from the MEX function retrieve this variable, convert it back to a pointer and access the functions. This might work, but it feels very nasty and I very much want to avoid it.

Instead of using the MEX interface, I could create a shared library, which can hold several functions that can share memory between them. But then I wouldn't have access to the MEX interface (mexEvalString, etc.)

Question 2: Is it possible to access MEX functions form a shared library? Would it be possible to load libmex as a shared library too? I assume that mex* functions need a specific environment in order to work correctly.

Question: What are my options for allowing for callbacks in such a situation?

Answer 1

[Philip Borghesani]

Any shared library directly or indirectly loaded by MATLAB can use mx and mex functions. To do this they must include the correct headers and link against the correct libraries. The mex compile command takes care of this but it can be done without using the mex command.

MATLABROOT/extern/examples/shrlib/yprime.c is an example of a mex file that has additional exports that can be called with calllib.

Without more information about the language you are trying to support it is difficult to give specific suggestions. Can MATLAB and the other language be loaded into the same process or must two process be used?

[Szabolcs]

The bridge is between Mathematica and MATLAB, but I didn't want to focus on that too much, and I don't expect people here to be familiar with Mathematica. I know Mathematica much better than MATLAB, so it's the MATLAB side I'm having more trouble with. The two systems must run in separate processes and would use a bidirectional communication protocol that's part of Mathematica. The way it works (simplified) is that each side can send a message, and will block until it receives a reply.

It's very useful to know that any shared library can use mex functions. Then this is probably the best way to go. Are there any differences between creating a mex file and calling that or calling a function from a shared library, that I need to be aware of?

In the meantime I did come up with another solution to share data between MEX files. It relies on passing a single pointer to a C++ object with virtual functions, encoded as an uint64 variable. I would appreciate it if people here could comment on this approach. Code is below:

Header for the C++ class

class Funcs {
    string str;
public:
    Funcs(string s) : str(s) {
        mxArray *handle = mxCreateNumericMatrix(1, 1, mxUINT64_CLASS, mxREAL);
        Funcs **ptr = reinterpret_cast<Funcs **>(mxMalloc(8));
        mxSetData(handle, ptr);
        *ptr = this;
        mexPutVariable("base", "testptr", handle);
        // mexLock();
        mexPrintf("inited\n");
    }
    ~Funcs() { mexPrintf("destroyed\n"); }
    virtual void print() const;
};

First MEX file has the implementation of the C++ class and creates a static (global) object:

void Funcs::print() const { mexPrintf("%s\n", str.c_str()); }

Funcs funcs("Hello!");

void mexFunction(int nlhs,mxArray *plhs[],int nrhs,const mxArray *prhs[])
{
  // do nothing
}

Second MEX file will call back to the first one:

void mexFunction(int nlhs,mxArray *plhs[],int nrhs,const mxArray *prhs[])
{
    // check arguments, etc.

      mxArray *fptr = mexGetVariable("base", "testptr");
      Funcs *funcs = *(reinterpret_cast<Funcs **>(mxGetData(fptr)));
      funcs->print();
  }

The first one needs to be run first, to initialize the C++ object and create a pointer to it in the MATLAB workspace. Then the second one can call the virtual functions of this object.

If using plain C, a struct with function pointers could accomplish the same in a bit more code.

Does this approach look reasonable to you? It does work and I do have a proof of concept of very basic bidirectional communication, but it feels ugly and unsafe to transfer a pointer through an integer variable.

(Sorry about the long code blocks, I removed just about everything that was not strictly necessary)