mxIsA and mxArrayClassIDName used to be part of the C API library interface (they appeared as undecorated names). In R2014a these functions were moved to the C++ API library interface (they appear only as decorated names). My guess is that you may be using a mex routine compiled in R2013b or earlier in version R2014a or later. I.e., whatever this function does ends up calling mxIsA and mxArrayClassIDName in the background, but it can't find them in the libmx.dll file for the version that you are using. If this is indeed the cause, you will need to use an earlier version of MATLAB or recompile the mex routine in your later version of MATLAB.
I don't know what you mean by "... I try to install libmx.dll ..."