Well, I'm going to be an heretic, but if you are going to use dotnet, maybe it is a better idea to directly use a windows executable for the convex hull and bypass Matlab.
Matlab's convhull is based on qHull ( http://www.qhull.org ). You can freely download the executable there. If you use that, you would be getting rid of the intermediary. Even the source code, in C++, is available, so Visual Studio might also be a better idea than Matlab.
As for how the convex hull is calculated, any book on computational geometry would give you both the mathematical definition and a few algorithms. A quick definition is: The convex hull of a set of points S is the smallest possible set containing S such that a line between any two points of S is inside the convex hull. There are many algorithms available to calculate that.
Cheers!
