I am unaware of any way to detect this in a mex function. It would be nice, of course, so that mex code could avoid some unnecessary calculations. But I think you are stuck and have to create that third output.
Typically, shared data copies of all plhs[ ] variables are returned to the caller, and then all the temp variables and memory allocated in the mex function are destroyed/freed. My belief is that the unwanted ~ variable gets destroyed at the caller level, but the mex function has no way of knowing that this is about to happen. And, as you noted, there is no specific documentation on how MATLAB actually handles inputs and outputs. In fact, this can change from release to release.
In particular, I don't think you can examine plhs[ ] upon entry to mexFunction( ). I don't think these values are specified to be anything in particular, since it is the programmer's job to fill them in downstream in the code. My guess is they would be all NULL upon entry, since that is how the caller would detect that you didn't fill it in with anything upon exit. If any of them were garbage and not NULL upon entry, and you didn't overwrite them with valid mxArray values, that would likely cause a MATLAB crash downstream, which is why I don't think MATLAB would do this. I just ran a quick check and all of the plhs[ ] values were NULL on entry. Again, there is no documentation about this, but this behaviour seems reasonable.
E.g., this
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
for( k=0; k<nlhs; k++ ) {
mexPrintf("plhs[%d] = %p\n",k,plhs[k]);
produces:
Building with 'MinGW64 Compiler (C)'.
MEX completed successfully.
>> [a,b,~,d,e] = tilde(1,2)
plhs[0] = 0000000000000000
plhs[1] = 0000000000000000
plhs[2] = 0000000000000000
plhs[3] = 0000000000000000
plhs[4] = 0000000000000000
One or more output arguments not assigned during call to "tilde".