My suspicion is that when you use clear for variables that are not "large", that MATLAB takes the time to try to return the memory to the free list in some kind of "best" order. It is known that "large" variables are handled differently: there is a free pool for small variables, and beyond a certain size, MATLAB requests an allocation of free space and returns the memory to the operating system afterwards.
What I do not know at the moment is why returning memory due to reaching the end of a function could that much faster than returning an individual variable.
I know that some implementations of some programming languages use the strategy of allocating a virtual memory heap for all local variables, and returning from the function can consist of deallocating the entire heap in one go, whereas clearing an individual variable could require maintaining a local free list -- more overhead. I do not know whether MATLAB does this; I have never heard that it does, and it has some implementation consequences that I have not seen evidence of. In particular, if you want to return a variable that might have portions allocated locally, then either you do a deep copy out of the local space, or else you have to do some kind of tracking of variables that are marked as return variables, and do a deep copy at the time one of them is assigned to, or else you have to do some kind of tracking of variables that are marked as return variables and do some crufty back tracking to find all places where memory might be returned (including through layers of struct) and allocate those in a different way (that last one seems unlikely.)