Dividing the work into several jobs and sending the jobs to the workers is very expensive. This is done at the line with parfor, before the loop starts. You're right in that there is no point to parallelize the code if this overhead is greater than the time needed to run the loop in serial.
I ran the code with for instead of parfor (it took me 308 sec for just x=3 and parfor was taking way too long to let it run to completion) and counted 6277 iterations of the while loop enclosing the parfor loop. That means that Parallel Computing Toolbox has to divide the loop and send the work to the workers 6277 times. That's a lot...
It's better to divide the work at a higher level, i.e. above the while loop. For example, you could do x=1, x=2, and x=3 on three workers instead of doing it in serial. Instead of taking 3 times 308 s, it should take just above ~308 s:
x = labindex;
Also, it would be nice to get rid of the inner for loop for j=2:N. You could see a speedup if you can vectorize this for loop somehow.