return does not mean 'restart'. return in the context of a script actually means 'abort' or 'return to the command line and stop executing this script'.
If you wish this code to repeat until a valid matrix is entered, wrap it in a while(true) loop and replace all of your return statments with continue (which will restart the loop, skipping any statements after that one).
Also, add a break after the last line of code for success (Q = R1'). This will terminate the while loop.
You should also add '\n' at the end of each of your error messages so the command prompt will start on a new line.
So, this should work better. Note that this is still a rather fragile way to do this. Using the input function leaves lots of opportunities for the user to enter stuff that doesn't make sense (especially leaving off the square brackets in this case).
fprintf('matlab inbuilt function "chol" is: ')
fprintf('matrix does not have positive eigenvalues, \n')
fprintf('Calculated value is: ')
fprintf('the input is not a symmetrical matrix, try again\n')
fprintf('the format of input matrix is wrong\n')