The double2rgb file exchange function in your code is converting the matrix of doubles to an RGB image with jet colormap. The im2gray function that you have used to convert the RGB image to back to grayscale is not meant for mapping jet colormap to grayscale colormap. Hence the large error.
Following is a modified version of your code with some additional lines to map the jet colormap to grayscale colormap.
mat = 100*peaks(100); minV = min(mat(:)); mat = mat - minV;
title('Matrix of Doubles')
matMin = min(mat(:)); matMax = max(mat(:));
matRGB = double2rgb(mat,jet);
matGray = zeros(size(matRGB));
mapColor = logical(prod(squeeze(matRGB(i,j,:))'==mapJet,2));
matGray(i,j,:) = mapGray(mapColor,:);
matGray = im2gray(matGray);
matDoub = rescale(matGray,matMin,matMax);
The conversion back to a matrix of doubles will not be perfect due to quantization losses. However the accuracy of conversion can be measured with the immse function. This function measures the mean squared error between images.
Modifying the code as shown above, the mean square error drops from 3.0496e+05 to 11.0156
Please contact the author of the file exchange submission for double2rgb for more details.