if you upload your images, batter for me to make a code then. Try this prototype
in order to calculate some RMS error for your images, you require two points
Then you can calculate RMS error as
RMS=sqrt(sum(YourOrginalimage (:)-Extimatedimage (:))^2/N)
where N may be total no of samples in YourOrginalimage
Check it and tell me?