How to calibrating and register two cameras of different resolutions?

Hi everyone, I'm experimenting on multi-modal imaging. We have two cameras, one IR and one RGB. I need to calibrate them and register their images with respect to each other, (i.e., to map the pixels in one to the other). The problem is these cameras are not the same resolution (they simply can't exist in the same resolution because of the technology, price and everything, so there's no way we can get an IR camera with high resolution). Now of course matlab's stereoCameraCalibrator only works if the cameras have same resolution. I want to avoid tricks like resizing images since it would result in error in the calibration parameters (the aspect ratios are different and one shouldn't crop images for calibration). Should I then calibrate them separately and somehow match their extrinsic parameters? Would that really work? (because then the correspondence btw. pair of images of the same scene captured by each of the cameras is not gonna be taken into account). I appreciate if anyone could help me out figuring how to do this. Thanks and best wishes. P.S: In case you wonder, the checkerboard is perfectly visible in both cameras.

Answer 1

[Dima Lisin]

You may be able to calibrate your cameras using the estimateCameraParameters function. It takes detected image points from each camera, and it does not care what the image sizes are. Unfortunately, the rectifyStereoImages function only supports images of the same size. You can still use undistortImage to undistort the two image individually, but you would have to write your own code to do the rectification.

[Shida]

Thanks a lot for your reply. I'm still confused about how to use the [stereoParams,pairsUsed,estimationErrors] =estimateCameraParameters(imagePoints,worldPoints); This is the same fucntion for single camera and stereo. Should I concatinate the imagePoints I get from each of the sets? How does it know which points are from what camera?

[Dima Lisin]

You are welcome.

Yes, the same function is used for single camera and stereo. In the case of stereo imagePoints is an M-by-2-by-numPairs-by-2, array, where numPairs is the number of stereo pairs of calibration images. So if you are detecting the points yourself, then you would concatenate the points from the two cameras along the 4th dimension. In other words imagePoints(:,:,:,1) are the points from camera 1, and imagePoints(:,:,:,2) are the points from camera 2.

So you would do something like this:

[pointsRGB, boardSizeRGB, imagesRGBused] = ...               
       detectCheckerboardPoints(imagesRGB);
[pointsIR, boardSizeIR, imagesIRused] = ...     
       detectCheckerboardPoints(imagesIR);

%...
% Remove pairs where the points were not detected correctly 
% in one or both images
%...

imagePoints = cat(4, poitnsRGB, pointsIR);

[Shida]

Thanks again. Now I'm calculating the calibration parameters. Haven't checked the results completely yet, but at least it runs ! One should be also very careful about the results of detectCheckerboardPoints when there are unused images in both sets as matlab removes the points from the point sets and therefore the indices of the two point sets no longer match. The corresponding indices btw. point sets need to be calculated carefully considering the images marked as used in each! It's not hard to code, just a bit annoying! I still of course need to do the rectification myself, but for now at least I have the stereo parameters. Considering how likely many people would need to calibrate two cameras of different resolutions with respect to each other (when working multi-modal the cameras are most likely different resolutions) I hope matlab fixes this in future versions. Anyway... Thanks a lot for helping me out. I hope your comments will be useful for others who are facing a similar problem. Best wishes.

[Shida]

P.S: undistortImage works with no problem as it's independently calculated for each camera. Now I'm off to write the rectification with the help of matlab !

Answer 2

[Shida]

For those who (like myself) need to use this to map depth and RGB images with different resolutions:

After calibration and undistorting images, one can use the depth info for mapping. It works well. I just used the calibration parameters from Matlab and the undistorted images (I implemented my own function for undistorting the depth to make sure values are not falsified by interpolation). My code is not fancy to share, but ask me if you're stuck. I might be able to help you.

[Yash Rathore]

Hey Shida, I am trying to do a similar thing and kind of stuck right now. I don't wish to ask stupid little things over here, so can I please contact you on your private email address or something? Thanks