Point cloud not accurate - Stereo Vision

I'm trying to do a distance estimation of an object with stereo vision.

I've done the calibration of the cameras using Matlab's Toolbox, and then I took a photo of a sheet where I printed a template in A3 format.

Then I tried to do a depth estimation using this code:

%% load
load params1309.mat
left=imread("406mm/left.png");
right=imread("406mm/right.png");
figure
imshowpair(left,right,"montage")

%% rectification
[leftRect,rightRect,reprojectionMatrix]=rectifyStereoImages(left,right,stereoParams1309);
figure
A=stereoAnaglyph(leftRect,rightRect);
imshow(A)

%% disparity map
disparityRange = [0 20*16];
disparityMap = disparityBM(leftRect,rightRect,'DisparityRange',disparityRange,'UniquenessThreshold',10,'BlockSize',9,'ContrastThreshold',0.5);
figure
imshow(disparityMap,disparityRange)
title('Disparity Map')
colormap parula
colorbar
filtered=medfilt2(disparityMap,[5 5]);
figure
imshow(filtered)
title('Disparity Map filtered')
colormap parula
colorbar

%% reconstruction
xyzPoints=reconstructScene(filtered,reprojectionMatrix);

%% xyzPoints=xyzPoints;
X=xyzPoints(:,:,1);
Y=xyzPoints(:,:,2);
Z=xyzPoints(:,:,3);
ptcloud=pointCloud(xyzPoints);
player=pcplayer([-500 500],[-500 500],[0 2000],'VerticalAxis','y');
view(player,ptcloud)

%% mask
mask=repmat(Z>0 & Z<500,[1,1]);
leftRect(~mask)=0;
figure
imshow(leftRect)

%% average
figure();imshow(xyzPoints(:,:,3)); colormap jet
distance=xyzPoints(:,:,3);
distance(distance>420)=[];
isfinite=mean(distance(isfinite(distance)))
average=mean(distance,'all',"omitnan")

Those are left and right photos:

And this is what the point cloud looks like:

The object was at 406mm from the left camera, but as you can see in the point cloud there were some correct points at 400mm circa and other wrong points far away from them (at like 1000mm), and I don't understand why. It looks like it couldn't understand that it was a 2D sheet.

That's why I tried to remove the points > 500mm with a mask before trying to calculate the average, and the result was this one:

Do you know what's wrong? Why does it split the points in two different regions at two different distances?

And if you don't know how to improve the point cloud, do you maybe know how can I get rid of all the points i don0t want in order to make the average more accurate (using only the points around 400mm)?

Thank you!

Answer 1

[Aishwarya]

Hi Luca,

As per my understanding you are getting an unexpected result in the point cloud generated through stereo vision depth estimation. After reviewing the information provided, here are some suggestions to improve the results:

1. Scene with repetitive pattern:

• The presence of uniform noise in the background of the point cloud could be due to difficulties in finding correspondences between the two images.
• This may be caused by the repetitive pattern and colour of the checkerboard scene in the images.
• To address this, I suggest testing the algorithm on images with more texture and variable depths in the scene. This can provide better features for the stereo vision algorithm to establish accurate correspondences.

2. Improvement in camera calibration:

• Follow the workflow outlined in the document provided below to enhance calibration accuracy: https://www.mathworks.com/help/vision/ug/using-the-stereo-camera-calibrator-app.html
• Ensure that the calibration data is captured under the same conditions as the rest of the experiment to maintain consistency.

3. Mismatched stereo pair: Verify that both cameras have the same intrinsic and extrinsic parameters to ensure accurate depth estimation.

4. Tuning disparity range and other parameters:

• Adjust the disparity range and other relevant parameters to optimize depth estimation.
• If the scene has limited texture, consider increasing the "BlockSize" parameter. A larger "BlockSize" can reduce noise in the disparity map, but it may come at a higher computational cost.
• For detailed instructions on tuning the disparity range using the "disparityBM" function, refer to the "Choosing Range of Disparity" subsection in the document provided: https://www.mathworks.com/help/vision/ref/disparitybm.html

5. Try different disparity mapping method:

• There are different disparity mapping methods like Block Matching (BM) and Semi-Global Matching (SGM).
• Semi-global matching is more computationally expensive than block matching, but it can produce high-quality results. It is recommended to try both algorithms and compare their results for the specific application.
• For instance, I have implemented the "disparityBM" algorithm using the parameters specified in your code and applied a "medfilt2" filter, as indicated in the code. Subsequently, I compared these results with those obtained using the "disparitySGM" method.

6. Post-processing and filtering: Consider applying some point cloud processing techniques to filter the noise. Some of the approaches could be:

• Filtering the noise in 3-D point cloud using “pcdenoise”.
• Apply median filtering of 3-D point cloud using “pcmedian”.

Please note that the provided links below contain documentation that you may find helpful for further reference:

• MATLAB documentation on “disparitySGM” function: https://in.mathworks.com/help/vision/ref/disparitysgm.html
• MATLAB documentation for ‘”pcdenoise” function: https://in.mathworks.com/help/vision/ref/pcdenoise.html
• MATLAB documentation for “pcmedian” function: https://in.mathworks.com/help/lidar/ref/pcmedian.html

Hope this helps!

Regards,

Aishwarya Palli