Rectification gone wrong!

[aaronlsmiles]

This is not an issue with the code (most likely), so I'm posting in Discussion instead of as an issue.

I have calibrated stereo rig with 30 images pairs (I understand I need to do this with more), checked that the checkerboard is being recognised (which it is), and got an RPE of 1.29 (not the best, I know).

Reprojection error: 1.2901758966507906
Centers: (array([0., 0., 0.]), array([146.05110854, 0.57357636, 1.35344688]))
Baseline: 146.05850579369803

I have then taken the following scene image pair:

When I rectify the images this is the output:

Can anyone intuit where I might be going wrong?

[decadenza]

Share your full code and calibration values, no magicians here...

[aaronlsmiles]

Ah my mistake, I should have said that it’s just using your boilerplate examples Rectification code, verbatim! Made no tweaks yet as learning how it all works with your examples before I dig into making any value changes etc.

[decadenza]

Well, my examples only work with photos from my cameras. You have to recalibrate with yours!

[aaronlsmiles]

I am using my stereo rig file calibrated using my photos/cameras, but rest of code is unchanged.

Full calib values:

Reprojection error: 1.2901758966507906
Centers: (array([0., 0., 0.]), array([146.05110854, 0.57357636, 1.35344688]))
Baseline: 146.05850579369803

[decadenza]

How is your stereo rig made? Are the two cameras well fixed with each other?
The above values are not the calibration values.
Share the stereRig.rig file (you can save it with rig.save("filename.json")) or you have to share all the intrinsic and extrinsic parameters.

[aaronlsmiles]

My rig consists of 2 cheapish actions cams clipped to a ruler (at exactly 10cm) clamped on to a makeshift mobile phone stand.

This chat won't accept json files, so I've copied the contents of the file below:

{"res1": [3840, 2160], "res2": [3840, 2160], "intrinsic1": [[2589.666734617553, 0.0, 1863.4422976417206], [0.0, 2588.8639983980333, 1232.0860745949783], [0.0, 0.0, 1.0]], "intrinsic2": [[2587.340230381427, 0.0, 1815.2650194192224], [0.0, 2587.9374721697814, 1167.6170381628453], [0.0, 0.0, 1.0]], "R": [[0.9998410728612688, -0.017756918009965185, 0.0015877286967296098], [0.017758019325179328, 0.9998420811697486, -0.0006822549321510139], [-0.0015753632195789685, 0.0007103414202064221, 0.9999985068217818]], "T": [[-146.0198610154371], [-3.166140792301913], [-1.123768750003275]], "distCoeffs1": [-0.46197315652128057, 0.2864239362020046, -0.01189615338138721, 0.009762834005990725, -0.09922063003612822], "distCoeffs2": [-0.4229369006826239, 0.23350481585755187, -0.0069605492057678835, 0.00914356589378856, 0.025213656887593097], "F": [[6.74659922425216e-10, 3.0338657530397775e-08, -0.0002604577626070416], [-3.660340550198532e-08, 3.345434925046434e-09, 0.010289317981265627], [8.1585897021156e-05, -0.010283012452427031, 1.0]], "E": [[0.02494372893179182, 1.121342244809824, -3.166902761461733], [-1.3536244710229939, 0.12367862500801918, 146.01785874186834], [0.5726140928328406, -146.053022632249, 0.10464974296361149]], "reprojectionError": 1.2901758966507906}

[decadenza]

There is something wrong. Your calibration shows a resolution of 3840x2160 for both sides, while the size of the above images is 5119x3674 for the left and 5120x3668 for the right one.

You have to use the same settings used during calibration. Once you calibrate the system, you cannot change any parameter that affects the camera.

This is the code I used to test your images:

import numpy as np
import cv2

import simplestereo as ss
"""
Rectify a couple of images starting from a calibrated stereo rig
"""

# Load stereo rig from file
rig = ss.StereoRig.fromFile("stereoRig.json")

# Rectify and obtain rectified stereo rig
rigRect = ss.rectification.directRectify(rig)

# Optionally save it to file, so you can load it directly as rectified.

# Read right and left image (please ensure the order!!!)
img1 = cv2.imread('left.png')
img2 = cv2.imread('right.png')

# ERROR # Image size is different from the one resulting from calibration
print(f"Expected image 1 resolution is {rigRect.res1}")
print(f"Input image 1 resolution is {img1.shape[:2][::-1]}")

print(f"Expected image 2 resolution is {rigRect.res2}")
print(f"Input image 2 resolution is {img2.shape[:2][::-1]}")

#rigRect.computeRectificationMaps(zoom=1.5) # May help removing unwanted border

# Simply rectify two images (it takes care of distortion too)
img1_rect, img2_rect = rigRect.rectifyImages(img1, img2)

# Show images together
visImg = np.hstack((img1_rect, img2_rect))

# Draw some horizontal lines as reference
# (after rectification all horizontal lines are epipolar lines)
for y in [289, 332, 362]:
    cv2.line(visImg, (0,y), (visImg.shape[1],y), color=(0,0,255), thickness=2)

cv2.imshow('Rectified images', visImg)
cv2.waitKey(0)
cv2.destroyAllWindows()

[aaronlsmiles]

Thanks! I'll have to identify why that has happened. I never changed anything intentionally.

[aaronlsmiles]

Seemed to be an issue with how I was getting the stills/frames from Premiere Pro. Had to get new scene images, which look better when rectified, but now I'm getting blank or blurry space added to the right and bottom of the images.

Rectified:

[decadenza]

Well, still not rectified evidently. If you use the code I posted, lines do not stop (internally it uses cv2.line with the image width).
It is definitely not an issue with the code, but something happened after calibration. Did you change camera settings? Also image resolution must be fixed.
Good luck.

[aaronlsmiles]

No, no settings were changed and I've used your code above to check the resolutions and they're identical as the calibrated images now.

[aaronlsmiles]

Solved it! Definitely not the code, but not something that was happening after calibration! Must of been an issue with the calib file. Tried with a new set of image pairs, which used a larger printed checkerboard. Got a marginally worse RPE (1.31) than the previous (1.29), but the images rectified nicely! 😃

Also set the lines to the image width.

Thanks for your help!