Skip to main content
added a link to a video showing the current state of the question
Source Link

edit2: Added the comparison between Unity and Processing screens while adjusting the calibration points. I am still aiming to obtain a stable matrix transformation, as seen in the Processing window.

edit2: Added the comparison between Unity and Processing screens while adjusting the calibration points. I am still aiming to obtain a stable matrix transformation, as seen in the Processing window.

Added pictures and results from the processes I am trying to debug or understand
Source Link

edited to make my steps clearer:

Starting a new unity scene. A camera calibration algorithm returns me a matrix \$C\$ such:

 0.00166351   -0.00917993    0.0216692   0.885074
 -0.0120409     0.0087226    0.0119709   0.464458
-1.5839E-05  -1.43059E-05  1.55468E-05  0.0013842

I add a new camera object to the unity scene and name it "projector".

After decomposing \$C\$ into \$R\$, \$K\$ and |T| I can apply rotation and transformation on newly added unity camera, and also extract the projection to apply on the same camera, following this and this steps.

\$R\$

 0.703018  -0.00551996  0.711151
-0.38204    0.8405      0.384195
-0.599843  -0.541784    0.588778

\$K\$

 629.808  -0.986886   633.74
  0        626.035    361.485
  0          0        1

applying position and rotation from R and |T|

projector.transform.position = new Vector3(-30.2851,-30.2294, 30.3633);
projector.transform.rotation = Quaternion.Euler(new Vector3(33.126,-45.329,-0.45));
//Ground truth camera position: 30,30,-30 ; rotation: 33,-45,0

ProjMat:

0.98408 0.00154 0.00978 0.00000
0.0     1.73899 0.00413 0.00000
0.0     0.0     1.001   0.10005
0.0     0.0     -1.0    0.00000

I apply this projection matrix to the unity as suggest:

Projector.projectionMatrix = ProjMat;

Although not constant for each calibration matrix, so far I obtain projector(pink) and ground_truth(green) cameras almost matching.

// Undo view matrix 'V'
var antiView = Projector.transform.localToWorldMatrix;

// Undo model matrix 'M' (assuming this object is not batched).
var antiModel = transform.localToWorldMatrix;
    

//the moment I enable the line below, my "projector" camera frustum collapses into a line and points somewhere away from the projection center (orange) and values go strange.

Projector.projectionMatrix = ProjMat * antiModel * antiView;

enter image description here

edited to make my steps clearer:

Starting a new unity scene. A camera calibration algorithm returns me a matrix \$C\$ such:

 0.00166351   -0.00917993    0.0216692   0.885074
 -0.0120409     0.0087226    0.0119709   0.464458
-1.5839E-05  -1.43059E-05  1.55468E-05  0.0013842

I add a new camera object to the unity scene and name it "projector".

After decomposing \$C\$ into \$R\$, \$K\$ and |T| I can apply rotation and transformation on newly added unity camera, and also extract the projection to apply on the same camera, following this and this steps.

\$R\$

 0.703018  -0.00551996  0.711151
-0.38204    0.8405      0.384195
-0.599843  -0.541784    0.588778

\$K\$

 629.808  -0.986886   633.74
  0        626.035    361.485
  0          0        1

applying position and rotation from R and |T|

projector.transform.position = new Vector3(-30.2851,-30.2294, 30.3633);
projector.transform.rotation = Quaternion.Euler(new Vector3(33.126,-45.329,-0.45));
//Ground truth camera position: 30,30,-30 ; rotation: 33,-45,0

ProjMat:

0.98408 0.00154 0.00978 0.00000
0.0     1.73899 0.00413 0.00000
0.0     0.0     1.001   0.10005
0.0     0.0     -1.0    0.00000

I apply this projection matrix to the unity as suggest:

Projector.projectionMatrix = ProjMat;

Although not constant for each calibration matrix, so far I obtain projector(pink) and ground_truth(green) cameras almost matching.

// Undo view matrix 'V'
var antiView = Projector.transform.localToWorldMatrix;

// Undo model matrix 'M' (assuming this object is not batched).
var antiModel = transform.localToWorldMatrix;
    

//the moment I enable the line below, my "projector" camera frustum collapses into a line and points somewhere away from the projection center (orange) and values go strange.

Projector.projectionMatrix = ProjMat * antiModel * antiView;

enter image description here

Source Link

Camera Calibration custom Model-View-Projection matrix in Unity?

Coming from this topic

I have a Model-View-Projection Matrix obtained from a Camera Calibration method, from where I want to render the scene:

-4.7415   -2.3964   -2.4075  253.0431
-2.7923    6.1067    6.1569  143.6573
 0.0000    0.0000    1.0000    1.0000
 0.0061   -0.0094    0.0145    1.0000

As far as I saw, all documentation shows methods to load ModelView and load Projection independently. I was hoping to find something like:

GL.PushMatrix();
GL.MultMatrix(theMVPMatrix);

-> draw/load all the scene 

GL.PopMatrix();

All examples I found show this would run OnPostRender(), but assuming there is a camera, at this point the camera has already rasterized the image. I guess?