How can I find the 3D coordinates of a projected rectangle?

Hello everybody. I have the following problem which is mainly algorithmic.

• Let ABCD be a rectangle with known dimensions d1, d2 lying somewhere in space.
• The rectangle ABCD is projected on a plane P (forming in the general case a trapezium KLMN). I know the projection matrix H.
• I can also find the 2D coordinates of the trapezium edge points K,L,M,N.

The Question is the following :

• Given the Projection Matrix H, The coordinates of the edges on the trapezium and the knowledge that our object is a rectangle with specified geometry (dimensions d1, d2), could we calculate the 3D coordinates of the points A, B, C, D ?

I am grabbing images of simple rectangles with a single camera and i want to reconstruct the rectangles on space. I could grab more than one image and use triangulation but this is not desired.

The projection Matrix alone isn't enough since a ray is projected to the same point. The fact that the object has known dimensions, makes me believe that the problem is solvable and there are finite solutions.

If I figure out how this reconstruction can be made I know how to program it. So I am asking for an algorithmic/math answer.

Any ideas are welcome Thanks

From stackoverflow

• You need to calculate the inverse of your projection matrix. (your matrix cannot be singular)

  mbeckish : @netclectic - Did the original projection map points to rays? No - and neither will the inverse. When you build the projection matrix, you decide the "depth" to which to project.

• m_oLogin is right. If I understand your goal, the image the camera snaps is the plane P, right? If so, you're measuring K,L,M,N off the 2D image. You need the inverse of the projection matrix to reconstruct A,B,C, and D.

  Now I've never done this before, but it ocurrs to me that you might run into the same problem GPS does with only 3 satellite fixes - there are two possible solutions, one 'behind' P and one 'in front' of it, right?

• I think this problem will generate a set of possible solutions, at least in 2D it does. For the 2D case:

             |   
  -----------+-----------
            /|\
           / | \
          /  |  \
         /---+---\VP
        /    |    \
       /     |     \
      /      |      \
     /       |       \
    /        |   --   \
   /         |    |    \
  /          |    |     \
  

  In the above diagram, the vertical segment and the horizontal segment would project to the same line on the view plane (VP). If you drew this out to scale you'd see that there are two rays from the eye passing through each end point of the unprojected line. This line can be in many positions and rotations - imagine dropping a stick into a cone, it can get stuck in any number of positions.

  So, in 2D space there are an infinite number of solutions within a well defined set.

  Does this apply to 3D?

  The algorithm would be along the lines of:

  1. Invert the projection matrix
  2. Calculate the four rays that pass through the vertices of the rectangle, effectively creating a skewed pyramid
  3. Try and fit your rectangle into the pyramid. This is the tricky bit and I'm trying to mentally visualise rectangles in pyramids to see if they can fit in more than one way.

  Skizz

  EDIT: If you knew the distance to the object it would become trivial.

  EDIT V2:

  OK, let Rn be the four rays in world space, i.e. transformed via the inverse matrix, expressed in terms of m.Rn, where |Rn| is one. The four points of the rectange are therefore:

  P1 = aR1
  P2 = bR2
  P3 = cR3
  P4 = dR4
  

  where P1..P4 are the points around the circumference of the rectangle. From this, using a bit of vector maths, we can derive four equations:

  |aR1 - bR2| = d1
  |cR3 - dR4| = d1
  |aR1 - cR3| = d2
  |bR2 - dR4| = d2
  

  where d1 and d2 are the lengths of the sides of the rectangle and a, b, c and d are the unknowns.

  Now, there may be no solution to the above in which case you'd need to swap d1 with d2. You can expand each line to:

  (a.R1x - b.R2x)² + (a.R1y - b.R2y)² + (a.R1z - b.R2z)² = d1²

  where R1? and R2? are the x/y/z components of rays 1 and 2. Note that you're solving for a and b in the above, not x,y,z.

  Mike Dunlavey : +1 If you haven't got it, you're close.

• The projection matrix encapsulates both the perspective and scale, so the inverse will give you the solution you are after. I think you are assuming that it only encapsulates the perspective, and you need something else to choose the correct scale.

  Skizz : The process of projecting from world space to view space loses information - namely the depth (an infinte set of 3D points all map to the same 2D point). The OP wants to know how to re-calculate the lost data.

  mbeckish : The set of projected points don't contain the depth info, but the projection matrix does - that's why you can use the inverse to get back to the original rectangle if you know the original projection matrix.

  Jacob : The projection matrix does not encapsulate scale.

  mbeckish : @Jacob - You are projecting the rectangle to a polygon on a specific target plane, not to the "shadow" which is cast from the rectangle in a given direction. Or else, how did you get the specific points K,L,M,N of the projected shape?

• I'm going to give a fairly brief answer here, but I think you'll get my general drift. I'm assuming you have a 3x4 projection matrix (P), so you should be able to get the camera centre by finding the right null vector of P: call it C.

  Once you have C, you'll be able to compute rays with the same direction as vectors CK,CL,CM and CN (i.e. the cross product of C and K,L,M or N, e.g. CxK)

  Now all you have to do is compute 3 points (u1,u2,u3) which satisfies the following 6 constraints (arbitrarily assuming KL and KN are adjacent and ||KL|| >= ||KN|| if d1 >= d2):

  1. u1 lies on CK, i.e. u1.CK = 0
  2. u2 lies on CL
  3. u3 lies on CN
  4. ||u1-u2|| = d1
  5. ||u1-u3|| = d2
  6. (u1xu2).(u1xu3) = 0 (orthogonality)

  where, A.B = dot product of vectors A and B ||A|| = euclidean norm of A AxB = cross product of A and B