/*
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ */
//
GJB: below are my intrinsic and extrinsic calibration results produced
//
using the calib_gui tool from the CVL. These are provided to illustrate
//
where your calibration numbers map to in Maya. Note that I’m only showing
//
one set of intrinsic & extrinsic computations. You will be make one set of
intrinsics
// to
be used with 2 sets of extrinsic calibrations. In Maya, this means you’ll make
a
// camera,
set its intrinsics, duplicate it, and then apply an extrinsic calibration to
each of them.
/*
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ */
Calibration
results after optimization:
Focal
Length: fc = [ 822.48804 748.83937 ]
Principal
point: cc = [ 356.67264 228.22535 ]
Skew: alpha_c = [ 0.00000 ] =>
angle of pixel = 90.00000 degrees
Distortion: kc = [ -0.23444 0.46020
-0.00117 -0.00099 -0.83313 ]
Pixel
error: err = [
0.11161 0.12090 ]
Extrinsic
parameters:
Translation
vector: Tc_ext = [ -106.514107 319.558805 2463.612936 ]
Rotation
vector: omc_ext = [ -1.220792 -0.462710 -0.571164
]
Rotation
matrix: Rc_ext = [ 0.772569 0.634324 -0.027771
-0.158799 0.235386 0.958841
0.614752 -0.736361 0.282582 ]
Pixel
error: err = [ 0.11180 0.14169 ]
Projection: proj_mat = [ 854.693853
259.083852 77.948219
791096.751336
21.387051
8.209970 782.510324
801557.135786
0.614752 -0.736361 0.282582 2463.612936
]
Figure
1: Before commenting on the shades, consider that I was dealing with two
halogen lights placed directly above the camera.
/*
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ */
//
GJB: hand-calculations for mapping the above intrinsics to Maya-compatible form
/*
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ */
You
will need to perform some calculations before mapping your cameras’ parameters
to Maya cameras:
First,
find your pixel aspect ratio from the focal lengths (fc):
748.83937
/ 822.48804 = 0.910456242 ( = b/a)
Now
express your camera aperture in inches (Maya measures this and a few other
things in inches, no matter what global units are selected in the preferences):
720mm
= 28.34646 inches
480mm
/ (b/a) =
480mm / .910456242 =
527.20820381mm =
20.7562284965 inches
-------------------------------------------------------------------------------------------------------
Look
at the extrinsic parameters from your "home" images.
Pick
two different home images, preferably ones that appear different from each other,
but where both saw the same pose of the calibration object, and both had small reprojection
errors.
For
each of the home images:
Make
a new matrix of size 4x4 with the Rc_ext as the first 3 rows and cols, the
Tc_ext as the first 3 elements of the 4th column, and (0 0 0 1) as the fourth
row. Note that when transposing the Translation vector to populate the fourth
column, you should also divide each value by 10 because we'll want to operate
in cm, and the calibration spit these out in mm.
I
get a matrix like this:
0.772569 0.634324 -0.027771 -10.6514107
-0.158799 0.235386 0.958841 31.9558805
0.614752 -0.736361 0.282582 246.3612936
0 0 0 1
Now
calculate the inverse of that matrix. (inverse is because we want to move the
camera in Maya, not the pattern)
Then
transpose it. (transpose is because Maya is column-major not row-major)
Save
it for when we apply the extrinsics to the virtual cameras. Hint - matlab is
pretty good for these operations.
After
transposing the whole matrix, mine looks like this:
0.772569 0.634323 -0.0277706 0
-0.158799 0.235386 0.958841 0
0.614752 -0.73636 0.282582 0
-138.148 180.645 -100.554 1
Get
out the info about the calib pattern you used for the extrinsic calcs:
I
used 5x5 of the squares on the calib pattern, where:
Each
Square width: 84.75mm
Square
height: 84.25mm
so...
the whole pattern (width, height) = ( 42.375cm, 42.125cm )
/*
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ */
// GJB: Now we'll start modeling our pattern
and our cameras in Maya,
// with the goal of rendering a replica from
two angles which looks
// like the raw footage.
/*
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$ */
Perform
the following actions:
Window->
Settings/Preferences -> Preferences -> Settings:
Linear
= centimeter (usually default)
Up
Axis: Z
Angular:
radians
Create->Polygon
Primitives->Plane (now a plane appears, centered at the origin).
Beware:
some Maya sessions may be not be set to create planes in the z=0 plane by
default. If your plane isn’t purely in the xy plane, create a new one, and use
the “custom” box at the right of the menu entry for ->Plane so you can
specify the Z-plane radio button.
Hit
Ctr-A to view its attributes
Go
to the PolyPlane tab, and adjust the settings for the overall width/height of
the plane, as well as the number of subdivisions (should be same as # of
squares in your pattern). While you're here, change the Texture setting to
"Preserve Aspect Ratio."
Go
to the pPlane settings tab and type into the Translate (x and y) fields amounts
that will move the plane so it fits wholly in the positive_x and positive_y
quadrant. That will be half of the pattern's width and height (in cm).
Go
to the pPlaneShape tab and the settings for Mesh Component Display: here you'll
want the backface culling to be off, since we'll be looking at the underside of
the plane
To
make the pattern look pretty, you'll want to do:
Window->Hypershade,
pick Rendering->Materials, right-click in the empty space on the right, and
Create->Materials->Lambert (left-click to put it down). Double-click the
new icon to bring up this material's settings. You want to change the color to
a special pattern, so click on the pattern-button to the right of the Color's
slider. Pick "checker," and under the Placement tab, give it a
RepeatUV equal to half of your #boxes_in_width and height.
Now
select the plane in the scene and right-click on the checker material's icon to
"Assign Texture's Material to Selection."
You
now have a calibration pattern which should be the same size and look as the
one you used to calibrate the extrinsics of your scene.
Create->Lights->Point Light will be needed at some point so your nice
checker model isn't pitch black. Move it around after you perform the steps
below to add the camera. So far, this was the easy part.
Save
your file now, and as frequently as you can remember.
Create->Camera
Window->Render
Globals...
Change
Image format to tif
Change
Camera to camera1
Resolution:
choose Custom, and set to 720 x 480
set
Pixel Aspect Ratio to: (b/a) (hit Tab to see how this auto-changes the Device
aspect ratio to 1.366).
Insure
'Channels' is set to "RGB Channel"
You'll
probably want to set the Anti-aliasing Quality set to High, but it's not
critical.
Select
camera, and hit ctr-A (get attributes window)
In
the cameraShape, set the Focal Length to a (the 1st or bigger focal length).
(Angle of View just changed automatically!)
Leave
Camera Scale and other fields not mentioned here at default.
Change
your Near Clip Plane to 0.01 and your Far Clip Plane to 50000. Bigger doesn't
hurt, and should be used if things are too far apart.
Camera
Aperture for (x,y) were calculated above, so these should be 28.346inches and
something near 20 (depending on your a and b).
Change
the Film Fit to "Horizontal"
Change
the Film Offset to:
{720/2
- cc(1), (cc(2) - 480/2) } =
(3.32736mm,
-11.77465mm) =
(0.1309984
inches, -0.463569 inches)
What
did we just do? We moved the location of the film to match the non-centeredness
of the real film/ccd w.r.t the actual lens.
Change
Display Options to have ON: Display Film Gate and Resolution
Add
Environment-> Image Plane: Create: Image Name = path to your home image.
Image Plane should be "Attached to Camera" What just happened? This
camera has a billboard showing your home-image mounted in front of it. If the
camera moves, so does the image plane, though objects, like the calib-pattern
object, can be placed between the two.
Optionally
make "Display" set to "in all views."
Size
= same as Camera Aperture
Offset
= same as Film Offset
Fit:
"To Size" and click on "Fit to Film Gate"
Note:
this should allow you to render through Maya the same image as you had as input.
If it’s somehow squished, try a Horizontal fit to match the camera settings.
The
depth of the Image Plane does not affect the size of the background image
attached to your camera. However, placing it too close to the camera could mean
that it occludes the plane sitting behind it, so type in a depth which
accommodates your distance from the origin: less than 50000, but more than the
largest translation component of your extrinsics.
In
main Maya app, pick a window, and Panel->Perspective->Camera1
Note:
If you only have one panel on the screen and want to have more, use
Window->View Arrangement->Four
In
the same window, pick Shading->"Smooth Shade All"
Also
under Shading, switch on Hardware Texturing if possible.
Fine
time to save your work.
You
should still not be seeing much in the camera's panel. I like to peek at the
state of the world by using a different panel and switching it to a perspective
view, which I can move around:
Panels->Perspective->persp.
Alt-Shift middle-mouse-button and Alt-shift left button allow you to translate
and rotate the scene, which should currently contain a camera and a pattern
both sitting at the origin.
If
you've saved your work, you might want to play with translating and rotating
the camera to make the CG "model" line-up with the image of the
model. Alt-r and Alt-t reveal rotation and translation handles you can mouse
around with to manipulate the camera or the model. Leave the model alone, but
do your best by moving the camera. This is hard, and once you're done wasting
your time, you'll proceed to the following instructions, because you'll realize
that a little math is easier than ALOT of clicking and dragging.
It's
important that you realize that what follows IS part of the assignment, but,
strictly speaking, isn't necessary for rendering CG to go with your footage.
Why? Your virtual camera now projects 3D objects in almost the same way as your
real camera did (more later on why not 100%). That means the two cameras'
intrinsics match. Since the camera didn't move, you could (theoretically) start
inserting CG flying elephants and they would enter and leave the field of view
in the right fashion, assuming you modeled them to scale (using cm units as we
specified earlier). This means that you could theoretically get by without
extrinsic calibration at all. So who needs the external parameters of the
camera (rotation and translation)? Doing a match-move certainly requires it,
and actually needs the R and t for each frame of footage. But even when the
camera is static, it's nice to do the following steps, because then you can
line-up the ground-planes (or wall planes if that's where you put the
calibration pattern) with the cardinal directions within Maya. You can texture
and place new CG objects in the scene, and they should line-up in each camera
for which you calculate the extrinsic parameters. That's more than just a
little useful if you intend to replace walls or match up foot-falls. It's especially
handy if you've calibrated multiple cameras and want them all in the same CG
scene.
To
remove ambiguity about rotation order pre/post multiplication, we're just going
to enter our transformation matrices directly. Note that Maya matrices are
transposed versions of the Foley et al. notation, so, for example, translation
in the x directions would go in the bottom left of a 4x4 matrix.
Window->General
Editor->Script Editor
Note:
type the following commands in the bottom panel, and use Edit->Execute or
Ctrl-Enter to execute them. The help contains more on MEL-scripting, but all
you'll need is given below.
Click-select
the camera.
type:
xform
-m
1 0
0 0
0 1
0 0
0 0
1 0
0 0
0 1;
setAttr
"camera1.rotateX" 3.1415926;
<execute
the above calls> (-m stands for matrix)
What
just happened? The camera was sent back to the origin, and back to pointing in
the default direction: -Z. Our calibration was done with the model origin in the
upper left and assumed the camera was looking toward +Z, so the second call
just rotated the camera about the x axis to make that true for the virtual
camera too: we're still sitting at (0,0,0), but facing +Z. Notice that the
panel showing the view through the camera has color-axes in the lower left, and
x and y are pointing to the right and down respectively. Now dust off the
inverted version of the camera's extrinsic transformation matrix. Transpose it,
and perform the following relative (hence the "-r") xform:
type
(substituting your own matrix in place of my 4x4):
xform
-r -m
0.772569
0.634323 -0.0277706 0
-0.158799
0.235386 0.958841 0
0.614752
-0.73636 0.282582 0
-138.148
180.645 -100.554 1;
Save
your work. You should be seeing a CG model (possibly with a checker-texture)
sitting almost precisely on top of the part of the image where the real
calibration model projected onto. If it's not perfect, don't get upset: the
calibration optimized the camera extrinsics after modeling lens-distortion (kc
in the intrinsic results). That's the one part of the calibration we can't get
Maya to reproduce. The absence of lens-distortion means that your CG object may
be SLIGHTLY off, but with the DV cameras we're using, won't explain gross
mismatches, so at this point, you really should be seeing a rather good match.
Render
this out (using Render->Render to New Window, and File-Save Image).
Figure
2: Alignment is off because the extrinsics were estimated to include lens distortion,
which is absent in this rendering.
Now
select your camera, and make a 2nd camera by doing
Edit->Duplicate. This camera has all the same parameters (intrinsic and
extrinsic) as the first, but you’ll reposition it to the other “home” camera
position:
First
clobber the extrinsics of camera2 (make sure #2 is the only one selected):
xform
-m
1 0
0 0
0 1
0 0
0 0
1 0
0 0
0 1;
setAttr
"camera2.rotateX" 3.1415926;
Then
apply the other set of extrinsics:
xform
-r -m
(the
4x4 matrix which comes from doing an extrinsic calibration of your camera when
it was placed at a second pose, looking at the same calib pattern)
Render
this out also.
To
get it perfect, render out a second pair of images after translating (ONLY!) very
slightly in the x and y directions (that's the dimension affected by
distortion). The hand-adjusted offset should be less than a cm for either axis.
Render these images out also: both are part of your submission for this part of
the assignment.
Figure
3: Minor (< 1cm) hand adjustment of the camera position (not rotation!) has
compensated for the lack of lens distortion. Lighting has also been adjusted
slightly, and now the scene is ready for new CG elements/action.
Now
model and render whatever you like, and render out a movie. Insert a simple CG
object (like torus or cone) into the scene so it “belongs”: on floor or wall
etc. Light and render to output a still/movie seen from both cameras, and
submit it to the HW2 directory. For Maya help in general, hit F1 or look
through the help-menu at the top right. The section on rendering (especially
Ch.10) will help you through rendering out a sequence of images. Doing a search
in the help on "background" will reveal a nice section which lays out
how to composite and texture 3D objects on top of live-action images (like the
home image) and image sequences (including movie files).
Total
submission:
2 x
image of calibration pattern as seen through camera placed purely
mathematically.
2 x
image of calibration pattern after tweaking for lens-distortion.
2 x
image/movie of new CG element added to the scene to look as real as possible.
--------------------------------------------------------------------------