Project 4
![[Logo]](mannequin.jpg) |
Project 4: Skinning and Mesh Interactivity
|
A reference implementation of the virtual mannequin, showing a model in one sculpted pose.
|
In this project, you will implement the first part of a skinning system in WebGL -- a way to animate a 3D model by manipulating "bones" embedded inside the model. To deform the model using skinning, you must implement a basic mouse-based bone selection and rotation interface, which is the focus of this assignment.
Installing TypeScript
As with Assignment 3, you will need to have TypeScript and HTTP-server installed to run the project in your browser. The first two steps do not need to be repeated if you completed them for Assignment 3, but you will need to install on additional package to handle offscreencanvas:
- Install the NPM package manager. You can get it here.
- Install the TypeScript compiler tsc, and http-server. If NPM is properly installed, you should be able to do this installation using the following commands (might require an Administrator Mode command line on Windows):
npm install -g typescript
npm install -g http-server
-
Install the additional packages for the TypeScript definitions for Off Screen Canvas:
npm install --save @types/offscreencanvas
Starter Code
As before, we have provided you with starter code, written in TypeScript and using WebGL and several helper libraries (primarily ThreeJS) as a starting point for implementing this project. We have also included code for reading and writing images to JPEG files (this requires a nontrivial amount of low-level parsing and byte manipulation). The starter code is a working example of how to set up WebGL and a shader program to draw a checkerboard floor, and move around the world using an orbital and FPS camera (as per assignment 2).
Assets are in the COLLADA file format (.dae extension), which is an open standard XML schema maintained by the Khronos Group for 3D models and animations. We have included several models for you to test your skinning algorithm on, but feel free to create additional .dae assets and add them to the assets/skinning/ directory. You will also want to add in additional keybinds for loading the model in GUI.ts.
Building the Starter Code
You will use the script make-skinning.py to invoke the tsc compiler on the TypeScript source files in src/. This does some post-processing to combine the Javascript files with static files necessary for the web site to function. The complete web package is installed in dist/. Do not directly edit the files in dist/. All development should take place in the source files in src/.
Thus, to compile the code, simply run make-skinning.py from the folder that contains it (the project root folder), and check that your updated file structure is correct.
Running the Starter Code
To run the starter code, first, you must launch the HTTP server. From the project root directory (containing dist/), run:
http-server dist -c-1
If successful, you should get a confirmation message that the server is now running. You can then launch the app by pointing your browser to http://127.0.0.1:8080.
Required Features
Bone Picking
As the user moves the mouse around the screen, you should highlight nearby bones; these highlighted bones are then the ones the user will rotate by clicking and dragging the mouse. Implement the functionality that determines, whenever the user moves the mouse, which bone (if any) should be highlighted. Most of this code will be handled in GUI.ts, but you'll also need to create a RenderPass in App.ts for the highlighting along with custom shader code in Shaders.ts.
- Ray generation: Each time the user moves the mouse, convert the position of the mouse cursor in screen coordinates to a ray in world coordinates, whose origin p is at the eye's position, and whose direction v is such that the ray from p in the v direction pierces the near plane at the pixel containing the mouse cursor. You will need to perform an operation similar to glm::unProject to get into world space (not camera space), as we want to intersect this ray with the geometry in the scene.
- Ray-Cylinder Intersection: The bones, when rendered as line segments, are very thin and hard to click on; to make the interface easier to use, we will "thicken" the bones into cylinders with a fixed radius (kCylinderRadius) for the purposes of detecting which bone the user intended to select.
The details of ray-cylinder intersection are up to you. One way to proceed is to first find the closest point c_0 on the ray (p + tv) to the infinite line passing through the cylinder's axis. If the distance between c_0 and the infinite line is greater than kCylinderRadius, the ray misses the cylinder. Also, if the value of t corresponding to c_0 is negative, the cylinder is behind the eye and the ray misses the cylinder. Otherwise, project c_0 onto the cylinder's axis, and check whether the projected point is between the two cylinder endcaps. If so, the ray hits the cylinder; otherwise, the ray misses. Note that there is a corner case where the ray hits one endcap, pierces the entire length of the cylinder, and leaves through the other endcap, which is declared a miss by this procedure. But this corner case is rare enough that you can ignore it.
If the ray hits one or more cylinders, the first such cylinder hit becomes the highlighted cylinder. Otherwise, no cylinders are highlighted.
- Cylinder Rendering: Visualize highlighted bones by drawing a wireframe prism approximating a cylinder of radius kCylinderRadius (and the same length as the bone) around the highlighted bone. One way to do is is by precomputing the vertices and faces of a single prism, scale it to match the length of the bone, then transform it to the correct position and orientation of the bone you are enclosing.
Notes:
There are three parts to bone picking:
- Ray generation.
Implement a ScreenToWorld function that generates a world space point given screen space (x, y). There are several ways to do this:
- Use the camera matrix:
- calculate the width and height (w, h) of the image plane in world coordinates (you need to know the field of view)
- calculate ndc.x, ndc.y
- use (ndc.x, ndc.y), (w, h), look, up, tangent to get the world space coordinates
- Use the projection matrix:
- unproject (x, y, 0) to get the world space coordinates
- Once you have this, subtract the eye position to get a world space ray.
- Another method is:
- let q be unproject (x, y, 0)
- let p be unproject (x, y, 1)
- take the ray through the two points p and q
- Cylinder intersection.
Implement a cylinder/ray intersection routine where the cylinder points straight up and is centered at the origin (or lying on its side), has radius R, and height H. Transform the ray into the cylinder's coordinates. Use the bone's orientation to do this.
Turn this problem into a 2D problem. Project the line onto the (x, z) plane and intersect a line with a circle of radius R. If the ray misses the circle, reject the ray. Else check the intersection times t0 and t1 and see if they are good. Compute the intersection points and take the closest good one. A point is good if it has a y-value between 0 and H. This algorithm does not handle intersection at the caps.
-
Cylinder drawing.
Generate a nxn two-dimensional grip of lines. Use a model transformation (which is the bone transformation with a scale along the y-axis applied) to stretch the cylinder out, and a scale in the x-z plane to widen the cylinder. The bone transformation won't contain a scale, since it has to be purely a rotation.
The vertex shader can wrap the line mesh around (0,0) using (cos(theta), sin(theta)).
Updating the Skeleton
All data related to a character is stored in the .dae file. This file contains the model itself - represented as a triangle mesh along with some information about the materials and textures to use for that mesh - as well as a skeleton rig and blending weights specifying how moving the bones affects the different parts of the model.
We have provided code for parsing and rendering the model mesh. AnimationFileLoader.ts handles loading the COLLADA data, and the mesh and skeleton data is already being rendered to the screen in App.ts. You will find some initial data structures to help you get started in Scene.ts. Please look over these classes to get an idea of how the data is stored, but you are welcome to change or modify these classes as you see fit.
MeshGeometry contains both the mesh information (such as vertex positions/normals) as well as the skinning information (which bones have what weight). Bone stores the information about the bones in world space. We will be working in world space for this assignment, so your main task will be to update the position of the bone joints (in world space) based on rotations applied to that bone by the animator. You will perform these updated within the Mesh class, which handles both the MeshGeometry and Bone data.
When the user clicks and drags the mouse outside of the mesh, the camera rotates as in assignment 2. Clicking and dragging the mouse on a selected bone rotates that bone and all child bones attached to that bone.
- Applying a rotation to a bone both rotates that bone and propogates that rotation through any child bones. We are not using an offset matrix (i.e. the inverse of the initial pose translation/rotations) to handle these transforms, so efficiency will not be a concern in this assignment. Note that the skeleton is represented using a forest of bones which connect pairs of joints, so several bones might share the same parent bone -- this represents a branch in the skeleton. For example, several finger joints might specify a wrist joint as its parent.
- The bone should rotate relative to the camera coordinate system. That is, convert the drag direction into a vector in world coordinates, take its cross product with the look direction, and rotate all basis vectors of the bone about this axis by rotation_speed radians. This calculation should remind you of the FPS camera controls.
- Pressing the left and right arrow keys while a bone is highlighted should roll the highlighted bone: spin the bone about its tangent axis by roll_speed radians.
Linear Blend Skinning
The data for the blend weights per mesh vertex is already loaded in MeshGeometry, but in order for the mesh to deform based on the skeleton's updated position/orientation, you will need to modify the sceneVSText shader. Currently the mesh is positioned based on the mesh's resting pose (passed in as vertPosition). Since we want the final vertex positions to be controlled based on the weights of associated bone, this world space position should be calculated based on the translations/rotations of these bones. These translations/rotations are being passed in via jTrans and jRots.
Note the Attributes in MeshGeometry: v0 through v3. These tell us the vertex's position in the coordinate system of the bone i corresponds to skinIndex[i] (i.e. v0 corresponds to skinIndex[0], v1 corresponds to skinIndex[1], etc). Thus to apply the linear blend skinning, we sum the weight of each bone times the vertex position within that bone's coordinate system.
Miscellaneous
Add a shader to handle texture mapping for models. You will also need to update the RenderPass for the model to do so. mapped_cube.dae is a good scene to check that texture mapping is working.
Extra Credit
Apply dual quaternion blend skinning instead of linear blend skinning. Only small adjustments to the math are required to apply dual quaternion blend skinning versus linear blend skinning, but you will want to write a couple helper functions to apply the dual to the vertices.
Design Document
Create a design document exploring how you will implement the math and code architecture necessary for Assignment 4. This design document is intended to provide you enough of a scaffold that you can begin implementing your solution in an organized, coherent way.
This design document should demonstrate your understanding of the existing code base, as well as the required functionality and steps needed to create an animated model. This means you should look carefully through the provided code base (including the shaders) before starting on this document. This is especially true if you are new to asynchronous programming, since a lot of the design is built around those principles.
This also means you should take time to understand the purpose of all the math we covered in class. It's easy to lose sight of the forest (i.e. creating an animated 3D model) when we're focusing on the leaves (i.e. affine transformations), but my experience is that students who are able to understand how the math relates to the tangible goals of the project have an easier time completing the assignment than those who are rotely going through the list of required features.
To accomplish all of the above, your design document should explain the following:
- Existing class and functions: Look through all of the existing functions and classes in main body of the skeleton code (App.ts, GUI.ts, Shaders.ts, and Scene.ts) and document what they do and how they do it.
- Bone/joint hierarchy: Determine the sorts of class structures you could implement to store the necessary data to create an animation hierarchy. This is your first approximation, so you may get it wrong, but document both the fields and their data types and methods and their signatures along with a brief explanation of how you'll use those fields/methods if it's not clear from context.
- Additional functionality: Document the high level functionality of what you'll be implementing. This should be high enough level that it's clear what you're trying to achieve, but low enough level that you can understand the specific functionality you'll need to put into place.
Once this is in place, describe both the math these functionalities will require and write out expected methods (and where to include them) in order to implement this math. This should include exploring how you will modify the shaders to allow for animation.
- Questions and Uncertainties: Include a section listing off any unanswered questions you have, or concerns about how to implement. These should be questions that were raised while writing the document, but that you were unable to answer even after taking time to think through the problem.
A4 Milestone
For your A4 milestone, submit a Gitlab link to the project on a branch called
a4-milestone. This should include the stable code you've written so far along with progress report as a .pdf document stating what has been accomplished (and by whom if working with a partner), and what the plan is for completing the assignment on time. Include some image artifacts of what's been implemented as well.
As a guideline of what we're expecting, you should have all of the bone-picking functionality completed. You should also be making good progress into mouse controls and skeleton rotation. Note that we aren't grading you on the quality of your controls, but you should have reasonable interactions to make grading easier.
In terms of your plan, please try to be specific -- this document is also intended to help you think through what needs to be done and what a reasonable timeline should be. Therefore, you should include:
-
A rough timeline of what you expect to complete the specific, required features
-
A description of the algorithm you'll be using to implement each of the specific, required features
-
Notes about any unknown questions and/or known issues that might slow down your progress implementing any of the required features
Submission Instructions
Submit your project via GitLab and provide a link to it via Canvas. Make sure to keep your repository private and give access to me (thesharkcs) and the TA (TA GitLab usernames are posted to Announcements on Canvas). We will be grading based on a branch called a4-code-freeze, so please make sure you have a working final version of the project on that branch. If commits are made after the deadline, we will treat that as a late submission, and we will deduct late slips accordingly and/or apply the late submission penalty as per the syllabus.
In addition to your complete source code, the project should include:
- A README file including your name (and your partner's name), and if you made any modifications to the included libraries/build script.
- A report explaining how you implemented the skeleton hierarchy, bone picking, and bone manipulation, as well as several screenshots of the final product. Please also include a section on issues you encountered, known bugs, and future work. This will help us understand where you ran into issues and better assess your work, even if we encounter issues while grading.
Grading
You'll be graded on how well you met the requirements of the assignment. D quality or below work has few or none of the features implemented/working. C quality work has implemented some or most of the features, but the features are built in a way that is not fully working and/or the logic is not well-considered. B quality work has implemented most or all of the features in a way that works but may have some issues and/or the logic is not fully considered. A quality work has implemented all of the features in robust, polished way that demonstrates your understanding of the assignment, the math, and the provided code base. Note that a well-written report can help us understand your thought process, which is a large part of what we are grading you on in these assignments.
Reminder about Collaboration Policy and Academic Honesty
You are free to talk to other students about the algorithms and theory involved in this assignment, or to get help debugging, but all code you (and your partner if you have one) submit (except for the starter code, and the external libraries included in the starter code) must be your own. Please see the syllabus for the complete collaboration policy.
Last modified: 01/10/24
by Sarah Abraham
theshark@cs.utexas.edu