Bryce
6feb83c9e3
Single command that runs RTMPose3D inference on a reference image and
retargets the keypoints onto a Mixamo-named armature in a .blend file:
./pose.sh <reference.png> <input.blend> <output.blend>
Components:
- pose.sh: shell wrapper that runs RTMPose3D in the rtmpose3d conda env
then invokes Blender headless with apply_pose.py.
- apply_pose.py: opens the input .blend, finds the character armature,
applies pure FK direction-matching for arms and legs, captures head
yaw via orient_bone with the horizontal ear axis, lands the rig on
the floor, saves the posed scene, and renders a front-view preview.
- inputs/: test references (basepose, basepose2, 0102, pose-test, squat)
and the rosella-hunyuan-online rig blend.
- README.md: usage, design notes, and the list of bones intentionally
left at rest (head, feet, spine, clavicles).
mmpose is treated as an external library at $MMPOSE_DIR; this repo is
only the retargeting glue.
66 lines
3.6 KiB
Markdown
66 lines
3.6 KiB
Markdown
# mimicrig
|
|
|
|
Mapping 2D photo poses to Mixamo Blender rigs.
|
|
|
|
Single command line: takes a reference image and a `.blend` containing a Mixamo-rigged character, and produces a posed `.blend` plus a front-view preview PNG.
|
|
|
|
```bash
|
|
./pose.sh <reference.png> <input.blend> <output.blend>
|
|
```
|
|
|
|
Outputs:
|
|
- `<output.blend>` — the posed scene
|
|
- `<output>.png` — front-view preview render (same basename)
|
|
|
|
## Example
|
|
|
|
```bash
|
|
./pose.sh inputs/pose-test.png inputs/rosella-hunyuan-online.blend out/pose-test_posed.blend
|
|
```
|
|
|
|
## How it works
|
|
|
|
1. **RTMPose3D inference** on the reference image, in the `rtmpose3d` conda env, produces 133 3D keypoint coordinates (COCO-WholeBody).
|
|
2. **`apply_pose.py`** runs inside Blender:
|
|
- Picks the first armature whose bone names start with `mixamorig:` and which has a mesh child (skips OpenPose debug rigs etc.).
|
|
- Clears any animation and resets to rest pose.
|
|
- **Pure FK direction matching**: for each major bone, rotates so its Y axis points from one keypoint to the next, processed parent-first.
|
|
- **Head yaw via `orient_bone()`**: the neck uses a full two-axis orientation (vertical Y + horizontal ear axis projection) to capture head turn around the vertical axis. `point_bone` alone can't represent rotation around the bone's own axis.
|
|
- **Forward-monotone arm chain**: mirrors arm joints whose Y is behind their parent — a safety net for monocular depth flips that put wrists behind the body.
|
|
- **Floor settle**: after rotations, translates the whole armature in Z so the lowest foot point sits at world Z=0. Fixes the "floating squat" problem for low-stance poses.
|
|
3. **Preview render**: front-view, framed to the posed character, lit and saved alongside the output `.blend`.
|
|
|
|
### Bones the script does NOT rotate
|
|
|
|
- **Hips, Spine, Spine1, Spine2**: keeping the spine straight avoids unnatural torso lean from monocular depth noise.
|
|
- **Head**: rest pose has the face pointing forward; rotating it from a single nose keypoint causes chin-down tilt because RTMPose3D places the nose well in front of the shoulders. Yaw is captured via the neck instead.
|
|
- **Feet/toes**: RTMPose3D's toe Z is consistently too low, which would make the feet point downward like ballet pointe. Rest pose (feet flat) is more accurate for standing/sitting poses.
|
|
- **Clavicle (Shoulder)**: doubly-rotating clavicle + upper arm causes the shoulder joint to over-extend.
|
|
|
|
## Best results
|
|
|
|
- Use a **head-on full-body reference image**. RTMPose3D's monocular depth estimation is much cleaner from a frontal view; angled views get the wrong front/back for individual joints.
|
|
- The character mesh and rig should be in **Mixamo bone-name convention** (`mixamorig:LeftArm` etc.).
|
|
- Rest pose should have the character **facing -Y** in world (Mixamo default).
|
|
|
|
## Layout
|
|
|
|
```
|
|
mimicrig/
|
|
├── pose.sh # CLI wrapper
|
|
├── apply_pose.py # Blender script (retargeting + preview render)
|
|
├── inputs/ # Test reference images and the rig blend
|
|
│ ├── *.png
|
|
│ └── rosella-hunyuan-online.blend
|
|
└── out/ # Outputs (created on first run)
|
|
```
|
|
|
|
## Dependencies
|
|
|
|
- **Blender 4.x** at `/opt/blender-4.3.2-linux-x64/blender` (override via `$BLENDER`)
|
|
- **Conda env `rtmpose3d`** with mmpose + mmdet + mmcv installed
|
|
- **mmpose checkout** with the `projects/rtmpose3d` directory, defaulting to `~/dev/playground/mmpose` (override via `$MMPOSE_DIR`)
|
|
- **RTMPose3D model weights** cached at `~/.cache/torch/hub/checkpoints/` — auto-downloaded by the demo script on first run
|
|
|
|
The mmpose checkout is treated as an external **library** here; this repo only holds the retargeting glue.
|