ScaleHP: Scale-Mediated Optimization of Coupled Errors for Metric-Space Hand Pose Estimation
*Equal contribution †Corresponding author
This work was done when Ruitao Jing was an intern at Visincept and IDEA Research.
Abstract
In this paper, we present ScaleHP, a unified framework that explicitly represents per-instance metric scale to resolve the coupled errors in calibrated camera-space hand pose estimation. Under the common root-relative-to-global paradigm, camera-space accuracy depends jointly on relative geometry, root localization, and their scale-dependent composition. ScaleHP treats scale as the shared interface among these terms rather than optimizing them in isolation. Its metric-aware 2D–3D Transformer decoder introduces a dedicated scale token that exchanges information with sparse, semantically indexed 2D/3D joint queries, allowing global metric reasoning and local joint geometry to mutually refine one another. A parameter-free calibrated module then recovers root translation from the predicted unit-scale joints, metric scale, and known camera intrinsics. ScaleHP demonstrates state-of-the-art CS-MPJPE on FreiHand (35.8 mm) and, under benchmark-specific training protocols, state-of-the-art PA-MPJPE on FreiHand, DexYCB, and HO3Dv3 (5.0/4.6/5.9 mm), showing that improved global metric localization is attained together with improved local articulation. Ablations further show that scale–pose interaction improves both camera-space accuracy and relative depth geometry.
Introduction
Hand pose estimation is a basic perception primitive for interaction, manipulation, and teleoperation. VR/AR, embodied AI, and robotic teleoperation need more than articulation: the hand must be placed at the correct physical location and size in the calibrated camera coordinate system.
In the common root-relative-to-global paradigm, Qcam = s(Jrel + tnorm). This factorization exposes three coupled error sources: root-relative geometry, root localization, and the scale-mediated composition of the two. Existing systems usually focus on relative geometry, delegate localization to learned translation or an external depth cue, or improve global positioning at the expense of local accuracy. Leaving scale implicit also makes these compensating errors difficult to diagnose.
Calibrated perspective correspondences can recover normalized translation but cannot determine metric scale: different hand-size–depth pairs can produce the same 2D projection. ScaleHP therefore learns an explicit per-instance metric scale from metric supervision and uses it as the interface between sparse, semantically indexed joint geometry and global camera-space coordinates. This is a learned statistical prior—not a claim that monocular projection uniquely identifies hand size.
| Coupled error source | Role in camera-space error | ScaleHP response |
|---|---|---|
| (a) Relative geometry | Unit-scale root-relative joint error | Sparse 2D/3D joint queries exchange information with the scale token |
| (b) Root localization | Normalized translation error | A parameter-free calibrated solver uses predicted joints and known intrinsics |
| (c) Scale-mediated coupling | Scale error multiplies the composed geometry and translation | A supervised per-instance scale participates throughout pose learning |
Contributions
Scale-Mediated Formulation
We expose relative geometry, root localization, and their scale-dependent composition as three coupled camera-space error sources, with per-instance metric scale as their shared interface.
Query-Coupled ScaleHP
A dedicated scale token interacts reciprocally with sparse semantic 2D/3D joint queries, allowing global metric reasoning and local joint geometry to refine one another.
Metric and Relative Accuracy
ScaleHP obtains 35.8 mm CS-MPJPE on FreiHand and 5.0/4.6/5.9 mm PA-MPJPE on FreiHand, DexYCB, and HO3Dv3 under the paper’s stated protocols.
Method
1 Frozen Detector
Grounding DINO provides robust hand detection and multi-scale image features while remaining frozen during training.
2 Metric 2D–3D Decoder
Sparse, semantically indexed 2D/3D joint queries predict image joints and unit-scale root-relative 3D joints while exchanging information with a dedicated scale stream.
3 Scale Token
The scale token predicts a supervised per-instance metric gauge and interacts reciprocally with joint tokens, so metric reasoning can improve local geometry and vice versa.
4 Analytic Translation Solver
A parameter-free least-squares solve under known camera intrinsics recovers normalized root translation and composes the final metric camera-space joints.
Results
CS-MPJPE directly measures 3D joint error in camera space without root or Procrustes alignment, making it the primary metric for absolute metric localization.
FreiHand: Camera-Space and Aligned Accuracy
| Method | CS-MPJPE ↓ | PA-MPJPE ↓ |
|---|---|---|
| CMR-PG | 48.8 | 6.9 |
| HandDGP | 46.3 | 7.4 |
| MobRecon | 50.2 | 5.7 |
| NVF | 42.4 | — |
| METRO | — | 6.7 |
| MaskHand | — | 5.5 |
| Hamba | — | 5.7 |
| HandOS | — | 5.0 |
| ScaleHP (Ours) | 35.8 | 5.0 |
Results follow the protocols reported by the respective methods. NVF’s 42.4 mm uses FreiHand’s reference-bone scale (47.2 mm without it); ScaleHP uses RGB and known intrinsics without test-time ground-truth scale.
Cross-Dataset Camera-Space Transfer
| Method | DexYCB CS-MPJPE ↓ | HO3Dv3 CS-MPJPE ↓ |
|---|---|---|
| CMR + oracle global scale | 183.2 | 152.3 |
| HandDGP + oracle global scale | 222.1 | 132.6 |
| ScaleHP (Ours) | 136.3 | 50.7 |
The same FreiHand-trained checkpoint is applied directly to both target datasets. The baseline oracle is a test-set-optimal dataset-level rescaling; ScaleHP uses neither target-dataset training nor test-time ground-truth scale.
State-of-the-Art Comparison (Aligned Metrics)
Root-relative and Procrustes-aligned metrics focus on articulated pose quality rather than absolute metric localization. Each dataset column below uses a checkpoint trained on that dataset.
| Method | FreiHand | DexYCB | HO3Dv3 | |
|---|---|---|---|---|
| PA-MPJPE ↓ | R-MPJPE ↓ | PA-MPJPE ↓ | PA-MPJPE ↓ | |
| CMR-PG | 6.9 | — | — | — |
| I2L-MeshNet | 7.4 | — | — | — |
| HandDGP | 7.4 | — | — | — |
| MobRecon | 5.7 | 14.2 | 6.4 | — |
| METRO | 6.7 | 15.2 | 7.0 | — |
| HandOccNet | — | 14.0 | 5.8 | — |
| H2ONet | — | 14.0 | 5.7 | — |
| Deformer | — | 13.6 | 5.2 | — |
| Zhou et al. | — | 12.4 | 5.5 | — |
| TI-Net | — | 16.8 | 4.9 | — |
| MaskHand | 5.5 | 11.7 | 5.0 | 7.0 |
| Hamba | 5.7 | — | — | 6.9 |
| HandOS | 5.0 | — | 5.2 | 6.8 |
| ScaleHP (Ours) | 5.0 | 10.3 | 4.6 | 5.9 |
Per-Instance Scale-Token Coupling
| Method | FreiHand CS-MPJPE ↓ | DexYCB CS-MPJPE ↓ | HO3Dv3 CS-MPJPE ↓ |
|---|---|---|---|
| No token + oracle scale | 44.3 | 40.2 | 42.0 |
| ScaleHP | 35.8 | 30.0 | 33.4 |
CS-MPJPE (mm); each dataset uses its own checkpoint. The no-token setting receives an oracle dataset-level global scale.
Depth-Axis Regularization
| Method | PA-MPJPE ↓ | X ↓ | Y ↓ | Z ↓ |
|---|---|---|---|---|
| No scale-token interaction | 5.6 | 2.3 | 2.4 | 3.5 |
| ScaleHP | 5.0 | 2.2 | 2.2 | 3.0 |
FreiHand aligned errors (mm). Scale-token interaction reduces every relative component, with the largest change on the depth axis.
Scale–Pose Co-Adaptation
| Training schedule | PA-MPJPE ↓ | CS-MPJPE ↓ | Scale L1 error (10−3 m) ↓ |
|---|---|---|---|
| Delayed introduction | 5.2 | 39.5 | 2.06 |
| Joint | 5.0 | 35.8 | 1.94 |
FreiHand results. Introducing scale from initialization improves relative pose, camera-space pose, and scale prediction together.
Demo Videos
Qualitative demonstrations of calibrated camera-space hand pose estimation from monocular RGB input with known camera intrinsics.
Frame-wise metric camera-space hand pose estimation from a single camera view.
Pick-and-place interaction demo using the predicted camera-space hand pose.
Qualitative Results
FreiHand: Comparison with Camera-Space Methods
Multi-method comparisons on the FreiHand evaluation set. Each plot overlays GT, ScaleHP, HandDGP, and CMR; top views reveal depth and scale errors in camera-space baselines.
In-the-Wild Results
Unconstrained real-world examples showing input images alongside metric-space reconstructions and multi-view renderings.
Metric-space reconstruction with accurate image-plane projection.
Robust pose under challenging hand-object interaction.
Consistent 3D geometry across varying viewpoints.
Reliable metric localization under severe occlusion.
Qualitative Comparison with HaMeR
Camera-space visualizations on challenging in-the-wild samples. Because metric ground truth is unavailable for these images, the views below are qualitative illustrations rather than a quantitative benchmark.
Qualitative visualization under hand–hand interaction and occlusion.
Qualitative visualization under viewpoint and articulation changes.
Potential Downstream Application: Hand-to-Robot Retargeting
Why Retargeting Requires Metric Hand Pose
Hand-to-robot retargeting maps human hand motion to robot end-effector trajectories for teleoperation and dexterous manipulation. Such a pipeline can benefit from more than relative articulation: it also needs a calibrated estimate of where the hand lies in camera space and its metric extent.
- Absolute 3D placement. Root-relative HPE only describes joint angles around a local origin; it cannot specify the hand’s position relative to objects or the robot base.
- Metric scale consistency. An explicit per-instance metric gauge reduces the need to absorb hand-size variation into translation or a fixed dataset scalar.
- Calibrated positioning. Known camera intrinsics and 2D/3D correspondences provide normalized translation without a separate learned translation branch.
How ScaleHP Addresses These Requirements
Metric-Space Input
Outputs metric camera-space joints that can serve as input to a separately calibrated retargeting or teleoperation pipeline.
Per-Instance Metric Scale
Predicts a supervised, data-driven scale prior for each hand rather than relying on one fixed scalar for an entire dataset.
Camera-Space Coordinates
Predictions are expressed in the calibrated camera frame and can be transformed into a robot frame when the camera-to-robot calibration is known.
This video illustrates a possible use of ScaleHP outputs; retargeting quality is not part of the paper’s quantitative evaluation.
Summary
ScaleHP places an explicit per-instance metric scale between unit-scale local geometry and global positioning. Its dedicated scale token interacts with sparse 2D/3D joint queries, while a parameter-free calibrated solver recovers normalized translation. ScaleHP achieves 35.8 mm CS-MPJPE on FreiHand and 5.0/4.6/5.9 mm PA-MPJPE on FreiHand, DexYCB, and HO3Dv3 under the stated protocols, showing that camera-space localization and local articulation can improve together.