Appearance, depth and geometry
RGB provides visual observations, stereo depth supplies observed metric geometry, and CAD provides the instrument model. Masks restrict the observed object support; errors can propagate into candidate ranking.
← Back to timeline
Individual research · Imperial College London
Zero-Shot 6-DoF Tool Pose Estimation
Large Needle Driver · RGB–Stereo–CAD Pipeline
Mar 2026 – Sep 2026Symmetry-aware geometric reranking raised top-1 ADD@5 mm from 44.11% to 57.75% across 968 valid pose outputs.
Completed MRes research
Role: Individual MRes research on geometric pose reranking and evaluation.
Problem
A foundation pose estimator can generate a valid candidate while ranking a geometrically inconsistent pose first. The object is symmetric, thin, partially occluded and evaluated at a strict 5 mm ADD threshold.
Constraints
- Zero-shot pipeline with no task-specific pose-network training
- Stereo depth and segmentation errors propagate downstream
- Symmetry makes visually plausible poses ambiguous
- CAD units and coordinate conventions must remain consistent
System representation
Inference and Evaluation Pipeline
The camera sees the tool. But where is it, and which way is it pointing?
A pose is an object's position and orientation. This project estimates both for the needle driver. My focus is the final choice: a useful pose can already be in the list, yet the estimator selects another one.
One recorded example: frame 125, sym252. Other frames can worsen; the full comparison remains below.
Three kinds of information meet at the pose estimator
WHERE TO LOOK
RGB → target mask
Mask R-CNN marks the pixels belonging to the target region. The pose pipeline uses these existing predictions.
HOW FAR AWAY
Stereo images → depth
The pipeline uses precomputed RAFT-Stereo depth supplied with the dataset.
WHAT SHAPE
CAD + camera calibration
CAD supplies the shape. Camera calibration relates 3D points to image pixels.
Any6D proposes and ranks possible placements
RGB, mask, depth, CAD and camera parameters enter together. First in the final ranking is not necessarily closest to the reference.
Choose an existing pose using geometric conditions
Check candidates in their original order and select the first that meets the conditions. If none qualifies, keep the original first choice.
REFERENCE USED AFTER SELECTION
Measure the 3D mismatch
ADD compares corresponding CAD points under the predicted and reference poses. A mean distance below 5 mm passes this project's threshold.
RGB + stereo + CAD · Inputs and observations
RGB, stereo frames and CAD feed segmentation and metric stereo depth. The pipeline converts CAD from millimetres to metres and centres it at its centre of mass.
Inputs: stereo RGB observations and the Large Needle Driver CAD model.
Segmentation · Mask R-CNN
Mask R-CNN (ResNet-101 + FPN) supplies the object mask used by the pose pipeline. GroundingDINO + SAM2 and its local-window variant are comparison methods.
Stereo depth · RAFT-Stereo
The pose pipeline reads RAFT-Stereo depth. The diagnostic figure compares FoundationStereo against that depth reference; the difference is between two estimates, not error against ground-truth depth.
Pose candidates · Any6D
Any6D generates and scores pose hypotheses from RGB, the object mask, depth and CAD.
Geometric reranking · z2 / z1 / y1
Geometric constraints reorder the Any6D candidates. The overlays show the original top-1 and the selected pose against GT, including rescued, harmed and unchanged cases.
Core Ideas / Concept Primer
The ideas behind the implementation
Initial rotation configurations
sym252 and sym126 describe Any6D initial rotation configurations. The reported evaluation uses ordinary ADD, which compares corresponding CAD points rather than ignoring axial differences.
Geometric reranking: z2 / z1 / y1
The ablation compares single and combined geometric constraints, with a higher top-1 score for the triple constraint.
Top-1, oracle and ADD@5 mm
Top-1 evaluates the selected hypothesis; the oracle ceiling asks whether a correct candidate existed at all. ADD@5 mm applies a strict 5 mm pose-evaluation threshold.
Technical Decisions & Trade-offs
Segmentation
Experimentally validated- Selected method
- Mask R-CNN
- Alternatives
- GroundingDINO + SAM2; local-window variant
- Trade-off / rationale
- Higher reported IoU across the same 970 frames.
- Comparison
- IoU ≈ 0.889 / 0.232 / 0.496
Candidate ranking
Experimentally validated- Selected method
- Triple geometric reranking
- Alternatives
- Raw Any6D top-1; single and paired constraints
- Trade-off / rationale
- Higher recorded top-1 ADD@5 mm under sym252; harmful reranking cases remain visible.
- Comparison
- 44.11% → 57.75% · 968 valid outputs
Initial rotation configuration
Engineering decision- Selected method
- sym252 with a validated 126-symmetry subset
- Trade-off / rationale
- Changes the initial rotation grid used by Any6D; ordinary ADD remains the evaluation metric.
Evaluation · ADD@5 mm
Evaluation uses top-1 ADD@5 mm under sym252, reports 968 valid outputs from 970 frames, and compares selected poses with an oracle candidate ceiling.
top-1 ADD@5 mm
57.75% triple geometric rerankingsym252; default top-1 versus triple geometric reranking
Evaluation details
- Sample size
- 968 valid pose outputs / 970 frames
- Protocol
- sym252 · ADD@5 mm
- Baseline
- 44.11% default top-1
valid Any6D outputs
968 / 970frames 1261 and 1262 failed
oracle candidate ceiling
86.36%sym252; a valid candidate existed but was not always ranked first
runtime per frame
8.92 s8,652 seconds total; peak GPU memory 4.25 GiB
Failure modes
Upstream mask failure
Empty or inaccurate segmentation changes the observed object support.
Report empty outputs explicitly and separate upstream failure from pose-ranking failure.
Stereo outlier
Thin reflective geometry and occlusion destabilise correspondence.
Use mask-restricted diagnostics and retain representative best, typical and worst cases.
Harmful reranking
A noisy geometric cue can demote a correct top-1 hypothesis.
Keep rescued, harmed and unchanged groups visible instead of reporting only aggregate gain.
Ownership
Designed
Defined the modular RGB–stereo–CAD research pipeline.
Implemented
Implemented the pipeline and geometric candidate reranking, retaining intermediate outputs for diagnosis.
Evaluated
Compared segmentation, stereo depth, symmetry handling and reranking; analysed quantitative results and failure cases.
Research context and support
Individual MRes work at the Hamlyn Centre, supervised by Dr Stamatia (Matina) Giannarou. Project 332 provided the dataset and research setting.
Engineering details
- PyTorch 2.8.0 + CUDA 12.8 on an RTX 5060
- CAD converted from millimetres to metres and centred at its centre of mass
- 252-symmetry evaluation set with a validated 126-symmetry subset
- Modular outputs retained for segmentation, depth, candidates and diagnostics