System Architecture
Two AgileX PiPER arms, multi-view cameras, a policy inference loop and an existing robot controller form the deployment platform. The controller, Ruckig integration and UMI hardware were provided by others.
← Back to timeline
Embodied AI internship · Latencore AI
Dual-arm Robot Learning / π0.5 Deployment
Aug 2026 – PresentIntegrated teleoperation, multi-view data capture, π0.5 inference and physical execution on two AgileX PiPER arms.
Deployment completed · comparative experiments in progress
Role: Data-workflow integration, π0.5 deployment and physical-system diagnosis.
Problem
A successful checkpoint is only one component of a real dual-arm system. Demonstrations, calibration, camera streams, inference timing, trajectory handoff and hardware behaviour must form a reliable loop.
Constraints
- Two physical arms and three camera views
- Policy inference and control operate at different rates
- Teleoperation and UMI produce different data distributions
- Deformable manipulation makes success and trajectory quality harder to define
System representation
System Architecture
From a person demonstrating a movement to two arms executing it
A recorded demonstration is not yet training-ready data. My work connects collection, data preparation and policy deployment on the shared robot platform.
- 01
Calibration
Establish the spatial relationships used by teleoperation.
- 02
Preflight + collection
Check the setup, then record human demonstrations.
- 03
Export Zarr
Export the collected data in Zarr format.
- 04
Convert to LeRobot
Prepare the dataset in the format used by the learning workflow.
- 05
Compute norm_stats
Prepare the dataset statistics used for normalisation.
- 06
Train π0.5
Use the prepared demonstrations in policy training.
- 07
Deploy
Run a trained checkpoint on the physical platform.
- 08
Validate
Observe task execution and diagnose failures.
Once deployed: two runtime responsibilities
30 Hz
Policy inference
The policy produces action outputs from observations.
200 Hz
Trajectory generation / control
The trajectory layer turns those outputs into smoother robot commands.
The low-level controller, Ruckig integration and UMI hardware were provided by other team members; my role is described below.
Technical walkthrough
Data / Training / Deployment Path
The teleoperation workflow proceeds through calibration, preflight and collection, Zarr export, LeRobot conversion, norm_stats computation, π0.5 training, deployment and validation. Rollout diagnosis informs further collection and evaluation.
Runtime Execution
Policy inference runs at 30 Hz, while the 200 Hz trajectory layer converts policy outputs into smoother robot commands.
Core Ideas / Concept Primer
The ideas behind the implementation
A checkpoint is one component
Deployment connects observations, predicted actions and physical execution. Camera, calibration, communication or handoff failures can break the system even when a model returns a plausible action.
Data collection changes the inputs
Teleoperation and UMI provide different data distributions. Data-mixture and continued-training comparisons are in progress.
Different clocks, explicit boundaries
Policy inference and trajectory generation run at different rates. Diagnosis follows model output, controller input and observed motion.
Implementation details
Demonstrate
PIKA teleoperation or UMI capture
- Input
- Human task execution
- Output
- Multi-view observations and end-effector trajectories
- Failure mode
- Operator difficulty, singular poses and tracking-frame drift
Prepare data
Zarr export, LeRobot conversion and norm_stats computation before training
- Input
- Raw camera and trajectory streams
- Output
- Training episodes
- Failure mode
- Misalignment or inconsistent frame conventions contaminate supervision
Train / continue
π0.5 fine-tuning
- Input
- Teleop and UMI episodes
- Output
- Policy checkpoints
- Failure mode
- More steps can overfit; new batches can cause forgetting
Infer at 30 Hz
VLA action prediction
- Input
- Multi-view observations and robot state
- Output
- Target end-effector actions
- Failure mode
- Latency or distribution shift produces unstable actions
Execute at 200 Hz
Ruckig trajectory generation through the existing controller
- Input
- Predicted targets
- Output
- Smooth joint-space execution
- Failure mode
- Unsafe or unreachable targets expose calibration and handoff errors
Evaluate and loop
Success, trajectory and failure analysis
- Input
- Physical rollouts
- Output
- Next data or training decision
- Failure mode
- A vague success definition hides where the system improved
Results and Demonstrations
physical robot platform
2 × PiPERdual-arm AgileX deployment
inference / trajectory rates
30 Hz / 200 Hzpolicy loop and Ruckig generation operate at different frequencies
PIKA capture
3 viewstwo wrist cameras plus one external panoramic view
Completed: physical deployment
Rigid-object pick-and-place and towel-manipulation rollouts have been executed on the dual-arm platform.
Completed: multi-view teleoperation
PIKA capture combines both wrist cameras with an external Insta360 view.
Ongoing: data and training comparison
The evaluation design compares training duration and teleop/UMI mixture under a common physical-task protocol.
Failure modes
Joint singularity
A j5 = 0 configuration was reproducible offline and disrupted teleoperation.
Trace the failure through recorded trajectories and constrain data collection away from the singular region.
Tracking-frame inconsistency
VIVE/libsurvive origin depends on calibration and accepted scene state; restart or base-station movement can change the frame.
Visualise axes, preserve calibration conditions and re-calibrate after moving base stations.
System-level handoff
A plausible model output can still fail at communication, coordinate conversion or trajectory execution.
Log the boundary between inference, controller input and physical motion instead of treating every failure as a model failure.
Ownership
Integrated and operated
Operated PIKA and colleague-provided UMI collection workflows; prepared LeRobot-compatible data and deployed π0.5 checkpoints.
Diagnosed and evaluated
Debugged camera, calibration, communication, inference and action-handoff failures.
Team platform and provided hardware
Internship on a shared platform. Low-level controller, Ruckig integration and UMI hardware were developed by other team members.
Engineering details
- π0.5 vision-language-action policy
- PIKA and 3D-printed UMI data capture with VIVE tracking
- LeRobot-compatible episode preparation
- 30 Hz policy loop with 200 Hz Ruckig trajectory generation
- Physical smoke tests and remote workstation operation