Research · 2024–2025 · Prototype · early robot trials

Zero-Shot Language-to-Motion Planning with Geometric Validation

Reproduction and extension of LMTG

A language-to-motion prototype that converts instructions into waypoint trajectories and checks them for collisions before execution.

Problem

LMTG generates manipulation trajectories from natural-language instructions without task-specific training, but a language model has no geometric model of the workspace, so generated waypoints can pass through objects or approach with an unusable end-effector orientation. This project asked whether a reproduction of LMTG could be made safe enough for real-robot trials by validating its output geometrically before execution.

Key result

Zero-shot

Planning setting. No task-specific model fine-tuning in the prototype.

Contribution

I implemented the task representation, waypoint parser, and collision checks, then connected the planner to simulation and initial real-robot trials.

My Role

Designed structured task and scene representations for language-model planning.
Implemented waypoint parsing, validation, and collision-aware trajectory generation.
Evaluated the planning loop in simulation and explored initial perception and real-robot execution integration.

Team Contribution

Laboratory mentors guided the research scope and reviewed prototype behavior.
Shared simulation assets and robot models supported evaluation.

Method

I reproduced LMTG on Franka and PyBullet, then extended it with object-aware waypoint and end-effector orientation refinement and height-based collision avoidance. Instructions are parsed into a structured task and scene representation, the language model proposes waypoints, and every trajectory is checked against scene geometry before execution. Initial sim-to-real perception integration connected the planner to early real-robot trials.

Diagram showing a natural-language instruction transformed into waypoint candidates, collision checks, and a simulated robot trajectory. — **Figure 1.** The prototype separates language reasoning from geometric validation before execution in simulation.

Natural-language task parsing
Structured waypoint generation
Collision-aware trajectory validation
Simulation and early real-robot integration

Evaluation results

Condition	Result	Note
Planning setting	Zero-shot	No task-specific model fine-tuning in the prototype.
Planning pipeline	3 stages	Instruction parsing, waypoint generation, and collision checking.

The planning loop was evaluated in simulation, and the real-robot clips on this page are prototype demonstrations rather than a controlled benchmark, so no quantitative success rate is claimed. The contribution is the validated language-to-motion pipeline and the perception integration that the VLM-grounded LMTG project extended.

Experiment evidence

**Video 1.** Early real-robot trial for the instruction “Pick up the box.”

**Video 2.** Simulation trace of the language-generated approach, gripper reorientation, and waypoint execution used before the early real-robot trials.

**Video 3.** Early placement trial for “Put the red box in the brown box,” connecting language-generated waypoints to physical execution.

Limitations

Real-robot clips are prototype demonstrations rather than a controlled quantitative benchmark.
The project has not been evaluated in a controlled real-robot benchmark.

Observed failure

**Video 4.** An inaccurate object bounding box produces an invalid approach near the brown box.

Materials

Stack. Python, LLMs, PyBullet, Motion Planning, Prompt Engineering.

Code and materials. Trajectory-generation research fork