Hyeonjun Cho

Research · 2025 · Evaluated on 15 scenarios

Grounded Language-to-Motion Planning with Memory and Gripper Constraints

Extension of LMTG

A tabletop manipulation planner that combines visual grounding, execution memory, and gripper-aware grasp selection for real-robot execution.

Problem

Language-model planners for tabletop manipulation often fail for reasons unrelated to reasoning: the object named in the instruction is not grounded in the image, the chosen grasp does not fit the gripper, or the same mistake is repeated after a failed attempt. The question was how much of a fixed scenario set a planner could complete once grounding, gripper constraints, and execution memory were made explicit.

Key result

12/15

Improved system. Successes in the defined 15-scenario evaluation.

Contribution

I connected visual grounding and execution memory to the planner, then evaluated the system on a fixed set of 15 scenarios. Success increased from 9/15 to 12/15.

My Role

  • Built the scene-grounding interface between visual observations and language-conditioned planning.
  • Added memory and gripper-aware grasp selection to reduce repeated execution failures.
  • Designed and ran a 15-scenario evaluation comparing the initial and improved systems.

Team Contribution

  • Research framing and experiment review were developed with laboratory mentors.
  • The work reused shared simulation and manipulation infrastructure.

Method

I integrated GPT-based visual grounding with Lang-SAM so that instructions resolve to segmented objects in the scene, added task memory and grasp-mode selection, and wired failure replanning for both PyBullet and RB10 execution. Task-order verification and explicit prompt constraints were enforced so the planner could not skip or reorder required steps.

Diagram of an RGB scene being grounded into objects, a gripper-aware grasp being selected, and execution feedback updating task memory.
Figure 1. The planner uses grounded objects and gripper constraints, then stores execution feedback for the next attempt.
  1. Visual scene grounding
  2. Grasp candidate selection
  3. Execution-memory update
  4. Language-conditioned trajectory generation

Evaluation results

ConditionResultNote
Improved system12/15Successes in the defined 15-scenario evaluation.
Initial system9/15Baseline before memory and gripper-aware improvements.
Additional successes+3Improvement within this evaluation protocol.

On the fixed 15-scenario evaluation set, success rose from 9/15 with the initial system to 12/15 with grounding, memory, and gripper-aware selection in place. The gain is specific to those scenarios; cluttered or previously unseen real-world scenes were not part of the evaluation.

Experiment evidence

Video 1. Real-robot trial for the prompt “Pick up the yellow bottle.”
Video 2. A second end-effector condition tests whether the planner can ground the red box and execute a suction grasp.

Limitations

  • The reported improvement is limited to the defined 15 scenarios.
  • The 15-scenario evaluation did not include cluttered or previously unseen real-world scenes.

Observed failure

Video 3. Zero-shot trial in which the robot approaches the target but fails to complete the grasp.

Materials

Stack. Python, Vision-Language Models, LLMs, PyBullet, Manipulation Planning.

Code and materials. VLM, memory, and gripper research fork