Abstract

The current paradigm for motion planning generates solutions from scratch for every new problem, which consumes significant amounts of time and computational resources. For complex, cluttered scenes, motion planning approaches can often take minutes to produce a solution, while humans are able to accurately and safely reach any goal in seconds by leveraging their prior experience. We seek to do the same by applying data-driven learning at scale to the problem of motion planning. Our approach builds a large number of complex scenes in simulation, collects expert data from a motion planner, then distills it into a reactive generalist policy. We then combine this with lightweight optimization to obtain a safe path for real world deployment. We perform a thorough evaluation of our method on 64 real-world motion planning tasks across four diverse environments with randomized poses, scenes and obstacles, in the real world, demonstrating an improvement of 23%, 17% and 79% motion planning success rate over state of the art sampling, optimization and learning based planning methods.

Neural Motion Planner

We present Neural Motion Planner, which consists of 3 main components. Left: Large Scale data generation in simulation using expert planners Middle: Training deep network models to perform fast reactive motion planning Right: Test-time optimization to ensure safe and reliable deployment

All videos play at 2x speed

Neural MP solves motion planning tasks with complex unseen object configurations.

Bins: 100%

Shelf: 100%

Articulated: 87.5%

In Hand: 81.25%

Neural MP generalizes to out of distribution obstacles and in-hand objects

Minecraft Sword

Shelf to Bookcase

Bookcase from Side

GPU into Cabinet Drawer

Long Pasta in Microwave

Bookcase from Behind

Neural MP extends directly to unstructured, in-the-wild scenes

Tidying Cabinet

Making Coffee

Drying Plates

Rearranging Ketchup

Cleaning Dishes

Defrosting Meat

Neural MP can motion plan in a dynamic, reactive manner

Sword Dodging

Multi-stage Obstacle Avoidance

Neural MP is trained using 6 classes of procedurally generated objects with significant size, shape and pose variation.

Cubby

Dishwasher

Microwaves

Boxes

Shelves

Cabinets

Neural MP Data Generation Process

Full procedural environment generation pipeline which randomly samples programmatic obstacles and objects to create diverse scenes.

Expert data generation involves sampling challenging motion planning configurations such as those inside obstacles.

Test-time Optimization Analysis

Low quality path, significant collisions. Objective Value (num scene points in collision): 1276

Medium-low quality path, some collisions. Objective Value (num scene points in collision): 233

Medium quality path, few minor collisions. Objective Value (num scene points in collision): 159

High quality path, zero collisions. Objective Value (num scene points in collision): 0

We visualize the range of paths induced by our model. As you can see, in general there is a range of trajectories produced by our model, with the best trajectory being collision free and the worst trajectory having multiple collisions. Test-time optimization is used to ensure we only deploy the best trajectories on the robot by optimizing for trajectories that have the fewest number of scene points in collision with the robot.

Failure Cases

Sometimes the pointcloud of the object is can have noise and holes, especially along edges. As a result, in this the top rung of the shelf is not incorporated in the pointcloud, resulting in the policy colliding with the top rung.

Tight spaces can also cause issues for our policy. Starting the end-effector at 50% into the microwave is not an issue. However, once we start the end-effector fully in the microwave, the policy fails to remove the hand without colliding with the microwave and dragging it along.

Neural MP solves motion planning tasks with significant variation across poses, objects, obstacles, backgrounds, scene arrangements, in-hand objects, and start/goal pairs.

BibTeX

@article{dalal2024neuralmp,
title={Neural MP: A Generalist Neural Motion Planner},
author={Murtaza Dalal and Jiahui Yang and Russell Mendonca and Youssef Khaky and Ruslan Salakhutdinov and Deepak Pathak},
journal = {arXiv preprint arXiv:2409.05864},
year={2024},
}