Can I read Semantic Mechanical Search with Large Vision and Language Models on EtoBox?

Semantic Mechanical Search with Large Vision and Language Models by Sharma, Satvik; Huang, Huang; Shivakumar, Kaushik; Chen, Lawrence Yunliang; Hoque, Ryan; Ichter, Brian; Goldberg, Ken is a scholarly article available to read on EtoBox.

What is Semantic Mechanical Search with Large Vision and Language Models about?

Moving objects to find a fully-occluded target object, known as mechanical search, is a challenging problem in robotics. As objects are often organized semantically, we conjecture that semantic information about object relationships can facilitate mechanical search and reduce search time. Large pretrained vision and language models (VLMs and LLMs) have shown promise in generalizing to uncommon objects and previously unseen real-world environments. In this work, we propose a novel framework called Semantic Mechanical Search (SMS). SMS conducts scene understanding and generates a semantic occupancy distribution explicitly using LLMs. Compared to methods that rely on visual similarities offered by CLIP embeddings, SMS leverages the deep reasoning capabilities of LLMs. Unlike prior work that uses VLMs and LLMs as end-to-end planners, which may not integrate well with specialized geometric planners, SMS can serve as a plug-in semantic module for downstream manipulation or navigation policies. For mechanical search in closed-world settings such as shelves, we compare with a geometric-based planner and show that SMS improves mechanical search performance by 24% across the pharmacy, kitche

Author

Sharma, Satvik; Huang, Huang; Shivakumar, Kaushik; Chen, Lawrence Yunliang; Hoque, Ryan; Ichter, Brian; Goldberg, Ken

Published

2023

Language

EN