Our Mission

Science discoveries take time and effort. We introduce novel robotic & AI-driven solutions to accelerate these discoveries.

Publications and Work

Natural Language Processing

Program Synthesis

Robotics

Errors are Useful Prompts: Instruction Guided Task Programming with Verifier-Assisted Iterative Prompting

Generating low-level robot task plans from high-level natural language instructions remains a challenging problem. Although large language models have shown promising results in generating plans, the accuracy of the output remains unverified. Furthermore, the lack of domain-specific language data poses a limitation on the applicability of these models. In this paper, we propose CLAIRIFY, a novel approach that combines automatic iterative prompting with program verification to ensure programs written in data-scarce domain-specific language are syntactically valid and incorporate environment constraints. Our approach provides effective guidance to the language model on generating structured-like task plans by incorporating any errors as feedback, while the verifier ensures the syntactic accuracy of the generated plans. We demonstrate the effectiveness of CLAIRIFY in planning chemistry experiments by achieving state-of-the-art results. We also show that the generated plans can be executed on a real robot by integrating them with a task and motion planner.

Under Review

Tools

Robotics

Hardware

3D Printing

Digital pipette: Open hardware for liquid transfer in self-driving laboratories

Self-driving laboratories promise to democratize automated chemical laboratories. Accurate liquid handling is an essential operation in the context of chemical labs, and consequently a self-driving laboratory will require a robotic liquid handling and transfer. Although many pipettes are available for human scientists, robots cannot manipulate these pipettes due to the limitations of current robot gripper morphology. We propose an intuitive yet elegant design for a 3D-printed digital pipette designed for robots to carry out chemical experiments. Performance-evaluation experiments were carried out liquid transfer tasks. Our results show that robots with digital pipette could transfer liquids within 0.5% error. This error is comparable to the baseline set by commercially available human-handled pipettes.

Under Review

Computer Vision

Robotics

Machine Learning

Data Set

MVTrans: Multi-View Transparent Object Perception

Transparent object perception is a crucial skill for applications such as robot manipulation in household and laboratory settings. Existing methods utilize RGB-D or stereo inputs to handle a subset of perception tasks including depth and pose estimation. However transparent object perception remains to be an open problem. In this paper, we forgo the unreliable depth map from RGB-D sensors and extend the stereo based method. Our proposed method, MVTrans, is an end-to-end multi-view architecture with multiple perception capabilities, including depth estimation, segmentation, and pose estimation. Additionally, we establish a novel procedural photo-realistic dataset generation pipeline and create a large-scale transparent object detection dataset, Syn-TODD, which is suitable for training networks with all three modalities, RGB-D, stereo and multi-view RGB.

ICRA 2023

Teams

Principal Investgators

Alán Aspuru-Guzik

Florian Shkurti

Animesh Garg

Current Members

Kourosh Darvish
Post Doc - Team Lead

Miroslav Bogdanovic
Post Doc

Haoping Xu
PhD

Naruki Yoshikawa
PhD

Marta Skreta
PhD