ManiSoft: Towards Vision-Language Manipulation for Soft Continuum Robotics

Most existing vision-language manipulation research targets rigid robotic arms, whose fixed morphology limits adaptability in cluttered or confined spaces. Soft robotic arms offer an appealing alternative due to their deformability, but confront challenges such as unreliable proprioception and distributed low-level actuation. To investigate these challenges, we introduce ManiSoft, a benchmark for vision-language manipulation with soft arms. ManiSoft features a tailored simulator that couples realistic soft-body dynamics with contact-rich interactions via an elastic force constraint. On this basis, ManiSoft defines four tasks, each highlighting distinct aspects of deformable control, from basic end-effector coordination to obstacle avoidance. To support policy training and evaluation, ManiSoft includes an automated pipeline that generates $6{,}300$ diverse scenes and corresponding expert trajectories. To produce high-quality trajectories at scale, we first employ a high-level planner to decompose each task into a sequence of waypoints, followed by a low-level reinforcement learning policy that generates torque commands to track waypoints. Benchmarking three representative policy models shows relatively promising results in clean scenes but substantial performance drop under randomization. Visualization analysis indicates that failures stem primarily from inaccurate visual estimation of proprioceptive state and limited exploitation of deformability for adaptive obstacle avoiding. We anticipate ManiSoft to serve as a valuable testbed, bridging the gap between rigid and soft arms in the context of vision-language manipulation.

Most robots used in today’s vision-language systems are built from rigid mechanical arms, which can struggle in crowded or narrow environments. Soft robotic arms, made from flexible materials, could adapt more naturally to these situations, but they are also much harder to control because they bend and deform continuously. To study these challenges, we introduce ManiSoft, a new benchmark for training and evaluating soft robotic arms on language-guided manipulation tasks. ManiSoft includes a realistic simulation environment where soft arms can interact with objects and obstacles in physically accurate ways. We design four representative tasks that test different abilities, such as precise reaching, coordinated movement, and navigating around obstacles. We also build an automated system to generate thousands of training scenes and expert demonstrations, enabling large-scale learning and evaluation. Our experiments with several representative robot learning methods show that current approaches can perform reasonably well in simple settings, but their performance drops significantly in more complex or randomized environments. Further analysis suggests that existing models struggle to accurately infer the shape and state of soft arms from visual input and do not fully take advantage of the flexibility of soft robots during interaction. We hope ManiSoft can serve as a useful platform for advancing research on flexible, adaptable robotic systems that can better operate in real-world environments.