Three-dimensional affordance segmentation for object point cloud driven by language instructions

Abstract

The location where a robot grasps an object is closely related to the task type. For the same object, different user requirements may necessitate different grasping strategies. Visual affordance serves as a reliable source of prior knowledge for manipulation. Existing methods learn affordance from images or videos, but planar affordance lacks the spatial information required for 6-degree-of-freedom (6-DoF) manipulation. Furthermore, current approaches are limited to affordances associated with predefined categories and cannot directly infer affordances from user instructions. To address such limitations, we propose a novel task: instruction-driven three-dimensional (3D) object affordance segmentation. To support this research, we introduce an instruction–affordance dataset (IAD), a challenging dataset consisting of 7,190 object instances across 20 common object categories, paired with 624 manipulation instructions that specify the corresponding affordances. To evaluate generalization to novel commands, our dataset includes both seen and unseen settings. Building on this, we design an instruction-driven 3D affordance segmentation (IDAS) network, which extracts point cloud features and integrates instruction features layer by layer. Given a user instruction, our method segments suggested manipulation regions on the object’s point cloud, thereby guiding the selection of optimal grasp poses. Experimental results show that our method outperforms other related approaches under both seen and unseen settings, demonstrating generalization ability to diverse user commands and unknown affordances.

1. Introduction

Service robots require intelligence to accomplish tasks assigned by users (Wang et al., 2022). After actively approaching the target object (Liu SP et al., 2022), the robot needs to analyze how to manipulate it. Manipulation strategies for service robots must align with human expectations regarding how they interact with objects. For instance, when picking up a cup, humans generally prefer that the robot grasp the body of the cup instead of the rim, which is the part that comes into contact with their mouth. This preference reflects a more acceptable and hygienic approach to service. Additionally, the appropriate grasping location on an object often varies depending on user requirements. For example, when the task is to open a backpack, the robot needs to operate the zipper. However, if the task is to lift a backpack, it should instead grasp the handle.

Affordance acts as an informative prior that facilitates decision-making in robot service. Affordance refers to what the environment offers to an animal, including the opportunities it provides for interaction (Gibson, 1978). In the context of robot manipulation, affordances describe how objects present action possibilities for human interaction. These affordances are crucial for understanding how to manipulate objects effectively in robotics, and have been the subject of extensive study. Early research primarily focused on learning affordances from static images (Song et al., 2015; Roy and Todorovic, 2016; Do et al., 2018; Ardón et al., 2019) or from videos of human–object interaction (Fang K et al., 2018; Nagarajan et al., 2019; Goyal et al., 2022). More recent studies (Deng et al., 2021; Nguyen et al., 2023; Yang YH et al., 2023; Li YC et al., 2024) have advanced the concept by extending affordance segmentation into the three-dimensional (3D) domain using point cloud data. However, these affordance learning methods do not directly provide knowledge for manipulation; rather, they offer a broader conceptual understanding of the environment, which is difficult to apply in practice.

With the prompt of affordance, grasping approaches can provide more task-oriented and spatially reasonable grasp poses. Traditional grasp-planning approaches (Mousavian et al., 2019; Sundermeyer et al., 2021; Fang HS et al., 2023) primarily focus on physical feasibility, aiming to maximize the success rates of grasps without taking into account the humanlikeness or task relevance of the grip. As a result, these general-purpose methods are usually limited to simple manipulation tasks (Qin et al., 2023), such as sequentially picking up and placing objects on a tabletop. Therefore, learning grasp-relevant affordances is essential for handling diverse tasks.