DeepNLP ICRA2022 Accepted Paper List AI Robotic and STEM Top Conference & Journal Papers
-
Multi-LiDAR system is an important part of V2X (Vehicle to Everything) to enhance the perception information for unmanned vehicles. To fuse the information from multiple 3D LiDARs, accurate extrinsic calibration between the LiDARs is essential. However, the existing multi-LiDAR calibration methods mainly focus on short baseline scenarios, where multiple LiDARs are closely mounted on a single platf...
-
The ability to detect objects, under image corruptions and different weather conditions is vital for deep learning models especially when applied to real-world applications such as autonomous driving. Traditional RGB-based detection fails under these conditions and it is thus important to design a sensor suite that is redundant to failures of the primary frame-based detection. Event-based cameras ...
-
Forecasting pedestrians' future motions is essential for autonomous driving systems to safely navigate in urban areas. However, existing prediction algorithms often overly rely on past observed trajectories and tend to fail around abrupt dynamic changes, such as when pedestrians suddenly start or stop walking. We suggest that predicting these highly non-linear transitions should form a core compon...
-
In the foreseeable future, connected and auto-mated vehicles (CAVs) and human-driven vehicles will share the road networks together. In such a mixed traffic environment, CAVs need to understand and predict maneuvers of surrounding vehicles for safer and more efficient interactions, especially when human drivers bring in a wide range of uncertainties. In this paper, we propose a learning-based lane...
-
Shape control involves bringing a deformable object to a desired shape. In the shape control literature, the positioning of the grippers on the object is usually predefined (user-defined) and therefore considered as input information. In this paper we address the gripper positioning problem for shape control. We propose a deformation process within a simulated fully-actuated scenario and introduce...
-
Contact adaptation is an essential capability when manipulating objects. Two key contact modes of non-prehensile manipulation are sticking and sliding. This paper presents a Trajectory Optimization (TO) method formulated as a Mathematical Program with Complementarity Constraints (MPCC), which is able to switch between these two modes. We show that this formulation can be applicable to both plannin...
-
As collaborative robots move closer to human environments, motion generation and reactive planning strategies that allow for elaborate task execution with minimal easy-to-implement guidance whilst coping with changes in the environment is of paramount importance. In this paper, we present a novel approach for generating real-time motion plans for point-to-point tasks using a single successful huma...
-
PyROBOCOP is a Python-based package for control, optimization and estimation of robotic systems described by nonlinear Differential Algebraic Equations (DAEs). In particular, the package can handle systems with contacts that are described by complementarity constraints and provides a general framework for specifying obstacle avoidance constraints. The package performs direct transcription of the D...
-
Generalizable manipulation requires that robots be able to interact with novel objects and environment. This requirement makes manipulation extremely challenging as a robot has to reason about complex frictional interaction with uncertainty in physical properties of the object. In this paper, we study robust optimization for control of pivoting manipulation in the presence of uncertainties. We pre...
-
This paper presents a unified controller for dual-arm robot dynamic multi-object nonprehensile transportation. The controller is composed of time-optimal path parameteri-zation (TOPP) and model predictive control (MPC) and aimed at efficiently and dynamically transporting objects using the dual-arm robot under physical constraints while avoiding the slippage of the objects. A force tracking contro...
-
We consider the problem of retrieving a target object from a confined space by two robotic manipulators where overhand grasps are not allowed. If other movable obstacles occlude the target, more than one object should be relocated to clear the path to reach the target object. With two robots, the relocation could be done efficiently by simultaneously performing relocation tasks. However, the prece...
-
In this paper, we propose a new framework for solving cable-routing problems with a dual-arm robot, where the objective is to clip a Deformable Linear Object (DLO) into several arbitrarily placed fixtures. The core of the framework is a task-space planner, which builds a roadmap from predefined tasks and employs a replanning strategy based on a genetic algorithm, if problems occur. The manipulatio...
-
Underwater thermocline, common in the lakes and ocean, plays a vital role in meteorological forecasting in the ocean and lakes dynamics research. This letter proposes a method for rapid and multipoint observation of thermocline variations with time and space using an airdropped micro-profiler array, named the DRAGONFLY system. It comprises specially designed disposable low-cost micro-profilers, a ...
-
Direct communication between humans and autonomous underwater vehicles (AUVs) is a relatively under-explored area in human-robot interaction research, although many tasks (e.g., surveillance, inspection, and search-and-rescue) require close diver-robot collaboration. Suboptimal AUV positioning relative to its human collaborators can lead to poor quality interaction and lead to excessive cognitive ...
-
Autonomous latching is essential for autonomous surface vessels (ASV) to reach full independence from human intervention. As part of the ASV Roboat project, a new solution for self-latching maneuvers has been developed and is presented here. We propose a system that has the key requirements of full integration with the navigation control system and zero-gap connection with the dock, the latter bei...
-
Navigating autonomous underwater vehicles (AUVs) in shallow and harbor waters is challenging and typically has higher accuracy requirements than navigation in the open sea. We investigate enhancements to underwater localization techniques based on Two-Way Ranging (TWR) using acoustic modems, which have great potential to meet localization accuracy requirements at lower cost and complexity than cur...
-
In this paper we present a complete framework for Underwater SLAM utilizing a single inexpensive sensor. Over the recent years, imaging technology of action cameras is producing stunning results even under the challenging conditions of the underwater domain. The GoPro 9 camera provides high definition video in synchronization with an Inertial Measurement Unit (IMU) data stream encoded in a single ...
-
Anthropomorphic robotic grippers are required for robots, prostheses, and orthosis to enable manipulation of a priori unknown and variable-shape objects. It has to meet a wide range of sometimes contradictory requirements in terms of adaptivity, dexterity, high payload to weight ratio, robustness, aesthetics, compactness, lightweight, etc. Within this paper, we utilize the morphological computatio...
-
Design and Optimization of a Magnetic Catcher for UAV Landing on Disturbed Aquatic Surface Platforms
In this paper, a new capture system for UAV precision landing in a disturbed environment is proposed. Compared with the traditional visual guided landing methods, perching mechanism based methods, and tethered landing methods, the proposed system takes into account the stability during landing process and retains the high accessibility of the UAV. The proposed system consists of a winch subsystem ...
-
Rolling spherical robots have been studied in the past few years as an alternative to legged and wheeled robots in unstructured environments. These systems are of uttermost interest for space exploration: fast, robust to collision and able to handle various terrain topologies. This paper introduces a novel barycentric spherical robot, dubbed the Autonomous Robotic Intelligent Explorer Sphere (ARIE...
-
Peripheral vascular intervention remains a challenging procedure mainly due to the tortuosity of the vessels needing to be traversed by guidewires and catheters. In addition, handling long guidewires while navigating tortuous vasculature requires extensive time and skill from the surgeon. In this work, a compact guidewire advancement mechanism is proposed that is able to dispense guidewires up to ...
-
Perching onto objects can allow flying robots to stay at a desired height at low or no cost of energy. This paper presents a novel passive mechanism for aerial perching onto smooth surfaces. This mechanism is made from a bistable mechanism and a soft suction cup. Different from existing designs, it can be easily attached onto and detached from a surface, but it can also hold a large weight when at...
-
To facilitate sensing and physical interaction in remote and/or constrained environments, high-extension, lightweight robot manipulators are easier to transport and reach substantially further than traditional serial chain manipulators. We propose a novel planar 3-degree-of-freedom manipulator that achieves low weight and high extension through the use of a pair of spooling bistable tapes, commonl...
-
Siamese network based trackers have achieved significant progress in visual object tracking. For the sake of speed, they mainly rely on offline training to learn a mono-level feature correlation between a target template and a search region. During the tracking period, they use a fixed strategy to infer target positions over sequences regardless of target states. However, such approaches are vulne...
-
Multi-perspective cameras are quickly gaining importance in many applications such as smart vehicles and virtual or augmented reality. However, a large system size or absence of overlap in neighbouring fields-of-view often complicate their calibration. We present a novel solution which relies on the availability of an external motion capture system. Our core contribution consists of an extension t...
-
Dense object tracking, the ability to localize specific object points with pixel-level accuracy, is an important computer vision task with numerous downstream applications in robotics. Existing approaches either compute dense keypoint embeddings in a single forward pass, meaning the model is trained to track everything at once, or allocate their full capacity to a sparse predefined set of points, ...
-
We propose a single-stage, category-level 6-DoF pose estimation algorithm that simultaneously detects and tracks instances of objects within a known category. Our method takes as input the previous and current frame from a monocular RGB video, as well as predictions from the previous frame, to predict the bounding cuboid and 6- DoF pose (up to scale). Internally, a deep network predicts distributi...
-
We present a novel self-supervised algorithm named MotionHint for monocular visual odometry (VO) that takes motion constraints into account. A key aspect of our approach is to use an appropriate motion model that can help existing self-supervised monocular VO (SSM-VO) algorithms to overcome issues related to the local minima within their self-supervised loss functions. The motion model is expresse...
-
Tracking of tissue in the surgical environment is often done via locating frame-to-frame keypoint correspondences, and then using these correspondences to warp a prior underlying model such as a spline, mesh, or embedded deformation. We introduce a novel learned model which takes keypoint correspondences as input and enables a prior-free estimation of deformation at any location. For fast point tr...
-
Vision-based 3D human pose estimation and shape reconstruction play important roles in robot-assisted healthcare monitoring and personal assistance. However, 3D data captured from a single viewpoint always encounter occlusions and exhibit substantial heterogeneity across different views, resulting in significant challenges for both tasks. Extensive approaches have been proposed to perform each tas...
-
The potential diagnostic applications of magnet-actuated capsules have been greatly increased in recent years. For most of these potential applications, accurate position control of the capsule have been highly demanding. However, the friction between the robot and the environment as well as the drag force from the tether play a significant role during the motion control of the capsule. Moreover, ...
-
This paper introduces an $L$1adaptive control aug-mentation for geometric tracking control of quadrotors. In the proposed design, the $L$ 1 augmentation handles nonlinear (time-and state-dependent) uncertainties in the quadrotor dynamics without assuming or enforcing parametric structures, while the baseline geometric controller achieves stabilization of the known nonlinear model of the system dyn...
-
The shape control of deformable linear objects (DLOs) is challenging, since it is difficult to obtain the deformation models. Previous studies often approximate the models in purely offline or online ways. In this paper, we propose a scheme for the shape control of DLOs, where the unknown model is estimated with both offline and online learning. The model is formulated in a local linear format, an...
-
Nonlinear dynamical effects are crucial to the operation of many agile robotic systems. Koopman-based model learning methods can capture these nonlinear dynamical system effects in higher dimensional lifted bilinear models that are amenable to optimal control. However, standard methods that lift the system state using a fixed function dictionary before model learning result in high dimensional mod...
-
We propose a method for robotic control of deformable objects using a learned nonlinear dynamics model. After collecting a dataset of trajectories from the real system, we train a recurrent neural network (RNN) to approximate its input-output behavior with a latent state-space model. The RNN internal state is low-dimensional enough to enable realtime nonlinear control methods. We demonstrate a clo...
-
We present a new data-driven technique for pre-dicting the motion of a low-cost omnidirectional mobile robot under the influence of motor torques and friction forces. Our method utilizes a novel differentiable physics engine for analytically computing the gradient of the deviation between predicted motion trajectories and real-world trajectories. This allows to automatically learn and fine-tune th...
-
Slipping may cause a vehicle out of control with serious accident potential. However, a kind of car slipping named “drifting” can be seen in professional contests. So, it is reasonable to apply drift maneuvers in autonomous driving. This article proposes a controller for the particular driving skill—drifting based on MPC(Model Predictive Control). Firstly, we analyze drift cornering mechanisms of ...
-
Assembly tasks are challenging for robot manipulation because the robot must reason over the composed effects of actions and execute multi-objective behaviors. Robots typically use predefined priorities provided by users to determine how to compose controller behaviors, but we want the robot to autonomously select these compositions based on their composed effects within the task. We present Compo...
-
Maneuvering an autonomous vehicle under drift condition is critical to the safety of autonomous vehicles when there is a sudden loss of traction due to external conditions such as rain or snow, which is a challenging control problem due to the presence of significant sideslip and nearly full saturation of the tires. In this paper, we focus on the control of drift maneuvers of autonomous vehicle to...
-
The recent successes in deep reinforcement learning largely rely on the capabilities of generating masses of data, which in turn implies the use of a simulator. In particular, current progress in multi body dynamic simulators are under-pinning the implementation of reinforcement learning for end-to-end control of robotic systems. Yet simulators are mostly considered as black boxes while we have th...
-
The field of physical human-robot interaction has dramatically evolved in the last decades. As a result, the robotic system's requirements have become more challenging, including personalized behavior for different tasks and users. Various machine learning techniques have been proposed to give the robot such adaptability features. This paper proposes a model-based evolutionary optimization algorit...
-
Impacts and other non-smooth behaviors are usually unwanted in robotic applications. However, several industrial tasks such as deburring, removing excess material, and assembling/fitting, involve impacts between objects, which can benefit from robotic automation due to the risks posed to human health. Towards this objective, in this paper, we propose a method for optimal impact planning and pre-co...
-
We present an algorithm, based on the Differential Dynamic Programming framework, to handle trajectory optimization problems in which the horizon is determined online rather than fixed a priori. This algorithm exhibits exact one-step convergence for linear, quadratic, time-invariant problems and is fast enough for real-time nonlinear model-predictive control. We show derivations for the nonlinear ...
-
Over the past decade, the Differential Dynamic Programming (DDP) method has gained in maturity and popularity within the robotics community. Several recent contributions have led to the integration of constraints within the original DDP formulation, hence enlarging its domain of application while making it a strong and easy-to-implement competitor against alternative methods of the state of the ar...
-
Trajectory Distribution Control for Model Predictive Path Integral Control using Covariance Steering
This paper presents a novel control approach for autonomous systems operating under uncertainty. We combine Model Predictive Path Integral (MPPI) control with Covariance Steering (CS) theory to obtain a robust controller for general nonlinear systems. The proposed Covariance-Controlled Model Predictive Path Integral (CC-MPPI) controller addresses the performance degradation observed in some MPPI i...
-
In this paper, we propose a formation control system for deforming and transporting simultaneously a de-formable object with a team of robots, modeled with double-integrator dynamics. The goal is to reach a target configuration, defined as a combination of shape, scale, orientation and position of the formation. We augment this controller with a set of control barrier functions (CBFs). The CBFs al...
-
Conventional methods to autonomous grasping rely on a pre-computed database with known objects to synthesize grasps, which is not possible for novel objects. On the other hand, recently proposed deep learning-based approaches have demonstrated the ability to generalize grasp for unknown objects. However, grasp generation still remains a challenging problem, especially in cluttered environments und...
-
Robotic manipulation behavior should be robust to disturbances that violate high-level task-structure. Such robustness can be achieved by constantly monitoring the environment to observe the discrete high-level state of the task. This is possible because different phases of a task are characterized by different sensor patterns and by monitoring these patterns a robot can decide which controllers t...
-
To continuously improve robotic grasping, we are interested in developing a contactless fingertip-mounted sensor for near-distance ranging and material sensing. Previously, we demonstrated a dual-modal and dual sensing mechanisms (DMDSM) pretouch sensor prototype based on pulse-echo ultrasound and optoacoustics. However, the complex system, the bulky and expensive pulser-receiver, and the omni-dir...
-
Efficient robotic manipulation of objects for sorting and searching often rely upon how well the objects are perceived and the available grasp poses. The challenge arises when the objects are irregular, have similar visual features (e.g., textureless objects) and the scene is densely cluttered. In such cases, non-prehensile manipulation (e.g., pushing) can facilitate grasping or searching by impro...
-
This paper presents a novel iterative closest points (ICP) variant, non-penetration iterative closest points (NPICP), which prevents interpenetration in 6DOF pose optimization and/or joint optimization of multiple object poses. This capability is particularly advantageous in cluttered scenarios, where there are many interactions between objects that constrain the space of valid poses. We use a sem...
-
In our daily life, there are many objects represented by cylindrical shapes and ellipsoids. The tops of these objects are formed by elliptic shape primitives. Thus, it is available for a robot to manipulate these objects by ellipse detection. In this work, we propose a novel approach to generating ground truth for training the model based on domain randomization. Using synthetic data generated in ...
-
Robotic grasping pose detection that predicts the configuration of the robotic gripper for object grasping is fundamental in robot manipulation. Based on point clouds, most of the existing methods predict grasp pose with the hierarchical PointNet++ backbone, while the non-local geometric information is underexplored. In this work, we address the 7-DoF (6- DoF with the grasp width) grasp detection ...
-
This work proposes a robotic pipeline for picking and constrained placement of objects without geometric shape priors. Compared to recent efforts developed for similar tasks, where every object was assumed to be novel, the proposed system recognizes previously manipulated objects and per-forms online model reconstruction and reuse. Over a lifelong manipulation process, the system keeps learning fe...
-
Prior work on 6-DoF object pose estimation has largely focused on instance-level processing, in which a textured CAD model is available for each object being detected. Category-level 6- DoF pose estimation represents an important step toward developing robotic vision systems that operate in unstructured, real-world scenarios. In this work, we propose a single-stage, keypoint-based approach for cat...
-
The ability to track the 6D pose distribution of an object when a mobile manipulator robot is still approaching the object can enable the robot to pre-plan grasps that combine base and arm motion. However, tracking a 6D object pose distribution from a distance can be challenging due to the limited view of the robot camera. In this work, we present a framework that fuses observations from external ...
-
We propose a framework for robust and efficient training of Dense Object Nets (DON) [1] with a focus on industrial multi-object robot manipulation scenarios. DON is a popular approach to obtain dense, view-invariant object descriptors, which can be used for a multitude of downstream tasks in robot manipulation, such as, pose estimation, state representation for control, etc. However, the original ...
-
Open challenges for deep reinforcement learning systems are their adaptivity to changing environments and their efficiency w.r.t. computational resources and data. In the application of learning lane-change behavior for autonomous driving, the number of required transitions imposes a bottleneck, since test drivers cannot perform an arbitrary amount of lane changes in the real world. In the off-pol...
-
Legged robots are physically capable of traversing a wide range of challenging environments, but designing controllers that are sufficiently robust to handle this diversity has been a long-standing challenge in robotics. Reinforcement learning presents an appealing approach for automating the controller design process and has been able to produce remarkably robust controllers when trained in a sui...
-
For model-free deep reinforcement learning of quadruped locomotion, the initialization of robot configurations is crucial for data efficiency and robustness. This work focuses on algorithmic improvements of data efficiency and robustness simultaneously through automatic discovery of initial states, which is achieved by our proposed K-Access algorithm based on accessibility metrics. Specifically, w...
-
Recent works in Reinforcement Learning (RL) combine model-free (Mf)-RL algorithms with model-based (Mb)-RL approaches to get the best from both: asymptotic performance of Mf-RL and high sample-efficiency of Mb-RL. Inspired by these works, we propose a hierarchical framework that integrates online learning for the Mb-trajectory optimization with off-policy methods for the Mf-RL. In particular, two ...
-
Visual-inertial sensors have a wide range of applications in robotics. However, good performance often requires different sophisticated motion routines to accurately calibrate camera intrinsics and inter-sensor extrinsics. This work presents a novel formulation to learn a motion policy to be executed on a robot arm for automatic data collection for calibrating intrinsics and extrinsics jointly. Ou...
-
Multi-agent formation as well as obstacle avoid-ance is one of the most actively studied topics in the field of multi-agent systems. Although some classic controllers like model predictive control (MPC) and fuzzy control achieve a certain measure of success, most of them require precise global information which is not accessible in harsh environments. On the other hand, some reinforcement learning...
-
Most human activities in daily living or professional work rely on upper body motion. Measuring upper body motion is essential for many applications such as health evaluation, rehabilitation, human power augmentation, skill transferring, etc. Computer vision-based systems have been widely used to directly capture upper limb motion but are usually constrained in a restricted area. Wearable sensors ...
-
Modern autonomous systems are purposed for many challenging scenarios, where agents will face unexpected events and complicated tasks. The presence of disturbance noise with control command and unknown inputs can negatively impact robot performance. Previous research of joint input and state estimation separately studied the continuous and discrete cases without any prior information. This paper c...
-
Inertial motion capture has become an attractive alternative to optical motion capture for human joint angle estimation outside the laboratory. Usually inertial sensors are assumed to be tightly fixed to the body segments, which can be cumbersome regarding setup-time and ease-of-use. However, integrating the sensors directly into loose clothing, usually, results in additional clothing motion relat...
-
Kalman-Particle Kernel Filter (KPKF) is a sub-class of Particle Filter (PF) that uses Gaussian kernels as particles, which enables a local Kalman update for each measurement in addition to the usual weight update. Besides, recent research about filtering on Lie groups brought powerful theoretical results, and showed the superiority of this approach. Hence, this paper extends the Euclidean KPKF to ...
-
Collaborative localization is an essential capability for a team of robots such as connected vehicles to collaboratively estimate object locations from multiple perspectives with reliant cooperation. To enable collaborative localization, four key challenges must be addressed, including modeling complex relationships between observed objects, fusing observations from an arbitrary number of collabor...
-
Manipulation tasks often require a robot to adjust its sensorimotor skills based on the state it finds itself in. Taking peg-in-hole as an example: once the peg is aligned with the hole, the robot should push the peg downwards. While high level execution frameworks such as state machines and behavior trees are commonly used to formalize such decision-making problems, these frameworks require a mec...
-
In this paper, we present a robotic device for mouse tail vein injection. We propose a mouse holding mechanism to realize vein injection without anesthetizing the mouse, which consists of a tourniquet, vacuum port, and adaptive tail-end fixture. The position of the target vein in 3D space is reconstructed from a high-resolution stereo vision. The vein is detected by a simple but robust vein line d...
-
This paper deals with the control of a redundant robotic system for middle ear surgery (i.e., cholesteatoma tissues removal). The targeted robotic system is a macro-micro-scale robot composed of a redundant seven degrees of freedom (DoFs) on which is attached a two DoFs robotized flexible fiberscope. Two different control architectures are proposed to achieve a defined surgical procedure to remove...
-
Robots that process navigation instructions in large outdoor environments will need to operate at different levels of abstraction. For example, a land-surveying aerial robot receiving the instruction “go to Boston and go through the state forest on the way” must reason about a long-range goal like “go to Boston” while also processing a finer-grained constraint like “go through the state forest.” E...
-
In this paper, we examine the problem of push recovery for bipedal robot locomotion and present a reactive decision-making and robust planning framework for locomotion resilient to external perturbations. Rejecting perturbations is an essential capability of bipedal robots and has been widely studied in the locomotion literature. However, adversarial disturbances and aggressive turning can lead to...
-
We want robots to complete assigned tasks even when unexpected task pressures arise, either from the robot or the environment. This paper presents a method of both learning sources of task failure in situ and rapidly planning new motions on-the-fly to accommodate them. This “risk-adaptive” approach to robot control uses a few encounters with a novel failure mode to generate a probabilistic failure...
-
This work explores the use of topological tools for achieving effective non-prehensile manipulation in cluttered, constrained workspaces. In particular, it proposes the use of persistent homology as a guiding principle in identifying the appropriate non-prehensile actions, such as pushing, to clean a cluttered space with a robotic arm so as to allow the retrieval of a target object. Persistent hom...
-
Task and motion planning (TAMP) algorithms aim to help robots achieve task-level goals, while maintaining motion-level feasibility. This paper focuses on TAMP domains that involve robot behaviors that take extended periods of time (e.g., long-distance navigation). In this paper, we develop a visual grounding approach to help robots probabilistically evaluate action feasibility, and introduce a TAM...
-
Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordances
We present a strategy for designing and building very general robot manipulation systems using a general-purpose task-and-motion planner with both engineered and learned modules that estimate properties and affordances of unknown objects. Such systems are closed-loop policies that map from RGB images, depth images, and robot joint encoder measurements to robot joint position commands. We show that...
-
Future robotic deployments will require robots to be able to repeatedly solve a variety of tasks in application domains. Task and motion planning addresses complex robotic problems that combine discrete reasoning over states and actions and geometric interactions during action executions. Moving beyond deterministic settings, stochastic actions can be handled by modeling the problem as a Markov De...
-
Retrieving objects from clutters is a complex task, which requires multiple interactions with the environment until the target object can be extracted. These interactions involve executing action primitives like grasping or pushing as well as setting priorities for the objects to manipulate and the actions to execute. Mechanical Search (MS) [1] is a framework for object retrieval, which uses a heu...
-
In this paper, we examine the problem of rearranging many objects on a tabletop in a cluttered setting using overhand grasps. Efficient solutions for the problem, which capture a common task that we solve on a daily basis, are essential in enabling truly intelligent robotic manipulation. In a given instance, objects may need to be placed at temporary positions (“buffers”) to complete the rearrange...
-
Prehensile object rearrangement in cluttered and confined spaces has broad applications but is also challenging. For instance, rearranging products in a grocery shelf means that the robot cannot directly access all objects and has limited free space. This is harder than tabletop rearrangement where objects are easily accessible with top-down grasps, which simplifies robot-object interactions. This...
-
Aerial Manipulators with Elastic Suspension (AMES) may be seen as a hybrid robot mixing properties of classical Aerial Manipulators (AMs) and Cable-Driven Parallel Robots (CDPRs). The optimal design and control of an AMES using unidirectional thrusters are considered in this paper. To maximize the workspace, an optimization algorithm is proposed. The position and orientation of the thrusters are o...
-
This paper solves the tracking control problem for a quadrotor system under the tasks of large-angle rotation and high-speed trajectory tracking. A quadrotor dynamic model is presented taking both disturbances and drag force into account. A reachability control strategy is developed for a quadrotor to track the planned attitude and position. Outdoor experiments of a circle trajectory tracking at d...
-
In this paper, we introduce a Modular Single Actuator Monocopter (M-SAM), which is capable of flying in both singular configuration and cooperative configuration. From singular mode, M-SAMs can be manually assembled into cooperative mode, using magnetic connectors built into the body of each M-SAM unit. The design of the connectors allow for passive separation of the units without the need for a d...
-
Recent advances in Model Predictive Control (MPC) algorithms and methodologies, combined with the surge of computational power of available embedded platforms, allows the use of real-time optimization-based control of fast mechatronic systems. This paper presents an implementation of an optimal guidance, navigation and control (GNC) system for the motion control of a small-scale electric prototype...
-
Inverted landing is a challenging feat to perform in aerial robots, especially without external positioning. However, it is routinely performed by biological fliers such as bees, flies, and bats. Our previous observations of landing behaviors in flies suggest an open-loop causal relationship between their putative visual cues and the kinematics of the aerial maneuvers executed. For example, the de...
-
We present SMORS, the first Soft fully actuated MultirOtoR System for multimodal locomotion. Unlike conventional hexarotors, SMORS is equipped with three rigid and three continuously soft arms, with each arm hosting a propeller. We create a bridge between the fields of soft and aerial robotics by mechanically coupling the actuation of a fully actuated flying platform with the actuation of a soft r...
-
This paper presents a modeling and control frame-work for multibody flying robots subject to non-negligible aero-dynamic forces acting on the centroidal dynamics. First, aero-dynamic forces are calculated during robot flight in different operating conditions by means of Computational Fluid Dynamics (CFD) analysis. Then, analytical models of the aerodynamics coefficients are generated from the data...
-
Cooperative transportation via multiple aerial robots has the potential to support various payloads and reduce the chances of them being dropped. Furthermore, autonomously controlled robots render the system scalable with respect to the payload. In this study, a cooperative transportation system was developed using rigidly attached single-rotor robots, and a decentralized controller was proposed t...
-
We present a design, model, and control for a novel jumping-flying robot that is called PogoDrone. The robot is composed of a quadrotor with a passive mechanism for jumping. The robot can continuously jump in place or fly like a normal quadrotor. Jumping in place allows the robot to quickly move and operate very close to the ground. For instance, in agricultural applications, the jumping mechanism...
-
Although manipulation capabilities of aerial robots greatly improved in the last decade, only few works addressed the problem of aerial physical interaction with dynamic environments, proposing strongly model-based approaches. However, in real scenarios, modeling the environment with high accuracy is often impossible. In this work, we aim at developing a control framework for Omnidirectional Micro...
-
Reorientation (turning in plane) plays a critical role for all robots in any field application, especially those that in confined spaces. While important, reorientation remains a relatively unstudied problem for robots, including limbless mechanisms, often called snake robots. Instead of looking at snakes, we take inspiration from observations of the turning behavior of tiny nematode worms C. eleg...
-
Despite advances in a diversity of environments, snake robots are still far behind snakes in traversing complex 3-D terrain with large obstacles. This is due to a lack of understanding of how to control 3-D body bending to push against terrain features to generate and control propulsion. Biological studies suggested that generalist snakes use contact force sensing to adjust body bending in real ti...
-
In most instances, flapping wing robots have emulated the “synchronous” actuation of insects in which the wingbeat timing is generated from a time-dependent, rhythmic signal. The internal dynamics of asynchronous insect flight muscle enable high-frequency, adaptive wingbeats with minimal direct neural control. In this paper, we investigate how the delayed stretch-activation (dSA) response of async...
-
Flapping Wing Rotorcraft (FWR) combines flapping and rotating wing motion in one element. Such a hybrid design integrates the high-efficiency characteristics of the rotating wing and the high-lift feature of the flapping wing under low Reynolds number, providing a broader range of simultaneous lift and power efficiency optimization. Nevertheless, the flight performance of the current FWRs is limit...
-
Bioinspired robots are useful tools to study complex biomechanical processes of animal locomotion. Key movements and kinematic parameters are under the control of experimenters, which is impossible to perform when experimenting with living animals. The primary challenge to test biological hypotheses is designing realistic robots taking inspiration from swimming snakes. Yet, underlying biomechanics...
-
This paper proposes a novel motion estimation algorithm using WiFi networks and IMU sensor data in large uncontrolled environments, dubbed “WiFi Structure-from-Motion” (WiFi SfM). Given smartphone sensor data through day-to-day activities from a single user over a month, our WiFi SfM algorithm estimates smartphone motion tra-jectories and the structure of the environment represented as a WiFi radi...
-
We address the problem of tracking 3D object poses from touch during in-hand manipulations. Specifically, we look at tracking small objects using vision-based tactile sensors that provide high-dimensional tactile image measurements at the point of contact. While prior work has relied on a-priori information about the object being localized, we remove this requirement. Our key insight is that an ob...
-
Highly accurate and robust relocalization or localization initialization ability is of great importance for autonomous vehicles (AVs). Traditional GNSS-based methods are not reliable enough in occlusion and multipath conditions. In this paper we propose a novel long-term semantic relocalization algorithm based on HD map and semantic features which are compact in representation. Semantic features a...
-
We present a novel real-time visual odometry framework for a stereo setup of a depth and high-resolution event camera. Our framework balances accuracy and robustness against computational efficiency towards strong performance in challenging scenarios. We extend conventional edge-based semi-dense visual odometry towards time-surface maps obtained from event streams. Semi-dense depth maps are genera...
-
Masking by Moving (MByM), provides robust and accurate radar odometry measurements through an exhaustive correlative search across discretised pose candidates. However, this dense search creates a significant computational bottleneck which hinders real-time performance when high-end GPUs are not available. Utilising the translational invariance of the Fourier Transform, in our approach, Fast Maski...
-
While complete localization approaches are widely studied in the literature, their data association and data representation subprocesses usually go unnoticed. However, both are a key part of the final pose estimation. In this work, we present DA-LMR (Delta-Angle Lane Marking Representation), a robust data representation in the context of localization approaches. We propose a representation of lane...
-
We present a novel framework of motion tracking from event data using implicit expression. Our framework uses pre-trained event generation MLP called the implicit event generator (IEG) and carries out motion tracking by updating its state (position and velocity) based on the difference between the observed event and generated event from the current state estimation. The difference is computed impl...
-
Global point cloud registration is an essential module for localization, of which the main difficulty exists in estimating the rotation globally without initial value. With the aid of gravity alignment, the degree of freedom in point cloud registration could be reduced to 4DoF, in which only the heading angle is required for rotation estimation. In this paper, we propose a fast and accurate global...
-
Retrieval-based place recognition is an efficient and effective solution for re-localization within a pre-built map, or global data association for Simultaneous Localization and Mapping (SLAM). The accuracy of such an approach is heavily dependant on the quality of the extracted scene-level representation. While end-to-end solutions - which learn a global descriptor from input point clouds - have ...
-
This paper presents a novel place recognition approach to autonomous vehicles by using low-cost, single-chip automotive radar. Aimed at improving recognition robustness and fully exploiting the rich information provided by this emerging automotive radar, our approach follows a principled pipeline that comprises (1) dynamic points removal from instant Doppler measurement, (2) spatial-temporal featu...
-
LiDAR is widely adopted in Simultaneous Localization And Mapping (SLAM) and High Definition (HD) map production. The accuracy of LiDAR Odometry (LO) is of great importance, especially in GPS-denied environments. However, we found typical LO results are prone to drift upwards along the vertical direction in underground parking lots, leading to poor mapping results. This paper proposes a Ground Cons...
-
We study a semantic SLAM problem where a robot is tasked with autonomous weeding under the corn canopy. The goal is to detect corn stalks and localize them in a global coordinate frame. This is a challenging scenario for existing algorithms because there is very little space between the camera and the plants, and the camera motion is primarily restricted to be along the row. To overcome these chal...
-
Automation of vineyards cultivation necessitates for mobile robots to retain accurate localization system. The paper introduces a stereo vision-based Graph-Simultaneous Localization and Mapping (Graph-SLAM) pipeline custom-tailored to the specificities of vineyard fields. Graph-SLAM is reinforced with a Loop Closure Detection (LCD) based on semantic segmentation of the vine trees. The Mask R-CNN n...
-
Monitoring individual plants and computing precise 3D reconstructions is highly relevant for crop breeding. In the conventional breeding approach, humans measure phenotypic traits by hand, requiring substantial manual labor. This paper addresses precise 3D plant reconstructions in a crop field or breeding plot based on UAV imagery. We explicitly address the challenges resulting from the thin struc...
-
Data augmentation can be a simple yet powerful tool for autonomous robots to fully utilise available data for self-supervised identification of atypical scenes or objects. State-of-the-art augmentation methods arbitrarily embed “structural” peculiarity on typical images so that classifying these artefacts can provide guidance for learning representations for the detection of anomalous visual signa...
-
Smart weeding systems to perform plant-specific operations can contribute to the sustainability of agriculture and the environment. Despite monumental advances in autonomous robotic technologies for precision weed management in recent years, work on under-canopy weeding in fields is yet to be realized. A prerequisite of such systems is reliable detection and classification of weeds to avoid mistak...
-
Robotic tree pruning requires highly precise manipulator control in order to accurately align a cutting implement with the desired pruning point at the correct angle. Simultaneously, the robot must avoid applying excessive force to rigid parts of the environment such as trees, support posts, and wires. In this paper, we propose a hybrid control system that uses a learned vision-based controller to...
-
Labour shortage, difficulties in labour management, the digitalization of fruit production pipeline to reduce the fruit production costs have made robotic systems for selective harvesting of strawberries an important industry and academic research. One of the important components of such technologies yet to be developed is fruit picking perception. For picking strawberries, a robot needs to infer ...
-
During postharvest storage, fruit firmness usually decreases due to respiration and bruise, the former of which indicates the fruit ripeness while the latter negatively influence consumers' taste preference. This paper presents a portable and low-cost device using vision-based tactile information to evaluate fruit firmness in a non-destructive manner. The device consists of a camera, LED lights, a...
-
Automated agricultural applications, i.e., fruit picking require spatial information about crops and, especially, their fruits. In this paper, we present a novel deep reinforcement learning (DRL) approach to determine the next best view for automatic exploration of 3D environments with a robotic arm equipped with an RGB-D camera. We process the obtained images into an octree with labeled regions o...
-
Recent innovations in miniature sensors are driving a shift from robotic to bio-hybrid systems for exploration of unstructured environments. The ubiquity of honey bees in modern agriculture and ecology along with their superior agility, olfactory sense, and collective foraging skills make them a promising complement to traditional robots. This paper explores the potential of such systems based on ...
-
Providing guidance about collision avoidance, pedestrian trajectory prediction is an important task for autonomous driving. In this paper, to produce plausible trajectory predictions in the first-person view circumstance, we propose a crossmodal transformer based generative framework which could leverage sequences of cues from multiple modalities as well as pedestrian attributes. For the encoder, ...
-
We present a lightweight encoder-decoder architecture for monocular depth estimation, specifically designed for embedded platforms. Our main contribution is the Guided Upsampling Block (GUB) for building the decoder of our model. Motivated by the concept of guided image filtering, GUB relies on the image to guide the decoder on upsampling the feature representation and the depth map reconstruction...
-
Uncertainty pervades through the modern robotic autonomy stack, with nearly every component (e.g., sensors, detection, classification, tracking, behavior prediction) producing continuous or discrete probabilistic distributions. Trajectory forecasting, in particular, is surrounded by uncertainty as its inputs are produced by (noisy) upstream perception and its outputs are predictions that are often...
-
Point clouds from 3D light detection and ranging (LiDAR) are widely used. Noise caused by falling snow reduces the availability of point clouds. Due to the sparseness of LiDAR point clouds and the fact that the snow point clouds are easily affected by multi factors such as wind or snowfall conditions, it is difficult to accurately remove the snow while preserving the details of the point clouds. T...
-
The image signal processing (ISP) pipeline, which transforms raw sensor measurement to a color image, is composed of a sequence of processing modules. Traditionally, the ISP pipeline is manually tuned by experts for human perception. The resulting handcrafted ISP configuration does not necessarily benefit the downstream high-level vision tasks. To mitigate these problems, this paper presents a sim...
-
Monocular depth estimation has been a popu-lar area of research for several years, especially since self-supervised networks have shown increasingly good results in bridging the gap with supervised and stereo methods. However, these approaches focus their interest on dense 3D reconstruction and sometimes on tiny details that are superfluous for autonomous navigation. In this paper, we propose to a...
-
In this paper we present a compositing image synthesis method that generates RGB canvases with well aligned segmentation maps and sparse depth maps, coupled with an in-painting network that transforms the RGB canvases into high quality RGB images and the sparse depth maps into pixel-wise dense depth maps. We benchmark our method in terms of structural alignment and image quality, showing an increa...
-
Real-time semantic segmentation is a crucial but challenging dense prediction task for scene parsing. However, the existing CNN-based methods commonly bias the model in favor of speed-boosting compromising spatial resolution due to business requirements and hardware constrains, which impedes the high-accuracy segmentation result. To address the dilemma, we provide a novel Holographic Segmentation ...
-
We present a system for automatic converting of 2D mask object predictions and raw LiDAR point clouds into full 3D bounding boxes of objects. Because the LiDAR point clouds are partial, directly fitting bounding boxes to the point clouds is meaningless. Instead, we suggest that obtaining good results requires sharing information between all objects in the dataset jointly, over multiple frames. We ...
-
Simulation has the potential to transform the development of robust algorithms for mobile agents deployed in safety-critical scenarios. However, the poor photorealism and lack of diverse sensor modalities of existing simulation engines remain key hurdles towards realizing this potential. Here, we present VISTA††Full code release for the VISTA data-driven simulation engine is available here: vista....
-
Deep imitation learning is a promising approach that does not require hard-coded control rules in autonomous robot manipulation. The current applications of deep imitation learning to robot manipulation have been limited to reactive control based on the states at the current time step. However, future robots will also be required to solve tasks utilizing their memory obtained by experience in comp...
-
A general-purpose robot should be able to master a wide range of tasks and quickly learn a novel one by leveraging past experiences. One-shot imitation learning (OSIL) approaches this goal by training an agent with (pairs of) expert demonstrations, such that at test time, it can directly execute a new task from just one demonstration. However, so far this framework has been limited to training on ...
-
Simulation is a crucial tool for accelerating the development of autonomous vehicles. Making simulation realistic requires models of the human road users who interact with such cars. Such models can be obtained by applying learning from demonstration (LfD) to trajectories observed by cars already on the road. However, existing LfD methods are typically insufficient, yielding policies that frequent...
-
The imitation learning research community has recently made significant progress towards the goal of enabling artificial agents to imitate behaviors from video demonstrations alone. However, current state-of-the-art approaches developed for this problem exhibit high sample complexity due, in part, to the high-dimensional nature of video observations. Towards addressing this issue, we introduce her...
-
Deep imitation learning requires many expert demonstrations, which can be hard to obtain, especially when many tasks are involved. However, different tasks often share similarities, so learning them jointly can greatly benefit them and alleviate the need for many demonstrations. But, joint multi-task learning often suffers from negative transfer, sharing information that should be task-specific. I...
-
Robust imitation learning using disturbance injections overcomes issues of limited variation in demonstrations. However, these methods assume demonstrations are optimal, and that policy stabilization can be learned via simple augmentations. In real-world scenarios, demonstrations are often of diverse-quality, and disturbance injection instead learns sub-optimal policies that fail to replicate desi...
-
We develop a method for learning periodic tasks from visual demonstrations. The core idea is to leverage periodicity in the policy structure to model periodic aspects of the tasks. We use active learning to optimize parameters of rhythmic dynamic movement primitives (rDMPs) and propose an objective to maximize the similarity between the motion of objects manipulated by the robot and the desired mo...
-
Learning from demonstrations in the wild (e.g. YouTube videos) is a tantalizing goal in imitation learning. However, for this goal to be achieved, imitation learning algorithms must deal with the fact that the demonstrators and learners may have bodies that differ from one another. This condition — “embodiment mismatch” — is ignored by many recent imitation learning algorithms. Our proposed imitat...
-
Enabling robots to solve multiple manipulation tasks has a wide range of industrial applications. While learning-based approaches enjoy flexibility and generalizability, scaling these approaches to solve such compositional tasks remains a challenge. In this work, we aim to solve multi-task learning through the lens of sequence-conditioning and weighted sampling. First, we propose a new suite of be...
-
While imitation learning for vision-based au-tonomous mobile robot navigation has recently received a great deal of attention in the research community, existing approaches typically require state-action demonstrations that were gathered using the deployment platform. However, what if one cannot easily outfit their platform to record these demonstration signals or-worse yet-the demonstrator does n...
-
We study the problem of generalizable task learning from human demonstration videos without extra training on the robot or pre-recorded robot motions. Given a set of human demonstration videos showing a task with different objects/tools (categorical objects), we aim to learn a representation of visual observation that generalizes to categorical objects and enables efficient controller design. We p...
-
Humans have the ability to pour various media, both liquid and granular, to desired ends in various containers. We do this by using multiple senses simultaneously in a constant feedback loop to complete a pouring task. Combining multiple sensing modalities, similar to humans, could aid in robotic pouring control outside of a structured or industrial setting. We present a multi-sensory pouring data...
-
Pilots operating aircraft in non-towered terminal airspace rely on their situational awareness and prior knowledge to predict the future trajectories of other agents. These predictions are conditioned on the past trajectories of other agents, agent-agent social interactions and environmental context such as airport location and weather. This paper provides a dataset, TrajAir, that captures this be...
-
Freespace detection is an essential component of autonomous driving technology and plays an important role in trajectory planning. In the last decade, deep learning based freespace detection methods have been proved feasible. However, these efforts were focused on urban road environments and few deep learning based methods were specifically designed for off-road freespace detection due to the lack...
-
Due to high complexity and occlusion, insufficient perception in the crowded urban intersection can be a serious safety risk for both human drivers and autonomous algorithms, whereas CVIS (Cooperative Vehicle Infrastructure System) is a proposed solution for full-participants perception under this scenario. However, the research on roadside multi-modal perception is still in its infancy, and there...
-
We present TartanDrive, a large scale dataset for learning dynamics models for off-road driving. We collected a dataset of roughly 200,000 off-road driving interactions on a modified Yamaha Viking ATV with seven unique sensing modalities in diverse terrains. To the authors' knowledge, this is the largest real-world multi-modal off-road driving dataset, both in terms of number of interactions and s...
-
Interactive 3D simulations have enabled break-throughs in robotics and computer vision, but simulating the broad diversity of environments needed for deep learning requires large corpora of photo-realistic 3D object models. To address this need, we present Google Scanned Objects, an open-source collection of over one thousand 3D-scanned household items released under a Creative Commons license; th...
-
There is a gap in holistic urban scene understanding between multi-modal datasets for segmentation and object detection on the one hand and traffic light datasets on the other hand. The role of traffic lights in the former is not labelled, making it difficult to use them for higher-level tasks and leave critical information of an intersection scene blank. Including traffic lights from traffic ligh...
-
Curbs are one of the essential elements of urban and highway traffic environments. Robust curb detection provides road structure information for motion planning in an autonomous driving system. Commonly, video cameras and 3D LiDARs are mounted on autonomous vehicles for curb detection. However, camera-based methods suffer from challenging illumination conditions. During the long period of time bef...
-
Employing Vehicle-to-Vehicle communication to enhance perception performance in self-driving technology has attracted considerable attention recently; however, the absence of a suitable open dataset for benchmarking algorithms has made it difficult to develop and assess cooperative perception technologies. To this end, we present the first large-scale open simulated dataset for Vehicle-to-Vehicle ...
-
Wireless tags are increasingly used to track and identify common items of interest such as retail goods, food, medicine, clothing, books, documents, keys, equipment, and more. At the same time, there is a need for labelled visual data featuring such items for the purpose of training object detection and recognition models for robots operating in homes, warehouses, stores, libraries, pharmacies, an...
-
Predicting the behaviors of other agents on the road is critical for autonomous driving to ensure safety and efficiency. However, the challenging part is how to represent the social interactions between agents and output different possible trajectories with interpretability. In this paper, we introduce a neural prediction framework based on the Transformer structure to model the relationship among...
-
Place recognition or loop closure detection is one of the core components in a full SLAM system. In this paper, aiming at strengthening the relevancy of local neighboring points and the contextual dependency among global points simultaneously, we investigate the exploitation of transformer-based network for feature extraction, and propose a Hierarchical Transformer for Place Recognition (HiTPR). T...
-
Diff-Net: Image Feature Difference Based High-Definition Map Change Detection for Autonomous Driving
Up-to-date High-Definition (HD) maps are essential for self-driving cars. To achieve constantly updated HD maps, we present a deep neural network (DNN), Diff-Net, to detect changes in them. Compared to traditional methods based on object detectors, the essential design in our work is a parallel feature difference calculation structure that infers map changes by comparing features extracted from th...
-
Understanding environment dynamics is necessary for robots to act safely and optimally in the world. In realistic scenarios, dynamics are non-stationary and the causal variables such as environment parameters cannot necessarily be precisely measured or inferred, even during training. We propose Implicit Identification for Dynamics Adaptation (IIDA), a simple method to allow predictive models to ad...
-
In this paper, we focus on a less explored, but more realistic and complex problem of domain adaptation in LiDAR semantic segmentation. There is a significant drop in performance of an existing segmentation model when training (source domain) and testing (target domain) data originate from different LiDAR sensors. To overcome this shortcoming, we propose an unsupervised domain adaptation framework...
-
Implicit Kinematic Policies: Unifying Joint and Cartesian Action Spaces in End-to-End Robot Learning
Action representation is an important yet often overlooked aspect in end-to-end robot learning with deep networks. Choosing one action space over another (e.g. target joint positions, or Cartesian end-effector poses) can result in surprisingly stark performance differences between various downstream tasks - and as a result, considerable research has been devoted to finding the right action space f...
-
Efficient exploration is a long-standing problem in reinforcement learning since extrinsic rewards are usually sparse or missing. A popular solution to this issue is to feed an agent with novelty signals as intrinsic rewards. In this work, we introduce SEMI, a self-supervised exploration policy by incentivizing the agent to maximize a new novelty signal: multisensory incongruity, which can be meas...
-
Flexible-joint manipulators are governed by complex nonlinear dynamics, defining a challenging control problem. In this work, we propose an approach to learn an outer-loop joint trajectory tracking controller with deep reinforcement learning. The controller represented by a stochastic policy is learned in under two hours directly on the real robot. This is achieved through bounded reference correc...
-
Shape and pose estimation is a critical perception problem for a self-driving car to fully understand its surrounding environment. One fundamental challenge in solving this problem is the incomplete sensor signal (e.g., LiDAR scans), especially for faraway or occluded objects. In this paper, we propose a novel algorithm to address this challenge, which explicitly leverages the sensor signal captur...
-
Real-time and precise morphological analysis of intraoperative AAA is a significant pre-imperative for robot-assisted minimally invasive surgery (RMIS). However, this task is frequently accompanied by the difficulties of ambiguous boundaries and obscured surfaces of aneurysms. To remedy these problems, we propose a Light-Weight Dual-Stream Boundary-Aware Network (DSB-Net) and a novel diagnosis alg...
-
We present an approach to robotic manipulation of unknown objects through regulation of the object's contact configuration: the location, geometry, and mode of all contacts between the object, robot, and environment. A contact configu-ration constrains the forces and motions that can be applied to the object; however, synthesizing these constraints generally requires knowledge of the object's pose...
-
Dexterous robotic cutting needs to demonstrate a skill level with smooth and efficient knife movements. The work performed by the knife mainly generates fracture and overcomes the blade-material friction. This paper presents a recursive least-squares method that repeatedly estimates relevant physical parameters such as Poisson's ratio, fracture toughness, and coefficient of friction, all varying w...
-
Variable impedance control in operation-space is a promising approach to learning contact-rich manipulation behaviors. One of the main challenges with this approach is producing a manipulation behavior that ensures the safety of the arm and the environment. Such behavior is typically implemented via a reward function that penalizes unsafe actions (e.g. obstacle collision, joint limit extension), b...
-
Controlling robotic manipulators with high-dimensional action spaces for dexterous tasks is a challenging problem. Inspired by human manipulation, researchers have studied generating and using postural synergies for robot hands to accomplish manipulation tasks, leveraging the lower dimensional nature of synergistic action spaces. However, many of these works require pre-collected data from an exis...
-
This paper presents Contact Mode Guided Manipulation Planning (CMGMP) for 3D quasistatic and quasi-dynamic rigid body motion planning in dexterous manipulation. The CMGMP algorithm generates hybrid motion plans including both continuous state transitions and discrete contact mode switches, without the need for pre-specified contact sequences or pre-designed motion primitives. The key idea is to us...
-
We show that a purely tactile dextrous in-hand manipulation task with continuous regrasping, requiring permanent force closure, can be learned from scratch and executed robustly on a torque-controlled humanoid robotic hand. The task is rotating a cube without dropping it, but in contrast to OpenAI's seminal cube manipulation task [1], the palm faces downwards and no cameras but only the hand's pos...
-
Finger-gaiting manipulation is an important skill to achieve large-angle in-hand re-orientation of objects. However, achieving these gaits with arbitrary orientations of the hand is challenging due to the unstable nature of the task. In this work, we use model-free reinforcement learning (RL) to learn finger-gaiting only via precision grasps and demonstrate finger-gaiting for rotation about an axi...
-
This paper addresses the generation mechanism and avoidance method of negative stiffness during in-Hand manipulation with underactuated compliant hands. Firstly, a planar hand with two three-jointed fingers manipulating a rectangular is set, and a quasi-static underactuated operation model is established. Secondly, based on this simulation model, we investigated the stiffness evolution during in-h...
-
In a long-term large-scenario application, the multi-agent collaborative SLAM is expected to improve the robustness and efficiency of executing tasks for mobile agents. In this paper, a multi-agent collaborative visual-inertial SLAM system is proposed based on a centralized client-server (CS) architecture, where the clients run on smart mobiles. In general, multi-agent collaborative SLAM relies on...
-
We consider the problem of completing a set of $n$ tasks with a human-robot team using minimum effort. In many domains, teaching a robot to be fully autonomous can be counterproductive if there are finitely many tasks to be done. Rather, the optimal strategy is to weigh the cost of teaching a robot and its benefit- how many new tasks it allows the robot to solve autonomously. We formulate this as ...
-
Humans engaged in collaborative activities are naturally able to convey their intentions to teammates through multi-modal communication, which is made up of explicit and implicit cues. Similarly, a more natural form of human-robot collaboration may be achieved by enabling robots to convey their intentions to human teammates via multiple communication channels. In this paper, we postulate that a be...
-
Human-robot collaboration aims to extend human ability through cooperation with robots. This technology is currently helping people with physical disabilities, has transformed the manufacturing process of companies, improved surgical performance, and will likely revolutionize the daily lives of everyone in the future. Being able to enhance the performance of both sides, such that human-robot colla...
-
The variety and complexity of tasks autonomous robots can tackle is constantly increasing, yet we seldom see robots collaborating with humans. Indeed, humans are either requested for punctual help or are given the lead on the whole task. We propose a human-aware task planning approach allowing the robot to plan for a task while also considering and emulating the human decision, action, and reactio...
-
Even though cobots have high potential in bringing several benefits in the manufacturing and logistic processes, their rapid (re-)deployment in changing environments is still limited. To enable fast adaptation to new product demands and to boost the fitness of the human workers to the allocated tasks, we propose a novel method that optimizes assembly strategies and distributes the effort among the...
-
Operating an articulated machine is a complex and hierarchical task, involving several levels of decision making. Motivated by the timber-harvesting applications of these machines, we are interested in developing a collaborative framework for operating an articulated machine/robot in order to increase its level of autonomy. In this paper, we consider two problems in the context of collaborative op...
-
Robotic engineers face major challenges to solve the complex actuation needs of Human-Robot Collaboration with existing act robotic gearboxes. Available technologies comprise high-ratio Planetary Gearheads, Cycloid Drives and Harmonic Drives, inherited from conventional industrial robotics. Alternative approaches include Direct-Drive and Quasi Direct-Drive actuation strategies, which propose to ca...
-
Autonomous navigation of mobile robots is an es-sential aspect in use cases such as delivery, assistance or logistics. Although traditional planning methods are well integrated into existing navigation systems, they struggle in highly dynamic en-vironments. On the other hand, Deep-Reinforcement-Learning-based methods show superior performance in dynamic obstacle avoidance but are not suitable for ...
-
Language allows humans to build mental models that interpret what is happening around them resulting in more accurate long-term predictions. We present a novel trajectory prediction model that uses linguistic intermediate representations to forecast trajectories, and is trained using trajectory samples with partially-annotated captions. The model learns the meaning of each of the words without dir...
-
We present a novel approach for risk-aware planning with human agents in multi-agent traffic scenarios. Our approach takes into account the wide range of human driver behaviors on the road, from aggressive maneuvers like speeding and overtaking, to conservative traits like driving slowly and conforming to the right-most lane. In our approach, we learn a mapping from a data-driven human driver beha...
-
Autonomous vehicle-based traffic smoothing con-trollers are often not transferred to real-world use due to challenges in calibrating many-agent traffic simulators. We show a pipeline to sidestep such calibration issues by collecting trajectory data and learning controllers directly from trajectory data that are then deployed zero-shot onto the highway. We construct a dataset of 772.3 kilometers of...
-
Driving automation is gradually replacing human driving maneuvers in different applications such as adaptive cruise control and lane keeping. However, contemporary driving automation applications based on expert systems or prede-fined control strategies are not in line with individual human driver's preference. To overcome this problem, we propose a Personalized Adaptive Cruise Control (P-ACC) sys...
-
Modeling multi-modal high-level intent is important for ensuring diversity in trajectory prediction. Existing approaches explore the discrete nature of human intent before predicting continuous trajectories, to improve accuracy and support explainability. However, these approaches often assume the intent to remain fixed over the prediction horizon, which is problematic in practice, especially over...
-
Accurate identification of important objects in the scene is a prerequisite for safe and high-quality decision making and motion planning of intelligent agents (e.g., autonomous vehicles) that navigate in complex and dynamic environments. Most existing approaches attempt to employ attention mechanisms to learn importance weights associated with each object indirectly via various tasks (e.g., traje...
-
Thermal infrared (TIR) image has proven effectiveness in providing temperature cues to the RGB features for multispectral pedestrian detection. Most existing methods directly inject the TIR modality into the RGB-based framework or simply ensemble the results of two modalities. This, however, could lead to inferior detection performance, as the RGB and TIR features generally have modality-specific ...
-
Multiple-camera systems have been widely used in self-driving cars, robots, and smartphones. In addition, they are typically also equipped with IMUs (inertial measurement units). Using the gravity direction extracted from the IMU data, the y-axis of the body frame of the multi-camera system can be aligned with this common direction, reducing the original three degree-of-freedom(DOF) relative rotat...
-
This paper presents the design of a novel expert system for robotic manipulators performing sanding tasks on work surfaces. The expert system adjusts the velocity of the robotic manipulator based on the observed surface quality. These observation are obtained by an analysis of the raw force data provided by a force-torque sensor at the end-effector level. The expert system consists of two governin...
-
In the peg insertion task, human pays attention to the seam between the peg and the hole and tries to fill it continuously with visual feedback. By imitating the human's behavior, we design architectures with position and orientation estimators based on the seam representation for pose alignment, which proves to be general to the unseen peg geometries. By putting the estimators into the closed-loo...
-
This paper discusses a low-cost, open-source and open-hardware design and performance evaluation of a low-speed, multi-fan wind system dedicated to micro air vehicle (MAV) testing. In addition, a set of experiments with a flapping wing MAV and rotorcraft is presented, demonstrating the capabilities of the system and the properties of these different types of drones in response to various types of ...
-
Compliant actuation bestows robots with the ability to cope with unstructured environments, move with agility, and interact safely with humans at the expense of reduced tracking accuracy. The inclusion of dampening components aims to reduce oscillatory dynamics and partially restore precision without sacrificing the previously obtained characteristics. This paper introduces the concept and design ...
-
The Modboat is a low-cost, underactuated, modular robot capable of surface swimming. It is able to swim individually, dock to other Modboats, and undock from them using only a single motor and two passive flippers. Undocking without additional actuation is achieved by causing intentional self-collision between the tails of neighboring modules; this becomes a challenge when group swimming as one co...
-
Unmanned underwater vehicles are crucial for deep-sea exploration and inspection without imposing any danger to human life due to extreme environmental conditions. But, designing a robust controller that can cope with model uncertainties, external disturbances, and time delays for such vehicles is a challenge. This paper implements a sliding mode position control algorithm with a time-delay estima...
-
Due to the difficulty and expense of underwater field trials, a high fidelity underwater simulator is a necessity for testing and developing algorithms. To fill this need, we present HoloOcean, an open source underwater simulator, built upon Unreal Engine 4 (UE4). HoloOcean comes equipped with multi-agent support, various sensor implementations of common underwater sensors, and simulated communica...
-
We present a flow-based control strategy that enables resource-constrained marine robots to patrol gyre-like flow environments on an orbital trajectory with a periodicity in a given range. The controller does not require a detailed model of the flow field and relies only on the robot's location relative to the center of the gyre. Instead of precisely tracking a pre-defined trajectory, the robots a...
-
In this work we present a novel method for reconstructing 3D surfaces using a multi-beam imaging sonar. We integrate the intensities measured by the sonar from different viewpoints for fixed cell positions in a 3D grid. For each cell we integrate a feature vector that holds the mean intensity for a discretized range of viewpoints. Based on the feature vectors and independent sparse range measureme...
-
Traditional robotic manipulator design methods require extensive, time-consuming, and manual trial and error to produce a viable design. During this process, engineers often spend their time redesigning or reshaping components as they discover better topologies for the robotic manipula-tor. Tactile sensors, while useful, often complicate the design due to their bulky form factor. We propose an int...
-
Autonomous underwater vehicles (AUVs) are spe-cialized robots that are commonly used for seafloor surveying and ocean water sampling. Computational design approaches have emerged to reduce the effort required to design both individual AUVs as well as fleets. As the number and scale of underwater missions increases beyond the capabilities of a single vehicle, fleet level design will become more imp...
-
Mobile manipulators for indoor human environments can serve as versatile devices that perform a variety of tasks, yet adoption of this technology has been limited. Reducing size, weight, and cost could facilitate adoption, but risks restricting capabilities. We present a novel design that reduces size, weight, and cost, while supporting a variety of tasks. The core design consists of a two-wheeled...
-
Continuum robots are typically slender and flexible with infinite freedoms in theory, which poses a challenge for their control and application. The shape reconstruction of continuum robots is vital to realize closed-loop control. This paper proposes a novel general real-time shape reconstruction framework of continuum robots based on the piecewise polynomial curvature (PPC) kinematics model. We i...
-
In this paper a novel full state feedback approach for control of compliant actuated robot with nonlinear spring characteristics is presented. A multi-DOF elastic robot arm is a multi-input multi-output (MIMO) under-actuated system. By the new novel controller, which is based on motor coordinate transformation and motor inertia shaping, the MIMO system can be converted into a set of decoupled sing...
-
We present a novel application of continuum robots acting as concrete hoses to support 3D printing of cementitious materials. An industrial concrete hose was fitted with a cable harness and remotely actuated via tendons. The resulting continuum hose robot exhibited non constant curvature. In order to account for this, a new geometric approach to modeling variable curvature inverse kinematics using...
-
In this paper, we propose a new tractable ordinary differential equation formulation for dynamic simulation of fabric- reinforced inflatable soft robots. The method performs a lumped-parameter discretization of the continuum robot into discrete discs (inertia), spring elements, and threads (representing the inextensible fabric reinforcement). Using the repetition in the structure of the Lagrangian...
-
Tensegrity robots, composed of rigid rods and flexible cables, are difficult to accurately model and control given the presence of complex dynamics and high number of DoFs. Differentiable physics engines have been recently proposed as a data-driven approach for model identification of such complex robotic systems. These engines are often executed at a high-frequency to achieve accurate simulation....
-
In this work, we consider the problem of controlling the end effector position of a continuum manipulator through local stiffness changes. Continuum manipulators offer the advantage of continuous deformation along their lengths, and recent advances in smart material actuators further enable local compliance changes, which can affect the manipulator's bulk motion. However, leveraging local stiffnes...
-
Wearable gait assistive devices have been im-proved by soft robotics in terms of safety and physical burden of the wearer. Currently, electrical sensors and computers around the wearer are bottlenecks in enhancing the wearer's activities. In this paper, we present a wearable gait assistive system using soft pneumatic system for all the actuation, sensing, and computation. Pneumatic artifical muscl...
-
Dielectric elastomer actuators (DEAs) have been widely employed to drive various soft robots, due to their quiet fast muscle-like behavior. It is significant but challenging to model and control these soft actuators, due to their viscoelastic property, irregular geometry, complex structure, etc. In this paper, we propose a data-driven sparse identification method to discover the hidden governing e...
-
Soft robots are made of compliant materials and perform tasks that are challenging for rigid robots. However, their continuum nature makes it difficult to develop model-based control strategies. This work presents a robust model-based control scheme for soft continuum robots. Our dynamic model is based on the Euler-Lagrange approach, but it uses a more accurate description of the robot's inertia a...
-
In unstructured urban canals, regulation-aware interactions with other vessels are essential for collision avoidance and social compliance. In this paper, we propose a regulations aware motion planning framework for Autonomous Surface Vessels (ASVs) that accounts for dynamic and static obstacles. Our method builds upon local model predictive contouring control (LMPCC) to generate motion plans sati...
-
This paper presents a novel strategy for autonomous teamed exploration of subterranean environments using legged and aerial robots. Tailored to the fact that subterranean settings, such as cave networks and underground mines, often involve complex, large-scale and multi-branched topologies, while wireless communication within them can be particularly challenging, this work is structured around the...
-
We present a motion planner for planning through space-time with dynamic obstacles, velocity constraints, and unknown arrival time. Our algorithm, Space-Time RRT*(ST-RRT*), is a probabilistically complete, bidirectional motion planning algorithm, which is asymptotically optimal with respect to the shortest arrival time. We experimentally evaluate ST-RRT* in both abstract (2D disk, 8D disk in clutt...
-
In this paper, we present a novel convex optimization approach to address the minimum-time speed planning problem over a fixed path with dynamic obstacle constraints and point-wise speed and acceleration constraints. The contributions of this paper are three-fold. First, we formulate the speed planning as an iterative convex optimization problem based on space discretization. Our formulation allow...
-
Coverage Path Planning is the work horse of contemporary service task automation, powering autonomous floor cleaning robots and lawn mowers in households and office sites. While steady progress has been made on indoor cleaning and outdoor mowing, these environments are small and with simple geometry compared to general urban environments such as city parking garages, highway bridges or city crossi...
-
Non-linear trajectory optimisation methods require good initial guesses to converge to a locally optimal solution. A feasible guess can often be obtained by allocating a large amount of time for the trajectory to be complete. However for unstable dynamical systems such as humanoid robots, this quasi-static assumption does not always hold. We propose a conservative formulation of the trajectory pro...
-
For a robot to perform a grasping and manipulation task, it has to determine possible robot placements in the workspace, from which target objects or environmental elements relevant to the given task are reachable. This work presents a novel approach for finding placements for the mobile base of a humanoid robot in an unknown environment with multiple support planes. We propose a novel type of rea...
-
In this research, we aim to answer the question: How to combine Closed-Loop State and Input Sensitivity-based with Observability-aware trajectory planning? These possibly op-posite optimization objectives can be used to improve trajectory control tracking and, at the same time, estimation performance. Our proposed novel Control & Observability-aware Planning (COP) framework is the first that uses ...
-
Heuristic search-based motion planning algorithms typically discretise the search space in order to solve the shortest path problem. Their performance is closely related to this discretisation. A fine discretisation allows for better approximations of the continuous search space, but makes the search for a solution more computationally costly. A coarser resolution might allow the algorithms to fin...
-
We propose a hybrid model predictive control algorithm, consensus complementarity control (C3), for systems that make and break contact with their environment. Many state-of-the-art controllers for tasks which require initiating contact with the environment, such as locomotion and manipulation, require a priori mode schedules or are so computationally complex that they cannot run at real-time rate...
-
In this paper, we present a novel maximum entropy formulation of the Differential Dynamic Programming algorithm and derive two variants using unimodal and multimodal value functions parameterizations. By combining the maximum entropy Bellman equations with a particular approximation of the cost function, we are able to obtain a new formulation of Differential Dynamic Programming which is able to e...
-
Derivative based optimization methods are efficient at solving optimal control problems near local optima. However, their ability to converge halts when derivative information vanishes. The inference approach to optimal control does not have strict requirements on the objective landscape. However, sampling, the primary tool for solving such problems, tends to be much slower in computation time. We...
-
This paper presents a novel planning and control strategy for competing with multiple vehicles in a car racing scenario. The proposed racing strategy switches between two modes. When there are no surrounding vehicles, a learning-based model predictive control (MPC) trajectory planner is used to guarantee that the ego vehicle achieves better lap timing performance. When the ego vehicle is competing...
-
Fast and accurate pose estimation is essential for many robotic applications such as SLAM, manipulation, and 3D point registration. Existing solutions to this problem suffer from either high computation overhead due to the nonlinear features or accuracy loss due to linear approximation. In this paper, we propose a dual quaternion feedback particle filter (DQFPF) that can capture the nonlinear fact...
-
The rapid development of affordable and compact high-fidelity sensors (e.g., cameras and LIDAR) allows robots to construct detailed estimates of their states and environments. However, the availability of such rich sensor information introduces two challenges: (i) the lack of analytic sensing models, which makes it difficult to design controllers that are robust to sensor failures, and (ii) the co...
-
Grasping a wide range of novel objects in densely cluttered scenes is difficult due to irregular shapes of objects and the uncertainty in sensing. In this paper, a novel vacuum cup grasping method, based on uncertainty modeling of perception data and grasp geometric heuristics, is proposed to grasp unknown objects in densely cluttered scenes. The probabilistic signed distance function is proposed ...
-
Symbols representing abstract states such as “dish in dishwasher” or “cup on table” allow robots to reason over long horizons by hiding details unnecessary for high-level planning. Current methods for learning to identify symbolic states in visual data require large amounts of labeled training data, but manually annotating such datasets is prohibitively expensive due to the combinatorial number of...
-
Detecting the onset/ongoing of slip, i.e. if a grasped object is slipping or will slip from the gripper while being lifted, is crucial. Conventionally, it is regarded as a tactile sensing related problem. However, recently multi-modal robotic learning has become popular and is expected to boost the performance. In this paper we propose a novel CNN-TCN model to fuse tactile and visual information f...
-
Navigation through uncontrolled intersections is one of the key challenges for autonomous vehicles. Identifying the subtle differences in hidden traits of other drivers can bring significant benefits when navigating in such environments. We propose an unsupervised method for inferring driver traits such as driving styles from observed vehicle trajectories. We use a variational autoencoder with rec...
-
Deformable object manipulation requires computationally efficient representations that are compatible with robotic sensing modalities. In this paper, we present VIRDO: an implicit, multi-modal, and continuous representation for deformable-elastic objects. VIRDO operates directly on visual (point cloud) and tactile (reaction forces) modalities and learns rich latent embeddings of contact locations ...
-
Vision-based reinforcement learning (RL) is a promising approach to solve control tasks involving images as the main observation. State-of-the-art RL algorithms still struggle in terms of sample efficiency, especially when using image observations. This has led to increased attention on integrating state representation learning (SRL) techniques into the RL pipeline. Work in this field demonstrates...
-
We present a new technique that enables manifold learning to accurately embed data manifolds that contain holes, without discarding any topological information. Manifold learning aims to embed high-dimensional data into a lower dimensional Euclidean space by learning a coordinate chart, but it requires that the entire manifold can be embedded in a single chart. This is impossible for manifolds wit...
-
In vision-based reinforcement learning (RL) tasks, it is prevalent to assign auxiliary tasks with a surrogate self-supervised loss so as to obtain more semantic representations and improve sample efficiency. However, abundant information in self-supervised auxiliary tasks has been disregarded, since the representation learning part and the decision-making part are separated. To sufficiently utiliz...
-
We consider real-world reinforcement learning (RL) of robotic manipulation tasks that involve both visuomotor skills and contact-rich skills. We aim to train a policy that maps multimodal sensory observations (vision and force) to a manipulator's joint velocities under practical considerations. We propose to use offline samples to learn a set of general value functions (GVFs) that make counterfact...
-
Recent work on using natural language to specify commands to robots has grounded that language to LTL. However, mapping natural language task specifications to LTL task specifications using language models require probability distributions over finite vocabulary. Existing state-of-the-art methods have extended this finite vocabulary to include unseen terms from the input sequence to improve output...
-
Reinforcement Learning is rapidly establishing itself as the foremost choice for optimization of sequential autonomous decision-making problems. Encumbered by its sample inefficiency, the extension of the field to large state space and dynamic environments remains an open problem. We present a novel concept that exploits abstract spatial symmetry in complex environments for extending the skills of...
-
Reproducing real world dynamics in simulation is critical for the development of new control and perception methods. This task typically involves the estimation of simu-lation parameter distributions from observed rollouts through an inverse inference problem characterized by multi-modality and skewed distributions. We address this challenging problem through a novel Bayesian inference approach th...
-
Kernel herding is a deterministic sampling algorithm designed to draw ‘super samples' from probability distributions when provided with their kernel mean embeddings in a reproducing kernel Hilbert space (RKHS). Empirical expectations of functions in the RKHS formed using these super samples tend to converge even faster than random sampling from the true distribution itself. Standard implementation...
-
In this paper, we deal with the problem of creating globally consistent pose graphs in a centralized multi-robot SLAM framework. For each robot to act autonomously, individual onboard pose estimates and maps are maintained, which are then communicated to a central server to build an optimized global map. However, inconsistencies between onboard and server estimates can occur due to onboard odometr...
-
In the presence of fast motion, point clouds obtained from mechanical spinning LiDAR can be easily distorted due to the slow scanning speed of the LiDAR. Existing LiDAR-only odometry algorithms generally ignore this distortion or compensate by linearly interpolating the estimated relative motion between scans. However, when there are abrupt and nonlinear motion changes, the linear interpolation me...
-
Depth acquisition with the active stereo camera is a challenging task for highly reflective objects. When setup permits, multi-view fusion can provide increased levels of depth completion. However, due to the slow acquisition speed of high-end active stereo cameras, collecting a large number of viewpoints for a single scene is generally not practical. In this work, we propose a next-best-view fram...
-
Sound is an essential navigation cue that intelligent robots can leverage for localizing sound-emitting targets. This work introduces a framework for the audio-aware navigation task of mobile robots equipped with a microphone array in a complex indoor environment. The robot initialized at a random starting position has to accurately localize a distant sound source and plan an optimal path towards ...
-
In this paper, we propose an efficient frontier detector method based on adaptive Rapidly-exploring Random Tree (RRT) for autonomous robot exploration. Robots can achieve real-time incremental frontier detection when they are exploring unknown environments. First, our detector adaptively adjusts the sampling space of RRT by sensing the surrounding environment structure. The adaptive sampling space...
-
From robotic space assistance to healthcare robotics, there is increasing interest in robots that offer adaptable levels of autonomy. In this paper, we propose an action representation and planning framework that is able to generate plans that can be executed with both shared control and supervised autonomy, even switching between them during task execution. The action representation - Constraint ...
-
Optical coherence tomography (OCT) has revolutionized diagnostics in ophthalmology. However, it requires pa-tient cooperation to fixate on multiple targets and stabilize their head utilizing both chin and forehead rests. Patient cooperation is particularly important for image montaging, where patients are asked to fixate on multiple targets to sequentially image different regions of interest on th...
-
This paper presents a novel, robust, and accurate three-dimensional (3D) rigid point set registration (PSR) method, which is achieved by generalizing the state-of-the-art (SOTA) Bayesian coherent point drift (BCPD) theory to the scenario that high-dimensional point sets(PSs) are aligned and that the anisotropic positional noise is considered. Our contributions in this paper are three folds. First,...
-
Despite significant developments in the design of surgical robots and automated techniques for objective evaluation of surgical skills, there are still challenges in ensuring safety in robot-assisted minimally-invasive surgery (RMIS). This paper presents a runtime monitoring system for the detection of executional errors during surgical tasks through the analysis of kinematic data. The proposed sy...
-
Real-time and precise prediction for traffic of networks is critically important for allocating the optimal computing/network resources based on users' business requirements, analyzing the network performance, and realizing intelligent congestion control and high-accuracy anomaly detection. The dramatic growth of users' applications significantly increases the volume, uncertainty, and complexity o...
-
Online state-time trajectory planning in highly dynamic environments remains an unsolved problem due to the curse of dimensionality of the state-time space. Existing state-time planners are typically implemented based on randomized sampling approaches or path searching on discrete graphs. The smoothness, path clearance, or planning efficiency is sometimes not satisfying. In this work, we propose a...
-
Minimum-time navigation within constrained and dynamic environments is of special relevance in robotics. Seeking time-optimality, while guaranteeing the integrity of time-varying spatial bounds, is an appealing trade-off for agile vehicles, such as quadrotors. State-of-the-art approaches, either assume bounds to be static and generate time-optimal trajectories offline, or compromise time-optimalit...
-
Ultrasound beam propulsion, a propulsion system that uses airborne ultrasound phased arrays (AUPAs) to propel a blimp in an indoor environment to propel a blimp, has advantages for operations near humans such as no audible noises and no risk of propeller strike. To achieve the high mobility with limited actuation force of AUPAs, the dynamics should be fully exploited. In this paper, we propose a t...
-
We present CineMPC, an algorithm to autonomously control a UAV-borne video camera in a nonlinear Model Predicted Control (MPC) loop. CineMPC controls both the position and orientation of the camera-the camera extrinsics-as well as the lens focal length, focal distance, and aperture-the camera intrinsics. While some existing solutions autonomously control the position and orientation of the camera,...
-
We present GTGraffiti, a graffiti painting system from Georgia Tech that tackles challenges in art, hardware, and human-robot collaboration. The problem of painting graffiti in a human style is particularly challenging and requires a system-level approach because the robotics and art must be designed around each other. The robot must be highly dynamic over a large workspace while the artist must w...
-
Neural Style Transfer (NST) refers to a class of algorithms able to manipulate an element, most often images, to adopt the appearance or style of another one. Each element is defined as a combination of Content and Style: the Content can be conceptually defined as the “what” and the Style as the “how” of said element. In this context, we propose a custom NST framework for transferring a set of sty...
-
In this paper, we propose a detumbling strategy that stabilizes the motion of a tumbling client satellite using an orbital servicing manipulator, which is the goal of the post-grasp phase. One of the critical aspects in this phase is ensuring that excessive contact forces are not generated at the grasp interface. In addition, space mission requirements might demand a nominal manipulator configurat...
-
Future Moon bases will likely be constructed using resources mined from the surface of the Moon. The difficulty of maintaining a human workforce on the Moon and communications lag with Earth means that mining will need to be conducted using collaborative robots with a high degree of autonomy. In this paper, we describe our solution for Phase 2 of the NASA Space Robotics Challenge, which provided a...
-
We present AstroLoc, an efficient and robust monocular visual-inertial graph-based localization system used by the Astrobee free-flying robots onboard the International Space Station (ISS). We provide a novel localization system that limits the traditionally higher computation times for graph-based localization systems and enables the resource constrained Astrobee robots to benefit from their incr...
-
We propose a system that uses semantic object detections to localize a microgravity free-flyer. Many applications require absolute localization in a known reference frame, such as the execution of waypoint trajectories defined by human operators. Classical geometric methods build a map of point features, which may not be able to be associated after lighting or environmental changes. By contrast, s...
-
Automotive mmWave radar has been widely used in the automotive industry due to its small size, low cost, and complementary advantages to optical sensors (e.g., cameras, LiDAR, etc.) in adverse weathers, e.g., fog, raining, and snowing. On the other side, its large wavelength also poses fundamental challenges to perceive the environment. Recent advances have made breakthroughs on its inherent drawb...
-
Semantic information and geometrical structures of a prior map can be leveraged in visual localization to bound drift errors and improve accuracy. In this paper, we propose SemLoc, a pure visual localization system, for accurate localization in a prior semantic map. To tightly couple semantic and structure information from prior maps, a hybrid constraint is presented by using the Dirichlet distrib...
-
We present a novel framework for floor plan-based, full six degree-of-freedom LiDAR localization. Our approach relies on robust ceiling and ground plane detection, which solves part of the pose and supports the segmentation of vertical structure elements such as walls and pillars. Our core contribution is a novel nearest neighbour data structure for an efficient look-up of nearest vertical structu...
-
In this paper, we present a low-latency odometry system designed for spinning lidars. Many existing lidar odometry methods wait for an entire sweep from the lidar before processing the data. This introduces a large delay between the first laser firing and its pose estimate. To reduce this latency, we treat the spinning lidar as a streaming sensor and process packets as they arrive. This effectivel...
-
Accurate camera pose estimation is a fundamental requirement for numerous applications, such as autonomous driving, mobile robotics, and augmented reality. In this work, we address the problem of estimating the global 6 DoF camera pose from a single RGB image in a given environment. Previous works consider every part of the image valuable for localization. However, many image regions such as the s...
-
Robust localization in dense urban scenarios using a low-cost sensor setup and sparse HD maps is highly relevant for the current advances in autonomous driving, but remains a challenging topic in research. We present a novel monocular localization approach based on a sliding-window pose graph that leverages predicted uncertainties for increased precision and robustness against challenging scenario...
-
Modular and truss robots offer the potential of high reconfigurability and great functional flexibility, but common implementations relying on rigid components often lead to highly complex actuation and control requirements. This paper introduces a new type of modular, compliant robot: TrussBot. TrussBot is composed of 3D-printed tetrahedral modules connected at the corners with compliant joints. ...
-
This paper outlines the design, specifications, and algorithms for a new modular self-reconfigurable robotic system; at its foundation is a novel modular magnetically geared linear actuator paired with a kinematic coupling connector. Motivating this work is the core idea that high performance actuators as well as inexpensive, precise and repeatable connectors are the key ingredients required for u...
-
Energy sharing in modular self-reconfigurable robots ensures the energy balance of the modules, thus allowing the system to work sustainably. This paper proposes an energy sharing mechanism for a novel modular self-reconfigurable robot that allows free connections among modules, termed as FreeBOT, such that each FreeBOT can share energy with peers through surface contact. Corresponding energy shar...
-
This paper proposes a novel freeform strut-node structured modular self-reconfigurable robot (MSRR) called FreeSN, consisting of strut and node modules. A node module is mainly a low-carbon steel spherical shell. A strut module contains two freeform connectors, which provide strong magnetic connections and flexible spherical motions. The FreeSN system shares the benefits of freeform connection and...
-
This paper introduces a cube-based reconfigurable robot that utilizes an electromagnet-based actuation framework to reconfigure in three dimensions via pivoting. While a variety of actuation mechanisms for self-reconfigurable robots have been explored, they often suffer from cost, complexity, assembly and sizing requirements that prevent scaled production of such robots. To address this challenge,...
-
This paper proposes a novel modular self-assembling, self-reconfiguring robot with the 3D continuous dock called “SnailBot”. SnailBot mainly consists of a spherical ferromagnetic shell and a six-wheel rocker chassis with embedded magnets. Unlike many other existing modular self-reconfigurable robots with fixed docking locations, SnailBot uses the 3D continuous dock to attach to its peers regardles...
-
In this paper, we present a heterogeneous robot swarm system that can physically couple with each other to form functional structures and dynamically decouple to perform individual tasks. The connection between robots can be formed with a passive coupling mechanism, ensuring minimum energy consumption during coupling and decoupling behavior. The heterogeneity of the system enables the robots to pe...
-
Modular robots are made up of a set of components which can be configured and reconfigured to form customized robots for a wide range of tasks. Fully utilizing the flexibility of modular robots is challenging, as it requires the identification of optimal modular designs for each given task, often with limited computation and time. Previous works in design automation achieve efficient run-times by ...
-
Dynamic movement primitives (DMPs) are a flexible trajectory learning scheme widely used in motion generation of robotic systems. However, existing DMP-based methods mainly focus on simple go-to-goal tasks. Motivated to handle tasks beyond point-to-point motion planning, this work presents temporal logic guided optimization of motion primitives, namely $\mathbf{PI}^{\mathbf{BB}-\mathbf{TL}}$ algor...
-
Current safety mechanisms implementing industry standards for human-robot coexistence separate humans and robots through caging. Other approaches allowing humans to enter the workspace of manipulators do not provide formal safety guarantees. Thus, this study aims to facilitate the widespread adoption of collaborative robots by presenting SaRA, an extensible tool that performs set-based reachabilit...
-
In this paper, we present an algorithm that computes funnels along trajectories of systems of ordinary differential equations. A funnel is a time-varying set of states containing the given trajectory, for which the evolution from within the set at any given time stays in the funnel. Hence it generalizes the behavior of single trajectories to sets around them, which is an important task, for exampl...
-
Inertial Navigation Systems (INS) are a key technology for autonomous vehicles applications. Recent advances in estimation and filter design for the INS problem have exploited geometry and symmetry to overcome limitations of the classical Extended Kalman Filter (EKF) approach that formed the mainstay of INS systems since the mid-twentieth century. The industry standard INS filter, the Multiplicati...
-
As robots gain capabilities to enter our humancentric world, they require formalism and algorithms that enable smart and efficient interactions. This is challenging, especially for robotic manipulators with complex tasks that may require collaboration with humans. Prior works approach this problem through reactive synthesis and generate strategies for the robot that guarantee task completion by as...
-
Planning problems can be solved not only by planners, but also by model checkers. While the former yield a plan that requires replanning as soon as any fault occurs, the latter provide a “universal” plan (a.k.a. strategy, policy, or controller) able to make decisions under all circumstances. One of the prohibitive aspects of the latter approach is stemming from this very advantage: since it is def...
-
Given a heterogeneous group of robots executing a complex task represented in Linear Temporal Logic, and a new set of tasks for the group, we define the task update problem and propose a framework for automatically updating individual robot tasks given their respective existing tasks and capabilities. Our heuristic, token-based, conflict resolution task allocation algorithm generates a near-optima...
-
The problem of combinatorial filter reduction arises from resource optimization in robots; it is one specific way in which automation can help to achieve minimalism, to build better robots. This paper contributes a new definition of filter minimization that is broader than its antecedents, allowing filters (input, output, or both) to be nondeterministic. This changes the problem considerably. Nond...
-
We are motivated by the problem of learning policies for robotic systems with rich sensory inputs (e.g., vision) in a manner that allows us to guarantee generalization to environments unseen during training. We provide a framework for providing such generalization guarantees by leveraging a finite dataset of real-world environments in combination with a (potentially inaccurate) generative model of...
-
This paper demonstrates how an efficient repre-sentation of the planned path using B-splines, and a construction procedure that takes advantage of the neural network's inductive bias, speed up both the inference and training of a DNN-based motion planner. We build upon our recent work on learning local car maneuvers from past experience using a DNN architecture, introducing a novel B-spline path c...
-
Optimal motion planning is a long-studied problem with a wide range of applications in robotics, from grasping to navigation. While sampling-based motion planning methods have made solving such problems significantly more feasible, these methods still often struggle in high-dimensional spaces wherein exploration is computationally costly. In this paper, we propose a new motion planning algorithm t...
-
Autonomous vehicle software is typically structured as a modular pipeline of individual components (e.g., perception, prediction, and planning) to help separate concerns into interpretable sub-tasks. Even when end-to-end training is possible, each module has its own set of objectives used for safety assurance, sample efficiency, regularization, or interpretability. However, intermediate objectives...
-
Geometric methods for solving open-world off-road navigation tasks, by learning occupancy and metric maps, provide good generalization but can be brittle in outdoor environments that violate their assumptions (e.g., tall grass). Learning-based methods can directly learn collision-free behavior from raw observations, but are difficult to integrate with standard geometry-based pipelines. This create...
-
Mixed integer convex and nonlinear programs, MICP and MINLP, are expressive but require long solving times. Recent work that combines learning methods on solver heuristics has shown potential to overcome this issue allowing for applications on larger scale practical problems. Gathering sufficient training data to employ these methods still present a challenge since getting data from traditional so...
-
Search missions require motion planning and navigation methods for information gathering that continuously replan based on new observations of the robot's surroundings. Current methods for information gathering, such as Monte Carlo Tree Search, are capable of reasoning over long horizons, but they are computationally expensive. An alternative for fast online execution is to train, offline, an info...
-
Aerial robots are increasingly being utilized for environmental monitoring and exploration. However, a key challenge is efficiently planning paths to maximize the information value of acquired data as an initially unknown environment is explored. To address this, we propose a new approach for informative path planning based on deep reinforcement learning (RL). Combining recent advances in RL and r...
-
Autonomous mobile manipulation robots that can collect trolleys are widely used to liberate human resources and fight epidemics. Most prior robotic trolley collection solutions only detect trolleys with 2D poses or are merely based on spe-cific marks and lack the formal design of planning algorithms. In this paper, we present a novel mobile manipulation system with applications in luggage trolley ...
-
When moving objects that are too bulky or heavy to be grasped or lifted, robotic manipulation can benefit from the object's interaction with the support surface and its natural dynamics under gravity. In this work, we show that such dynamic, underactuated manipulation capability can be acquired through reinforcement learning and deployed on real robot systems. First, we present a framework to lear...
-
This paper describes the development of a modular end-effector system (MEES) for autonomous robotic maintenance and repair tasks. The design consists of the following major components: Robot Side Mating Socket Module (RSMS), End-Effector Side Mating Socket Module (EEMS), the Modular Camera System (MCS), and Tool Holder/Changer unit. Multiple prototypes for each component have been manufactured, te...
-
ReachBot is a new concept for planetary exploration, consisting of a small body and long, lightweight extending arms loaded primarily in tension. The arms are equipped with spined grippers for anchoring on rock surfaces. The design and testing of a planar prototype is presented here. Experiments with rock grasping and coordinated locomotion illustrate the advantages of low inertia passive grippers...
-
Detection of rare objects (e.g., traffic cones, traffic barrels and traffic warning triangles) is an important perception task to improve the safety of autonomous driving. Training of such models typically requires a large number of annotated data which is expensive and time consuming to obtain. To address the above problem, an emerging approach is to apply data augmentation to automatically gener...
-
Liquid state estimation is important for robotics tasks such as pouring; however, estimating the state of transparent liquids is a challenging problem. We propose a novel segmentation pipeline that can segment transparent liquids such as water from a static, RGB image without requiring any manual annotations or heating of the liquid for training. Instead, we use a generative model that is capable ...
-
We propose a semi-supervised learning framework for monocular depth estimation. Compared to existing semi-supervised learning methods, which inherit limitations of both sparse supervised and unsupervised loss functions, we achieve the complementary advantages of both loss functions, by building two separate network branches for each loss and distilling each other through the mutual distillation lo...
-
Scene classification is a well-established area of computer vision research that aims to classify a scene image into pre-defined categories such as playground, beach and airport. Recent work has focused on increasing the variety of pre-defined categories for classification, but so far failed to consider two major challenges: changes in scene appearance due to lighting and open set classification (...
-
Autonomous vehicles must reason about spatial occlusions in urban environments to ensure safety without being overly cautious. Prior work explored occlusion inference from observed social behaviors of road agents, hence treating people as sensors. Inferring occupancy from agent behaviors is an inherently multimodal problem; a driver may behave similarly for different occupancy patterns ahead of th...
-
Underwater image enhancement has become an attractive topic as a significant technology in marine engi-neering and aquatic robotics. However, the limited number of datasets and imperfect hand-crafted ground truth weaken its robustness to unseen scenarios, and hamper the application to high-level vision tasks. To address the above limitations, we develop an efficient and compact enhancement network...
-
Human motion prediction is key to understand social environments, with direct applications in robotics, surveil-lance, etc. We present a simple yet effective pedestrian trajectory prediction model aimed at pedestrians' positions prediction in urban-like environments conditioned by the environment: map and surround agents. Our model is a neural-based architecture that can run several layers of atte...
-
Autonomous Underwater Vehicles (AUVs) are a vital element for ocean exploration in various applications; however, energy sustainability still limits long-term operations. An option to overcome this problem is using underwater docking for power and data transfer. To robustly guide an AUV into a docking station, we propose an underwater vision algorithm for short-distance detection. In this paper, w...
-
Constructing HD semantic maps is a central component of autonomous driving. However, traditional pipelines require a vast amount of human efforts and resources in annotating and maintaining the semantics in the map, which limits its scalability. In this paper, we introduce the problem of HD semantic map learning, which dynamically constructs the local semantics based on onboard sensor observations...
-
Seemingly in defiance of basic physics, cats consistently land on their feet after falling. In this paper, we design a controller that lands the Mini Cheetah quadruped robot on its feet as well. Specifically, we explore how trajectory optimization and machine learning can work together to enable highly dynamic bioinspired behaviors. We find that a reflex approach, in which a neural network learns ...
-
Haptic sensing has recently been used effectively for legged robot localization in extreme scenarios where cam-eras and LiDAR might fail, such as dusty mines and foggy sewers. However, existing haptic sensing mainly relies on supervised classification, with training and evaluation executed over explicit terrain classes. Defining classes is a significant limitation to real-world applications, where...
-
Four-legged animals are able to change their gaits adaptively for lower energy consumption. However, designing a robust controller for their robot counterparts with multi-modal locomotion remains challenging. In this paper, we present a hierarchical control framework that decomposes this challenge into two kinds of problems: high-level decision-making for gait selection and robust low-level contro...
-
Centroidal dynamics, which describes the overall linear and angular motion of a robot, is often used in locomotion generation and control of legged robots. However, the equation of centroidal dynamics contains nonlinear terms mainly caused by the robot's angular motion and needs to be linearized for deriving a linear model-predictive motion controller. This paper proposes a new linearization of th...
-
This paper presents a systematical approach to develop a novel reduced complexity quadruped (RCQ) robot designed for serpentine robotic tail research purposes. The critical design requirements are determined based on careful dynamic analysis and synthesis results. Guided by formulated design requirements and principles, a robot prototype was designed and built. The robot has an overall weight of 5...
-
Automatic recognition of surgical workflow is a growing area of interest with significant potential to become part of context-aware decision-support systems in future enhanced ORs and clinical suites. Applications range from post-operative analysis to intra-operative monitoring to providing automated assistance to the clinical staff. This work proposes, for the first time, automatic tool presence ...
-
This paper presents a novel manipulation method utilizing dynamic deformation of a flexible body with a structural anisotropy. Employing a spiral flexible body, a dynamic underactuated manipulation using its various vibrational patterns is proposed. First, the orbit of the tip of flexible body for the vibrational input to its root is theoretically derived. Subsequently, for flexible bodies with an...
-
Connection mechanisms are crucial in reconfigurable robots. In this work, we present a novel approach, based on the self-healing property of a hydrogel synthesized by our group, which allows us to easily attach and detach robotic modules using water as the only trigger element. Our connection mechanism does not need external energy to work and it is reversible and soft, being useful for soft modul...
-
Origami/kirigami robotics are opening a path that leads to lightweight, compact, and expandable robots. However, it is generally challenging to design agile motions for origami/kirigami robots due to their size and the intrinsic limitation of the materials. In this paper, we propose to use the bistability of the waterbomb base structure to generate the swift motion of the robots. We evaluate the b...
-
Soft robotics has seen an exponential growth in the past decade, in part because the transition to soft materials has made a wider range of applications possible. Tasks involving contact with fragile objects or unstructured environments are particularly amenable to devices based on soft materials. To date, research has primarily focused on the development of soft analogs to traditional sensors and...
-
Soft robot serial chain manipulators with the capability for growth, stiffness control, and discrete joints have the potential to approach the dexterity of traditional robot arms, while improving safety, lowering cost, and providing an increased workspace, with potential application in home environments. This paper presents an approach for design optimization of such robots to reach specified targ...
-
Repeated jumping is crucial to the mobility of jumping robots. In this paper, we extend upon the REBOund jumping robot design, an origami-inspired jumping robot that uses the Reconfigurable Expanding Bistable Origami (REBO) pattern as its body. The robot design takes advantage of the pattern's bistability to jump with controllable timing. For jump repeatedly, we also add self-righting legs that ut...
-
Recent research has shown that guiding sampling-based planners with sampling distributions, learned from previous experiences via density estimation, can significantly decrease computation times for motion planning. We propose an algorithm that can estimate the density from the experiences of a robot with different kinematic structure, on the same task. The method allows to generalize collected da...
-
Motion planning is a core problem in robotics, with a range of existing methods aimed to address its diverse set of challenges. However, most existing methods rely on complete knowledge of the robot environment; an assumption that seldom holds true due to inherent limitations of robot perception. To enable tractable motion planning for high-DOF robots under partial observability, we introduce BLIN...
-
Recent work has demonstrated that motion planners' performance can be significantly improved by retrieving past experiences from a database. Typically, the experience database is queried for past similar problems using a similarity function defined over the motion planning problems. However, to date, most works rely on simple hand-crafted similarity functions and fail to generalize outside their c...
-
One of the fundamental challenges in realizing the potential of legged robots is generating plans to traverse challenging terrains. Control actions must be carefully selected so the robot will not crash or slip. The high dimensionality of the joint space makes directly planning low-level actions from onboard perception difficult, and control stacks that do not consider the low-level mechanisms of ...
-
In order to safely and efficiently collaborate with humans, industrial robots need the ability to alter their motions quickly to react to sudden changes in the environment, such as an obstacle appearing across a planned trajectory. In Realtime Motion Planning, obstacles are detected in real time through a vision system, and new trajectories are planned with respect to the current positions of the ...
-
Tactical planning is crucial for safe and efficient driving on the highway. However, the problem is complicated by the uncertain intention of surrounding vehicles, as well as observation noise caused by measurement noise and perception errors. Rule-based tactical planning methods are ineffective in handling dynamic scenarios with uncertainty, and susceptible to observation noise. To tackle this pr...
-
We study the problem of safe multi-agent motion planning in cluttered environments. Existing multi-agent reinforcement learning-based motion planners only provide approximate safety enforcement. We propose a safe reinforcement learning algorithm that leverages single-agent reinforcement learning for target regulation and a subsequent convex optimization-based filtering that ensures the collective ...
-
Efficient Object Manipulation to an Arbitrary Goal Pose: Learning-Based Anytime Prioritized Planning
We focus on the task of object manipulation to an arbitrary goal pose, in which a robot is supposed to pick an assigned object to place at the goal position with a specific orientation. However, limited by the execution space of the manipulator with gripper, one-step picking, moving and releasing might be failed, where a reorientation object pose is required as a transition. In this paper, we prop...
-
Learning from demonstration (LfD) provides a fast, intuitive and efficient framework to program robot skills, which has gained growing interest both in research and industrial applications. Most complex manipulation tasks are long-term and involve a set of skill primitives. Thus it is crucial to have a reliable coordination scheme that selects the correct sequence of skill primitive and the correc...
-
This paper is focused on safe mapless navigation of mobile robots in unknown and possibly complex environments containing both internal and dynamic obstacles. We present a novel modular approach that combines the strengths of artificial potential functions (APF) with deep reinforcement learning. Differing from related work, the robot learns how to adjust the two input parameters of the APF control...
-
Next best view (NBV) is a technology that finds the best view sequence for sensor to perform scanning based on partial information, which is the core part for robot active reconstruction. Traditional works are mostly based on the evaluation of candidate views through time-consuming volu-metric transformation and ray casting, which heavily limits the applications of NBV. Recent deep learning based ...
-
In realistic applications of object search, robots will need to locate target objects in complex environments while coping with unreliable sensors, especially for small or hard-to-detect objects. In such settings, correlational information can be valuable for planning efficiently. Previous approaches that consider correlational information typically resort to ad-hoc, greedy search strategies. We i...
-
In this paper we address mobile manipulation planning problems in the presence of sensing and environmental uncertainty. In particular, we consider mobile sensing manipulators operating in environments with unknown geometry and uncertain movable objects, while being responsible for accomplishing tasks requiring grasping and releasing objects in a logical fashion. Existing algorithms either do not ...
-
Planning the motion of bearings-only sensors is critical for enabling accurate tracking of the positions of moving targets. In this paper, we demonstrate planning the observer's motion over horizons greater than one step for estimating an unknown and varying number of indistinguishable, maneuvering targets of interest using a probability hypothesis density (PHD) filter, with a Rériyi divergence re...
-
Hyperdimensional computing (HDC) is a brain-inspired computing paradigm that operates on pseudo-random hypervectors, an information-rich, hardware-efficient representation that is robust to noise and facilitates learning with limited training data. This work explores how robot navigation tasks can leverage the high-capacity hypervector representation to enable behavioral prioritization through a w...
-
We describe a task and motion planning architecture for highly dynamic systems that combines a domain-independent sampling-based deliberative planning algorithm with a global reactive planner. We leverage the recent development of a reactive, vector field planner that provides guarantees of reachability to large regions of the environment even in the face of unknown or unforeseen obstacles. The re...
-
Collaborative robots can help industry workers to improve their ergonomics. They can propose a safe and ergonomic posture to the workers to reduce the risk of musculoskeletal disorders. Proposing an ergonomic stance needs postural evaluation and optimization. To optimize the workers' posture, we need to run the optimization on a cost function representing the ergonomic status. The tabular ergonomi...
-
Much of what we do as humans is engage socially with other agents, a skill that robots must also eventually possess. We demonstrate that a rich theory of social interactions originating from microsociology can be formalized by extending a nested MDP where agents reason about arbitrary functions of each other's rewards. This extended Social MDP allows us to encode the five basic interactions that u...
-
This paper describes the development of a 3- stage signalling framework to trigger a social robot's bottom- up reactive behavior inspired by a biological model. In the first stage, low-level firing of stimuli due to external sources is constructed through perception grounding. This is followed by a saliency classifier which fires-up high level salient signals that require attention and are used to...
-
Robot-mediated interventions are one promising and novel approach for encouraging motor exploration in young children, but knowledge about the effectiveness of toy-like features for child-robot interaction is limited. We were interested in understanding the characteristics of current toys to inform the design of interactive abilities for assistive robots. This work first provides a systematic revi...
-
When humans control robot arms these robots often need to infer the human's desired task. Prior research on assistive teleoperation and shared autonomy explores how robots can determine the desired task based on the human's joystick inputs. In order to perform this inference the robot relies on an internal mapping between joystick inputs and discrete tasks: e.g., pressing the joystick left indicat...
-
A fruitful collaboration is based on the mutual knowledge of each other skills and on the possibility of communicating their own limits and proposing alternatives to adapt the execution of a task to the capabilities of the collaborators. This paper aims at reproducing such a scenario in a human-robot collaboration setting by proposing a novel communication control architecture. Exploiting control ...
-
We can make it easier for disabled users to control assistive robots by mapping the user's low-dimensional joystick inputs to high-dimensional, complex actions. Prior works learn these mappings from human demonstrations: a non-disabled human either teleoperates or kinesthetically guides the robot arm through a variety of motions, and the robot learns to reproduce the demonstrated behaviors. But th...
-
In recent years, a rising trend of floor cleaning robots could be observed in the consumer electronic market. Area coverage performance is a crucial factor that determines the overall productivity of a floor cleaning robot. Nevertheless, the area coverage performance of commercially available floor cleaning robots is limited due to narrow spaces resulting from complex furniture arrangements. Tradi...
-
Can a robot autonomously learn to design and construct a bridge from varying-sized blocks without a blueprint? It is a challenging task with long horizon and sparse reward - the robot has to figure out physically stable design schemes and feasible actions to manipulate and transport blocks. Due to diverse block sizes, the state space and action trajectories are vast to explore. In this paper, we p...
-
Realistic manipulation tasks require a robot to interact with an environment with a prolonged sequence of motor actions. While deep reinforcement learning methods have recently emerged as a promising paradigm for automating manipulation behaviors, they usually fall short in long-horizon tasks due to the exploration burden. This work introduces Manipulation Primitive-augmented reinforcement Learnin...
-
Applying reinforcement learning (RL) methods on robots typically involves training a policy in simulation and deploying it on a robot in the real world. Because of the model mismatch between the real world and the simulator, RL agents deployed in this manner tend to perform suboptimally. To tackle this problem, researchers have developed robust policy learning algorithms that rely on synthetic noi...
-
Present-day Deep Reinforcement Learning (RL) systems show great promise towards building intelligent agents surpassing human-level performance. However, the computational complexity associated with the underlying deep neural networks (DNNs) leads to power-hungry implementations. This makes deep RL systems unsuitable for deployment on resource-constrained edge devices. To address this challenge, we...
-
In this work, we address the problem of solving complex collaborative robotic tasks subject to multiple varying parameters. Our approach combines simultaneous policy blending with system identification to create generalized policies that are robust to changes in system parameters. We employ a blending network whose state space relies solely on parameter estimates from a system identification techn...
-
Building assistive interfaces for controlling robots through arbitrary, high-dimensional, noisy inputs (e.g., webcam images of eye gaze) can be challenging, especially when it involves inferring the user's desired action in the absence of a natural ‘default’ interface. Reinforcement learning from online user feedback on the system's performance presents a natural solution to this problem, and enab...
-
World models learn behaviors in a latent imagination space to enhance the sample-efficiency of deep reinforcement learning (RL) algorithms. While learning world models for high-dimensional observations (e.g., pixel inputs) has become practicable on standard RL benchmarks and some games, their effectiveness in real-world robotics applications has not been explored. In this paper, we investigate how...
-
Graph-based Cluttered Scene Generation and Interactive Exploration using Deep Reinforcement Learning
We introduce a novel method to teach a robotic agent to interactively explore cluttered yet structured scenes, such as kitchen pantries and grocery shelves, by leveraging the physical plausibility of the scene. We propose a novel learning framework to train an effective scene exploration policy to discover hidden objects with minimal interactions. First, we define a novel scene grammar to represen...
-
A practical approach to learning robot skills, often termed sim2real, is to train control policies in simulation and then deploy them on a real robot. Popular sim2real techniques build on domain randomization (DR) - training the policy on diverse randomly generated domains for better generalization to the real world. Due to the large number of hyper-parameters in both the policy learning and DR al...
-
Quality Diversity (QD) algorithms in population-based reinforcement learning aim to optimize agents' returns and diversity among the population simultaneously. It is conducive to solving exploration problems in reinforcement learning and potentially getting multiple good and diverse strategies. However, previous methods typically define behavioral embedding in action space or outcome space, which ...
-
This paper introduces an efficient direct visual odometry (VO) algorithm using points and lines. Pixels on lines are generally adopted in direct methods. However, the original photometric error is only defined for points. It seems difficult to extend it to lines. In previous works, the collinear constraints for points on lines are either ignored [1] or introduce heavy computational load into the r...
-
Scene graphs represent the key components of a scene in a compact and semantically rich way, but are difficult to build during incremental SLAM operation because of the challenges of robustly identifying abstract scene elements and optimising continually changing, complex graphs. We present a distributed, graph-based SLAM framework for incrementally building scene graphs based on two novel compone...
-
We present a novel interval-based visual-inertial LiDAR SLAM (i-VIL SLAM) method that solely assumes sensor errors to be bounded and propagates the error from the input sources to the estimated map and trajectory using interval analysis. The method allows us to restrict the solution set of the robot poses and the position of the landmarks to the set that is consistent with the measurements. If the...
-
In existing self-supervised depth and ego-motion estimation methods, ego-motion estimation is usually limited to only leveraging RGB information. Recently, several methods have been proposed to further improve the accuracy of self-supervised ego-motion estimation by fusing information from other modalities, e.g., depth, acceleration, and angular velocity. However, they rarely focus on how differen...
-
We present the HD Ground Database, a comprehensive database for ground texture based localization. It contains sequences of a variety of textures, obtained using a downward facing camera. In contrast to existing databases of ground images, the HD Ground Database is larger, has a greater variety of textures, and has a higher image resolution with less motion blur. Also, our database enables the fir...
-
Recently, the community has witnessed numerous datasets built for developing and testing state estimators. However, for some applications such as aerial transportation or search-and-rescue, the contact force or other disturbance must be perceived for robust planning and control, which is beyond the capacity of these datasets. This paper introduces a Visual-Inertial-Dynamical (VID) dataset, not onl...
-
This paper presents a long-range magnetic actuated and guided endoscope for uniport video-assisted thoracic surgery (VATS). In VATS, the incision is quite narrow and part of the chest wall may be very thick. So, the magnetic endoscope system is required to produce sufficient attractive force at a considerable distance with a compact dimension. In this paper, a magnetic endoscope system is develope...
-
Presented is a new 10-mm diameter wrist designed for robotic surgery. Featuring greater dexterity than current designs, entirely new procedures may be possible. An innovative, parallel mechanism, it offers 180 degrees of singularity-free pitch/yaw motion. The wrist is also a new form of high angulation, constant velocity, universal joint and is capable of continuous 360 degrees of roll rotation at...
-
Automatic laparoscope motion control is fundamentally important for surgeons to efficiently perform operations. However, its traditional control methods based on tool tracking without considering information hidden in surgical scenes are not intelligent enough, while the latest supervised imitation learning (IL)-based methods require expensive sensor data and suffer from distribution mismatch issu...
-
One of the major functions brought by robots in Minimally Invasive Surgery is endoscope holding. This consists, for the user, in placing the camera at a desired location which the robot will maintain still once he/she releases it. This behavior is usually achieved with rigid position servoing, leading to possibly high forces generated and safety issues. Model-based weight compensation is an altern...
-
Human-robot shared control, which integrates the advantages of both humans and robots, is an effective approach to facilitate efficient surgical operation. Learning from demonstration (LfD) techniques can be used to automate some of the surgical sub tasks for the construction of the shared control mechanism. However, a sufficient amount of data is required for the robot to learn the manoeuvres. Us...
-
Retinal surgery is a complex medical procedure that requires exceptional expertise and dexterity. For this purpose, several robotic platforms are currently under development to enable or improve the outcome of microsurgical tasks. Since the control of such robots is often designed for navigation inside the eye in proximity to the retina, successful trocar docking and insertion of the instrument in...
-
Maneuverable multicamera systems offer potential benefits in abdominal minimally-invasive procedures, including multi-view scene reconstruction and optimal viewpoint capture. Effective autonomous movement and re-positioning of such systems, however, remains an open challenge due to dynamic motion constraints, deforming surgical scenes, and visual artifacts such as motion blur, specular reflections...
-
Recent advances in medical technology have produced a number of flexible instruments that are capable of traversing non-linear paths. This is of special interest in the field of neurosurgery. However, the non-rigid instruments have the disadvantage that path planning becomes increasingly difficult. In addition to anatomical risk factors, the mechanical properties and constraints of the specific in...
-
ComOpT is an open-source research tool for coverage-driven testing of autonomous driving systems, focusing on planning and control. Starting with (i) a meta-model characterizing discrete conditions to be considered and (ii) constraints specifying the impossibility of certain combinations, ComOpT first generates constraint-feasible abstract scenarios while maximally increasing the coverage of k-way...
-
Data-driven simulators promise high data-efficiency for driving policy learning. When used for modelling interactions, this data-efficiency becomes a bottleneck: small underlying datasets often lack interesting and challenging edge cases for learning interactive driving. We address this challenge by proposing a data-driven simulation engine† that uses inpainted ado vehicles for learning robust dri...
-
In this paper, a Deep Neural Network is trained using Reinforcement Learning in order to drift on arbitrary trajectories which are defined by a sequence of waypoints. In a first step, a highly accurate vehicle simulation is used for the training process. Then, the obtained policy is refined and validated on a self-built model car. The chosen reward function is inspired by the scoring process of re...
-
Visual perception of an autonomous vehicle is a crucial component of autonomous driving technologies. While visual perception research has achieved promising performance in recent years, modern methods are mostly trained, applied, and tested on single clean images. Recently, deep learning-based perception methods have addressed multiple degrading effects to reflect real-world bad weather cases, bu...
-
3D LiDAR is widely used in autonomous systems such as self-driving cars and autonomous robots because it provides accurate 3D point clouds of the surrounding environment under harsh conditions. However, a high-resolution LiDAR is expensive and bulky. Although a low-resolution LiDAR is compact and affordable, the obtained point clouds are so sparse that it is difficult to extract features that are ...
-
Predicting the motion of surrounding vehicles is essential for autonomous vehicles, as it governs their own motion plan. Current state-of-the-art vehicle prediction models heavily rely on map information. In reality, however, this information is not always available. We therefore propose CRAT-Pred, a multi-modal and non-rasterization-based trajectory prediction model, specifically designed to effe...
-
Accurately predicting the possible behaviors of traffic participants is an essential capability for autonomous vehicles. Since autonomous vehicles need to navigate in dynamically changing environments, they are expected to make accurate predictions regardless of where they are and what driving circumstances they encountered. Therefore, generalization capability to unseen domains is crucial for pre...
-
Predicting the future behavior of road users is one of the most challenging and important problems in autonomous driving. Applying deep learning to this problem requires fusing heterogeneous world state in the form of rich perception signals and map information, and inferring highly multi-modal distributions over possible futures. In this paper, we present MultiPath++, a future prediction model th...
-
This paper develops an adaptive PID autotuner for multicopters, and presents simulation and experimental results. The autotuner consists of adaptive digital control laws based on retrospective cost adaptive control implemented in the PX4 flight stack. A learning trajectory is used to optimize the autopilot during a single flight. The autotuned autopilot is then compared with the default PX4 autopi...
-
Flying robots are usually rather delicate and require protective enclosures when facing the risk of collision, while high complexity and reduced payload are recurrent problems with collision-resilient flying robots. Inspired by arthropods' exoskeletons, we design a simple, open source, easily manufactured, semi-rigid structure with soft joints that can withstand high-velocity impacts. With an exos...
-
The visible capability is critical in many robot applications, such as inspection and surveillance, etc. Without the assurance of the visibility to targets, some tasks end up not being complete or even failing. In this paper, we propose a visibility guaranteed planner by star-convex constrained optimization. The visible space is modeled as star convex polytope (SCP) by nature and is generated by f...
-
Robot grasping applications are faced with challenges and limitations leading to errors that affect their performance and accuracy. Although these errors are reduced in expensive industrial systems, low-cost robots are still prone to inaccurate perception and execution due to their limited hardware and software capabilities. To mitigate these challenges and limitations, this work develops a Joint-...
-
Human-Robot Collaboration (HRC) has become a major trend in robotics in recent years with the idea of combining the strengths from both humans and robots. In order to share the work to be done, many task planning approaches have been implemented. However, they don't fully satisfy the required adaptability in human-robot collaborative tasks, with most approaches not considering neither the state of...
-
The problem addressed in this work is the arbitration of the role between a robot and a human during physical Human-Robot Interaction, sharing a common task. The system is modeled as a Cartesian impedance, with two separate external forces provided by the human and the robot. The problem is then reformulated as a Cooperative Differential Game, which possibly has multiple solutions on the Pareto fr...
-
Cell fusion has been widely applied in scientific research for cancer immunotherapy, antibody production, and nuclear reprogramming of somatic cells, and therefore the cell fusion technique that enable us to precisely control the fusion process with high throughput manner has been desired. Here, we present a novel microfluidic method for automatic cell pairing by microdroplets, separation of dropl...
-
Soft tissue cutting is used for incision, separation and removal of tissues or cells. Due to high deformation of soft tissues resulting from their viscosity and elasticity, it is challenging to accurately cut the tissue along a desired path and control the force applied to the tissue for reducing invasiveness, especially at the microscale. This paper presents a robotic biopsy system for cutting an...
-
Macrophage immunotherapy is a promising clinical approach to treat cancer. However, low targeting efficiency severely limits the immunotherapeutic effect of macrophages. Here, we report a unique macrophage robot that can target and kill cancer cells using a combination of external acoustic and magnetic fields. First, the inactive macrophages (Mø) are magnetized by endocytosis of the $\gamma$-Fe2O3...
-
Wire arc additive manufacturing is a metal additive manufacturing process in which the material is deposited using arc welding technology. It is gaining popularity due to high material deposition rates and faster build time. It is en-abled using robotic manipulators and can build relatively large-scale parts faster when compared with other metal additive manufacturing processes. However, the size ...
-
We revisit the problem of efficiently leveraging prior map information within a visual-inertial estimation framework. The use of traditional landmark-based maps with 2D-to-3D measurements along with the recently introduced keyframe-based maps with 2D-to-2D measurements are inves-tigated. The full joint estimation of the prior map is compared within a visual-inertial simulator to the Schmidt-Kalman...
-
3D environment mapping has been actively stud-ied recently with the development of autonomous driving and augmented reality. Although many image-based methods are proposed due to their convenience and flexibility compared to other complex sensors, few works focus on fixing the inherent scale ambiguity of image-based methods and registering the reconstructed structure to the real-world 3D map, whic...
-
Emergence of massive dynamic objects will diversify spatial structures when robots navigate in urban environments. Therefore, the online removal of dynamic objects is critical. In this paper, we introduce a novel online removal framework for highly dynamic urban environments. The framework consists of the scan-to-map front-end and the map-to-map back-end modules. Both the front- and back-ends deep...
-
Long-term 3D map management is a fundamental capability required by a robot to reliably navigate in the non-stationary real-world. This paper develops open-source, modular, and readily available LiDAR-based lifelong mapping for urban sites. This is achieved by dividing the problem into successive subproblems: multi-session SLAM (MSS), high/low dynamic change detection, and positive/negative change...
-
Computing consumes a significant portion of energy in many robotics applications, especially the ones involving energy-constrained robots. In addition, memory access accounts for a significant portion of the computing energy. For mapping a 3D environment, prior approaches reduce the map size while incurring a large memory overhead used for storing sensor measurements and temporary variables during...
-
Global registration using 3D point clouds is a crucial technology for mobile platforms to achieve localization or manage loop-closing situations. In recent years, numerous researchers have proposed global registration methods to address a large number of outlier correspondences. Unfortunately, the degeneracy problem, which represents the phenomenon in which the number of estimated inliers becomes ...
-
For robotic interaction in environments shared with other agents, access to volumetric and semantic maps of the scene is crucial. However, such environments are inevitably subject to long-term changes, which the map needs to account for. We thus propose panoptic multi-TSDFs as a novel representation for multi-resolution volumetric mapping in changing environments. By leveraging high-level informat...
-
It is well known that visual SLAM systems based on dense matching are locally accurate but are also susceptible to long-term drift and map corruption. In contrast, feature matching methods can achieve greater long-term consistency but can suffer from inaccurate local pose estimation when feature information is sparse. Based on these observations, we propose an RGB-D SLAM system that leverages the ...
-
Dynamic Object-aware SLAM (DOS) exploits object-level information to enable robust motion estimation in dynamic environments. Existing methods mainly focus on identifying and excluding dynamic objects from the optimization. In this paper, we show that feature-based visual SLAM systems can also benefit from the presence of dynamic articulated objects by taking advantage of two observations: (1) The...
-
Environment modeling is the backbone of how autonomous agents understand the world, and therefore has significant implications for decision-making and verification. Motivated by the success of relational mapping tools such as Lanelet2, we present the Dynamic Relation Graph (DRG). The DRG is a novel method for extending prior relational maps to include online observations, creating a unified en-vir...
-
This paper presents a physical interface for collaborative mobile manipulators in industrial manufacturing and logistics applications. The proposed work builds on our earlier MOCA-MAN interface, through which an operator could be physically coupled to a mobile manipulator to be assisted in performing daily activities. The previous interface was based on a magnetic clamp attached to one arm of the ...
-
Despite the advances in the field of human-robot interface (HRI) based on biological neural signal, the use of the sole electroencephalography (EEG) signal to help robotic exoskeleton predict the limb movement is currently no mature in rehabilitation training, due to its unreliability. Multimodal HRI represents a very recent solution to enhance the performance of single modal HRI. These HRI normal...
-
Parkinson's disease is a severe neurodegenerative disorder that affects sensorimotor control. In particular, several gait impairments are reported, including a decrease of long-range autocorrelations in stride duration time series. This complex statistics is potentially a biomarker of the risk of falling. This paper aims at developing model-based predictions about the loss of long-range autocorrel...
-
Robot grasping and manipulation allow robots to interact with their environments and execute a plethora of complex tasks that require increased dexterity (e.g., open a door, push buttons, collect and transpose objects, etc.). Collecting data of such activities is of paramount importance as it allows roboticists to create new methods and models that will facilitate the execution of sophisticated ta...
-
In order to provide therapy in a functional context, controls for wearable robotic orthoses need to be robust and intuitive. We have previously introduced an intuitive, user-driven, EMG-based method to operate a robotic hand orthosis, but the process of training a control that is robust to concept drift (changes in the input signal) places a substantial burden on the user. In this paper, we explor...
-
MyoSim: Fast and physiologically realistic MuJoCo models for musculoskeletal and exoskeletal studies
Owing to the restrictions of live experimentation, musculoskeletal simulation models play a key role in biological motor control studies and investigations. Successful results of which are then tried on live subjects to develop treatments as well as robot aided rehabilitation procedures for addressing neuromusculoskeletal anomalies ranging from limb loss, to tendinitis, from sarcopenia to brain an...
-
Wearable robots are limited by their actuators performances because they must bear the weight of their own power system and energy source. This paper explores the idea of leveraging hybrid modes to meet multiple operating points with a lightweight and efficient system by using hydraulic valves to dynamically reconfigure the connections of a hydrostatic actuator. The analyzed opportunities consist ...
-
This paper describes rubble recognition using a depth image sensor and an automatic rubble crushing system using a construction machine for automatic rubble crushing at a building demolition site. Depth Distribution Split Labeling (DDSL) is proposed to recognize irregularly shaped rubble using depth images and to identify the largest rubble in the workspace. In DDSL, we focused on the fact that th...
-
Even if UAVs undoubtedly had a profound effect on the visual inspection capabilities of transmission lines, rolling robots, especially for bundled configurations, will still play an extensive role in the maintenance of these strategic assets. As such, LineRanger is among the most efficient and capable wheeled platform, that can travel at an average speed of 8 km/h. In this paper, LineRanger mechan...
-
Traversing soft soils represents a major concern of planetary rover missions. In this paper, we present a new chassis mechanism capable of a crawling gait that enhances trafficability on soft soil while relying on as few actuators as possible. Articulated by two actuated joints, MARCEL is a four-wheeled rover chassis which name stands for Mobile Active Rover Chassis for Enhanced Locomotion. MARCEL...
-
Heterogeneous robotic systems in the field often encounter bodies of water with unknown traversability properties. One approach to measuring depth, current, soil composition, etc. is via an in situ underwater sensor being dragged by cable attached to a maneuvering airborne multicopter - which entails a novel motion planning and control problem with mixed resistive media. In this work we propose a ...
-
Robotic systems such as unmanned ground vehicles (UGVs) often depend on GPS for navigation in outdoor environments. In GPS-denied environments, one approach to maintain a global state estimate is localizing based on preexisting georeferenced aerial or satellite imagery. However, this is inherently challenged by the significantly differing perspectives between the UGV and reference images. In this ...
-
Fast, autonomous flight in unstructured, cluttered environments such as forests is challenging because it requires the robot to compute new plans in realtime on a computationally-constrained platform. In this paper, we enable this capability with a search-based planning framework that adapts sampling density in realtime to find dynamically-feasible plans while remaining computationally tractable. ...
-
Testing and evaluation of field robotic systems requires both experimentation in representative conditions and human supervision to effectively assess components, manage risk, and interpret results. Due to the complexity of robotic sys-tems, we argue this experimentation should be done adaptively by using insights gained from previous trials. Furthermore, we envision an advisory system that could ...
-
Stabilising an inverted pendulum on a cart is a well-known control problem. This paper proposes the mechan-ical and control design for solving the oscillation problem of a variable-length flexible beam mounted on a mobile robot. The system under consideration is the robot PovRob, used at the European Organization for Nuclear Research (CERN) for visual and remote inspection tasks of particle accele...
-
This paper considers the problem of prolonged occlusions on navigation sensors due to dust, smudges, soils, etc. Such uncontrollable occlusions often cause lower visibility as well as higher uncertainty that require considerably sophisticated behavior. To secure visibility (i.e., confidence about the world), we propose a confidence-based navigation method that encourages the robot to explore the u...
-
6-DoF grasp pose detection of multi-grasp and multi-object is a challenge task in the field of intelligent robot. To imitate human reasoning ability for grasping objects, data driven methods are widely studied. With the introduction of large-scale datasets, we discover that a single physical metric usually generates several discrete levels of grasp confidence scores, which cannot finely distinguis...
-
In this work, we focus on multi-step manipulation tasks that involve long-horizon planning and considers progress reversal. Such tasks interlace high-level reasoning that consists of the expected states that can be attained to achieve an overall task and low-level reasoning that decides what actions will yield these states. We propose a sample efficient Previous Action Conditioned Robotic Manipula...
-
Robots need the capability of placing objects in arbitrary, specific poses to rearrange the world and achieve various valuable tasks. Object reorientation plays a crucial role in this as objects may not initially be oriented such that the robot can grasp and then immediately place them in a specific goal pose. In this work, we present a vision-based manipulation system, ReorientBot, which consists...
-
While deep learning has enabled significant progress in designing general purpose robot grasping systems, there remain objects which still pose challenges for these systems. Recent work on Exploratory Grasping has formalized the problem of systematically exploring grasps on these adversarial objects and explored a multi-armed bandit model for identifying high-quality grasps on each object stable p...
-
Manipulating deformable objects, such as ropes and clothing, is a long-standing challenge in robotics, because of their large degrees of freedom, complex non-linear dynamics, and self-occlusion in visual perception. The key difficulty is a suitable representation, rich enough to capture the object shape, dynamics for manipulation and yet simple enough to be estimated reliably from visual observati...
-
If robots could reliably manipulate the shape of 3D deformable objects, they could find applications in fields ranging from home care to warehouse fulfillment to surgical assistance. Analytic models of elastic, 3D deformable objects require numerous parameters to describe the potentially infinite degrees of freedom present in determining the object's shape. Previous attempts at performing 3D shape...
-
This paper introduces the task of Planar Robot Casting (PRC): where one planar motion of a robot arm holding one end of a cable causes the other end to slide across the plane toward a desired target. PRC allows the cable to reach points beyond the robot workspace and has applications for cable management in homes, warehouses, and factories. To efficiently learn a PRC policy for a given cable, we p...
-
The food packaging industry handles an immense variety of food products with wide-ranging shapes and sizes, even within one kind of food. Menus are also diverse and change frequently, making automation of pick-and-place difficult. A popular approach to bin-picking is to first identify each piece of food in the tray by using an instance segmentation method. However, human annotations to train these...
-
Manipulating objects with dexterity requires timely feedback that simultaneously leverages the senses of vision and touch. In this paper, we focus on the problem setting where both visual and tactile sensors provide pixel-level feedback for Visuotactile reinforcement learning agents. We investigate the challenges associated with multimodal learning and propose several improvements to existing RL m...
-
Recently simulation methods have been developed for optical tactile sensors to enable the Sim2Real learning, i.e., first training models in simulation before deploying them on a real robot. However, some artefacts in real objects are unpredictable, such as imperfections caused by fabrication processes, or scratches by natural wear and tear, and thus cannot be represented in the simulation, resulti...
-
In the literature on object grasping, the robot often determines the grasp point and posture from visual information. They predict the grasping point uniquely from the object's shape characteristics. However, as a practical matter, there are cases where there are constraints on grasp point due to the object states, the limitation of the robot's hardware and the surrounding environment. In this stu...
-
This paper proposes a quadratic programming (QP)-based variable impedance control (VIC) algorithm to solve contact-rich trajectory tracking problems with impedance, position and velocity constraints. To the best of our knowledge, the impedance constraints which are significant to ensure the worst contact compliance have never been considered in other previous works. To handle the impedance constra...
-
Stable contact and safe responses to the collision have been studied to develop interactive robots such as service and collaborative robots. Stable and safe interactions are usually achieved through the inherent compliance of a motion controller with external torque estimation. However, a fixed control gain would sacrifice either compliance or position tracking performance. Additionally, external ...
-
The hierarchical quadratic programming (HQP) is commonly applied to consider strict hierarchies of multi-tasks and robot's physical inequality constraints during whole-body compliance. However, for the one-step HQP, the solution can oscillate when it is close to the boundary of constraints. It is because the abrupt hit of the bounds gives rise to unrealizable jerks and even infeasible solutions. T...
-
This paper proposes a parameterization method to represent SO (3) over multiple turns. This method is called a memory-based parameterization, because the idea is to integrate the past trajectory of exponential coordinates. The parameterization is consistent in the sense that the true rotation matrix can be reconstructed by using the exponential map. As an application of the proposed method, a 6D i...
-
In this work, a robot control method is proposed for dragging an object by applying top contact forces under unknown friction and object dynamics. This is a non-prehensile manipulation of an object that can enhance the grasping capabilities of a robotic manipulator in a plethora of grasping scenarios. In the proposed method, an initializing controller generates reference contact force trajectories...
-
Large efforts have focused on ensuring that the controllers for mobile service robots follow proxemics and other social rules to ensure both safe and socially acceptable distance to pedestrians. Nonetheless, involuntary contact may be unavoidable when the robot travels in crowded areas or when encountering adversarial pedestrians. Freezing the robot in response to contact might be detrimental to b...
-
We believe that the future of robot motion planning will look very different than how it looks today: instead of complex collision avoidance trajectories with a brittle dependence on sensing and estimation of the environment, motion plans should consist of smooth, simple trajectories and be executed by robots that are not afraid of making contact. Here we present a “contact-aware” controller which...
-
Ultrasound scanning is an imaging technique that aids medical professionals in diagnostics and interventional procedures. However, a trained human-in-the-loop (HITL) with a radiologist is required to perform the scanning procedure. We seek to create a novel ultrasound system that can provide imaging in the absence of a trained radiologist, say for patients in the field who suffered injuries after ...
-
State-of-the-art autonomous driving systems rely on high definition (HD) maps for localization and navigation. However, building and maintaining HD maps is time-consuming and expensive. Furthermore, the HD maps assume structured environment such as the existence of major road and lanes, which are not present in rural areas. In this work, we propose an end-to-end transformer networks based approach...
-
Autonomous robots deal with unexpected scenarios in real environments. Given input images, various visual perception tasks can be performed, e.g., semantic segmentation, depth estimation and normal estimation. These different tasks provide rich information for the whole robotic perception system. All tasks have their own characteristics while sharing some latent correlations. However, some of the ...
-
In this paper, we are interested in the problem of generating target grasps by understanding freehand sketches. The sketch is useful for the persons who cannot formulate language and the cases where a textual description is not available on the fly. However, very few works are aware of the usability of this novel interactive way between humans and robots. To this end, we propose a method to genera...
-
Super-resolution of LiDAR range images is crucial to improving many downstream tasks such as object detection, recognition, and tracking. While deep learning has made a remarkable advances in super-resolution techniques, typical convolutional architectures limit upscaling factors to specific output resolutions in training. Recent work has shown that a continuous representation of an image and lear...
-
Incremental few-shot learning is highly expected for practical robotics applications. On one hand, robot is desired to learn new tasks quickly and flexibly using only few annotated training samples; on the other hand, such new additional tasks should be learned in a continuous and incremental manner without forgetting the previous learned knowledge dramatically. In this work, we propose a novel Cl...
-
We propose CLA-NeRF - a Category-Level Articulated Neural Radiance Field that can perform view synthesis, part segmentation, and articulated pose estimation. CLA-NeRF is trained at the object category level using no CAD models and no depth, but a set of RGB images with ground truth camera poses and part segments. During inference, it only takes a few RGB views (i.e., few-shot) of an unseen 3D obje...
-
Manipulating an articulated object requires perceiving its kinematic hierarchy: its parts, how each can move, and how those motions are coupled. Previous work has explored perception for kinematics, but none infers a complete kinematic hierarchy on never-before-seen object instances, without relying on a schema or template. We present a novel perception system that achieves this goal. Our system i...
-
Camera calibration is integral to robotics and computer vision algorithms that seek to infer geometric properties of the scene from visual input streams. In practice, calibration is a laborious procedure requiring specialized data collection and careful tuning. This process must be repeated whenever the parameters of the camera change, which can be a frequent occurrence for mobile robots and auton...
-
This paper studies the task of any objects grasping from the known categories by free-form language instructions. This task demands the technique in computer vision, natural language processing, and robotics. We bring these disciplines together on this open challenge, which is essential to human-robot interaction. Critically, the key challenge lies in inferring the category of objects from linguis...
-
Optical Coherence Tomography (OCT) is a rapidly growing and promising imaging technique, enabling non-invasive high-resolution visualization of biological tissues. Segmentation of tissue structures from OCT scans is essen-tial for disease diagnosis but remains challenging for the blurry boundaries and large volumes. Deep learning-based OCT segmentation algorithms always require large numbers of an...
-
As a basic part of the centroidal dynamics, an-gular momentum plays a critical role in humanoid motion control. Therefore, how to explicitly express and control an-gular momentum through whole-body motion is an important topic for researchers. This study discusses the selection of the generalized velocity corresponding to whole-body angular momentum. Based on the discussion, we present a method th...
-
Collaborative robots that operate alongside humans require the ability to understand their intent and forecast their pose. Among the various indicators of intent, the eye gaze is particularly important as it signals action towards the gazed object. By observing a person's gaze, one can effectively predict the object of interest and subsequently, forecast the person's pose. We leverage this and pre...
-
Inverse Optimal Control (IOC) is a popular method for human motion analysis. In the context of these methods it is necessary to pay attention to the reliability of the results. This paper proposes an approach based on the evaluation of Karush-Kuhn-Tucker conditions relying on a complete analysis with Singular Value Decomposition and provides a detailed analysis of reliability. With respect to a gr...
-
Music to dance for humanoid robots is an interesting task. Robot dance generation is challenging when considering music pieces, human dancer motions, and robot stability simultaneously. Previous methods rely on human-designed motion library or stability constraints for robot postures. Hence, dance generation for humanoid robots requires expert design, which can be time-consuming across different h...
-
The humanoid robot that can transform itself into a form according to its purpose requires whole-body motions with complex contact state transitions such as recovery from a fall and transition to the target form. To make the robot behavior in simulations closer to that in the real world for planning complex target trajectories, we need a platform that can measure the body stiffness during the moti...
-
Humanoid lower limbs with tactile cognition are crucial for future bipedal robots developing advanced bionic intelligence, such as owning autonomous reflexes and performing human-like actions. Most existing robotic lower limbs focus on providing physical support and mobility, with little work on more bionic DoFs or tactile sensing abilities that are more than significant for a fully humanoid syste...
-
It is well established that a stiff structure along with an optimal mass distribution are key features to perform dynamic movements, and parallel designs provide these characteristics to a robot. This work presents the new upper-body design of the humanoid robot RH5 named RH5 Manus with series-parallel hybrid design. The new design choices allow us to perform dynamic motions including tasks that i...
-
As the study of humanoid robots becomes a world-wide interdisciplinary research field, the demand for a cost-effective bipedal robot system capable of dynamic behaviors is growing exponentially. This paper presents a miniature bipedal robot named Bipedal Robot Unit with Compliance Enhanced (BRUCE). Each leg of BRUCE has five degrees of freedom (DoFs), which includes a spherical hip joint, a knee j...
-
Although human navigational intent inference has been studied in the literature, none have adequately considered both the dynamics that describe human motion and internal human parameters that may affect human navigational behaviour. In this paper, we propose a general probabilistic framework to infer the probability distribution over future navigational states of a human. Our framework incorporat...
-
In this paper, we revisit the problem of learning a stabilizing controller from a finite number of demonstrations by an expert. By focusing on feedback linearizable systems, we show how to combine expert demonstrations into a stabilizing controller, provided that demonstrations are sufficiently long and there are at least $n+1$ of them, where $n$ is the number of states of the system being control...
-
The ability to apply a previously-learned skill (e.g., pushing) to a new task (context or object) is an important requirement for new-age robots. An attempt is made to solve this problem in this paper by proposing a deep meta-imitation learning framework comprising of an attentive-embedding net-work and a control network, capable of learning a new task in an end-to-end manner while requiring only ...
-
This work aims to learn how to perform complex robot manipulation tasks that are composed of several, consecutively executed low-level sub-tasks, given as input a few visual demonstrations of the tasks performed by a person. The sub-tasks consist of moving the robot's end-effector until it reaches a sub-goal region in the task space, performing an action, and triggering the next sub-task when a pr...
-
When a robot observes another agent unexpectedly modifying their behavior, inferring the most likely cause is a valuable tool for maintaining safety and reacting appropriately. In this work, we present a novel method for inferring constraints that works on continuous, possibly sub-optimal demonstrations. We first learn a representation of the continuous-state maximum entropy trajectory distributio...
-
This work presents the dual benefit of integrating imitation learning techniques, based on the dynamical systems formalism, with the visual servoing paradigm. On the one hand, dynamical systems allow to program additional skills without explicitly coding them in the visual servoing law, but leveraging few demonstrations of the full desired behavior. On the other, visual servoing allows to consider...
-
$A$ core challenge for an autonomous agent acting in the real world is to adapt its repertoire of skills to cope with its noisy perception and dynamics. To scale learning of skills to long-horizon tasks, robots should be able to learn and later refine their skills in a structured manner through trajectories rather than making instantaneous decisions individually at each time step. To this end, we ...
-
The occupancy grid map is a critical component of autonomous positioning and navigation in the mobile robotic system, as many other systems' performance depends heavily on it. To guarantee the quality of the occupancy grid maps, researchers previously had to perform tedious manual recognition for a long time. This work focuses on automatic abnormal occupancy grid map recognition using the residual...
-
Deployment of deep neural networks (DNNs) for monocular depth estimation in safety-critical scenarios on resource-constrained platforms requires well-calibrated and efficient uncertainty estimates. However, many popular uncertainty estimation techniques, including state-of-the-art ensembles and popular sampling-based methods, require multiple inferences per input, making them difficult to deploy i...
-
Exploiting internal spatial geometric constraints of sparse LiDARs is beneficial to depth completion, however, has been not explored well. This paper proposes an efficient method to learn geometry-aware embedding, which encodes the local and global geometric structure information from 3D points, e.g., scene layout, object's sizes and shapes, to guide dense depth estimation. Specifically, we utiliz...
-
Depth images usually contain pixels with invalid measurements. This paper presents a deep learning approach that receives as input a partially-known volumetric model of the environment and a camera pose, and it predicts the probability that a pixel would contain a valid depth measurement if a camera was placed at the given pose. The proposed network architecture consists of a 3D Convolutional Neur...
-
In Human Robot Interaction (HRI) scenarios, robot systems would benefit from an understanding of the user's state, actions and their effects on the environments to enable better interactions. While there are specialised vision algorithms for different perceptual channels, such as objects, scenes, human pose, and human actions, it is worth considering how their interaction can help improve each oth...
-
Depth information is considered valuable as it describes geometric structures, which benefits various robotic tasks. However, the depth acquired by RGB-D sensors still suffers from two deficiencies, i.e., incompletion and noises. Previous methods complete depth by exploring hand-tuned models or raising surface assumptions, while nowadays, deep approaches intend to solve this problem with rendered ...
-
Due to overly relying on appearance information or adopting direct static feature fusion, most of the existing action recognition methods based on multi-modality have poor robustness and insufficient consideration of modality differences. To address these problems, we propose a two-stream adaptive weight integration network with a three-dimensional parallel attention module, PA-AWCNN. Firstly, a t...
-
In this paper, we propose a novel variable rate deep compression architecture that operates on raw 3D point cloud data. The majority of learning-based point cloud compression methods work on a downsampled representation of the data. Moreover, many existing techniques require training multiple networks for different compression rates to generate consolidated point clouds of varying quality. In cont...
-
With the development of sensing and communication technologies in networked cyber-physical systems (CPSs), multi-agent reinforcement learning (MARL)-based methodologies are integrated into the control process of physical systems and demonstrate prominent performance in a wide array of CPS domains, such as connected autonomous vehicles (CAVs). However, it remains challenging to mathematically chara...
-
Graph Neural Networks (GNNs) are a paradigm-shifting neural architecture to facilitate the learning of complex multi-agent behaviors. Recent work has demonstrated remarkable performance in tasks such as flocking, multi-agent path planning and cooperative coverage. However, the policies derived through GNN-based learning schemes have not yet been deployed to the real-world on physical multi-robot s...
-
This paper develops a decentralized approach to mobile sensor coverage by a multi-robot system. We consider a scenario where a team of robots with limited sensing range must position itself to effectively detect events of interest in a region characterized by areas of varying importance. Towards this end, we develop a decentralized control policy for the robots-realized via a Graph Neural Network-...
-
In this paper, we propose a novel decentralized method based on deep reinforcement learning using robot-level and target-level relational graphs, to solve the problem of multi-target encirclement with collision avoidance (MECA). Specifically, the robot-level relational graphs, composed of three heterogeneous relational graphs between each robot and other robots, targets and obstacles, are modeled ...
-
The problem of multi-robot navigation of connectivity maintenance is challenging in multi-robot applications. This work investigates how to navigate a multi-robot team in unknown environments while maintaining connectivity. We propose a reinforcement learning (RL) approach to develop a decentralized policy, which is shared among multiple robots. Given range sensor measurements and the positions of...
-
It is of great challenge, though promising, to coordinate collective robots for hunting an evader in a decentralized manner purely in light of local observations. In this paper, this challenge is addressed by a novel hybrid cooperative pursuit algorithm that combines reinforcement learning with the artificial potential field method. In the proposed algorithm, decentralized deep reinforcement learn...
-
This paper presents a novel graph reinforcement learning (RL) architecture to solve multi-robot task allocation (MRTA) problems that involve tasks with deadlines and workload, and robot constraints such as work capacity. While drawing motivation from recent graph learning methods that learn to solve combinatorial optimization (CO) problems such as multi-Traveling Salesman and Vehicle Routing Probl...
-
In operations of multi-agent teams ranging from homogeneous robot swarms to heterogeneous human-autonomy teams, unexpected events might occur. While efficiency of operation for multi-agent task allocation problems is the primary objective, it is essential that the decision-making framework is intelligent enough to manage unexpected task load with limited resources. Otherwise, operation effectivene...
-
We introduce a visually-guided task-and-motion planning benchmark, which we call the ThreeDWorld Trans-port Challenge. In this challenge, an embodied agent is spawned randomly in a simulated physical home environment and required to transport a small set of objects scattered around the house with containers. We build this benchmark challenge using the ThreeDWorld simulation: a virtual 3D environme...
-
In order to obtain a high-precision contact model that can properly describe the target soft tissue, this paper proposes a hybrid soft contact model based on a clustering algorithm ART-II, which selects the most suitable soft contact model according to the surgical environment. The least-square method is used to identify the parameters of the model online. In the experiments, different parts of an...
-
Manipulation involves a broad spectrum of skills, e.g., polishing, peeling, flipping, screwing, etc., requiring complex and delicate control over both force and position. This paper aims at designing an optimal haptic interface for providing a robot with direct demonstrations of human's innate intelligence in performing a wide range of force-based bimanual manipulation tasks. Based on the proprioc...
-
In both teleoperation in real space and exploration in virtual space, ‘passive’ and ‘active’ haptic feedback can help to improve the performance of the task, especially in object handover and exploring. However, the current wearable haptic devices are hard to display continuous omnidirectional motion feedback simultaneously, which makes it not yet achieved. In this study, we thus propose a cutaneo...
-
Virtual environments designed for haptic applications are usually rendered as a combination of spring and damper elements. The resulting haptic experience can be greatly enhanced by also adding virtual inertia, for example when interacting with mobile virtual objects. This paper analyzes the impact of implementing virtual inertia on haptic rendering stability. It describes the methodology followed...
-
Interactive robotic grasping using natural language is one of the most fundamental tasks in human-robot interaction. However, language can be a source of ambiguity, particularly when there are ambiguous visual or linguistic contents. This paper investigates the use of object attributes in disambiguation and develops an interactive grasping system capable of effectively resolving ambiguities via di...
-
Autonomous Exploration Development Environment is an open-source repository released to facilitate development of high-level planning algorithms and integration of com-plete autonomous navigation systems. The repository contains representative simulation environment models, fundamental navigation modules, e.g., local planner, terrain traversability analysis, waypoint following, and visualization t...
-
This article studies the tracking control of a quadrotor unmanned aerial vehicle (UAV) under time-varying external disturbances. An event-triggered sliding mode control (SMC) strategy is proposed by introducing a new triggering condition form of desired trajectory, quadrotor position, and velocity. In the sense of Lyapunov theory, the stability of the entire closed-loop control system is analyzed,...
-
Predicting the future motion of traffic agents is crucial for safe and efficient autonomous driving. To this end, we present PredictionNet, a deep neural network (DNN) that predicts the motion of all surrounding traffic agents together with the ego-vehicle's motion. All predictions are probabilistic and are represented in a simple top-down rasterization that allows an arbitrary number of agents. C...
-
Robustness and safety concerns of perception systems are of great importance for autonomous vehicle navigation applications. Recent researches demonstrate that the surrounding dynamic object detection results of current perception systems can be easily interfered or attacked to mislead the navigation performance of the victim vehicle. In this paper, we develop a GNN based relation learning network...
-
One of the challenges in vision-based driving trajectory generation is dealing with out-of-distribution scenarios. In this paper, we propose a domain generalization method for vision-based driving trajectory generation for autonomous vehicles in urban environments, which can be seen as a solution to extend the Invariant Risk Minimization (IRM) method in complex problems. We leverage an adversarial...
-
We introduce a motion forecasting (behavior prediction) method that meets the latency requirements for autonomous driving in dense urban environments without sacrificing accuracy. A whole-scene sparse input representation allows StopNet to scale to predicting trajectories for hundreds of road agents with reliable latency. In addition to predicting trajectories, our scene encoder lends itself to pr...
-
In this paper, we propose a novel network, Fusion-Net, which can estimate the extrinsic calibration matrix between LiDAR and a monocular RGB camera with high accuracy and robustness. FusionNet is a coarse-to-fine method, providing an online and end-to-end solution that can automatically detect and correct the decalibration without any specially designed targets or environments. First, the network ...
-
Cyclops, introduced in this paper, is an open research platform for everyone who wants to validate novel ideas and approaches in self-driving heavy-duty vehicle platooning. The platform consists of multiple 1/14 scale semi-trailer trucks equipped with associated computing, communication and control modules that enable self-driving on our scale proving ground. The perception system for each vehicle...
-
Providing safety guarantees for Autonomous Vehicle (AV) systems with machine-learning based controllers remains a challenging issue. In this work, we propose Simplex-Drive, a framework that can achieve runtime safety assurance for machine-learning enabled controllers of AVs. The proposed Simplex-Drive consists of an unverified Deep Reinforcement Learning (DRL)-based advanced controller (AC) that a...
-
Planning a safe trajectory for an ego vehicle through an environment with occluded regions is a challenging task. Existing methods use some combination of metrics to evaluate a trajectory, either taking a worst case view or allowing for some probabilistic estimate, to eliminate or minimize the risk of collision respectively. Typically, these approaches assume occluded regions of the environment ar...
-
Temporal patterns (how vehicles behave in our observed past) underline our reasoning of how people drive on the road, and can explain why we make certain predictions about interactions among road agents. In this paper we propose the ConceptNet trajectory predictor - a novel prediction framework that is able to incorporate agent interactions as explicit edges in a temporal knowledge graph. We demon...
-
Real-time kinodynamic trajectory planning in dy-namic environments is critical yet challenging for autonomous driving. In this paper, we propose an efficient trajectory plan-ning system for autonomous driving in complex dynamic sce-narios through iterative and incremental path-speed optimization. Exploiting the decoupled structure of the planning prob-lem, a path planner based on Gaussian process ...
-
This paper presents a novel gradient ascent al-gorithm and nonlinear programming algorithm for finite state controller policies in constrained partially observable Markov decision processes (CPOMDPs). A key component of the gradient ascent algorithm is a constraint projection to ensure constraints are satisfied. Both an optimal and an approximate projection are formally defined. A theoretical anal...
-
Agile navigation through uncertain and obstacle-rich environments remains a challenging task for autonomous mobile robots (AMR). For most AMR, obstacles are identified using onboard sensors, e.g., lidar or cameras. The effectiveness of these sensors may be severely limited, however, by occlusions introduced from the presence of other obstacles. The occluded area may contain obstacles, static or dy...
-
In this paper, we propose GOHOME, a method leveraging graph representations of the High Definition Map and sparse projections to generate a heatmap output representing the future position probability distribution for a given agent in a traffic scene. This heatmap output yields an unconstrained 2D grid representation of agent future possible locations, allowing inherent multimodality and a measure ...
-
We approach instantaneous mapping, converting images to a top-down view of the world, as a translation problem. We show how a novel form of transformer network can be used to map from images and video directly to an overhead map or bird's-eye-view (BEV) of the world, in a single end-to-end network. We assume a 1–1 correspondence between a vertical scanline in the image, and rays passing through th...
-
Panoptic segmentation aims to address semantic and instance segmentation simultaneously in a unified framework. However, an efficient solution of panoptic segmentation in applications like autonomous driving is still an open research problem. In this work, we propose a novel LiDAR-based panoptic system, called SMAC-Seg. We present a learnable sparse multi-directional attention clustering to segmen...
-
3D point cloud semantic segmentation is a challenging topic in the computer vision field. Most of the existing methods in literature require a large amount of fully labeled training data, but it is extremely time-consuming to obtain these training data by manually labeling massive point clouds. Addressing this problem, we propose a superpoint-guided semi-supervised segmentation network for 3D poin...
-
A key proficiency an autonomous mobile robot must have to perform high-level tasks is a strong understanding of its environment. This involves information about what types of objects are present, where they are, what their spatial extend is, and how they can be reached, i.e., information about free space is also crucial. Semantic maps are a powerful instrument providing such information. However, ...
-
This article investigates the real-time semantic segmentation in robot engineering applications based on the Broad Learning System (BLS), and a novel Multi-level Enhancement Layers Network (MELNet) based on BLS framework is proposed for real-time vision tasks in a complex street scene on the unmanned mobile robot. This network mainly solves two problems: (1) mitigating the contradiction between ac...
-
In this paper, we present a complete pipeline for 3D semantic mapping solely based on a stereo camera system. The pipeline comprises a direct sparse visual odometry frontend as well as a back-end for global optimization including GNSS integration, and semantic 3D point cloud labeling. We propose a simple but effective temporal voting scheme which improves the quality and consistency of the 3D poin...
-
LiDAR point cloud panoptic segmentation, including both semantic and instance segmentation, plays a critical role in meticulous scene understanding for autonomous driving. Existing 3D voxelized approaches either utilize 3D sparse convolution that only focuses on local scene understanding, or add extra and time-consuming PointNet branch to capture global feature structures. To address these limitat...
-
For achieving significant levels of autonomy, legged robot behaviors require perceptual awareness of both the terrain for traversal, as well as structures and objects in their surroundings for planning, obstacle avoidance, and high-level decision making. In this work, we present a perception engine for legged robots that extracts the necessary information for developing semantic, contextual, and m...
-
Referring expressions are commonly used when referring to a specific target in people's daily dialogue. In this paper, we develop a novel task of audio-visual grounding referring expression for robotic manipulation. The robot leverages both the audio and visual information to understand the referring expression in the given manipulation instruction and the corresponding manipulations are implement...
-
We present a framework for dynamic trajectory generation for autonomous navigation, which does not rely on HD maps as the underlying representation. High Definition (HD) maps have become a key component in most autonomous driving frameworks, which include complete road network information annotated at a centimeter-level that include traversable waypoints, lane information, and traffic signals. Ins...
-
Representations are crucial for a robot to learn effective navigation policies. Recent work has shown that mid-level perceptual abstractions, such as depth estimates or 2D semantic segmentation, lead to more effective policies when provided as observations in place of raw sensor data (e.g., RGB images). However, such policies must still learn latent three-dimensional scene properties from mid-leve...
-
This work presents a method for tactile classification of materials for virtual reality (VR) based robot teleoperation. In our system, a human-operator uses a remotely controlled robot-manipulator with an optical fibre-based tactile and proximity sensor to scan surfaces of objects in a remote environment. Tactile and proximity data and the robot's end-effector state feedback are used for the class...
-
In this paper, we present a semi-autonomous teleoperation framework for a pick-and-place task using an RGB-D sensor. In particular, we assume that the target object is located in a cluttered environment where both prehensile grasping and non-prehensile manipulation are combined for efficient teleoperation. A trajectory-based reinforcement learning is utilized for learning the non-prehensile manipu...
-
Bilateral teleoperation offers an intriguing solution towards shared autonomy with aerial vehicles in contact-based inspection and manipulation tasks. Omnidirectional aerial robots allow for full pose operations, making them particularly attractive in such tasks. Naturally, the question arises whether standard bilateral teleoperation methodologies are suitable for use with these vehicles. In this ...
-
Robot teleoperation via human motion tracking has been proven to be easy to learn, intuitive to operate, and facilitate faster task execution than existing baselines. However, precise control while performing the dexterous telemanipulation tasks is still a challenge. In this paper, we implement sensory augmentation in terms of haptic and augmented reality visual cues to represent four types of inf...
-
The ongoing COVID-19 pandemic has enforced governments across the world to impose social restrictions on the movement of people and confined them to their homes to avoid the spread of the disease. This not only forbids them from leaving their homes but also greatly reduces their physical activities. This situation has brought attention to virtual technologies such as virtual tours or telepresence ...
-
Movement primitives have the property to accom-modate changes in the robot state while maintaining attraction to the original policy. As such, we investigate the use of primitives as a blending mechanism by considering that state deviations from the original policy are caused by user inputs. As the primitive recovers from the user input, it implicitly blends human and robot policies without requir...
-
This paper proposes a teleoperation framework to exploit the maximum manipulation capability during teleoperation. Here, exploiting maximum manipulation capacity means that the robot moves with its maximum control input while not violating the given constraints, and it is a nonlinear optimization problem with nonlinear constraints which is hard to be solved. The proposed framework relaxes the opti...
-
The robustness of visual navigation policies trained through imitation often hinges on the augmentation of the training image-action pairs. Traditionally, this has been done by collecting data from multiple cameras, by using standard data augmentations from computer vision, such as adding random noise to each image, or by synthesizing training images. In this paper we show that there is another pr...
-
Vision-and-language navigation (VLN) has been an important task in the field of Robotics and Computer Vision. However, most existing vision-and-language navigation models only use features extracted from RGB observation as input, while robots can utilize depth sensors in the real world. Existing research has also shown that simply adding a depth stream to neural models could only provide a margina...
-
In this paper, we address the problem of sampling-based motion planning under motion and measurement un-certainty with probabilistic guarantees. We generalize traditional sampling-based, tree-based motion planning algorithms for deterministic systems and propose belief-A, a framework that extends any kinodynamical tree-based planner to the belief space for linear (or linearizable) systems. We intr...
-
Unmapped areas and aerodynamic disturbances render autonomous navigation with quadrotors extremely challenging. To fly safely and efficiently, trajectory planners and trackers must be able to navigate unknown environments with unpredictable aerodynamic effects in real-time. When encountering aerodynamic effects such as strong winds, most current approaches to quadrotor trajectory planning and trac...
-
We address the risk bounded trajectory optimization problem of stochastic nonlinear robotic systems. More precisely, we consider the motion planning problem in which the robot has stochastic nonlinear dynamics and uncertain initial locations, and the environment contains multiple dynamic uncertain obstacles with arbitrary probabilistic distributions. The goal is to plan a sequence of control input...
-
A new belief space planning algorithm, called covariance steering Belief RoadMap (CS-BRM), is introduced, which is a multi-query algorithm for motion planning of dynamical systems under simultaneous motion and observation uncertainties. CS-BRM extends the probabilistic roadmap (PRM) approach to belief spaces and is based on the recently developed theory of covariance steering (CS) that enables gua...
-
Unresolved data association in ambiguous and perceptually aliased environments leads to multi-modal hypotheses on both the robot's and the environment state. To avoid catastrophic results, when operating in such ambiguous environments, it is crucial to reason about data association within Belief Space Planning (BSP). However, explicitly considering all possible data associations, the number of hyp...
-
We present a planning framework for min-imising the deterministic worst-case error in sparse Gaus-sian process (GP) regression. We first derive a univer-sal worst-case error bound for sparse GP regression with bounded noise using interpolation theory on reproducing kernel Hilbert spaces (RKHSs). By exploiting the conditional inde-pendence (CI) assumption central to sparse GP regression, we show th...
-
Although experts carefully specify the high-level decision-making models in autonomous systems, it is infeasible to guarantee safety across every scenario during operation. We therefore propose a safety metareasoning system that optimizes the severity of the system's safety concerns and the interference to the system's task: the system executes in parallel a task process that completes a specified...
-
Autonomous robotic surgery requires deliberation, i.e. the ability to plan and execute a task adapting to uncer-tain and dynamic environments. Uncertainty in the surgical domain is mainly related to the partial pre-operative knowledge about patient-specific anatomical properties. In this paper, we introduce a logic-based framework for surgical tasks with deliberative functions of monitoring and le...
-
We present CCO-VOXEL: the very first chance-constrained optimization (CCO) algorithm that can compute trajectory plans with probabilistic safety guarantees in real-time directly on the voxel-grid representation of the world. CCO-VOXEL maps the distribution over the distance to the closest obstacle to a distribution over collision-constraint violation and computes an optimal trajectory that minimiz...
-
Efficient and reliable generation of global path plans are necessary for safe execution and deployment of autonomous systems. In order to generate planning graphs which adequately resolve the topology of a given environment, many sampling-based motion planners resort to coarse, heuristically-driven strategies which often fail to generalize to new and varied surroundings. Further, many of these app...
-
This paper investigates the usefulness of reasoning about the uncertain presence of obstacles during path planning, which typically stems from the usage of probabilistic occupancy grid maps for representing the environment when mapping via a noisy sensor like a stereo camera. The traditional planning paradigm prescribes using a hard threshold on the occupancy probability to declare that a cell is ...
-
LiDAR semantic segmentation essential for advanced autonomous driving is required to be accurate, fast, and easy-deployed on mobile platforms. Previous point-based or sparse voxel-based methods are far away from real-time applications since time-consuming neighbor searching or sparse 3D convolution are employed. Recent 2D projection-based methods, including range view and multi-view fusion, can ru...
-
The high performance of RGB-D based road segmentation methods contrasts with their rare application in commercial autonomous driving, which is owing to two reasons: 1) the prior methods cannot achieve high inference speed and high accuracy in both ways; 2) the different properties of RGB and depth data are not well-exploited, limiting the reliability of predicted road. In this paper, based on the ...
-
Panoptic segmentation of point clouds is one of the key challenges of 3D scene understanding, requiring the simultaneous prediction of semantics and object instances. Tasks like autonomous driving strongly depend on these information to get a holistic understanding of their 3D environment. This work presents a novel proposal free framework for lidar-based panoptic segmentation, which exploits thre...
-
Object rearrangement has recently emerged as a key competency in robot manipulation, with practical solutions generally involving object detection, recognition, grasping and high-level planning. Goal-images describing a desired scene configuration are a promising and increasingly used mode of instruction. A key outstanding challenge is the accurate inference of matches between objects in front of ...
-
Surgical instrument segmentation - in general a pixel classification task - is fundamentally crucial for promoting cognitive intelligence in robot-assisted surgery (RAS). However, previous methods are struggling with discriminating instrument types and instances. To address above issues, we explore a mask classification paradigm that produces per-segment predictions. We propose TraSeTR, a novel Tr...
-
A particular challenge for both autonomous and human driving is dealing with risk associated with dynamic occlusion, i.e., occlusion caused by other vehicles in traffic. Based on the theory of hypergames, we develop a novel multi-agent dynamic occlusion risk (DOR) measure for assessing situational risk in dynamic occlusion scenarios. Furthermore, we present a white-box, scenario-based, accelerated...
-
In physical human-robot interaction (pHRI), safety is a key requirement. As collisions between humans and robots can generally not be avoided, it must be ensured that the human is not harmed. The robot reflected mass, the contact geometry, and the relative velocity between human and robot are the parameters that have the most significant influence on human injury severity during a collision. The r...
-
Circular economy and agile manufacturing require a safe and efficient industrial robot system working in close human proximity. Although, close proximity local sensing enables safe collaboration with small cobots. However, they cannot ensure safety at high velocities with a heavy-duty industrial robot. Stereo-camera and 3D LiDAR-based touch-less global sensing methods exist, but they do not addres...
-
Human Motion Prediction (HMP) plays a crucial role in safe Human-Robot-Interaction (HRI). Currently, the majority of HMP algorithms are trained by massive pre-collected data. As the training data only contains a few pre-defined motion patterns, these methods cannot handle the unfamiliar motion patterns. Moreover, the pre-collected data are usually non-interactive, which does not consider the real-...
-
An outstanding challenge with safety methods for human-robot interaction is reducing their conservatism while maintaining robustness to variations in human behavior. In this work, we propose that robots use confidence-aware game-theoretic models of human behavior when assessing the safety of a human-robot interaction. By treating the influence between the human and robot as well as the human's rat...
-
How do we make concrete progress towards de-signing robots that can navigate ethically sensitive contexts? Almost two decades after the word ‘roboethics’ was coined, translating interdisciplinary roboethics discussions into techni-cal design still remains a daunting task. This paper describes our first attempt at addressing these challenges through a roboethics-themed design competition. The desig...
-
Multi-Robot Systems (MRS) present many advantages over single robots, e.g. improved stability and payload capacity. Being able to operate or teleoperate these systems is therefore of high interest in industries such as construction or logistics. However, controlling the collective motion of a MRS can place a significant cognitive burden on the operator. We present a Mixed Reality (MR) control inte...
-
Human-Swarm Interaction (HSI) is a fast-growing research area in swarm robotics. One challenging aspect of HSI is facilitating effective handling of the many degrees-of-freedom present in robot swarms by humans. One emergent option is the use of Augmented Reality (AR) systems to encode information. AR based interfaces can help provide human operators with visual cues about the swarm's states and c...
-
In this paper we investigate the influence interfaces and feedback have on human-robot trust levels when operating in a shared physical space. The task we use is specifying a “no-go” region for a robot in an indoor environment. We evaluate three styles of interface (physical, AR, and map-based) and four feedback mechanisms (no feedback, robot drives around the space, an AR “fence”, and the region ...
-
Simulation has been a necessary, safe testbed for robotics systems (RS). However, testing in simulation alone is not enough for robotic systems operating in close proximity, or interacting directly with, humans, because simulated humans are very limited. Furthermore, testing with real humans can be unsafe and costly. As recent advances in machine learning are being brought to physical robotic syst...
-
This paper presents a reinforcement learning method for object goal navigation (ObjNav) where an agent navigates in 3D indoor environments to reach a target object based on long-term observations of objects and scenes. To this end, we propose Object Memory Transformer (OMT) that consists of two key ideas: 1) Object-Scene Memory (OSM) that enables to store long-term scenes and object semantics, and...
-
We consider the problems of exploration and pointgoal navigation in previously unseen environments, where the spatial complexity of indoor scenes and partial observability constitute these tasks challenging. We argue that learning occupancy priors over indoor maps provides significant advantages towards addressing these problems. To this end, we present a novel planning framework that first learns...
-
Autonomous mobile robots deployed in outdoor environments must reason about different types of terrain for both safety (e.g., prefer dirt over mud) and deployer preferences (e.g., prefer dirt path over flower beds). Most existing solutions to this preference-aware path planning problem use semantic segmentation to classify terrain types from camera images, and then ascribe costs to each type. Unfo...
-
This paper presents a novel design and experimental investigation for a self-propelled capsule robot that can be used for painless colonoscopy during a retrograde progression from the patient's rectum. The steerable robot is driven forward and backward via its internal vibration and impact with orientation control by using an electromagnetic actuator. The actuator contains four sets of coils and a...
-
Actuated universal joints, or equivalent joint systems, are found in a number of robotic applications, in particular mobile snake robots, continuum robots and robotic tails. These joints have two degrees of freedom on two axes, each perpendicular to a third axis and to themselves. Such joints use a variety of actuation methods, including direct drive motors, linear screw drives, cable based system...
-
This paper reports on preliminary experimental results of recently proposed mechanism kinematics for a legged robot. The proposed kinematics creates a mapping from a series-elastic actuator to a foot motion that includes a pair of singularities within a fully rotatable kinematic circuit. Such a circuit is less common and only possible with certain multi-loop linkages. A slice of the configuration ...
-
Robots performing close physical interaction with humans would require a continuously variable transmission to operate in the region around the peak efficiency or peak power of the driving system. Conventional continuously variable transmission (CVT) has shown advantages in energy-efficient driving systems. However, these CVT designs are heavy and large for robotic applications. This paper present...
-
Collaborative robots need to work closely and safely with users while being fast and strong. Fulfilling both these needs simultaneously presents a significant challenge, if not a roadblock, for conventional geared motor technology. Magnetorheological (MR) actuation is an alternative technology that has the potential to exhibit both safety and speed at the same time in a compact and cost-effective ...
-
Robots are typically designed as occupants of rooms, adapting to, and navigating within them. “Robot surfaces,” an emerging robot typology, are not occupants of but integral with rooms, physically shaping rooms to support human activity. We report on an advancement of robot surfaces formed by weaving McKibben Pneumatic Air Muscles that, when actuated, morph a 2D planar surface to generate 3D geome...
-
This paper presents a proposal of new hydraulic circuit, designated as a modular hydraulic servo booster (MHSB), aimed at the realization of modular hydraulic robots. The modular robots, however, have important shortcomings compared to non-modular robots, such as separate power sources and power imbalance between axes when applied to serially configured robots. To mitigate those difficulties, we t...
-
Damping properties in biological muscle are crit-ical for absorbing shock, maintaining posture, and positioning limbs and appendages. When creating biomimetic robots, the ability to replicate the dynamics of biological muscle is neces-sary to reproduce behaviors seen in an animal model. However, the damping properties of existing soft artificial muscles are difficult to predict and tune to match s...
-
The long-term goal of this work is to enable agents with low-information sensors to perform tasks usually restricted to ones with more sophisticated, high-information sensing capabilities. Our approach is to regulate the motion of these low-information agents to obtain “high-information” results. As a first step, we consider a multi-agent system tasked with locating and tracking a moving target us...
-
With the increasing prevalence of complex vision-based sensing methods for use in obstacle identification and state estimation, characterizing environment-dependent measurement errors has become a difficult and essential part of modern robotics. This paper presents a self-supervised learning approach to safety-critical control. In particular, the uncertainty associated with stereo vision is estima...
-
Industrial robots can solve tasks in controlled environments, but modern applications require robots able to operate also in unpredictable surroundings. An increasingly popular reactive policy architecture in robotics is Behavior Trees (BTs) but as other architectures, programming time drives cost and limits flexibility. The two main branches of algorithms to generate policies automatically, autom...
-
Behavior trees (BTs) are hierarchical agent control architectures popular for robot task-level planning that can be autonomously learned from robot demonstrations via decision tree (DT) intermediaries, making them accessible to non-expert users. Conversion algorithms from DTs to BTs, such as the BT-Espresso algorithm, focus on replicating DT logic in a BT format but do not exploit the strengths of...
-
CPG-Based oscillator networks are increasingly being used to drive multi-limbed robots. To produce effective gaits with these networks, the relationship between the CPG parameters and the characteristics of the gait must be determined. However, due to the nonlinear nature of the oscillators, this relationship is challenging to ascertain. In this work a reinforcement learning algorithm is used to d...
-
In this paper, we consider the European Rail Traffic Management System (ERTMS) as a System-of-Systems (SoS) and propose modeling it using colored Petri nets. We formally control the European rail transport, while guaranteeing a set of cross-border security properties. This becomes an essential and challenging task since each of them have mainly developed safety and trackside rules regardless of it...
-
Collaborative robots (cobots) are machines designed to work safely alongside people in human-centric environments. Providing cobots with the ability to quickly infer the inertial parameters of manipulated objects will improve their flexibility and enable greater usage in manufacturing and other areas. To ensure safety, cobots are subject to kinematic limits that result in low signal-to-noise ratio...
-
Aerial robots have been widely used as sensor carrying platforms in a wide range of application, mainly because using this type of systems for physical interaction seems to be an unsuitable operation. This is not only due to the risk of collision and damage of the platform, but also because it is unclear whether a consumer-grade UAV can withstand physical contact with the environment. In this pape...
-
This work presents a novel Aspect Ratio-Modular Vertical Take-Off and Landing (ARM-VTOL) aerial robot, which is a meta-aircraft composed of two or more TiltRotor hybrid aircraft systems capable of magnetically being coupled during hovering flight, and of executing VTOL / Fixed-Wing hybrid missions once combined. The proposed meta-aircraft system carries the advantage of improved aerodynamic effici...
-
This paper proposes Elastic Tracker, a flexible trajectory planning framework that can deal with challenging tracking tasks with guaranteed safety and visibility. Firstly, an object detection and intension-free motion prediction method is designed. Then an occlusion-aware path finding method is proposed to provide a proper topology. A smart safe flight corridor generation strategy is designed with...
-
Aerial Manipulation Using Contact with the Environment by Thrust Vectorable Multilinked Aerial Robot
In recent years, an increasing number of research works have been focusing on the manipulation by aerial robots. Previous works using aerial robots with robotic arms have two problems: underactuation and external disturbances. We propose the fully-actuated control method and motion strategy using contact with the environment to solve these problems, along with the mechanical approach required. Fir...
-
Robotic airships offer significant advantages in terms of safety, mobility, and extended flight times. However, their highly restrictive weight constraints pose a major challenge regarding the available computational resources to perform the required control tasks. Neuromorphic computing stands for a promising research direction for addressing such problem. By mimicking the biological process for ...
-
With the advancement in computing and robotics, it is necessary to develop fluent and intuitive methods for inter-acting with digital systems, augmented/virtual reality (AR/VR) interfaces, and physical robotic systems. Hand movement recognition is widely used to enable such interaction. Hand configuration classification and metacarpophalangeal (MCP) joint angle detection are important for a compre...
-
SAGCI-System: Towards Sample-Efficient, Generalizable, Compositional, and Incremental Robot Learning
Building general-purpose robots to perform a diverse range of tasks in a large variety of environments in the physical world at the human level is extremely challenging. According to [1], it requires the robot learning to be sample-efficient, generalizable, compositional, and incremental. In this work, we introduce a systematic learning framework called SAGCI-system towards achieving these above f...
-
While the design of autonomous robots often emphasizes developing proficient robots, another important attribute of autonomous robot systems is their ability to evaluate their own proficiency and limitations. A robot should be able to assess how well it can perform a task before, during, and after it attempts the task. Thus, we consider the following question: How can we design autonomous robots t...
-
This paper presents a solution to the problem of executing robotic tasks over time horizons that exceed the robot's total battery capacity. In the presented robotics application, the robot's mission is to satisfy two tasks: environmental exploration and environmental monitoring, both of which need to be executed over long time periods. These tasks need therefore to be persistified. Ensuring the lo...
-
Practical robot designs must strike a compromise between fabrication/manufacture cost and anticipated execution performance. Compared to parsimonious designs, more capable (and hence more expensive) robots generally achieve their ends with greater efficiency. This paper examines how the roboticist might explore the space of designs to gain an understanding of such trade-offs. We focus, specificall...
-
In this work we present our results on dynamic visual servoing for the case of moving targets while also exploring the possibility of using such a controller for interaction with the environment. We illustrate the derivation of a feature space impedance controller for tracking a moving object as well as an Extended Kalman Filter based on the visual servoing kinematics for increasing the rate of th...
-
This paper presents a simple but compact design of a bicycle-like robot for inspecting complex-shaped ferromagnetic structures. The design concept for versatile locomotion relies on two independently steered magnetic wheels formed in a bicycle-like configuration, allowing the robot to possess multi-directional mobility. The key feature of a reciprocating mechanism enables the robot to change its s...
-
This paper presents CCRobot-V, the fifth version of CCRobot, a cooperative serial multi-robot system for bridge cable inspection and maintenance that uses silkworm-like locomotion to climb the entire length of super-long stay cable at high speeds while carrying heavy inspection/maintenance equipment. CCRobot-V consists of one climbing precursor robot, one inspection/maintenance robot, several cabl...
-
In this work, we investigate a form of dynamic contact-rich locomotion in which a robot pushes off from obstacles in order to move through its environment. We present a reflex-based approach that switches between optimized hand-crafted reflex controllers and produces smooth and predictable motions. In contrast to previous work, our approach does not rely on periodic movements, complex models of ro...
-
Quadrupedal landing is a complex process involving large impacts, elaborate contact transitions, and is a crucial recovery behavior observed in many biological animals. This work presents a real-time, optimal landing controller that is free of pre-specified contact schedules. The controller determines optimal touchdown postures and reaction force profiles and is able to recover from a variety of f...
-
This paper studies a passive biped walker with knees and biomimetic feet and its behavior, as a function of key parameters. The model includes a continuous dynamic representation of the knee joint's interaction with a viscoelastic kneecap, as well as a complete kinematic description of feet that are designed to mimic the human rollover shape. First, the analytical model is derived and studied nume...
-
This paper mathematically conceives a novel walking support device that leverages passive dynamics and coupling effects. In this model, a passive human walker is flexibly connected to an active humanoid, where the coupling effect induces a stable walking gait of the human. To understand the key mechanism of such indirect gait regulation, different actuation modes are designed for the humanoid and ...
-
We present simplified 2D dynamic models of the 3D, passive dynamic inspired walking gait of a physical quasi-passive walking robot. Quasi-passive walkers are robots that integrate passive walking principles and some form of actuation. Our ultimate goal is to better understand the dynamics of actuated walking in order to create miniature, untethered, bipedal walking robots. At these smaller scales ...
-
We present a whole-body control architecture for the generation of stable task-oriented motions in Wheeled Inverted Pendulum (WIP) robots. Controlling WIP systems is challenging because the successful execution of tasks is subordinate to the ability to maintain balance. Our feedback control approach relies both on partial feedback linearization and Model Predictive Control (MPC). The partial feedb...
-
This paper introduces a novel approach for whole-body motion planning and dynamic occlusion avoidance. The proposed approach reformulates the visibility constraint as a likelihood maximization of visibility probability. In this formulation, we augment the primary cost function of a whole-body model predictive control scheme through a relaxed log barrier function yielding a relaxed log-likelihood m...
-
Parallel mechanisms are becoming increasingly popular as subsystems in various robots due to their superior stiffness, payload-to-weight ratio, and dynamic properties. The serial connection of parallel subsystems leads to series-parallel hybrid robots, which are more difficult to model and control than serial or tree-type systems. At the same time, Whole-Body Control (WBC) has become the method of...
-
Omni-Roach: A Legged Robot Capable of Traversing Multiple Types of Large Obstacles and Self-Righting
Robots excel at avoiding obstacles but struggle to traverse complex 3-D terrain with cluttered large obstacles. By contrast, insects like cockroaches excel at doing so. Recent research in our lab elucidated how locomotor transitions emerge from locomotor-environment interaction for diverse locomotor challenges abstracted from complex 3-D terrain and the strategies to overcome them. Here we built o...
-
We present a fast collision checking/avoidance algorithm for collaborative robot arms that work in close proximity. We formulate forward kinematics and separating distance function using DH convention and Taylor models (the tight enclosure of a function), and then compute their tight bounds for determining interference between robot arms. Our algorithm allows the collaborative robot arms to perfor...
-
Safe UAV navigation is challenging due to the complex environment structures, dynamic obstacles, and uncertainties from measurement noises and unpredictable moving obstacle behaviors. Although plenty of recent works achieve safe navigation in complex static environments with sophisticated mapping algorithms, such as occupancy map and ESDF map, these methods cannot reliably handle dynamic environme...
-
We propose factor graph optimization for simultaneous planning, control, and trajectory estimation for collision-free navigation of autonomous systems in environments with moving objects. The proposed online probabilistic motion planning and trajectory estimation navigation technique generates optimal collision-free state and control trajectories for autonomous vehicles when the obstacle motion mo...
-
When multiple quadrotors fly in a cluttered environment, collision-free flight must be assured. In this paper, we propose a novel elastic safety clearance based model predictive control (ESC-MPC) for multiple maneuverable quadrotors to avoid collisions in the presence of disturbance. This is accomplished through leveraging tube based model predictive control to maintain the quadrotor in a tube of ...
-
In this paper we present a novel strategy for reactive collision-free feasible motion planning for robotic manipulators operating inside an environment populated by moving obstacles. The proposed strategy embeds the Dynamical System (DS) based obstacle avoidance algorithm into a constrained non-linear optimization problem following the Model Predictive Control (MPC) approach. The solution of the p...
-
Existing navigation policies for autonomous robots tend to focus on collision avoidance while ignoring human-robot interactions in social life. For instance, robots can pass along the corridor safer and easier if pedestrians notice them. Sounds have been considered as an efficient way to attract the attention of pedestrians, which can alleviate the freezing robot problem. In this work, we present ...
-
Obstacle avoidance between polytopes is a chal-lenging topic for optimal control and optimization-based tra-jectory planning problems. Existing work either solves this problem through mixed-integer optimization, relying on simpli-fication of system dynamics, or through model predictive control with dual variables using distance constraints, requiring long horizons for obstacle avoidance. In either...
-
Collision avoidance algorithms are of central interest to many drone applications. In particular, decentralized approaches may be the key to enabling robust drone swarm solutions in cases where centralized communication becomes computationally prohibitive. In this work, we draw biological inspiration from flocks of starlings (Sturnus vulgaris) and apply the insight to end-to-end learned decentrali...
-
Motion planning is a ubiquitous problem that is often a bottleneck in robotic applications. We demonstrate that motion planning problems such as minimum constraint removal, belief-space planning, and visibility-aware motion planning (VAMP) benefit from a path-dependent formulation, in which the state at a search node is represented implicitly by the path to that node. A naïve approach to computing...
-
Automatic disassembly planning for complex industrial products like vehicles checks the expandability of components already at early stages of design. For a fast computation of collision-free disassembly paths, sampling-based rigid body motion planning is used in the literature. However, in real-world scenarios there are circumstances that prevent the finding of plausible collision-free disassembl...
-
Road extraction is an essential step in building autonomous navigation systems. Detecting road segments is challenging as they are of varying widths, bifurcated throughout the image, and are often occluded by terrain, cloud, or other weather conditions. Using just convolution neural networks (ConvNets) for this problem is not effective as it is inefficient at capturing distant dependencies between...
-
Neural network used for the LiDAR semantic segmentation task needs the point-wise labeled point clouds for training, which is more expensive than bounding box annotations. Enhancing the diversity of training data through object insertion is an effective method to reduce labeling costs. The existing object insertion methods are mainly divided into two categories. First is “copy” the clusters from a...
-
Monocular image-based 3D perception has become an active research area in recent years owing to its applications in autonomous driving. Approaches to monocular 3D perception including detection and tracking, however, often yield inferior performance when compared to LiDAR-based techniques. Through systematic analysis, we identified that per-object depth estimation accuracy is a major factor boundi...
-
Finding relative pose for cameras is of vital importance in computer vision and robotics. We investigate the problem of relative motion estimation between successive frames from a minimal number of correspondences. Existing approximated methods use a first-order approximation to relative pose in order to simplify the problem and produce an estimate quickly. Our solution uses Cayley parameterizatio...
-
Event cameras are bio-inspired sensors that capture per-pixel asynchronous intensity change rather than the synchronous absolute intensity frames captured by a classical camera sensor. Such cameras are ideal for robotics applications since they have high temporal resolution, high dynamic range and low latency. However, due to their high temporal resolution, event cameras are particularly sensitive...
-
Industrial bin picking is a challenging task that requires accurate and robust segmentation of individual object instances. Particularly, industrial objects can have irregular shapes, that is, thin and concave, whereas in bin-picking scenarios, objects are often closely packed with strong occlusion. To address these challenges, we formulate a novel part-aware instance segmentation pipeline. The ke...
-
Stereo matching is an indispensable function that enables machine vision system to obtain depth information of its environment. However, most of existing algorithms rely on conventional camera, which follows the frame-based scheme and has several shortcomings: low dynamic range, low temporal resolution and high power consumption. To address these issues, we propose two novel patch-based stereo mat...
-
We discuss the problem of 3D pose estimation for multi-view videos. With previous frame-by-frame multi-view methods, it has been difficult to achieve stable estimation under challenging settings such as low-resolution or with only a few views. Temporal approaches are effective ways of addressing such problems, but enforcing temporal consistency with neigh-boring frames sometimes damages the precis...
-
Many robotic tasks are composed of a lot of temporally correlated sub-tasks in a highly complex environment. It is important to discover situational intentions and proper actions by deliberating on temporal abstractions to solve problems effectively. To understand the intention separated from changing task dynamics, we extend an empowerment-based regularization technique to situations with multipl...
-
Existing learning from demonstration algorithms usually assume access to expert demonstrations. However, this assumption is limiting in many real-world applications since the collected demonstrations may be suboptimal or even consist of failure cases. We therefore study the problem of learning from imperfect demonstrations by learning a confidence predictor. Specifically, we rely on demonstrations...
-
Inverse Reinforcement Learning (IRL) is attractive in scenarios where reward engineering can be tedious. However, prior IRL algorithms use on-policy transitions, which require intensive sampling from the current policy for stable and optimal performance. This limits IRL applications in the real world, where environment interactions can become highly expensive. To tackle this problem, we present Of...
-
Animals are able to imitate each others' behavior, despite their difference in biomechanics. In contrast, imitating other similar robots is a much more challenging task in robotics. This problem is called cross domain imitation learning (CDIL). In this paper, we consider CDIL on a class of similar robots. We tackle this problem by introducing an imitation learning algorithm based on invariant repr...
-
We propose a demonstration-efficient strategy to compress a computationally expensive Model Predictive Controller (MPC) into a more computationally efficient representation based on a deep neural network and Imitation Learning (IL). By generating a Robust Tube variant (RTMPC) of the MPC and leveraging properties from the tube, we introduce a data augmentation method that enables high demonstration...
-
Correspondence learning is a fundamental problem in robotics, which aims to learn a mapping between state, action pairs of agents of different dynamics or embodiments. However, current correspondence learning methods either leverage strictly paired data-which are often difficult to collect-or learn in an unsupervised fashion from unpaired data using regularization techniques such as cycle-consiste...
-
2D&3D object detection always suffers from a dramatic performance drop when transferring the model trained in the source domain to the target domain due to various domain shifts. In this paper, we propose a Joint Self-Training (JST) framework to improve 2D image and 3D point cloud detectors with aligned outputs simultaneously during the transferring. The proposed framework contains three novelties...
-
We present a novel method for generating, predicting, and using Spatiotemporal Occupancy Grid Maps (SOGM), which embed future information of dynamic scenes. Our au-tomated generation process creates groundtruth SOGMs from previous navigation data. We build on prior work to annotate lidar points based on their dynamic properties, which are then projected on time-stamped 2D grids: SOGMs. We design a...
-
We propose a novel method that estimates the Kinematic Structure (KS) of arbitrary articulated rigid objects from event-based data. Event cameras are emerging sensors that asynchronously report brightness changes with a time resolution of microseconds, making them suitable candidates for motion-related perception. By assuming that an articulated rigid object is composed of body parts whose shape c...
-
Remote controlling robots without any automated help is difficult due to various limitations. Autocomplete mitigates this difficulty by automatically detecting and completing the intended motions on robots from the input of the user. Such an approach can improve the system performance and reduce the load on the operator. Usually, recognizing intended motions is achieved using pre-trained Deep Lear...
-
The ability to perceive object slip via tactile feedback enables humans to accomplish complex manipulation tasks including maintaining a stable grasp. Despite the utility of tactile information for many applications, tactile sensors have yet to be widely deployed in industrial robotics settings; part of the challenge lies in identifying slip and other events from the tactile data stream. In this p...
-
Spatio-temporal graphs (ST-graphs) have been used to model time series tasks such as traffic forecasting, human motion modeling, and action recognition. The high-level structure and corresponding features from ST-graphs have led to improved performance over traditional architectures. However, current methods tend to be limited by simple features, despite the rich information provided by the full g...
-
In this study, working with the task of object retrieval in clutter, we have developed a robot learning framework in which Monte Carlo Tree Search (MCTS) is first applied to enable a Deep Neural Network (DNN) to learn the intricate interactions between a robot arm and a complex scene containing many objects, allowing the DNN to partially clone the behavior of MCTS. In turn, the trained DNN is inte...
-
Real-time video denoising finds applications in several fields like mobile robotics, satellite television, and surveillance systems. Traditional denoising approaches are more common in such systems than their deep learning-based counterparts despite their inferior performance. The large size and heavy computational requirements of neural network-based denoising models pose a serious impediment to ...
-
This paper introduces an intelligent system which composes music following the users' instructions. Current auto-matic music generation models are lack of stability. Meanwhile, they cannot satisfy the preference of different people. To overcome these challenges, we train a Transformer-based neural network to generate short music segments using a dataset. A user can compose music pieces by interact...
-
Predicting future trajectories of road agents is a critical task for autonomous driving. Recent goal-based trajectory prediction methods, such as DenseTNT and PECNet [1], [2], have shown good performance on prediction tasks on public datasets. However, they usually require complicated goal-selection algorithms and optimization. In this work, we propose KEMP, a hierarchical end-to-end deep learning...
-
Behavior prediction models have proliferated in recent years, especially in the popular real-world robotics application of autonomous driving, where representing the distribution over possible futures of moving agents is essential for safe and comfortable motion planning. In these models, the choice of coordinate frames to represent inputs and outputs has crucial trade offs which broadly fall into...
-
Depth completion refers to inferring the dense depth map from a sparse depth map with or without corre-sponding color image. Numerous neural networks have been proposed to accomplish this task. However, insufficient uti-lization of heteromorphic data and the fact that predicted dense depth prefers a sparse depth enormously damage the performance of approaches. To reduce data preference and fully u...
-
This paper emphasizes the importance of a robot's ability to refer to its task history, especially when it exe-cutes a series of pick-and-place manipulations by following language instructions given one by one. The advantage of referring to the manipulation history can be categorized into two folds: (1) the language instructions omitting details but using expressions referring to the past can be i...
-
3D Multi-Object Tracking (MOT) plays a crucial role in efficient and safe operation of automatic driving, especially in scenarios of occlusion or poor visibility. Most 3D MOT methods leverage only positional distance, which is insufficient for scenes with high density of objects or drastic changes in the motion state. In order to address this, we propose a new 3D MOT model which fuses information ...
-
A multi-fingered hand that can grasp and manipulate a variety of objects is an option for assisting people in their daily lives. However, the range of torque output that can be handled by the multi-fingered hand is very limited compared to the capability of the human hand. In this paper, we introduce a new multi-fingered hand consisting of a dynamic pulley and a linkage mechanism, aiming to achiev...
-
Grasping in cluttered environments is one of the most fundamental skills in robotic manipulation. Most of the current works focus on estimating grasp poses for parallel-jaw or suction-cup end effectors. However, the study for dexterous anthropomorphic hand grasping in clutter remains a great challenge. In this paper, we propose HGC-Net, a single-shot network that learns to predict dense hand grasp...
-
Robotic grasping for a diverse set of objects is essential in many robot manipulation tasks. One promising approach is to learn deep grasping models from large training datasets of object images and grasp labels. However, empirical grasping datasets are typically sparsely labeled (i.e., a small number of successful grasp labels**Labels refer to marking the image to indicate a successful robotic gr...
-
Multi-Dimensional Compliance of Soft Grippers Enables Gentle Interaction with Thin, Flexible Objects
In this paper, we discuss the role of gripper compliance in successful grasping and manipulation of thin, flexible materials. We show, both conceptually and empirically, that each axis of compliance in a planar gripper provides unique benefits in this domain. Vertical compliance allows robust grasping of thin materials in the presence of large uncertainty in positioning. Lateral compliance increas...
-
Handling object deformations for robotic grasping is still a major problem to solve. In this paper, we propose an efficient learning-free solution for this problem where generated grasp hypotheses of a region of an object are adapted to its deformed configurations. To this end, we investigate the applicability of functional map (FM) correspondence, where the shape matching problem is treated as se...
-
Robots in the real world frequently come across identical objects in dense clutter. When evaluating grasp poses in these scenarios, a target-driven grasping system requires knowledge of spatial relations between scene objects (e.g., proximity, adjacency, and occlusions). To efficiently complete this task, we propose a target-driven grasping system that simultaneously considers object relations and...
-
We present a data-driven approach for effective bin picking from clutter. Recent bin picking solutions usually lead to a direct pinch grasp on a target object without addressing any other potential contact interaction in clutter. However, appropriate physical interaction can be essential to successful singulation and subsequent secure picking, the goal of bin picking. In this work, we contribute a...
-
Grasping a particular object may require a dedicated grasping movement that may also be specific to the robot end-effector. No generic and autonomous method does exist to generate these movements without making hypotheses on the robot or on the object. Learning methods could help to autonomously discover relevant grasping movements, but they face an important issue: grasping movements are so rare ...
-
Grasping unknown objects with multi-fingered hands at high success rates and in real-time is an unsolved problem. Existing methods are limited in the speed of grasp synthesis or the ability to synthesize a variety of grasps from the same observation. We introduce Five-finger Hand Net (FFHNet), an ML model which can generate a wide variety of high-quality multi-fingered grasps for unseen objects fr...
-
Grasping fragile objects in the presence of un-certainty is a crucial task for robots, that becomes inherently challenging if the manipulator in use is an industrial robot platform that does not provide compliant control inputs. This requires not only to estimate the alignment error during object contact but also to alter the robot configuration to decrease this error while taking interaction cons...
-
This paper proposes 12 multi-object grasps (MOGs) types from a human and robot grasping data set. The grasp types are then analyzed and organized into a MOG taxonomy. This paper first presents three MOG data collection setups: a human finger tracking setup for multi-object grasping demonstrations, a real system with Barretthand, UR5e arm, and a MOG algorithm, a simulation system with the same sett...
-
Non-invasive brain-computer interfaces (BCIs) provide us with the unique ability to classify the psychological state of a person using only neurophysiological signals, such as those captured with an electroencephalogram (EEG). With this ability, new avenues for innovation in the field of healthcare arise, especially as it is used for robotics. EEGNet is a novel deep learning technique for the clas...
-
In this paper, we develop a novel adaptive motion imitation algorithm (AMI) for robotic systems. Although AMI can be used in a variety of human-robot interaction scenarios, we are particularly interested in robotic rehabilitation where the robot plays the role of demonstrating and practicing challenging motion physiotherapy. During therapy, the robot first demonstrates a reference trajectory to th...
-
Accurate, long-term forecasting of pedestrian trajectories in highly dynamic and interactive scenes is a longstanding challenge. Recent advances in using data-driven approaches have achieved significant improvements in terms of prediction accuracy. However, the lack of group-aware analysis has limited the performance of forecasting models. This is especially nonnegligible in highly crowded scenes,...
-
With the ever-increasing elderly population, elder walking assistance is in strong demand. Instead of receiving assistance from a human carer, a smart walker can bring an elder user a more convenient and autonomous walking experience. Towards intelligent and safe walking assistance, we propose a close-proximity front-following model for smart walkers, which analyzes the walking gait and detects th...
-
In this work, we propose A2DIO, a novel hybrid neural network model with a set of carefully designed attention mechanisms for pose invariant inertial odometry. The key idea is to extract both local and global features from the window of IMU measurements for velocity prediction. A2DIO leverages the convolutional neural network (CNN) to capture the sectional features and long-short term memory (LSTM...
-
Vital signs such as heart rate, oxygen saturation, and blood pressure are crucial information for healthcare workers to identify clinical deterioration of ward patients. Currently, medical devices monitor these vital signs and trigger alarms when the vital signs are not in the normal ranges based on predefined thresholds, which suggests the presence of clinical deterioration. However, such thresho...
-
Assistance robots have gained widespread attention in various industries such as logistics and human assistance. The tasks of guiding or following a human in a crowded environment such as airports or train stations to carry weight or goods is still an open problem. In these use cases, the robot is not only required to intelligently interact with humans, but also to navigate safely among crowds. Th...
-
Robust dynamic interactions are required to move robots in daily environments alongside humans. Optimisation and learning methods have been used to mimic and reproduce human movements. However, they are often not robust and their generalisation is limited. This work proposed a hierarchical control architecture for robot manipulators and provided capabilities of reproducing human-like motions durin...
-
Human motion prediction is a crucial step for achieving human-robot interactions. While recent transformer-based methods have shown great potentials in 3D human motion prediction, they still suffer from mode collapse to non-plausible poses and quadratically computational complexity with respect to the increasing length of input sequences. In this paper, we propose a novel spatio-temporal deformabl...
-
Modeling stochastic traffic dynamics is critical to developing self-driving cars. Because it is difficult to develop first principle models of cars driven by humans, there is great potential for using data driven approaches in developing traffic dynamical models. While there is extensive literature on this subject, previous works mainly address the prediction accuracy of data-driven models. Moreov...
-
The proposed RMS-FlowNet is a novel end-to-end learning-based architecture for accurate and efficient scene flow estimation which can operate on point clouds of high density. For hierarchical scene flow estimation, the existing methods depend on either expensive Farthest-Point-Sampling (FPS) or structure-based scaling which decrease their ability to handle a large number of points. Unlike these me...
-
We propose a novel and pragmatic framework for traffic scene perception with roadside cameras. The proposed framework covers a full-stack of roadside perception pipeline for infrastructure-assisted autonomous driving, including object detection, object localization, object tracking, and multi-camera information fusion. Unlike previous vision-based perception frameworks rely upon depth offset or 3D...
-
In this paper we present the first safe system for full control of self-driving vehicles trained from human demonstrations and deployed in challenging, real-world, urban environments. Current industry-standard solutions use rule-based systems for planning. Although they perform reasonably well in common scenarios, the engineering complexity renders this approach incompatible with human-level perfo...
-
Recently, vehicle re-identification methods based on deep learning constitute remarkable achievement. However, this achievement requires large-scale and well-annotated datasets. In constructing the dataset, assigning globally available identities (Ids) to vehicles captured from a great number of cameras is labour-intensive, because it needs to consider their subtle appearance differences or viewpo...
-
Ride-sharing has important implications for improving the efficiency of mobility-on-demand systems. However, it remains a challenge due to the complex dynamics between vehicles and requests. This paper presents a decentralized ride-sharing algorithm suitable for shared autonomous vehicles (SAVs) deployment. The ride-sharing problem is formulated as a multi-agent reinforcement learning problem. We ...
-
We demonstrate a multi-lidar calibration frame-work for large mobile platforms that jointly calibrate the extrinsic parameters of non-overlapping Field-of-View (FoV) lidar sensors, without the need for any external calibration aid. The method starts by estimating the pose of each lidar in its corresponding sensor frame in between subsequent timestamps. Since the pose estimates from the lidars are ...
-
Learning controllers that reproduce legged locomotion in nature has been a longtime goal in robotics and computer graphics. While yielding promising results, recent approaches are not yet flexible enough to be applicable to legged systems of different morphologies. This is partly because they often rely on precise motion capture references or elaborate learning environments that ensure the natural...
-
To traverse complex scenarios reliably a legged robot needs to move its base aided by the ground reaction forces, which can only be generated by the legs that are momentarily in contact with the ground. A proper selection of footholds is crucial for maintaining balance. In this paper, we propose a foothold evaluation criterion that considers the transition feasibility for both linear and angular d...
-
In this paper, we present a real-time whole-body planner for collision-free legged mobile manipulation. We enforce both self-collision and environment-collision avoidance as soft constraints within a Model Predictive Control (MPC) scheme that solves a multi-contact optimal control problem. By penalizing the signed distances among a set of representative primitive collision bodies, the robot is abl...
-
This paper presents QuadRunner, a transformable quasi-wheel legged robot that achieves both quadruped locomotion and wheel locomotion by exploiting a novel semicircular leg-wheel design with a Trotting Wheel gait. We built upon the Stanford Doggo open architecture platform and integrated it with a transformable leg-wheel design to enhance its locomotion capabilities. On its gait control, improveme...
-
In this work, a non-gaited framework for legged system locomotion is presented. The approach decouples the gait sequence optimization by considering the problem as a decision-making process. The redefined contact sequence problem is solved by utilizing a Monte Carlo Tree Search (MCTS) algorithm that exploits optimization-based simulations to evaluate the best search direction. The proposed scheme ...
-
In this paper, we present a framework rooted in control and planning that enables quadrupedal robots to traverse challenging terrains with discrete footholds using visual feedback. Navigating discrete terrain is challenging for quadrupeds because the motion of the robot can be aperiodic, highly dynamic, and blind for the hind legs of the robot. Additionally, the robot needs to reason over both the...
-
Navigation in close proximity with pedestrians is a challenge on the way to fully automated vehicles. Pedestrian-friendly navigation requires an understanding of pedestrian reaction and intention. Merely safety based reactive systems can lead to sub-optimal navigation solutions resulting in the freezing of the vehicle in many scenarios. Moreover, a strictly reactive method can produce unnatural dr...
-
This paper presents a deep reinforcement learning (DRL) sframework for safe and efficient navigation in crowded environments. Here, the robot learns cooperative behavior using a new reward function that penalizes robot actions interfering with the pedestrian's movement. Also, we propose a simulated pedestrian policy reflecting data from actual pedestrian movements. Furthermore, we introduce a coll...
-
Strategies for safe human-robot interaction (HRI), such as the well-established Safe Motion Unit, provide a velocity scaling for biomechanically safe robot motion. In addition, psychologically-based safety approaches are required for trustworthy HRI. Such schemes can be very conservative and robot motion complying with such safety approaches should be time efficient within the robot motion plannin...
-
As technology advances, the need for safe, efficient, and collaborative human-robot-teams has become increasingly important. One of the most fundamental collaborative tasks in any setting is the object handover. Human-to-robot handovers can take either of two approaches: (1) direct hand-to-hand or (2) indirect hand-to-placement-to-pick-up. The latter approach ensures minimal contact between the hu...
-
Collision-free navigation while moving amongst static and dynamic obstacles with a limited sensor range is still a great challenge for modern mobile robots. Therefore, the ability to avoid collisions with obstacles in crowded, partially observable environments is one of the most important indicators to measure the navigation performance of a mobile robot. In this paper, we propose a novel deep rei...
-
Robot-assisted feeding in household environments is challenging because it requires robots to generate trajectories that effectively bring food items of varying shapes and sizes into the mouth while making sure the user is comfortable. Our key insight is that in order to solve this challenge, robots must balance the efficiency of feeding a food item with the comfort of each individual bite. We for...
-
The use of drones in human-populated areas is increasing day by day. Such robots flying in close proximity to humans and potentially interacting with them, as in object handover or delivery, need to carefully plan their navigation considering the presence of humans. We propose a humanaware 3D reactive planner based on stochastic optimization for drone navigation. Besides considering the kinematics...
-
The increasing deployment of robots in co-working scenarios with humans has revealed complex safety and efficiency challenges in the computation of the robot behavior. Movement among humans is one of the most fundamental —and yet critical—problems in this frontier. While several approaches have addressed this problem from a purely navigational point of view, the absence of a unified paradigm for c...
-
This paper is concerned with the problem of estimating (interpolating and smoothing) the shape (pose and the six modes of deformation) of a slender flexible body from multiple camera measurements. This problem is important in both biology, where slender, soft, and elastic structures are ubiquitously encountered across species, and in engineering, particularly in the area of soft robotics. The prop...
-
Grasping objects whose physical properties are unknown is still a great challenge in robotics. Most solutions rely entirely on visual data to plan the best grasping strategy. However, to match human abilities and be able to reliably pick and hold unknown objects, the integration of an artificial sense of touch in robotic systems is pivotal. This paper describes a novel model-based slip detection p...
-
Vision-based tactile sensors typically utilize a deformable elastomer and a camera mounted above to provide high-resolution image observations of contacts. Obtaining accurate volumetric meshes for the deformed elastomer can provide direct contact information and benefit robotic grasping and manipulation. This paper focuses on learning to synthesize the volumetric mesh of the elastomer based on the...
-
Parallel kinematics machines (PKM) operate with maximal acceleration being designed for highly dynamic manipulation tasks. This leads to extreme loads of the joints, which is usually not accounted for in the motion planning. In this paper an extended inverse dynamics method is introduced, which allows computing the joint reaction forces along with the actuation torques, and provides a basis for ti...
-
In this paper we introduce a comprehensive framework to control an aerial manipulator, i.e., an aerial vehicle with a robotic arm, in physical interaction with a human operator or co-worker. The framework uses an admittance control paradigm in order to attain human ergonomy and safety; an interaction supervisor to automatically shape the compliance based on the interaction regions defined around t...
-
Collaborative robots are gradually taking over the leading position in automating the production and manufacturing of the SMEs, where the human-robot collaboration is highly emphasized. Therefore, estimating the force and simulating the performance of robots are of great importance. As a newly introduced technology, digital twin, has gained more attentions for simulation, process evaluation, real-...
-
This paper investigates the cooperative control problem of choke-point navigation for multiple quadcopters when only their subgroup is equipped with obstacle detecting sensors. We define a quadcopter as a leader if it is equipped with an obstacle detecting sensor; otherwise, it is a follower. In addition, we introduce a virtual leader agent to create the group motion. First, we apply the leader-fo...
-
In this paper we analyze the equilibrium points of a collaborative transportation task, composed of two unmanned aerial vehicles and a payload - in this case, a bar. Moreover, centralized and decentralized linear model predictive controllers are designed, where the nonlinear dynamics are linearized around the equilibrium points previously analyzed. A comparison between the centralized and decentra...
-
In this study, we explore the application of game theory, in particular Stackelberg games, to address the issue of effective coordination strategy generation for heterogeneous robots with one-way communication. To that end, focusing on the task of multi-object rearrangement, we develop a theoretical and algorithmic framework that provides strategic guidance for a pair of robot arms, a leader and a...
-
We study how information flows through a multi-robot network in order to better understand how to provide resilience to malicious information. While the notion of global resilience is well studied, one way existing methods provide global resilience is by bringing robots closer together to improve the connectivity of the network. However, large changes in network structure can impede the team from ...
-
Multi-robot teams are becoming an increasingly popular approach for information gathering in large geographic areas, with applications in precision agriculture, surveying the aftermath of natural disasters or tracking pollution. These robot teams are often assembled from untrusted devices not owned by the user, making the maintenance of the integrity of the collected samples an important challenge...
-
Implementing and conducting reproducible experiments on multi-robot hardware platforms are challenging tasks due to variations in hardware, software, and most importantly the intensive implementation effort. In this paper, we aim to present the Driving Swarm software framework which is developed to facilitate the implementation, deployment, supervision, and analysis of multi-robot experiments. We ...
-
The objective of foraging robot swarms is to search for and collect resources in an unknown arena as quickly as possible. To avoid the congestion near the central collection zone, we previously proposed an extension to the multiple-place foraging in which robot chains are deployed dynamically so that foraging robots can deliver to the robot chains instead of the central collection zone. However, a...
-
This paper studies the distributed multi-robot constrained navigation problem. While the multi-robot collision avoidance has been extensively studied in the literature with safety being the primary focus, the individual robot's destination convergence is not necessarily guaranteed. In particular, robots may get stuck in the local equilibria or periodic orbits of the multi-robot system, some of whi...
-
For aerial swarms, navigation in a prescribed formation is widely practiced in various scenarios. However, the associated planning strategies typically lack the capability of avoiding obstacles in cluttered environments. To address this deficiency, we present an optimization-based method that ensures collision-free trajectory generation for formation flight. In this paper, a novel differentiable m...
-
This paper proposes a method to compute camera 6 DoF poses to achieve a user defined coverage. The camera placement problem is modeled as a combinatorial optimization where given the maximum number of cameras, a camera set is selected from a larger pool of possible camera poses. We propose to minimize the squared error between the desired and the achieved coverage, and formulate the non-linear cos...
-
3D object detection with only image inputs is an interesting and important problem in computer vision and autonomous driving. Nowadays, most existing monocular 3D object detection algorithms rely solely on the approximation power of convolutional neural networks to learn a mapping from pixels to 3D predictions without knowing the projection matrix of the camera. To endow the networks with camera p...
-
To achieve real-time dynamic simulation analysis and optimization design, a dynamic digital twin of a nonholonomic mobile manipulator (one UR5e mounted on an industrial mobile robot MIR 200) has been developed in this paper. First, the digital twin integrated with dynamics of a mobile manipulator is established. The framework of the dynamic digital twin is presented in detail. Then, the dynamic mo...
-
The ultrasonic mapping of plate-based facilities is an essential step towards the robotic inspection of large metal structures such as storage tanks or ship hulls. This work proposes a novel framework that exploits ultrasonic echoes to recover grid-based and feature-based spatial representations jointly. We aim to improve on a previous mapping method [1] subject to errors due to interference, and ...
-
Class imbalance is a fundamental problem in computer vision applications such as semantic segmentation. Specifically, uneven class distributions in a training dataset often result in unsatisfactory performance on under-represented classes. Many works have proposed to weight the standard cross entropy loss function with pre-computed weights based on class statistics, such as the number of samples a...
-
Self-driving cars must detect other traffic participants like vehicles and pedestrians in 3D in order to plan safe routes and avoid collisions. State-of-the-art 3D object detectors, based on deep learning, have shown promising accuracy but are prone to over-fit domain idiosyncrasies, making them fail in new environments-a serious problem for the robustness of self-driving cars. In this paper, we p...
-
Mean field approximation methodology has laid the foundation of modern Continuous Random Field (CRF) based solutions for the refinement of semantic segmentation. In this paper, we propose to relax the hard constraint of mean field approximation - minimizing the energy term of each node from probabilistic graphical model, by a global optimization with the proposed dilated sparse convolution module ...
-
Instance-aware segmentation of unseen objects is essential for a robotic system in an unstructured environment. Although previous works achieved encouraging results, they were limited to segmenting the only visible regions of unseen objects. For robotic manipulation in a cluttered scene, amodal perception is required to handle the occluded objects behind others. This paper addresses Unseen Object ...
-
LiDAR sensors are becoming crucial for achieving higher levels of autonomy. With the current sensor technology, LiDAR sensors are still susceptible to erroneous measurements in adverse weather conditions due to weather artifacts observed in the point cloud data. In this work, we analyze the performance of deep learning LiDAR object detectors in adverse weather conditions. We study the under-resear...
-
Existing state-of-the-art 3D point clouds understanding methods only perform well in a fully supervised manner. To the best of our knowledge, there exists no unified framework which simultaneously solves the downstream high-level understanding tasks, especially when labels are extremely limited. This work presents a general and simple framework to tackle point clouds understanding when labels are ...
-
While multi-class 3D detectors are needed in many robotics applications, training them with fully labeled datasets can be expensive in labeling cost. An alternative approach is to have targeted single-class labels on disjoint data samples. In this paper, we are interested in training a multi-class 3D object detection model, while using these single-class labeled data. We begin by detailing the uni...
-
Semantic segmentation is a core ability required by autonomous agents, as being able to distinguish which parts of the scene belong to which object class is crucial for navigation and interaction with the environment. Approaches which use only one time-step of data cannot distinguish between moving objects nor can they benefit from temporal integration. In this work, we extend a backbone LatticeNe...
-
This paper presents a novel robot arm that is capable of switching between a rigid robot arm and a continuum robot arm. Therefore, the novel robot arm can perform adaptive physical interaction and manipulation against complex working environments and tasks. The switch-ability of the robot arm is achieved with two types of joints: knee-like flexible joints and continuum flexible joints, with which ...
-
Perception and portability are critical issues for wearable gloves in hand assistive engineering. However, available wearable gloves either lack flexible sensing or are bulky. In this paper, we present a tendon-driven lightweight wearable glove with soft joint sensing, Sen-Glove. Sen-Glove is equipped with 14 soft strain sensors, which enables full bending motion monitoring of 14 joints of five fi...
-
Foldable, origami-inspired, and laminate mechanisms are highly susceptible to deformation under external loading, which can lead to position or orientation errors if idealized kinematic models are used. According to dimensional scaling laws, laminate devices can often be treated as rigid bodies at millimeter and smaller scale deformations. However, foldable mechanisms enter the territory of soft r...
-
Soft robots have drawn great interest due to their ability to take on a rich range of shapes and motions, compared to traditional rigid robots. However, the motions, and underlying statics and dynamics, pose significant challenges to forming well-generalized and robust models necessary for robot design and control. In this work, we demonstrate a five-actuator soft robot capable of complex motions ...
-
Soft robots can be incredibly robust and safe but typically fail to match the strength and precision of rigid robots. This dichotomy between soft and rigid is recently starting to break down, with emerging research interest in hybrid soft-rigid robots. In this work, we draw inspiration from Nature, which achieves the best of both worlds by coupling soft and rigid tissues-like muscle and bone-to pr...
-
In this paper we introduce a novel technique that aims to control a two-module bio-inspired soft-robotic arm in order to qualitatively reproduce human demonstrations. The main idea behind the proposed methodology is based on the assumption that a complex trajectory can be derived from the composition and asynchronous activation of learned parameterizable simple movements constituting a knowledge b...
-
Trajectory scaling has long been used to address velocity and acceleration constraints in robotic motion planning. In later years, reactive motion planning based on dynamical systems has become popular. The traditional scaling techniques are not always suitable to adopt directly when online modifications of the trajectories are made leading to feasibility problems. In this paper, we propose an app...
-
Indirect trajectory optimization methods such as Differential Dynamic Programming (DDP) have found considerable success when only planning under dynamic feasibility constraints. Meanwhile, nonlinear programming (NLP) has been the state-of-the-art approach when faced with additional constraints (e.g., control bounds, obstacle avoidance). However, a naïve implementation of NLP algorithms, e.g., shoo...
-
High-speed motions in pick-and-place operations are critical to making robots cost-effective in many automation scenarios, from warehouses and manufacturing to hospitals and homes. However, motions can be too fast-such as when the object being transported has an open-top, is fragile, or both. One way to avoid spills or damage, is to move the arm slowly. We propose an alternative: Grasp-Optimized M...
-
Prioritization of tasks is a common approach to resolve conflicts in instantaneous control of redundant robots. However, the idea of prioritization has not yet been satisfactorily extended to model predictive control (MPC) to allow for real-time robot control. The standard sequential approach for prioritization is unsuitable because of the computational burden involved in solving a nonlinear probl...
-
Informative planning seeks a sequence of actions that guide the robot to collect the most informative data to build a large-scale environmental model or learn a dynamical system. Existing work in informative planning mainly focuses on proposing new planners and applying them to various robotic applications such as environmental monitoring, autonomous exploration, and system identification. The inf...
-
Quality-Diversity (QD) algorithms are powerful exploration algorithms that allow robots to discover large repertoires of diverse and high-performing skills. However, QD algorithms are sample inefficient and require millions of evaluations. In this paper, we propose Dynamics-Aware Quality-Diversity (DA-QD), a framework to improve the sample efficiency of QD algorithms through the use of dynamics mo...
-
We achieved contact-rich flexible object manipulation, which was difficult to control with vision alone. In the unzipping task we chose as a validation task, the gripper grasps the puller, which hides the bag state such as the direction and amount of deformation behind it, making it difficult to obtain information to perform the task by vision alone. Additionally, the flexible fabric bag state con...
-
Comprehensive and correct state estimation with meaningful uncertainties is the basis of object-based perception for automated mobile platforms. According to fatality statistics, the most endangered group of vulnerable road users are single-track two-wheelers (ST2W), consisting mainly of cyclists, motorcyclists, and scooter riders. Due to counter-steering, they need more time to adjust their drivi...
-
The free energy principle from neuroscience provides a brain-inspired perception scheme through a data-driven model learning algorithm called Dynamic Expectation Maximization (DEM). This paper aims at introducing an exper-imental design to provide the first experimental confirmation of the usefulness of DEM as a state and input estimator for real robots. Through a series of quadcopter flight exper...
-
We propose a motion generation model that can achieve robust behavior against environmental changes based on language instructions at a low cost. Conventional robots that communicate with humans use a restricted environment and language to build up a mapping between language and motion, and thus need to prepare a huge training set in order to achieve versatility. Our method trains pairs of languag...
-
Development and Evaluation of a Gait Assistance System Based on Haptic Cane and Active Knee Orthosis
Post-stroke gait rehabilitation is necessary to aid social re-integration. An active knee orthosis (AKO) can aid gait training through the provision of bodyweight support and assistive knee torque. However, its use may cause instability and it does not ensure improved gait symmetry and speed. Use of a speed regulation device such as a robotic cane in conjunction with and AKO may overcome these lim...
-
Due to hemiparesis, stroke survivors frequently develop a dysfunctional gait that is often characterized by an overall decrease in walking speed and a unilateral decrease in step length. With millions currently affected by this dys-functional gait, robust and effective rehabilitation protocols are needed. Although robotic devices have been used in numerous rehabilitation protocols for gait, the la...
-
Robot-aided rehabilitation is pushing forward novel robotic architectures to provide physical therapy. This paper presents a patient-tailored control architecture for upper-limb robot-aided orthopaedic rehabilitation capable of i) adapting the robot workspace on the basis of patient Range of Motion (RoM); ii) generating a tunnel, around the desired path to be followed by the patient, which guarant...
-
Hip exoskeletons may hold potential to augment walking performance and mobility in individuals with disabilities. The purpose of this study was to design and validate a novel autonomous hip exoskeleton with a user-adaptive control strategy capable of reducing the energy cost of level and incline walking in individuals with and without walking impairment. First, in a small cohort of three unimpaire...
-
A pressing challenge in the design of actuated knee prostheses is the ability to address the high variation of speed and torque requirements for the different types and phases of locomotion. This manuscript presents a novel over-actuated knee prosthesis which makes use of a dual motor actuation architecture to address this issue. It utilizes a high speed/low torque motor to enable natural and high...
-
Powered knee prosthesis commonly improves locomotion of above-knee amputees by generating net positive mechanical work at the knee joint which is especially required for movements with active knee extension and flexion such as sit-to-stand maneuvers, level-ground walking with various walking speed, stair/slope ascent ambulation and so forth. These studies tend to refer and trace normal human locom...
-
In order to meet the mobility and physical activity needs of people with impaired limbs function, a novel limbs-free variable structure wheelchair system controled by face-computer interface (FCI) was developed in this study. FCI used facial electromyography (fEMG) as a human intention recognition method from 6 facial movements, and the accuracy of intent recognition reached 97.6% under a series o...
-
Physicians perform minimally invasive percuta-neous procedures under Computed Tomography (CT) image guidance both for the diagnosis and treatment of numerous diseases. For these procedures performed within Computed Tomography Scanners, robots can enable physicians to more accurately target sub-dermal lesions while increasing safety. However, existing robots for this application have limited dexter...
-
In a multirobot system, a number of cyber-physical attacks (e.g., communication hijack, observation per-turbations) can challenge the robustness of agents. This robust-ness issue worsens in multiagent reinforcement learning because there exists the non-stationarity of the environment caused by simultaneously learning agents whose changing policies affect the transition and reward functions. In thi...
-
Communication is an important factor that en-ables agents to work cooperatively in multi-agent reinforcement learning (MARL) contexts. Prior work used continuous message communication whose high representational capacity comes at the expense of interpretability. Allowing agents to learn their own discrete emergent message communication protocols can increase the interpretability for human designer...
-
Safety is of great importance in multi-robot navigation problems. In this paper, we propose a control barrier function (CBF) based optimizer that ensures robot safety with both high probability and flexibility, using only sensor measurement. The optimizer takes action commands from the policy network as initial values and provides refinement to drive the potentially dangerous ones back into safe r...
-
Recent Multi-Agent Deep Reinforcement Learning approaches factorize a global action-value to address non-stationarity and favor cooperation. These methods, however, hinder exploration by introducing constraints (e.g., additive value-decomposition) to guarantee the factorization. Our goal is to enhance exploration and improve sample efficiency of multi-robot mapless navigation by incorporating a pe...
-
Distributed model predictive control (DMPC) concerns how to online control multiple robotic systems with constraints effectively. However, the nonlinearity, nonconvexity, and strong interconnections of dynamic system models and constraints can make the real-time and real-world DMPC implementations nontrivial. Reinforcement learning (RL) algorithms are promising for control policy design. However, ...
-
We propose an adaptive optimisation approach for tuning stochastic model predictive control (MPC) hyper-parameters while jointly estimating probability distributions of the transition model parameters based on performance rewards. In particular, we develop a Bayesian optimisation (BO) algorithm with a heteroscedastic noise model to deal with varying noise across the MPC hyper-parameter and dynamic...
-
An oft-ignored challenge of real-world reinforcement learning is that the real world does not pause when agents make learning updates. As standard simulated environments do not address this real-time aspect of learning, most available implementations of RL algorithms process environment interactions and learning updates sequentially. As a consequence, when such implementations are deployed in the ...
-
We present an automated learning framework for a robotic sketching agent that is capable of learning stroke-based rendering and motor control simultaneously. We formulate the robotic sketching problem as a deep decoupled hierarchical reinforcement learning; two policies for stroke-based rendering and motor control are learned independently to achieve sub-tasks for drawing, and form a hierarchy whe...
-
Achieving robust performance is crucial when applying deep reinforcement learning (RL) in safety critical systems. Some of the state of the art approaches try to address the problem with adversarial agents, but these agents often require expert supervision to fine tune and prevent the adversary from becoming too challenging to the trainee agent. While other approaches involve automatically adjusti...
-
One of the key challenges to deep reinforcement learning (deep RL) is to ensure safety at both training and testing phases. In this work, we propose a novel technique of unsupervised action planning to improve the safety of on-policy reinforcement learning algorithms, such as trust region policy optimization (TRPO) or proximal policy optimization (PPO). We design our safety-aware reinforcement lea...
-
Multi-beam LiDAR sensors are increasingly used in robotics, particularly with autonomous cars for localization and perception tasks, both relying on the ability to build a precise map of the environment. For this, we propose a new real-time LiDAR-only odometry method called CT-ICP (for Continuous-Time ICP), completed into a full SLAM with a novel loop detection procedure. The core of this method, ...
-
In endoscopy, many applications (e.g., surgical navigation) would benefit from a real-time method that can simultaneously track the endoscope and reconstruct the dense 3D geometry of the observed anatomy from a monocular endoscopic video. To this end, we develop a Simultaneous Localization and Mapping system by combining the learning-based appearance and optimizable geometry priors and factor grap...
-
In this paper, we propose a novel direct visual odometry algorithm to take the advantage of a 360-degree camera for robust localization and mapping. Our system extends direct sparse odometry by using a spherical camera model to process equirectangular images without rectification to attain omnidirectional perception. After adapting mapping and optimization algorithms to the new model, camera param...
-
Monocular visual odometry (VO) has attracted extensive research attention by providing real-time vehicle motion from cost-effective camera images. However, state-of-the-art optimization-based monocular VO methods suffer from the scale inconsistency problem for long-term predictions. Deep learning has recently been introduced to address this issue by leveraging stereo sequences or ground-truth moti...
-
In this work we present the first initialization methods equipped with explicit performance guarantees that are adapted to the pose-graph simultaneous localization and mapping (SLAM) and rotation averaging (RA) problems. SLAM and rotation averaging are typically formalized as large-scale nonconvex point estimation problems, with many bad local minima that can entrap the smooth optimization methods...
-
In this paper, we propose a tightly-coupled SLAM system fused with RGB, Depth, IMU and structured plane information. Traditional sparse points based SLAM systems always maintain a mass of map points to model the environment. Huge number of map points bring us a high computational complexity, making it difficult to be deployed on mobile devices. On the other hand, planes are common structures in ma...
-
This paper presents a real-time 3D mapping framework based on global matching cost minimization and LiDAR-IMU tight coupling. The proposed framework comprises a preprocessing module and three estimation modules: odometry estimation, local mapping, and global mapping, which are all based on the tight coupling of the GPU-accelerated voxelized GICP matching cost factor and the IMU preintegration fact...
-
Modern SLAM engines typically rely on high-end sensor rigs and robust algorithms to guarantee the high-quality requirements that self-driving cars and other complex autonomous systems require from 3D point cloud maps. Nonetheless, multiple factors can impact the reconstruction quality and it is not uncommon to end up with generally consistent maps affected by local distortions and artifacts, espec...
-
Unified Representation of Geometric Primitives for Graph-SLAM Optimization Using Decomposed Quadrics
In Simultaneous Localization And Mapping (SLAM) problems, high-level landmarks have the potential to build compact and informative maps compared to traditional point-based landmarks. In this work, we focus on the param-eterization of frequently used geometric primitives including points, lines, planes, ellipsoids, cylinders, and cones. We first present a unified representation based on quadrics, a...
-
Keypoint extraction including both keypoint detection and description is a fundamental step in a wide range of geometric multimedia applications. In recent years, many learning-based approaches for keypoint extraction emerge and achieve promising results. However, they usually design network architectures empirically and lack of considerations about the comprehensive performance, which leads to li...
-
Achieving instinctive multi-grasp control of prosthetic hands typically still requires a large number of sensors, such as electromyography (EMG) electrodes mounted on a residual limb, that can be costly and time consuming to position, with their signals difficult to classify. Deep-learning-based EMG classifiers however have shown promising results over traditional methods, yet due to high computat...
-
Passive prostheses cannot provide the net positive work required at the knee and ankle for step-over stair ascent. Powered prostheses can provide this net positive work, but user synchronization of joint motion and power input are critical to enabling natural stair ascent gaits. In this work, we build on previous phase variable-based control methods for walking and propose a stair ascent controlle...
-
Wrist-driven orthotics have been designed to assist people with C6-7 spinal cord injury, however, the kinematic constraint imposed by such a control strategy can impede mobility and lead to abnormal body motion. This study characterizes body compensation using the novel Tenodesis Grasp Emulator, an adaptor orthotic that allows for the investigation of tenodesis grasping in subjects with unimpaired...
-
Data-driven methods based on neural networks are becoming more widespread for predicting human lower-limb motion. Until now, however, actual examples have focused on only a handful, steady locomotion behaviors. Here we explore if neural network predictors can simultaneously cover many more behaviors including transient ones. Training four common types of predictor networks on a large data set of h...
-
Grasping and manipulation are two of the most important hand functions that allow people to efficiently execute activities of daily living. Over the last years, many robotic devices have been proposed to assist people who suffer from neurological conditions by enhancing their grasping capabilities. In this work, we focus on the development of a robotic exoskeleton glove that can increase the grasp...
-
Lower limb exoskeletons have been successfully used in robotic-assisted rehabilitation. However, the design limitations of exoskeletons mechanics, such as weight and the lack of kinematic compatibility relative to the user's joints, limit the outcomes of treatment. To address these shortcomings, this work presents the design of a magneto-rheological fluid-based actuator for a knee exoskeleton, nam...
-
Certain wheeled mobile robots e.g., electric wheelchairs, can operate through indirect joystick controls from users. Correct steering angle becomes essential when the user should determine the vehicle direction and velocity, in particular for differential wheeled vehicles since the vehicle velocity and direction are controlled with only two actuating wheels. This problem gets more challenging when...
-
Tracks, wheels, and legs are all useful locomotion modes for Unmanned Ground Vehicles (UGVs), and ground robots that combine these mechanisms have the potential to climb over large obstacles. As robot morphologies include more degrees of freedom and obstacles become increasingly large and complex, UGVs will need to rely on automatic motion planning to compute the joint trajectories for traversal. ...
-
Nowadays, wheeled mobile robots constitute a considerable portion of robots in industrial applications. Generally, regardless of their purpose, these systems are not designed to physically interact with humans, other robots, or the environment. In this study, we present a novel safe autonomous mobile - SAM - robot, which is a torque-controlled compliant robot that is conceived for safe human-robot...
-
Search-based techniques have shown great success in motion planning problems such as robotic navigation by discretizing the state space and precomputing motion primitives. However in domains with complex dynamic constraints, constructing motion primitives in a discretized state space is non-trivial. This requires operating in continuous space which can be challenging for search-based planners as t...
-
The gaits of locomoting systems are typically designed to maximize some sort of efficiency, such as cost of transport or speed. Equally important is the ability to modulate such a gait to effect turning maneuvers. For drag-dominated systems, geometric mechanics provides an elegant and practical framework for both ends—gait design and gait modulation. Within this framework, “constraint curvature” m...
-
We focus on the problem of analyzing multiagent interactions in traffic domains. Understanding the space of behavior of real-world traffic may offer significant advantages for algorithmic design, data-driven methodologies, and bench-marking. However, the high dimensionality of the space and the stochasticity of human behavior may hinder the identification of important interaction patterns. Our key...
-
Velocity Planning for self-driving vehicles in a complex environment is one of the most challenging tasks. It must satisfy the following three requirements: safety with regards to collisions; respect of the maximum velocity limits defined by the traffic rules; comfort of the passengers. In order to achieve these goals, the jerk and dynamic objects should be considered, however, it makes the proble...
-
The shapes of most real-world objects are symmetric with respect to at least one plane of symmetry. This information is unconsciously used by humans when they attempt to estimate the shape of an object in presence of uncertainty or missing evidence, for example if the object is partially occluded or if they are exploring the object by touch (i.e. haptic exploration). In robotics, this concept has ...
-
In this work, we carry out structural and al-gorithmic studies of a problem of barrier forming: selecting the minimum number of straight line segments (barriers) that separate several sets of mutually disjoint objects in the plane. The problem models the optimal placement of line sensors (e.g., infrared laser beams) for isolating many types of regions in a pair- wise manner for practical purposes ...
-
Pavement cleaning is a labor-intensive, repetitive task and can be automated. Several autonomous pavement cleaning robots have been developed, pushing research towards their design and autonomous capabilities. Advances in design have been reported in earlier works on a self-reconfigurable robot with four independent steering drive (4ISD) capabilities, Panthera, for pavement cleaning and maintenanc...
-
This paper addresses the problem of visual-inertial odometry (VIO) with a downward facing monocular camera when a micro aerial vehicle (MAV) flying at high altitude (over 100 meters). It is important to note that large scene depth causes visual motion constraints significantly less informative than that in near-sighted scenarios as considered in most existing VIO methods. To cope with this challen...
-
Insulator visual aiming is difficult for washing drone due to the complex washing environment, strong dis-turbance, lack of debugging environment, and other factors. Conventional visual servo control methods often fail to consider these complex factors adequately and fall short in reliable insulator visual aiming. To address these problems, we propose a novel multi-feature fusion-based drone visua...
-
Aerial robots can enhance their safe and agile navigation in complex and cluttered environments by efficiently exploiting the information collected during a given task. In this paper, we address the learning model predictive control problem for quadrotors. We design a learning receding-horizon nonlinear control strategy directly formulated on the system nonlinear manifold configuration space SO(3)...
-
Jet-powered vertical takeoff and landing (VTOL) drones require precise thrust estimation to ensure adequate stability margins and robust maneuvering. Small-scale turbojets have become good candidates for powering heavy aerial drones. However, due to limited instrumentation available in these turbojets, estimating the precise thrust using classical techniques is not straightforward. In this paper, ...
-
Disturbance mainly caused by recoil force in-evitably makes washing drone seriously deviate from the desired position, thereby reducing the cleaning efficiency. It is neces-sary to develop an effective anti-disturbance control method. Although some progresses have been made, the position error thereof is still large, rendering existing methods inapplicable in washing drone. In this paper, we propo...
-
Visual tracking is adopted to extensive unmanned aerial vehicle (UAV)-related applications, which leads to a highly demanding requirement on the robustness of UAV trackers. However, adding imperceptible perturbations can easily fool the tracker and cause tracking failures. This risk is often overlooked and rarely researched at present. Therefore, to help increase awareness of the potential risk an...
-
Drop mitigation is an important function of micro aerial vehicles (MAVs) that are used for internal inspections of enclosed and cluttered structures (height: 5–10 m). The mechanism also allows continuous operation of drones, prevents the downtime required for maintenance and repair and also provides a safer environment for workers who are working below the drones. Some solutions include parachutes...
-
Although there has been tremendous progress in autonomous driving, navigating environments and predicting the behavior of other drivers in the presence of occlusions remains challenging. Cities have started investing in infrastructure sensors that could provide information about occluded spaces. We propose a framework that integrates infrastructure-to-vehicle communication in autonomous vehicle de...
-
The environments of such large industrial machines as waste cranes in waste incineration plants are often weakly observable, where little information about the environ-mental state is contained in the observations due to technical difficulty or maintenance cost (e.g., no sensors for observing the state of the garbage to be handled). Based on the findings that skilled operators in such environments...
-
Knowledge of the robot's load inertial parameters is indispensable for accurate and safe operation, especially in collaborative robotics. However, an intuitive method for online inertial payload identification, usable while the robot is executing another online generated task, is still lacking. In this work, we propose an online payload identification approach based on the momentum observer using ...
-
Dynamic models play a key role in robot motion generation and control and the identification of inertial parameters is a critical component for obtaining an accurate dynamic model of a robot. This paper presents a novel iterative primitive shape division method for the inertia parameter identification of floating-base robots. Describing a robot by a set of primitive shapes with uniform mass distri...
-
Disparity and optical flow estimation are respectively 1D and 2D dense correspondence matching (DCM) tasks in nature. Unsupervised domain adaptation (UDA) is crucial for their success in new and unseen scenarios, enabling networks to draw inferences across different domains without manually-labeled ground truth. In this paper, we propose a general UDA framework (UnDAF) for disparity or optical flo...
-
This work presents a technique for localization of a smart infrastructure node, consisting of a fisheye camera, in a prior map. These cameras can detect objects that are outside the line of sight of the autonomous vehicles (AV) and send that information to AVs using V2X technology. However, in order for this information to be of any use to the AV, the detected objects should be provided in the ref...
-
Automated laboratory experiments have the potential to propel new discoveries, while increasing reproducibility and improving scientists' safety when handling dangerous materials. However, many automated laboratory workflows have not fully leveraged the remarkable advancements in robotics and digital lab equipment. As a result, most robotic systems used in the labs are programmed specifically for ...
-
We present a novel approach to improve 6 degree-of-freedom state propagation for unmanned aerial vehicles in a classical filter through pre-processing of high-speed inertial data with AI algorithms. We evaluate both an LSTM-based approach as well as a Transformer encoder architecture. Both algorithms take as input short sequences of fixed length N of high-rate inertial data provided by an inertial...
-
There is an increasing interest in soft robotic grippers as they exhibit an ability to grip objects of differing shapes, sizes, textures, and even deformable materials, all of which present a difficult challenge to traditional rigid grippers. An ideal soft gripper would exhibit universal gripping with high gripping force and consists of low-cost materials with simple fabrication processes. This pa...
-
Large-area and high-precision tactile sensing information can not only improve the stability of robot grasping but also compensate for the lack of visual information in specific environments such as turbid underwater, dimness, and smoke. In this paper, we devise a universal jamming gripper with high-quality tactile sensing capability. The gripper adopts the particle jamming mechanism for grasping,...
-
Shelves are common in homes, warehouses, and commercial settings due to their storage efficiency. However, this efficiency comes at the cost of reduced visibility and accessibility. When looking from a side (lateral) view of a shelf, most objects will be fully occluded, resulting in a constrained lateral-access mechanical search problem. To address this problem, we introduce: (1) a novel bluction ...
-
In this paper, we present a methodology to design soft-rigid grippers able to perform different manipulation tasks. The main idea is the introduction of wave-shaped hinges whose geometrical parameters can be designed to achieve different three-dimensional impedance characteristics. This allows one to use the same tendon-driven actuation to perform different tasks including grasping objects with di...
-
Accurately simulating whether an object will be lifted securely or dropped during grasping is a longstanding Sim2Real challenge. Soft compliant jaw tips are almost universally used with parallel-jaw robot grippers due to their ability to increase contact area and friction between the jaws and the object to be manipulated. However, interactions between the compliant surfaces and rigid objects are n...
-
Increasing the performance of tactile sensing in robots enables versatile, in-hand manipulation. Vision-based tactile sensors have been widely used as rich tactile feedback has been shown to be correlated with increased performance in manipulation tasks. Existing tactile sensor solutions with high resolution have limitations that include low accuracy, expensive components, or lack of scalability. ...
-
We present a novel method for the direct transfer of grasps and manipulations between objects and hands through utilization of contact areas. Our method fully preserves contact shapes, and in contrast to existing techniques, is not dependent on grasp families, requires no model training or grasp sampling, makes no assumptions about manipulator morphology or kinematics, and allows user control over...
-
A substantially advance skill-set is a prerequisite in the domain of retinal surgery, given that the surgical instruments, constrained by small incisions made on the sclera, should be manipulated in a confined intraocular space. Therefore, robotic technologies with a snake-like architecture may be critical in retinal surgery to overcome this problem. These robots are expected to approach a target ...
-
Autonomous exploration and mapping of unknown terrains employing single or multiple robots is an essential task in mobile robotics and has therefore been widely investigated. Nevertheless, given the lack of unified data sets, metrics, and platforms to evaluate the exploration approaches, we develop an autonomous robot exploration benchmark en-titled Explore-Bench. The benchmark involves various ex...
-
Particle robots are novel biologically-inspired robotic systems where locomotion can be achieved collectively and robustly, but not independently. While its control is currently limited to a hand-crafted policy for basic locomotion tasks, such a multi-robot system could be potentially controlled via Deep Reinforcement Learning (DRL) for different tasks more efficiently. However, the particle robot...
-
This paper presents recent advances to the Affordance Template (AT) task description language. Affordance Templates provide standardized, easy-to-use tools for defining robot manipulation tasks that provide a high level of augmented reality capabilities to facilitate human-in-the-loop operation, but can also be used to support robot autonomy when coupled with various planning tools. While initiall...
-
The Xacro XML macro language can be used to augment the Universal Robot Description Format (URDF) and is part of a critical toolchain from geometric representations to simulation, visualization, and system execution. However, mem-bers of the robotics community, especially newcomers, struggle to troubleshoot and understand the interplay between systems and the Xacro preprocessing pipeline. To bette...
-
We introduce GRiD: a GPU-accelerated library for computing rigid body dynamics with analytical gradients. GRiD was designed to accelerate the nonlinear trajectory opti-mization subproblem used in state-of-the-art robotic planning, control, and machine learning, which requires tens to hundreds of naturally parallel computations of rigid body dynamics and their gradients at each iteration. GRiD leve...
-
This paper presents an intelligent machine which can automatically convert the captured portrait into a physical gadget made up of LEGO bricks. On the contrary to synthesising a 2D image or a virtual 3D object, generating physical 3D assembly object needs to take physical properties and assembly process into consideration, leading to more challenges. To generate brick models for arbitrary portrait...
-
Bionic underwater robots have demonstrated their superiority in many applications. Yet, training their intelligence for a variety of tasks that mimic the behavior of underwater creatures poses a number of challenges in practice, mainly due to lack of a large amount of available training data as well as the high cost in real physical environment. Alternatively, simulation has been considered as a v...
-
In this paper, we study the problem of autonomously seating a teddy bear on a previously unseen chair. To achieve this goal, we present a novel method for robots to imagine the sitting pose of the bear by physically simulating a virtual humanoid agent sitting on the chair. We also develop a robotic system which leverages motion planning to plan SE(2) motions for a humanoid robot to walk to the cha...
-
StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objects
Geometric organization of objects into semantically meaningful arrangements pervades the built world. As such, assistive robots operating in warehouses, offices, and homes would greatly benefit from the ability to recognize and rearrange objects into these semantically meaningful structures. To be useful, these robots must contend with previously unseen objects and receive instructions without sig...
-
We address the problem of devising the means for a robot to rapidly and safely learn insertion skills with just a few human interventions and without hand-crafted rewards or demonstrations. Our InsertionNet version 2.0 provides an improved technique to robustly cope with a wide range of use-cases featuring different shapes, colors, initial poses, etc. In particular, we present a regression-based m...
-
Grasping made impressive progress during the last few years thanks to deep learning. However, there are many objects for which it is not possible to choose a grasp by only looking at an RGB-D image, might it be for physical reasons (e.g., a hammer with uneven mass distribution) or task constraints (e.g., food that should not be spoiled). In such situations, the preferences of experts need to be ta...
-
Deep reinforcement learning (RL) has shown promising results in the motion planning of manipulators. However, no method guarantees the safety of highly dynamic obstacles, such as humans, in RL-based manipulator control. This lack of formal safety assurances prevents the application of RL for manipulators in real-world human environments. Therefore, we propose a shielding mechanism that ensures ISO...
-
Robots deployed in many real-world settings need to be able to acquire new skills and solve new tasks over time. Prior works on planning with skills often make assumptions on the structure of skills and tasks, such as subgoal skills, shared skill implementations, or task-specific plan skeletons, which limit adaptation to new skills and tasks. By contrast, we propose doing task planning by jointly ...
-
Picking cluttered general objects is a challenging task due to the complex geometries and various stacking configurations. Many prior works utilize pose estimation for picking, but pose estimation is difficult on cluttered objects. In this paper, we propose Cluttered Objects Descriptors (CODs), a dense cluttered objects descriptor which can represent rich object structures, and use the pre-trained...
-
Many possible fields of application of robots in real world settings hinge on the ability of robots to grasp objects. As a result, robot grasping has been an active field of research for many years. With our publication we contribute to the endeavor of enabling robots to grasp, with a particular focus on bin picking applications. Bin picking is especially challenging due to the often cluttered and...
-
Robots operating in human-centered environments should have the ability to understand how objects function: what can be done with each object, where this interaction may occur, and how the object is used to achieve a goal. To this end, we propose a novel approach that extracts a self-supervised visual affordance model from human teleoperated play data and leverages it to enable efficient policy le...
-
We present a system for accurately predicting stable orientations for diverse rigid objects. We propose to overcome the critical issue of modelling multimodality in the space of rotations by using a conditional generative model to accurately classify contact surfaces. Our system is capable of operating from noisy and partially-observed pointcloud observations captured by real world depth cameras. ...
-
Reinforcement learning (RL) can in principle let robots automatically adapt to new tasks, but current RL methods require a large number of trials to accomplish this. In this paper, we tackle rapid adaptation to new tasks through the framework of meta-learning, which utilizes past tasks to learn to adapt with a specific focus on industrial insertion tasks. Fast adaptation is crucial because prohibi...
-
We present Neural Descriptor Fields (NDFs), an object representation that encodes both points and relative poses between an object and a target (such as a robot gripper or a rack used for hanging) via category-level descriptors. We employ this representation for object manipulation, where given a task demonstration, we want to repeat the same task on a new object instance from the same category. W...
-
Task-relevant grasping is critical for industrial assembly, where downstream manipulation tasks constrain the set of valid grasps. Learning how to perform this task, however, is challenging, since task-relevant grasp labels are hard to define and annotate. There is also yet no consensus on proper representations for modeling or off-the-shelf tools for performing task-relevant grasps. This work pro...
-
We present a transformer-based network, Human Mesh Deformer (HMD-former), to tackle the problem of 3D human mesh reconstruction from a single RGB image. HMD-former applies a pre-trained CNN to extract image grid features and a transformer decoder to gradually warp the template 3D mesh to the deformed mesh. On each decoder layer, the fine-grained local information of grid features is well utilized ...
-
With recent advancements in CV (computer vision) and AI (Artificial Intelligence) technologies, pointing gesture is becoming an emerging trend for human-robot interaction. Its intuitive and deictic nature makes it an ideal way for giving commands, especially referring spatial information to the robots. In this paper, we propose an augmented pointing gesture estimation method to enable richer and p...
-
Hand pose estimation constitutes prime attainment for human-machine interaction-based applications. Real-time operation is vital in such tasks. Thus, a reliable estimator should exhibit low computational complexity and high precision at the same time. Previous works have explored the regression techniques, including the coordinate regression and heatmap regression methods. Primarily incorporating ...
-
The success of motion prediction for autonomous driving relies on integration of information from the HD maps. As maps are naturally graph-structured, investigation on graph neural networks (GNNs) for encoding HD maps is burgeoning in recent years. However, unlike many other applications where GNNs have been straightforwardly deployed, HD maps are heterogeneous graphs where vertices (lanes) are co...
-
A predictive model for mobility systems capable of understanding the trajectory a user intends to follow in the environment is proposed. Understanding user intention is paramount for any shared-control navigation strategy between a user and an active robotic agent. Equally important however is being able to go beyond simple sample generation to assign probabilistic meaning to the set of possible f...
-
This work investigates how arm swing movements measured by Inertial Motion Unit (IMU) sensors can be used to identify and maintain the walking state in a self-balanced lower-limb exoskeleton for medical use. When an exoskeleton is in a dynamical state during gait, short patterns in IMU signals (e.g. a braking movement) can be hard to extract. Therefore, by relying on a threshold-based classifier c...
-
Goal inference is of great importance for a variety of applications that involve interaction, coordination, and/or competition with goal-oriented agents. Typical goal inference approaches use as many pointwise measurements of the agent's trajectory as possible to pursue a most accurate a-posteriori estimate of the goal. However, taking frequent measurements may not be preferred in situations where...
-
This paper presents a data-driven multiple model framework for estimating the intention of a target from observations. Multiple model (MM) state estimation methods have been extensively used for intention estimation by mapping one intention to one dynamic model assuming one-to-one relations. However, intentions are subjective to humans and it is difficult to establish the one-to-one relations expl...
-
In providing physical assistance to elderly people, ensuring cooperative behavior from the elderly persons is a critical requirement. In sit-to-stand assistance, for example, an older adult must lean forward, so that the body mass can shift towards the feet before a caregiver starts lifting the body. An experienced caregiver guides the older adult through verbal communications and physical interac...
-
Thin, reflective objects such as forks and whisks are common in our daily lives, but they are particularly chal-lenging for robot perception because it is hard to reconstruct them using commodity RGB-D cameras or multi-view stereo techniques. While traditional pipelines struggle with objects like these, Neural Radiance Fields (NeRFs) have recently been shown to be remarkably effective for performi...
-
Standard frame-based cameras that sample light intensity frames are heavily impacted by motion blur for high-speed motion and fail to perceive scene accurately in high-dynamic range environments. Event-based cameras, on the other hand, overcome these limitations by asynchronously detecting the variation in individual pixel intensities. However, event cameras only capture pixels in motion, leading ...
-
Monocular depth estimation in the wild inherently predicts depth up to an unknown scale. To resolve scale ambiguity issue, we present a learning algorithm that leverages monocular simultaneous localization and mapping (SLAM) with proprioceptive sensors. Such monocular SLAM systems can provide metrically scaled camera poses. Given these metric poses and monocular sequences, we propose a self-superv...
-
Covariance estimation for the Iterative Closest Point (ICP) point cloud registration algorithm is essential for state estimation and sensor fusion purposes. We argue that a major source of error for ICP is in the input data itself, from the sensor noise to the scene geometry. Benefiting from recent developments in deep learning for point clouds, we propose a data-driven approach to learn an error ...
-
We propose an efficient method for unsupervised learning of metric depth estimation from a single image in the context of unconstrained videos captured from UAVs. We combine the accuracy of an analytical solution based on odometry with the power of deep learning. First, we show how to correct the noisy odometric measurements by optimizing the alignment between the derotated optical flow and the pr...
-
While modern deep neural networks are performant perception modules, performance (accuracy) alone is insufficient, particularly for safety-critical robotic applications such as self-driving vehicles. Robot autonomy stacks also require these otherwise blackbox models to produce reliable and calibrated measures of confidence on their predictions. Existing approaches estimate uncertainty from these n...
-
The performance of monocular depth estimation generally depends on the amount of parameters and computational cost. It leads to a large accuracy contrast between light-weight networks and heavy-weight networks, which limits their application in the real world. In this paper, we model the majority of accuracy contrast between them as the difference of depth distribution, which we call 'Distribution...
-
Robots capable of traversing flights of stairs play an important role in both indoor and outdoor applications. The capability of accurately identifying a staircase is one of the vital technical functions in these robots. This paper presents a vision-based ascending stair detection algorithm using RGB-Depth (RGB-D) data based on an interpretable model. The method follows the four steps: 1) pre-proc...
-
In this paper, we aim to apply deep saliency prediction to automatic drone exploration, which should consider not only one single image, but multiple images from different view angles or localizations in order to determine the exploration direction. However, little attention has been paid to such saliency prediction problem over multiple-discontinuous-image and none of existing methods take tempor...
-
This paper presents a convex model predictive control framework for versatile dynamic legged motions with negligible leg dynamics. The framework utilizes the single rigid body model linearly approximated around the operating point. With ground reaction forces as direct control inputs to the system, no reference control trajectory needs to be specified in advance. By using the rotation matrix for t...
-
Embedding elastic elements into legged robots through mechanical design enables highly efficient oscillating patterns that resemble natural gaits. However, current trajectory planning techniques miss the opportunity of taking advantage of these natural motions. This work proposes a locomotion planning method that aims to unify traditional trajectory generation with modal oscillations. Our method u...
-
Many controllers for legged robotic systems leverage open- or closed-loop control at discrete hybrid events to enhance stability. These controllers appear in several well studied phenomena such as the Raibert stepping controller, paddle juggling, and swing leg retraction. This work introduces hybrid event shaping (HES): a generalized method for analyzing and designing stable hybrid event controlle...
-
Humans noticeably swing their arms for balancing and locomotion. Although the underlying biomechanical mechanisms have been studied, it is unclear how robots can fully take advantage of these appendages. Most controllers that exploit arms for balance and locomotion rely on feedback and cannot anticipate incoming disturbances and future states. Model predictive controllers readily address these dra...
-
This paper introduces a design method for an efficient and agile quadruped robot. A mixed-integer optimization formulation including the number of gear teeth is derived to obtain the optimal gear ratio that minimizes cost for a running-trot with the target speed of 3 m/s. With the inclusion of integer constraints related to the number of gear teeth, detailed design considerations of gear trains ca...
-
Dynamic jumping with legged robots poses a challenging problem in planning and control. Formulating the jump optimization to allow fast online execution is difficult; efficiently using this capability to generate long-horizon motion plans further complicates the problem. In this work, we present a hierarchical planning framework to address this problem. We first formulate a real-time tractable tra...
-
This paper tackles the problem of nonprehensile object transportation through a legged manipulator. A whole-body control architecture is devised to prevent sliding of the object placed on the tray at the manipulator's end-effector and retain the legged robot balance during walking. The controller solves a quadratic optimization problem to realize the sought transportation task while maintaining th...
-
Navigating natural environments with deformable terrain is a difficult challenge in robotics. Understanding the interaction dynamics between robots and such terrain is an important first step in enabling them to explore these environments. Terramechanics models are largely developed and tested on wheeled and tracked platforms, but with the advent of readily available lightweight legged robots, dev...
-
Visual servoing enables robotic systems to perform accurate closed-loop control, which is required in many applications. However, existing methods require either precise calibration of the robot kinematic model and cameras or use neural architectures that require large amounts of data to train. In this work, we present a method for unsupervised learning of visual servoing that does not require any...
-
This paper presents results from a study on the perception of robot attributes by human observers in their peripheral field of view depending on types of eye-movements of the latter. A between-subjects design is used, where a picture of a robot head is presented on a screen in the peripheral field of view of the participants with two conditions of eye-movements: static and pursuit. The two conditi...
-
Human-in-the-loop (HIL) optimization usually optimizes assistive torque of exoskeletons to minimize the human's energetic expenditure in walking, quantified by metabolic cost. This formulation can, however, result in altered gait pattern of the human joint from the natural pattern, which is undesired. In this paper, we proposed a novel concept of HIL optimization of a hip exoskeleton. The optimiza...
-
For Underwater Vehicle Manipulator Systems (UVMS), the ability to keep a fixed end effector pose is required for intervention tasks. Maintaining a static configuration in a dynamic underwater environment requires significant amounts of energy over time, limiting the operational time for battery powered systems. In this work we consider learning the periodic components of the dynamic flow in order ...
-
ROS is popular in robotic-software development, and thus detecting bugs in ROS programs is important for modern robots. Fuzzing is a promising technique of runtime testing. But existing fuzzing approaches are limited in testing ROS programs, due to neglecting ROS properties, such as multi-dimensional inputs, temporal features of inputs and the distributed node model. In this paper, we develop a ne...
-
This paper presents algorithms for performing data-driven reachability analysis under temporal logic side information. In certain scenarios, the data-driven reachable sets of a robot can be prohibitively conservative due to the inherent noise in the robot's historical measurement data. In the same scenarios, we often have side information about the robot's expected motion (e.g., limits on how much...
-
In this work, we address the problem of formal safety verification for stochastic cyber-physical systems (CPS) equipped with ReLU neural network (NN) controllers. Our goal is to find the set of initial states from where, with a predetermined confidence, the system will not reach an unsafe configuration within a specified time horizon. Specifically, we consider discrete-time LTI systems with Gaussi...
-
Quadratic Program(QP) based state-feedback controllers, whose inequality constraints bound the rate of change of control barrier (CBFs) and lyapunov function with a class-$\mathcal{K}$ function of their values, are sensitive to the parameters of these class-$\mathcal{K}$ functions. The construction of valid CBFs, however, is not straightforward, and for arbitrarily chosen parameters of the QP, the...
-
Many robot control scenarios involve assessing system robustness against a task specification. If either the controller or environment are composed of “black-box” components with unknown dynamics, we cannot rely on formal verification to assess our system. Assessing robustness via exhaustive testing is also often infeasible if the number of possible environments is large compared to experiment cos...
-
We study the class of reach-avoid dynamic games in which multiple agents interact noncooperatively, and each wishes to satisfy a distinct target criterion while avoiding a failure criterion. Reach-avoid games are commonly used to express safety-critical optimal control problems found in mobile robot motion planning. Here, we focus on finding time-consistent solutions, in which future motion plans ...
-
This paper considers a multi-robot team tasked with monitoring an environmental field of interest over long time horizons. The approach is based on a control-theoretic measure of the information collected by the robots, namely a norm of the constructability Gramian. This measure is leveraged in order to learn a distributed multi-robot control policy using the reinforcement learning paradigm. The l...
-
We apply a novel framework for decomposing and reasoning about free space in an environment to a multi-agent persistent monitoring problem. Our decomposition method represents free space as a collection of ellipsoids associated with a weighted connectivity graph. The same ellipsoids used for reasoning about connectivity and distance during high level planning can be used as state constraints in a ...
-
Developing reusable software for mobile robots is still challenging. Even more so for swarm robots, despite the desired simplicity of the robot controllers. Prototyping and experimenting are difficult due to the multi-robot setting and often require robot-robot communication. Also, the diversity of swarm robot hardware platforms increases the need for hardware-independent software concepts. The ma...
-
MOSAIX is a new robot swarm platform built to be used in social settings. Consisting of up to 100 individual robot Tiles, MOSAIX is a social swarm system, aimed at helping humans in social tasks such as opinion-mixing and brainstorming. MOSAIX also has the potential to be used as a platform to study human-swarm interaction and swarm expressivity. MOSAIX is intended to be used outside laboratory se...
-
Involving human operators to support swarms of robots can be beneficial to address increasingly complex scenarios. However, the shared control between multiple operators remains a challenge, especially where communication between the operators is not available. This paper studies the problem of forming a dynamic chain of robots connecting two operators moving within an environment. The robot chain...
-
Potential field approaches have been often used to describe and model interactions within a swarm of robots performing collective motion, also called flocking. Despite the high number of proposed approaches, most have only been tested in simulation and among the minority tested on real robots, even fewer abandoned the laboratory boundaries in favor of real-world scenarios. In this work, we propose...
-
Understanding decentralized dynamics from collective behaviors in swarms is crucial for informing robot controller designs in artificial swarms and multi-agent robotic systems. However, the complexity in agent-to-agent interactions and the decentralized nature of most swarms pose a significant challenge to the extraction of single-robot control laws from collective behaviors. In this work, we cons...
-
Exploration of unknown environments is an important challenge in the field of robotics. While a single robot can achieve this task alone, evidence suggests it could be accomplished more efficiently by groups of robots, with advantages in terms of terrain coverage as well as robustness to failures. Exploration can be guided through belief maps, which provide probabilistic information about which pa...
-
We introduce a new simulation benchmark “Han-doverSim” for human-to-robot object handovers. To simulate the giver's motion, we leverage a recent motion capture dataset of hand grasping of objects. We create training and evaluation environments for the receiver with standardized protocols and metrics. We analyze the performance of a set of baselines and show a correlation with a real-world evaluati...
-
Physical human-robot interaction can improve human ergonomics, task efficiency, and the flexibility of automation, but often requires application-specific methods to detect human state and determine robot response. At the same time, many potential human-robot interaction tasks involve discrete modes, such as phases of a task or multiple possible goals, where each mode has a distinct objective and ...
-
Human-robot handover is a fundamental yet challenging task in human-robot interaction and collaboration. Recently, remarkable progressions have been made in human-to-robot handovers of unknown objects by using learning-based grasp generators. However, how to responsively generate smooth motions to take an object from a human is still an open question. Specifically, planning motions that take human...
-
For immersive interaction in a virtual reality (VR) environment, an omnidirectional treadmill (ODT) can support performance of various locomotive motions (curved walk, side walk, moving with shooting stance) in any direction. When a user performs lateral locomotive motions on an ODT, a control scheme to achieve immersive and safe interaction with the ODT should satisfy robustness in terms of posit...
-
Wheelchair-mounted robotic manipulators have the potential to help the elderly and individuals living with disabilities carry out their activities of daily living (ADLs) independently. Robotics researchers focus on assistive tasks from the perspective of various control schemes and motion types, whereas, health research focuses on clinical assessment and rehabilitation, arguably leaving important ...
-
In this paper, to achieve the goal of aiding the blind and visually impaired (BVI) to read any text not written in Braille, a custom-built, finger-wearable, and electro-tactile based Braille reading system with its Rapid Optical Character Recognition (R-OCR) method is developed. The R-OCR is capable of processing text information in real time using a miniature fish-eye imaging device mounted at th...
-
This paper presents an optimization of an admittance control model for gait balance assistance offered by a walker-type assistive robot. We previously introduced the notion of quasi-passive physical Human-Robot Interaction (pHRI) where a non-wearable assistive device adaptively achieves supportability for providing physical assistance and operability to follow the user's intuitive operation. Aimin...
-
Clustering objects from the LiDAR point cloud is an important research problem with many applications such as autonomous driving. To meet the real-time requirement, existing research proposed to apply the connected-component-labeling (CCL) technique on LiDAR spherical range image with a heuristic condition to check if two neighbor points are connected. However, LiDAR range image is different from ...
-
In this paper, we propose a method for segmentation and shape estimation of multiple deformed cloths stacked on a floor from an image using a CNN-based landmark detector and clustering. The proposed method first estimates landmark positions from the heatmaps generated from the landmark detector and then clusters them using their attributes also given from the landmark detector. The contributions o...
-
Despite its importance, unsupervised domain adaptation (UDA) on LiDAR semantic segmentation is a task that has not received much attention from the research community. Only recently, a completion-based 3 $D$ method has been proposed to tackle the problem and formally set up the adaptive scenarios. However, the proposed pipeline is complex, voxel-based and requires multi-stage inference, which inhi...
-
Knowledge of 3-D object shape is of great importance to robot manipulation tasks, but may not be readily available in unstructured environments. While vision is often occluded during robot-object interaction, high-resolution tactile sensors can give a dense local perspective of the object. However, tactile sensors have limited sensing area and the shape representation must faithfully approximate n...
-
Object classification is a key element that enables effective decision-making in many autonomous systems. A more sophisticated system may also utilize the probability distribution over the classes instead of basing its decision only on the most likely class. This paper introduces new performance metrics: the absolute class error (ACE), expectation of absolute class error (EACE) and variance of abs...
-
Drawing inspiration from biology, we describe the way in which visual sensing with a monocular camera can provide a reliable signal for navigation of mobile robots. The work takes inspiration from the classic paper [3] which described a behavioral strategy pursued by diving sea birds based on a visual cue called time-to-contact. A closely related concept of time-to-transit, $\tau$, is defined, and...
-
Augmented reality (AR) technology has been in-troduced into the robotics field to narrow the visual gap between indoor and outdoor environments. However, without signals from satellite navigation systems, flight experiments in these indoor AR scenarios need other accurate localization approaches. This work proposes a real-time centimeter-level indoor localization method based on psycho-visually in...
-
Robot navigation traditionally relies on building an explicit map that is used to plan collision-free trajectories to a desired target. In deformable, complex terrain, using geometric-based approaches can fail to find a path due to mischaracterizing deformable objects as rigid and impassable. Instead, we learn to predict an estimate of traversability of terrain regions and to prefer regions that a...
-
Previous incremental estimation methods consider estimating a single line, requiring as many observers as the number of lines to be mapped. This leads to the need for having at least 4N state variables, with N being the number of lines. This paper presents the first approach for multi-line incremental estimation. Since lines are common in structured environments, we aim to exploit that structure t...
-
Visual navigation by mobile robots is classically tackled through SLAM plus optimal planning, and more recently through end-to-end training of policies implemented as deep networks. While the former are often limited to waypoint planning, but have proven their efficiency even on real physical environments, the latter solutions are most frequently employed in simulation, but have been shown to be a...
-
Event cameras open up new possibilities for robotic perception due to their low latency and high dynamic range. On the other hand, developing effective event-based vision algorithms that fully exploit the beneficial properties of event cameras remains work in progress. In this paper, we focus on event-based visual odometry (VO). While existing event-driven VO pipelines have adopted continuous-time...
-
How can a robot navigate successfully in rich and diverse environments, indoors or outdoors, along office corridors or trails on the grassland, on the flat ground or the staircase? To this end, this work aims to address three challenges: (i) complex visual observations, (ii) partial observability of local visual sensing, and (iii) multimodal robot behaviors conditioned on both the local environmen...
-
We present a novel method for reliable robot navigation in uneven outdoor terrains. Our approach employs a fully-trained Deep Reinforcement Learning (DRL) network that uses elevation maps of the environment, robot pose, and goal as inputs to compute an attention mask of the environment. The attention mask is used to identify reduced stability regions in the elevation map and is computed using chan...
-
Accurate and robust initialization is crucial in visual-inertial system, which significantly affects the localization accuracy. Most of the existing feature-based initialization methods rely on point features to estimate initial parameters. However, the performance of these methods often decreases in real scene, as point features are unstable and may be discontinuously observed especially in low t...
-
Methods for state estimation that rely on visual information are challenging on legged robots due to rapid changes in the viewing angle of onboard cameras. In this work, we show that by leveraging structure in the way that the robot locomotes, the accuracy of visual-inertial SLAM in these challenging scenarios can be increased. We present a method that takes advantage of the underlying periodic pr...
-
A navigation system which can output drift-free global trajectory estimation with local consistency holds great potential for autonomous vehicles and mobile devices. We propose a tightly-coupled GNSS-aided visual-inertial navigation system (GAINS) which is able to leverage the complementary sensing modality from a visual-inertial sensing pair, which provides high-frequency local information, and a...
-
We propose a continuous-time spline-based formulation for visual-inertial odometry (VIO). Specifically, we model the poses as a cubic spline, whose temporal derivatives are used to synthesize linear acceleration and angular velocity, which are compared to the measurements from the inertial measurement unit (IMU) for optimal state estimation. The spline boundary conditions create constraints betwee...
-
In this paper, we propose a novel object-based visual-inertial navigation system fully embedded in a matrix Lie group and built upon the invariant Kalman filtering theory. Specifically, we focus on relative pose measurements of objects and derive an error equation at the associated tangent space. We prove that the observability property does not suffer from the filter inconsistency and nonlinear e...
-
In this paper, we propose a novel consistent state estimator design for visual-inertial systems. Motivated by first-estimates Jacobian (FEJ) based estimators - which uses the first-ever estimates as linearization points to preserve proper observability properties of the linearized estimator thereby improving the consistency - we carefully model measurement linearization errors due to its Jacobian ...
-
We propose a novel method to tackle the visual-inertial localization problem for constrained camera movements. We use residuals from the different modalities to jointly optimize a global cost function. The residuals emerge from IMU measurements, stereoscopic feature points, and constraints on possible solutions in SE(3). In settings where dynamic disturbances are frequent, the residuals reduce the...
-
Heart rhythm disorders are becoming increasingly prevalent with population aging. Atrial fibrillation ablation (AFA) is a procedure used to treat an irregular heart rhythm (arrhythmia) that starts in the heart's upper chambers. The AFA works by scarring or destroying heart tissue to disrupt aberrant conduction pathways causing the arrhythmia. In hospital cardiac units, a flexible catheter with int...
-
Soft continuum robots have been widely used as guide wires or catheters for minimally invasive surgery (MIS), and the miniaturization and dexterity are very important characteristics of soft continuum robots. As a representative actuation method for soft continuum robots, the steering using an external magnetic field has been actively studied. The magnetic actuation method is appropriate for the m...
-
Tip force sensing for cable-driven continuum robots are vital to provide the force information for safe and reliable human-robot interaction. However, traditional triaxial force sensors usually have a complicated structure occupying its inner lumen, without enough space for additional instrumental tools. To solve this, this paper proposes a fixed and sliding fiber Bragg grating (FBG) sensors-based...
-
Despite the recent advances in continuum robots for minimally invasive surgery or interventions, their applications to endovascular neurosurgery have remained technically challenging due to the difficulty of miniaturization. Aimed at enabling robotic applications to neurovascular interventions for endovascular treatments of stroke or brain aneurysms, we present a telerobotically controlled magneti...
-
Continuum surgical robots can navigate anatomical pathways to reach pathological locations deep inside the human body. Their flexibility, however, generally comes with reduced dexterity at their tip and limited workspace. Building on recent work on eccentric tube robots, this paper proposes a new continuum robot architecture and theoretical framework that combines the flexibility of push/pull actu...
-
The promising automation of flexible surgical instruments and robots is impeded by the lack of sensory means, which allow for sensing of an instrument's position to the surrounding tissue. This work presents a novel sensory method utilizing capacitive proximity sensing to derive a relative localization of a flexible instrument inside a hollow organ. The method is evaluated by exemplary integration...
-
Robotic Surgical Assistants (RSAs) are commonly used to perform minimally invasive surgeries by expert surgeons. However, long procedures filled with tedious and repetitive tasks such as suturing can lead to surgeon fatigue, motivating the automation of suturing. As visual tracking of a thin reflective needle is extremely challenging, prior work has modified the needle with nonreflective contrasti...
-
Medical steerable needles can follow 3D curvilinear trajectories inside body tissue, enabling them to move around critical anatomical structures and precisely reach clinically significant targets in a minimally invasive way. Automating needle steering, with motion planning as a key component, has the potential to maximize the accuracy, precision, speed, and safety of steerable needle procedures. I...
-
This paper contributes a method to design a novel navigation planner exploiting a learning-based collision prediction network. The neural network is tasked to predict the collision cost of each action sequence in a predefined motion primitives library in the robot's velocity-steering angle space, given only the current depth image and the estimated linear and angular velocities of the robot. Furth...
-
The next generation of Mars rotorcrafts requires on-board autonomous hazard avoidance landing. To this end, this work proposes a system that performs continuous multi-resolution height map reconstruction and safe landing spot detection. Structure-from-Motion measurements are aggregated in a pyramid structure using a novel Optimal Mixture of Gaus-sians formulation that provides a comprehensive unce...
-
Flying insect robots weighing less than a gram (FIRs) have advantages over their larger counterparts due to their low materials cost, small size, and low weight, allowing for deployment in large numbers. Control autonomy in such aircraft introduces challenges arising from their small size such as high-speed dynamics, limited power and payload capacity. Previous work has produced and characterized ...
-
Despite recent progress of robotic exploration, most methods assume that drift-free localization is available, which is problematic in reality and causes severe distortion of the reconstructed map. In this work, we present a systematic exploration mapping and planning framework that deals with drifted localization, allowing efficient and globally consistent reconstruction. A real-time re-integrati...
-
Relative localization is an important ability for multiple robots to perform cooperative tasks in GPS-denied environments. This paper presents a novel autonomous positioning framework for monocular relative localization of multiple tiny flying robots. This approach does not require any groundtruth data from external systems or manual labeling. Instead, the proposed framework is able to label real-...
-
Recent research has enabled fixed-wing unmanned aerial vehicles (UAVs) to maneuver in constrained spaces through the use of direct nonlinear model predictive control (NMPC) [1]. However, this approach has been limited to a priori known maps and ground truth state measurements. In this paper, we present a direct NMPC approach that leverages NanoMap [2], a light-weight point cloud mapping framework,...
-
To certify UAV operations in populated areas, risk mitigation strategies - such as Emergency Landing (EL) - must be in place to account for potential failures. EL aims at reducing ground risk by finding safe landing areas using on-board sensors. The first contribution of this paper is to present a new EL approach, in line with safety requirements introduced in recent research. In particular, the p...
-
Micro-scale robots introduce great prospective into many different medical applications such as targeted drug delivery, minimally invasive surgery and localized bio-metric diagnostics. This research presents a method for object detection and tracking system of a chain-like magnetic microsphere robots using ultrasound imaging in an in-vitro environment. The method estimates the position of the micr...
-
Jumping motion is an effective way to overcome large obstacles, especially for the miniature robots. However, controlling of the jumping trajectory on a centimeter scale robot is not easy due to the limitation of size and payload. None of the jumping robots lighter than 90 g achieved the feedback control of their jumping height and take-off angle independently. In this work, we proposed a miniatur...
-
To date, untethered micro/nanorobots have attracted considerable attention in various aspects due to their unique potential for in-vivo applications such as the targeted therapy. One of the most promising types of micro/nanorobots is the class of ferromagnetic microrobots which can be efficiently actuated via gradient/rotational magnetic field generated by less costly electromagnetic coil systems....
-
Beetles can walk smoothly on the meshed surface without slipping or getting stuck in the meshed surface due to its stiffness-variable tarsi and expandable hooks on the tip of tarsi. In this study, we find that beetles bend and open their claws proactively to walk freely. Inspired by the mechanism, we designed a centimeter-scale climbing robot, equipping an artificial claw to open and bend in the s...
-
In this paper, we fabricated a soft sensor based on PEDOT:PSS for thin film structure. The developed soft sensor can measure the contraction force at real time to be embedded in a modular bio-actuator [1]. The modular actuator generated contraction forces at 0.3 mN when applying electric pulse stimulation. To measure millinewton contraction forces and make a built in sensor, we fabricated a soft s...
-
This paper presents four data-driven system models for a magnetically controlled swimmer. The models were derived directly from experimental data, and the accuracy of the models was experimentally demonstrated. Our previous study successfully implemented two non-model-based control algorithms for 3D path-following using PID and model reference adaptive controller (MRAC). This paper focuses on syst...
-
A key problem in robotic locomotion is in finding optimal shape changes to effectively displace systems through the world. Variational techniques for gait optimization require estimates of body displacement per gait cycle; however, these estimates introduce error due to unincluded high order terms. In this paper, we formulate existing estimates for displacement, and describe the contribution of lo...
-
The task space control of robot manipulators requires solving the thorny problem of stabilizing the compound system {outer controller - inner controller - robot manipulator}. To stabilize this compound system, both controllers must be designed by the user to achieve convergence of the tracking error. This problem is tricky to solve in the case of the control of an industrial robot manipulator beca...
-
Complex assembly tasks remain huge challenge for robots because the traditional control methods rely on complicated contact state analysis. Reinforcement learning (RL) becomes one of the preferred embodiments to construct the control strategy of complex tasks. In this paper, the method of model-driven RL (MDRL) is employed to construct the control strategy. Then a completely innovative action dime...
-
The automatic generation of (dis)assembly sequences for complex technical products is a challenging field. Complex products like vehicles consist of numerous different components. Determining the sequence using a brute-force-approach by testing all components for disassembly one after another in a loop until all components are disassembled is laborious and costly. In industrial scenarios, a large ...
-
This paper introduces a discrete-continuous action space to learn insertion primitives for robotic assembly tasks. Primitives are sequences of elementary actions with certain exit conditions, such as “pushing down the peg until contact”. Since the primitive is an abstraction of robot control commands and encodes human prior knowledge, it reduces the exploration difficulty and yields better learnin...
-
We present the first realization of an assembly sequence planning framework for large-scale and complex 3D real-world CAD scenarios. Other than in academic benchmark data sets, in our scenario each assembled part is allowed to contain flexible fastening elements and the number of assembled parts is quite high. With our framework we are able to derive a meaningful assembly priority graph for the pa...
-
This paper presents the design and development of a robotic system to give physical assistance to the elderly or people with neurological disorders such as Ataxia or Parkin-son's. In particular, we propose using a mobile collaborative robot with an interaction-assistive whole-body interface to help people unable to maintain balance. The robotic system consists of an Omni-directional mobile base, a...
-
Collaborative heterogeneous robots are expected to perform comprehensive perception, mapping and coordination in search and rescue scenarios. The challenge of collaboration between heterogeneous robots lies in their huge differences in perception, mobility and processing capabilities. In this paper, a novel collaborative UAV-UGV mapping framework is proposed in GNSS-denied and unknown environments...
-
Enabling autonomous operation of large-scale construction machines, such as excavators, can bring key benefits for human safety and operational opportunities for applications in dangerous and hazardous environments. To facilitate robot autonomy, robust and accurate state-estimation remains a core component to enable these machines for operation in a diverse set of complex environments. In this wor...
-
In autonomous vehicles or robots, point clouds from LiDAR can provide accurate depth information of objects compared with 2D images, but they also suffer a large volume of data, which is inconvenient for data storage or transmission. In this paper, we propose a Range image-based Point Cloud Compression method, R-PCC, which can reconstruct the point cloud with uniform or non-uniform accuracy loss. ...
-
This paper presents a real-time control framework for consensus in operational space for robotic manipulators while satisfying task and input constraints. Consensus in operational space, as compared to joint space, enables heterogeneous robotic manipulators to achieve consensus. However, traditional frameworks tend to ignore task and input constraints while achieving consensus in operational space...
-
Robots need object-level scene understanding to manipulate objects while reasoning about contact, support, and occlusion among objects. Given a pile of objects, object recognition and reconstruction can identify the boundary of object instances, giving important cues as to how the objects form and support the pile. In this work, we present a system, SafePicking, that integrates object-level mappin...
-
We propose a method that actively estimates contact location between a grasped rigid object and its environment and uses this as input to a peg-in-hole insertion policy. An estimation model and an active tactile feedback controller work collaboratively to estimate the external contacts accurately. The controller helps the estimation model get a better estimate by regulating a consistent contact mo...
-
As service robots begin to be deployed to assist humans, it is important for them to be able to perform a skill as ubiquitous as pouring. Specifically, we focus on the task of pouring an exact amount of water without any environmental instrumentation, that is, using only the robot's own sensors to perform this task in a general way robustly. In our approach we use a simple PID controller which use...
-
For collaborative tasks, involving handovers, humans are able to exploit visual, non-verbal cues, to infer physical object properties, like mass, to modulate their actions. In this paper, we investigate how the different levels of liquid inside a cup can be inferred from the observation of the movement of the person handling the cup. We model this mechanism from human experiments and incorporate i...
-
Ex-DoF: Expansion of Action Degree-of-Freedom with Virtual Camera Rotation for Omnidirectional Image
Inter-robot transfer of training data is a little explored topic in learning- and vision-based robot control. Here we propose a transfer method from a robot with a lower Degree-of-Freedom (DoF) to one with a higher DoF utilizing the omnidirectional camera image. The virtual rotation of the robot camera enables data augmentation in this transfer learning process. As an experimental demonstration, a...
-
Previous studies suggest that bipedal robots that have prismatic compliance in the legs can achieve efficient walking. However, how this efficiency can be achieved still remains an open research problem. In this study, we developed a 2-degree-of-freedom planar bipedal robot comprising Neidhart springs and five-bar parallel mechanisms. The five-bar parallel mechanism allows the robot to achieve pri...
-
Experimental demonstration of complex robotic behaviors relies heavily on finding the correct controller gains. This painstaking process is often completed by a domain expert, requiring deep knowledge of the relationship between parameter values and the resulting behavior of the system. Even when such knowledge is possessed, it can take significant effort to navigate the nonintuitive landscape of ...
-
This paper presents a Non-Linear Model Predictive Controller for humanoid robot locomotion with online step adjustment capabilities. The proposed controller considers the Centroidal Dynamics of the system to compute the desired contact forces and torques and contact locations. Differently from bipedal walking architectures based on simplified models, the presented approach considers the reduced ce...
-
In recent years, the capabilities of legged locomotion controllers have been significantly advanced enabling them to traverse basic types of uneven terrain without visual perception. However, safely and autonomously traversing longer distances over difficult uneven terrain requires appropriate motion planning using online collected environmental knowledge. In this paper, we present such a novel me...
-
Recently, work on reinforcement learning (RL) for bipedal robots has successfully learned controllers for a variety of dynamic gaits with robust sim-to-real demonstrations. In order to maintain balance, the learned controllers have full freedom of where to place the feet, resulting in highly robust gaits. In the real world however, the environment will often impose constraints on the feasible foot...
-
This paper presents an online walking synthesis methodology to enable dynamic and stable walking on constrained footholds for underactuated bipedal robots. Our approach modulates the change of angular momentum about the foot-ground contact pivot at discrete impact using pre-impact vertical center of mass (COM) velocity. To this end, we utilize the underactuated Linear Inverted Pendulum (LIP) model...
-
Gait generation frameworks for humanoid robots typically assume a constant centroidal angular momentum (CAM) throughout the walking cycle, which induces undesirable contact torques in the feet and results in performance degradation. In this work, we present a novel algorithm to learn the CAM online and include the obtained knowledge within the closed-form solutions of the Divergent Component of Mo...
-
Recent work on sim-to-real learning for bipedal locomotion has demonstrated new levels of robustness and agility over a variety of terrains. However, that work, and most prior bipedal locomotion work, have not considered locomotion under a variety of external loads that can significantly influence the overall system dynamics. In many applications, robots will need to maintain robust locomotion und...
-
In this paper, we propose a multi-domain control parameter learning framework that combines Bayesian Optimization (BO) and Hybrid Zero Dynamics (HZD) for locomotion control of bipedal robots. We leverage BO to learn the control parameters used in the HZD-based controller. The learning process is firstly deployed in simulation to optimize different control parameters for a large repertoire of gaits...
-
Quadrotors are highly nonlinear dynamical systems that require carefully tuned controllers to be pushed to their physical limits. Recently, learning-based control policies have been proposed for quadrotors, as they would potentially allow learning direct mappings from high-dimensional raw sensory observations to actions. Due to sample inefficiency, training such learned controllers on the real pla...
-
Learning performant robot manipulation policies can be challenging due to high-dimensional continuous actions and complex physics-based dynamics. This can be alleviated through intelligent choice of action space. Operational Space Control (OSC) has been used as an effective task-space controller for manipulation. Nonetheless, its strength depends on the underlying modeling fidelity, and is prone t...
-
Kinematics and instantaneous kinematics are fundamental in many robotic tasks, such as positioning and collision avoidance. Existing learning methods mainly concern a single robot, and small-scale networks are sufficient for considerable approximation accuracy. A question is: Can we learn a kinematics model that can generalize to various robots rather than a single robot? This paper studies the ki...
-
The challenge of mapping indoor environments is addressed. Typical heuristic algorithms for solving the motion planning problem are frontier-based methods, that are especially effective when the environment is completely unknown. However, in cases where prior statistical data on the environment's architectonic features is available, such algorithms can be far from optimal. Furthermore, their calcu...
-
Sampling-based Model Predictive Control (MPC) is a flexible control framework that can reason about non-smooth dynamics and cost functions. Recently, significant work has focused on the use of machine learning to improve the performance of MPC, often through learning or fine-tuning the dynamics or cost function. In contrast, we focus on learning to optimize more effectively. In other words, to imp...
-
The ability of Gaussian processes (GPs) to predict the behavior of dynamical systems as a more sample-efficient alternative to parametric models seems promising for real-world robotics research. However, the computational complexity of GPs has made policy search a highly time and memory consuming process that has not been able to scale to larger problems. In this work, we develop a policy optimiza...
-
Quadruped locomotion is rapidly maturing to a degree where robots now routinely traverse a variety of unstructured terrains. However, while gaits can be varied typically by selecting from a range of pre-computed styles, current planners are unable to vary key gait parameters continuously while the robot is in motion. The synthesis, on-the-fly, of gaits with unexpected operational characteristics o...
-
We develop a task-independent predictive framework that estimates hip, knee and ankle future behavior from sonomyographic sensing of quadriceps musculature. Two regression models, support vector regression and Gaussian process regression, were trained and tested such that no ambulation mode recognition was required. Sonomyography features of the anterior thigh musculature were extracted during the...
-
Fast and accurate tracking of an object's motion is one of the key functionalities of a robotic system for achieving reliable interaction with the environment. This paper focuses on the instance-level six-dimensional (6D) pose tracking problem with a symmetric and textureless object under occlusion. We propose a Temporally Primed 6D pose tracking framework with Auto-Encoders (TP-AE) to tackle the ...
-
We propose a novel framework to estimate the confidence of a disparity map taking into account, for the first time, the uncertainty affecting the confidence estimation process itself. Conversely to other tasks such as disparity estimation, the uncertainty of confidence directly hints that the confidence should be increased if initially low, but with high uncertainty, decreased otherwise. By modell...
-
This paper studies the complex task of simultaneous multi-object 3D reconstruction, 6D pose and size estimation from a single-view RGB-D observation. In contrast to instance- level pose estimation, we focus on a more challenging problem where CAD models are not available at inference time. Existing approaches mainly follow a complex multi-stage pipeline which first localizes and detects each objec...
-
We present an approach to synthesize novel views from dynamics scenes captured by multi-view videos of cameras mounted on a driving vehicle. We unify existing methods and propose a new training loss to explicitly disentangle the static background from the dynamic foreground objects using scene flow's magnitude, learnt only from proxy 2D optical flow supervision. We obtain high quality static and d...
-
Scene flow prediction is a challenging task that aims at jointly estimating the 3D structure and 3D motion of dynamic scenes. The previous methods concentrate more on point-wise estimation instead of considering the correspondence between objects as well as lacking the sensation of high-level semantic knowledge. In this paper, we propose a concise yet effective method for scene flow prediction. Th...
-
Loop closure detection is an important building block that ensures the accuracy and robustness of simultaneous localization and mapping (SLAM) systems. Due to their generalization ability, CNN-based approaches have received increasing attention. Although they normally benefit from training on datasets that are diverse and reflective of the environments, new environments often emerge after the mode...
-
In this paper, we propose a novel LiDAR-Inertial-Visual sensor fusion framework termed R3LIVE, which takes advantage of measurement of LiDAR, inertial, and visual sensors to achieve robust and accurate state estimation. R3LIVE consists of two subsystems, a LiDAR-Inertial odometry (LIO) and a Visual-Inertial odometry (VIO). The LIO subsystem (FAST-LIO) utilizes the measurements from LiDAR and inert...
-
Multi-agent pathfinding (MAPF) has been widely used to solve large-scale real-world problems, e.g., automation warehouses. The learning-based, fully decentralized framework has been introduced to alleviate real-time problems and simultaneously pursue optimal planning policy. However, existing methods might generate significantly more vertex conflicts (or collisions), which lead to a low success ra...
-
Autonomous navigation of mobile robots is a well-studied problem in robotics. However, the navigation task becomes challenging when multi-robot systems have to cooperatively navigate dynamic environments with deadlock-prone layouts. We present a Distributed Timed Elastic Band (DTEB) Planner that combines Prioritized Planning with the online TEB trajectory Planner, in order to extend the capabiliti...
-
We perform a systematic exploration of the principle of Space Utilization Optimization (SUO) as a heuristic for planning better individual paths in a decoupled multi-robot path planner, with applications to both one-shot and life-long multi-robot path planning problems. We show that the heuristic set, SU - I, preserves single path optimality and significantly reduces congestion that naturally happ...
-
This paper studies a multi-robot visibility-based pursuit-evasion problem in which a group of pursuer robots are tasked with detecting an evader within a two dimensional polygonal environment. The primary contribution is a novel formulation of the pursuit-evasion problem that modifies the pursuers' objective by requiring that the evader still be de-tected, even in spite of the failure of any singl...
-
Multi-agent interactions are important to model for forecasting other agents' behaviors and trajectories. At a certain time, to forecast a reasonable future trajectory, each agent needs to pay attention to the interactions with only a small group of most relevant agents instead of unnecessarily paying attention to all the other agents. However, existing attention modeling works ignore that human a...
-
We formalize and study the multi-goal task assignment and path finding (MG-TAPF) problem from theoretical and algorithmic perspectives. The MG-TAPF problem is to compute an assignment of tasks to agents, where each task consists of a sequence of goal locations, and collision-free paths for the agents that visit all goal locations of their assigned tasks in sequence. Theoretically, we prove that th...
-
This paper is focused on the cooperative trajectory planning problem for multiple car-like robots in a cluttered and unstructured environment narrowed by static obstacles. The concerned multi-vehicle trajectory planning (MVTP) problem is challenging because i) the scenario is nonconvex and tiny; ii) the vehicle kinematics is nonconvex; and iii) a feasible homotopy class is unavailable a priori. We...
-
We present a novel path-planning algorithm to reduce localization error for a network of robots cooperatively localizing via inter-robot range measurements. The quality of localization with range measurements depends on the configuration of the network, and poor configurations can cause substantial localization errors. To reduce the effect of network configuration on localization error for moving ...
-
Disaster areas involving floods and earthquakes are commonly large, with the rescue time being quite tight, suggesting multi-Unmanned Aerial Vehicles (UAV) exploration rather than employing a single UAV. For such scenarios, current UAV exploration is modeled as a Coverage Path Planning (CPP) problem to achieve full area coverage in the presence of obstacles. However, the UAV's endurance capability...
-
In this paper, we present a data acquisition method to realize a distributed tactile sensor system that can provide wide-range, and high-sensitivity with a small data size for communication. Since the data size is proportional to the number of acquired data and the resolution of the data, we propose systems to increase the resolution of the sensor output values without increasing the amount of dat...
-
Tactile sensing typically involves active exploration of unknown surfaces and objects, making it especially effective at processing the characteristics of materials and textures. A key property extracted by human tactile perception in material classification is surface roughness, which relies on measuring vibratory signals using the multi-layered fingertip structure. Existing robotic systems lack ...
-
This work presents a new version of tactile-sensing finger, GelSlim 3.0, which integrates the ability to sense high-resolution shape, force, and slip in a more compact form factor than previous implementations, designed for cluttered bin-picking scenarios. The novel design integrates real-time model-based algorithms to measure shape, estimate the 3-D contact force distribution, and detect incipien...
-
As robots move toward more complex environments, imbuing them with a sense of touch similar to humans becomes increasingly important. To fulfill that goal, there has been significant research conducted in the past few decades to develop a tactile sensor that matches human level touch capabilities. Recently, the progress in capacitive touch screens has made capacitive sensing a very appealing optio...
-
Collision risk and smoothness are the most important factors in global path planning. Currently, planning methods that reduce global path collision risk and improve its smoothness through numerical optimization have achieved good results. However, these methods cannot always optimize the path. The reason is all points on the path are considered as decision variables, which leads to the high dimens...
-
3D meshes offer a computationally efficient but still quite accurate path to the understanding of a robot's environment. While mesh reconstructions are often employed in indoor regions where regular planar surfaces dominate the scene, their use in urban outdoor environments has been under-explored. This is as outdoor urban environments produce a significant contrast between preserving discontinuit...
-
Most autonomous driving and robotic applications require retrieving map data around the vehicle's current location. Those maps can cover large areas and are often stored in a compressed form to save memory and allow for efficient transmission. In this paper, we address the problem of place recognition in a compressed point cloud map. To this end, we propose a novel deep neural network architecture...
-
In this paper, a novel adjustable-stiffness force sensor is developed for multitask measurements requiring different force resolutions and ranges. The applied force of the force sensor is indirectly measured through the linear deformation instead of the structure strain through an optical linear encoder. The main structure of the force sensor is actually a linear variable stiffness mechanism with ...
-
Soft robotics is an emerging field that yields promising results for tasks that require safe and robust interactions with the environment or with humans, such as grasping, manipulation, and human-robot interaction. Soft robots rely on intrinsically compliant components and are difficult to equip with traditional, rigid sensors which would interfere with their compliance. We propose a highly flexib...
-
For assembly tasks, it is essential to fix target parts firmly and accurately estimate their poses. Several rigid jigs for individual parts are frequently used in assembly factories to achieve a precise and time-efficient product assembly. However, providing customized jigs is time-consuming. In this study, to address the lack of versatility in the shapes for which jigs can be used, we developed a...
-
Handed Shearing Auxetics (HSA) are a promising structure for making electrically driven robots with distributed compliance that convert a motors rotation and torque into extension and force. These structures expand and contract by changing an internal angle between links, the evolution of the structure as this angle changes is known as the auxetic trajectory. We overcome past limitations on the ra...
-
Proprioception and variable stiffness are two trending topics in soft robotics research. The former could endow soft robots with the ability to perceive the environment as well as their internal states without the need of dedicated sensors, while the latter could strengthen the otherwise excessive compliance, enabling soft robots for tasks which require a higher force. Both directions have been ex...
Introduction
Conference ICRA2022 accepted paper complete List. Top ranking conferences for AI and Robotics communities. Total Accepted Paper Count 935