DeepNLP ICRA2024 Accepted Paper List AI Robotic and STEM Top Conference & Journal Papers
-
Model-predictive control (MPC) is a powerful tool for controlling highly dynamic robotic systems subject to complex constraints. However, MPC is computationally demanding, and is often impractical to implement on small, resource-constrained robotic platforms. We present TinyMPC, a high-speed MPC solver with a low memory footprint targeting the microcontrollers common on small robots. Our approach ...
-
This study proposes a novel movable microfluidic chip in which a microfluidic chip is integrated into a robotic manipulator for manipulating oocytes. The microfluidic device has the ability to release a single oocyte with a gap effect. The robotic manipulator can control the position of the microfluidic chip. The microfluidic chip with a pipette tip is directly fabricated using 3D printing. Xenopu...
-
We propose an interpretable framework for reading analog gauges that is deployable on real world robotic systems. Our framework splits the reading task into distinct steps, such that we can detect potential failures at each step. Our system needs no prior knowledge of the type of gauge or the range of the scale and is able to extract the units used. We show that our gauge reading algorithm is able...
-
Parameter convergence in adaptive control is crucial for improving the stability and robustness of robotic systems. Nevertheless, a stringent condition named persistent excitation (PE) needs to be satisfied to ensure parameter convergence in the conventional adaptive robot control. Composite learning robot control (CLRC) is an innovative methodology that guarantees parameter convergence under a co...
-
Estimating the region of attraction (RoA) for a robot controller is essential for safe application and controller composition. Many existing methods require a closed-form expression that limit applicability to data-driven controllers. Methods that operate only over trajectory rollouts tend to be data-hungry. In prior work, we have demonstrated that topological tools based on Morse Graphs (directed...
-
Autonomous robots must navigate reliably in unknown environments even under compromised exteroceptive perception, or perception failures. Such failures often occur when harsh environments lead to degraded sensing, or when the perception algorithm misinterprets the scene due to limited generalization. In this paper, we model perception failures as invisible obstacles and pits, and train a reinforce...
-
Understanding how humans leverage semantic knowledge to navigate unfamiliar environments and decide where to explore next is pivotal for developing robots capable of human-like search behaviors. We introduce a zero-shot navigation approach, Vision-Language Frontier Maps (VLFM), which is inspired by human reasoning and designed to navigate towards unseen semantic objects in novel environments. VLFM...
-
Geometric regularity, which leverages data symmetry, has been successfully incorporated into deep learning architectures such as CNNs, RNNs, GNNs, and Transformers. While this concept has been widely applied in robotics to address the curse of dimensionality when learning from high-dimensional data, the inherent reflectional and rotational symmetry of robot structures has not been adequately explo...
-
Reinforcement learning (RL) for bipedal locomotion has recently demonstrated robust gaits over moderate terrains using only proprioceptive sensing. However, such blind controllers will fail in environments where robots must anticipate and adapt to local terrain, which requires visual perception. In this paper, we propose a fully-learned system that allows bipedal robots to react to local terrain w...
-
Robotic learning for navigation in unfamiliar environments needs to provide policies for both task-oriented navigation (i.e., reaching a goal that the robot has located), and task-agnostic exploration (i.e., searching for a goal in a novel setting). Typically, these roles are handled by separate models, for example by using subgoal proposals, planning, or separate navigation strategies. In this pa...
-
Safe corridor-based Trajectory Optimization (TO) presents an appealing approach for collision-free path planning of autonomous robots, because its convex formulation can guarantee global optimality. The safe corridor is constructed based on the obstacle map, however, the non-ideal perception induces uncertainty, which is rarely considered in the context of trajectory generation. In this paper, we ...
-
Safety is a core challenge of autonomous robot motion planning, especially in the presence of dynamic and uncertain obstacles. Many recent results use learning and deep learning-based motion planners and prediction modules to predict multiple possible obstacle trajectories and generate obstacle-aware ego robot plans. However, planners that ignore the inherent uncertainties in such predictions incu...
-
Partially observable Markov decision processes (POMDPs) have been widely used in many robotic applications for sequential decision-making under uncertainty. POMDP online planning algorithms such as Partially Observable Monte-Carlo Planning (POMCP) can solve very large POMDPs with the goal of maximizing the expected return. But the resulting policies cannot provide safety guarantees which are imper...
-
Environments with regions of uncertain traversability can be modeled as roadmaps with probabilistic edges for efficient planning under uncertainty. We would like to generate roadmaps that enable planners to efficiently find paths with expected low costs through uncertain environments. The roadmap must be sparse so that the planning problem is tractable, but still contain edges that are likely to c...
-
In many robotic systems, the holding state consumes power, limits operating time, and increases operating costs. Electrostatic clutches have the potential to improve robotic performance by generating holding torques with low power consumption. A key limitation of electrostatic clutches has been their low specific shear stresses which restrict generated holding torque, limiting many applications. H...
-
This paper presents a bionic foldable wing that imitates the hind wing of ladybirds. Based on the folding mechanism of the hind wing of ladybirds and the theory of origami, the motion model of the bionic foldable wing is established, yield the motion law of the crease angles and the variation relationship between the panels are obtained. Bionic foldable wings utilise shape memory alloy to drive wi...
-
Variable Stiffness Mechanisms (VSM) are becoming ubiquitous in mechatronics given the benefit they provide in terms of safety and performance. Despite these assets, VSMs remain fairly complex mechanical devices lacking in compactness, ease of manufacturing and accessibility. In addition, the scarcity of commercially available VSMs requires that such systems are mostly designed in-house. We propose...
-
Plant cells expand and elongate. Their cumulative actuation defines organ morphing. Inspired by this modular transformability, this study proposes a modular concept for growing robots that will be able to grow by adding at their tip Transformable Modules (TMs). We provide a two-module implementation to evaluate the concept viability. We designed and characterized Shape-Retention Bellows (SRBs) tha...
-
This paper presents the design and experimental results of a proprioceptive, high-bandwidth quasi-direct drive (QDD) actuator for highly dynamic robotic applications. A comprehensive review of the mechanical design of the PULSE115-60 actuator is presented, with particular focus on the design parameters affecting the dynamic performance of the actuator and a full specification is provided. Fundamen...
-
With the development of society, aging population and the number of stroke patients is increasing year by year. Rehabilitation exoskeleton can help patients to carry out rehabilitation training and improve their activities of daily living (ADL). First of all, a reconfigurable exoskeleton for upper limb rehabilitation is designed in this paper. The exoskeleton combines gravity compensation with lef...
-
A support structure for flexible displays such as OLED or flexible LEDs was developed using the flexible omnidirectional driving gear mechanism. It is a gear mechanism having two degrees of freedom on one surface. This flexible display mechanism is expected to be placed inside a car dashboard as a human interface and for workspace optimization. In this study, we propose a novel flexible omnidirect...
-
Flexible manufacturing lines are required to meet the demand for customized and small batch-size products. Even though state-of-the-art tactile robots may provide the versatility for increased adaptability and flexibility, their potential is yet to be fully exploited. To support robotics deployment in manufacturing, we propose a task-based tactile robot programming paradigm that uses an object-cen...
-
In this paper, we consider the problem of safety analysis of a closed-loop control system with anytime perception sensor. We formalize the framework and present a general procedure for safety analysis using reachable set computation. We instantiate the procedure for two concrete classes, namely, the classical discrete-time linear system with linear state feedback controller and an extension with v...
-
In the realm of robotics, numerous downstream robotics tasks leverage machine learning methods for processing, modeling, or synthesizing data. Often, this data comprises variables that inherently carry geometric constraints, such as the unit-norm condition of quaternions representing rigid-body orientations or the positive definiteness of stiffness and manipulability ellipsoids. Handling such geom...
-
In this work, we address the problem of control synthesis for a homogeneous team of robots given a global temporal logic specification and formal user preferences for relaxation in case of infeasibility. The relaxation preferences are represented as a Weighted Finite-state Edit System and are used to compute a relaxed specification automaton that captures all allowable relaxations of the mission s...
-
This paper introduces an iterative approach to multi-agent route planning under chance constraints. A heterogeneous team of agents with various capabilities is tasked with a Capability Temporal Logic (CaTL) mission, a fragment of Signal Temporal Logic. The agents’ motion is modeled as a finite weighted graph, where the weights represent travel durations. Given the probability distribution over the...
-
Reviewing the previous work of diversity Reinforcement Learning, diversity is often obtained via an augmented loss function, which requires a balance between reward and diversity. Generally, diversity optimization algorithms use Multi-armed Bandits algorithms to select the coefficient in the pre-defined space. However, the dynamic distribution of reward signals for MABs or the conflict between qua...
-
This work is focused on reinforcement learning (RL)-based navigation for drones, whose localisation is based on visual odometry (VO). Such drones should avoid flying into areas with poor visual features, as this can lead to deteriorated localization or complete loss of tracking. To achieve this, we propose a hierarchical control scheme, which uses an RL-trained policy as the high-level controller ...
-
We propose a novel approach to multi-robot collaboration that harnesses the power of pre-trained large language models (LLMs) for both high-level communication and low-level path planning. Robots are equipped with LLMs to discuss and collectively reason task strategies. They generate sub-task plans and task space waypoint paths, which are used by a multi-arm motion planner to accelerate trajectory...
-
End-to-end deep reinforcement learning (DRL) for quadrotor control promises many benefits – easy deployment, task generalization and real-time execution capability. Prior end-to-end DRL-based methods have showcased the ability to deploy learned controllers onto single quadrotors or quadrotor teams maneuvering in simple, obstacle-free environments. However, the addition of obstacles increases the n...
-
Rollout algorithms are renowned for their abilities to correct for the suboptimalities of offline-trained base policies. In the multiagent setting, performing online rollout can require an exponentially large number of optimizations with respect to the number of agents. One-agent-at-a-time algorithms offer computationally efficient approaches to guaranteed policy improvement; however, this improve...
-
It is common for us to feel pressure in a competition environment, which arises from the desire to obtain success comparing with other individuals or opponents. Although we might get anxious under the pressure, it could also be a drive for us to stimulate our potentials to the best in order to keep up with others. Inspired by this, we propose a competitive learning framework which is able to help ...
-
Multi-modal 3D Human Tracking for Robots in Complex Environment with Siamese Point-Video Transformer
Tracking a specific person in 3D scene is gaining momentum due to its numerous applications in robotics. Currently, most 3D trackers focus on driving scenarios with neglected jitter and uncomplicated surroundings, which results in their severe degeneration in complex environments, especially on jolting robot platforms (only 20-60% success rate). To improve the accuracy, a Point-Video-based Transfo...
-
As intelligent systems become increasingly important in our daily lives, new ways of interaction are needed. Classical user interfaces pose issues for the physically impaired and are partially not practical or convenient. Gesture recognition is an alternative, but often not reactive enough when conventional cameras are used. This work proposes a Spiking Convolutional Neural Network, processing eve...
-
Control barrier functions have become an increasingly popular framework for safe real-time control. In this work, we present a computationally low-cost framework for synthesizing barrier functions over point cloud data for safe vision-based control. We take advantage of surface geometry to locally define and synthesize a quadratic CBF over a point cloud. This CBF is used in a CBF-QP for control an...
-
Visual odometry (VO) system is challenged by complex illumination environments. Image quality and its consistency in the time domain directly determine feature detection and tracking performance, which further affect the robustness and accuracy of the entire system. In this paper, an image acquisition scheme with image bracketing patterns is proposed. Images with different exposure levels are cont...
-
Accurate robotic control over interactions with the environment is fundamentally grounded in understanding tactile contacts. In this paper, we introduce MagicTac, a novel high-resolution grid-based tactile sensor. This sensor employs a 3D multi-layer grid-based design, inspired by the Magic Cube structure. This structure can help increase the spatial resolution of MagicTac to perceive external int...
-
In the last few years, mobile robots such as floor cleaners, assistive robots, and home telepresence have become an essential part of our day-to-day activities. In human-robot interaction, speech is the preferred way of communication, especially in indoor environments. This paper proposes a speech module to rotate the mobile robot. It has two components, namely, a distant automatic speech recogniz...
-
When used in a real-world noisy environment, the capacity to generalize to multiple domains is essential for any autonomous scene text spotting system. However, existing state-of-the-art methods employ pretraining and fine-tuning strategies on natural scene datasets, which do not exploit the feature interaction across other complex domains. In this work, we explore and investigate the problem of d...
-
Point cloud is a popular and widely used geometric representation, which has attracted significant attention in 3D vision. However, the geometric variability of point cloud representations across different datasets can cause domain discrepancies, which hinder knowledge transfer and model generalization, resulting in degraded performance in target domain. In this paper, we present a novel approach ...
-
In scenarios of Human-Robot Interaction (HRI), it is often assumed that the robot should cooperate with the closest individual or that only one person is present. However, in real-life situations, such as shop floor operations, this assumption may not hold. Thus, it becomes necessary for a robot to recognize a specific target in a crowded environment. To address this problem, we propose a person r...
-
This paper presents Action-SGFA, a novel action feature alignment approach to learn unified joint embeddings across four action modalities incorporating scene graph (SG) comprehension. A new training paradigm for Action-SGFA is also devised to improve pre-trained VL models using datasets with SG annotation. When learning from image-SG pairs, it captures structure-associated action knowledge for vi...
-
With innovation in fields such as autonomous driving and augmented reality, point cloud-based place recognition has gained significant attention. Many methods try to address this problem by extracting and matching global descriptors in a database, but they often must balance the extraction of comprehensive contextual information and large model sizes. To overcome this challenge, we propose a light...
-
Correlation filter (CF)-based approaches have gained widespread attention in the field of unmanned aerial vehicle (UAV) visual tracking due to their light-weight characteristics. However, CFs are prone to generating low-quality response in challenging UAV scenarios, e.g., fast motion and background clutter. In this paper, in order to model the tracker more robustly, we first conduct an effective r...
-
As robotic systems increasingly encounter complex and unconstrained real-world scenarios, there is a demand to recognize diverse objects. The state-of-the-art 6D object pose estimation methods rely on object-specific training and therefore do not generalize to unseen objects. Recent novel object pose estimation methods are solving this issue using task-specific fine-tuned CNNs for deep template ma...
-
Lifelong learning or continual learning is the problem of training an AI agent continuously while also preventing it from forgetting its previously acquired knowledge. Streaming lifelong learning is a challenging setting of lifelong learning with the goal of continuous learning in a dynamic non-stationary environment without forgetting. We introduce a novel approach to lifelong learning, which is ...
-
Continual reinforcement learning, which aims to help robots acquire skills without catastrophic forgetting, obviating the need to re-learn all tasks from scratch. In order to enable lifelong acquisition of skills in robots, replay-based continual reinforcement learning has emerged as a promising research direction. These techniques replay data from previous tasks to mitigate forgetting when learni...
-
Open-world robotic tasks such as autonomous driving pose significant challenges to robot control due to unknown and unpredictable events that disrupt task performance. Neural network-based reinforcement learning (RL) techniques (like DQN, PPO, SAC, etc.) struggle to adapt in large domains and suffer from catastrophic forgetting. Hybrid planning and RL approaches have shown some promise in handling...
-
Large Language Models (LLMs) have emerged as a new paradigm for embodied reasoning and control, most recently by generating robot policy code that utilizes a custom library of vision and control primitive skills. However, prior arts fix their skills library and steer the LLM with carefully handcrafted prompt engineering, limiting the agent to a stationary range of addressable tasks. In this work, ...
-
Large Language Models (LLMs) have been shown to act like planners that can decompose high-level instructions into a sequence of executable instructions. However, current LLM-based planners are only able to operate with a fixed set of skills. We overcome this critical limitation and present a method for using LLM-based planners to query new skills and teach robots these skills in a data and time-ef...
-
The sense of touch is essential for robots to perform various daily tasks. Artificial Neural Networks have shown significant promise in advancing robotic tactile learning. However, due to the changing of tactile data distribution as robots encounter new tasks, ANN-based robotic tactile learning suffers from catastrophic forgetting. To solve this problem, we introduce a novel continual learning (CL...
-
We introduce LOTUS, a continual imitation learning algorithm that empowers a physical robot to continuously and efficiently learn to solve new manipulation tasks throughout its lifespan. The core idea behind LOTUS is constructing an ever-growing skill library from a sequence of new tasks with a small number of human demonstrations. LOTUS starts with a continual skill discovery process using an ope...
-
Safety certificates based on energy functions can provide demonstrable safety for complex robotic systems. However, all recent studies on learning-based energy function synthesis only consider the feasibility of the control policy, which might cause over-conservativeness and even fail to achieve the control goal. To solve the problem of over-conservative controllers, we proposed the magnitude regu...
-
A robot in a human-centric environment needs to account for the human’s intent and future motion in its task and motion planning to ensure safe and effective operation. This requires symbolic reasoning about probable future actions and the ability to tie these actions to specific locations in the physical environment. While one can train behavioral models capable of predicting human motion from pa...
-
Various robots have been developed so far; however, we face challenges in modeling the low-rigidity bodies of some robots. In particular, the deflection of the body changes during tool-use due to object grasping, resulting in significant shifts in the tool-tip position and the body’s center of gravity. Moreover, this deflection varies depending on the weight and length of the tool, making these mo...
-
In this study, we present an implementation strategy for a robot that performs peg transfer tasks in Fundamentals of Laparoscopic Surgery (FLS) via imitation learning, aimed at the development of an autonomous robot for laparoscopic surgery. Robotic laparoscopic surgery presents two main challenges: (1) the need to manipulate forceps using ports established on the body surface as fulcrums, and (2)...
-
Most existing robotic datasets capture static scene data and thus are limited in evaluating robots’ dynamic performance. To address this, we present a mobile robot oriented large-scale indoor dataset, denoted as THUD (Tsinghua University Dynamic) robotic dataset, for training and evaluating their dynamic scene understanding algorithms. Specifically, the THUD dataset construction is first detailed,...
-
In collaborative human-robot manipulation, a robot must predict human intents and adapt its actions accordingly to smoothly execute tasks. However, the human’s intent in turn depends on actions the robot takes, creating a chicken-or-egg problem. Prior methods ignore such inter-dependency and instead train marginal intent prediction models independent of robot actions. This is because training cond...
-
Machine Learning (ML) has replaced handcrafted methods for perception and prediction in autonomous vehicles. Yet for the equally important planning task, the adoption of ML-based techniques is slow. We present nuPlan, the world’s first real-world autonomous driving dataset and benchmark. The benchmark is designed to test the ability of ML-based planners to handle diverse driving situations and to ...
-
Social navigation and pedestrian behavior research has shifted towards machine learning-based methods and converged on the topic of modeling inter-pedestrian interactions and pedestrian-robot interactions. For this, large-scale datasets that contain rich information are needed. We describe a portable data collection system, coupled with a semi-autonomous labeling pipeline. As part of the pipeline,...
-
We present a scalable, bottom-up and intrinsically diverse data collection scheme that can be used for high-level reasoning with long and medium horizons and that has 2.2x higher throughput compared to traditional narrow top-down step-by-step collection. We collect realistic data by performing any user requests within the entirety of 3 office buildings and using multiple embodiments (robot, human,...
-
A key challenge for robotic manipulation in open domains is how to acquire diverse and generalizable skills for robots. Recent progress in one-shot imitation learning and robotic foundation models have shown promise in transferring trained policies to new tasks based on demonstrations. This feature is attractive for enabling robots to acquire new skills and improve their manipulative ability. Howe...
-
Aquatic organisms, due to soft body structure and high agility, have inspired many biomimetic robots. However, considering the issues of insulation and waterproofing, as well as the driving module of soft materials, their control systems are usually larger and heavier. Therefore, small underwater robots often tethered, i.e., it cannot integrate energy and control systems onto the body, which great...
-
Research in the area of photo-actuation is growing rapidly, yet there are few examples of photo-actuators with practical use cases. One potential application is for the control of intelligent electromagnetic surfaces, or two-dimensional arrays that could shape and control an incident electromagnetic field in ideally any manner. A promising concept to realize such a surface leverages signal refract...
-
Many robotic hands currently rely on extremely dexterous robotic fingers and a thumb joint to envelop themselves around an object. Few hands focus on the palm even though human hands greatly benefit from their central fold and soft surface. As such, we develop a novel structurally compliant soft palm, which enables more surface area contact for the objects that are pressed into it. Moreover, this ...
-
The inherent elasticity of soft materials can be used to create robotic grippers that deform and comply to a variety of irregular shapes. To date, several soft adaptive grasping strategies have been reported, however, most of them focus on adapting to the overall shape of the structure, while the adaptive grasping of small surface asperities is overlooked. In this paper, we propose a novel method ...
-
Origami designs and structures have been widely used in many fields, such as morphing structures, robotics, and metamaterials. However, the design and fabrication of origami structures rely on human experiences and skills, which are both time and labor-consuming. In this paper, we present a rapid design and fabrication method for string-driven origami structures and robots. We developed an origami...
-
Point cloud primitive segmentation aims to segment the surface point cloud into various geometric types of primitives, which plays a vital role in robot operation and industrial automation. However, differences in object structures and shapes across industrial datasets create domain shift issues, compounded by privacy concerns preventing dataset sharing. To address these challenges, we propose a n...
-
Cross-modality point cloud registration is confronted with significant challenges due to inherent differences in modalities between sensors. To deal with this problem, we propose FF-LOGO: a cross-modality point cloud registration framework with Feature Filtering and LOcal-Global Optimization. The cross-modality feature correlation filtering module extracts geometric transformation-invariant featur...
-
The ability to estimate joint parameters is essential for various applications in robotics and computer vision. In this paper, we propose CAPT: category-level articulation estimation from a point cloud using Transformer. CAPT uses an end-to-end transformer-based architecture for joint parameter and state estimation of articulated objects from a single point cloud. The proposed CAPT methods accurat...
-
Efficient interest point detection and description in images play a crucial role in many tasks such as multi-robot SLAM and collaborative localization. To facilitate fast detection and generate compact descriptions on edge devices, we introduce EdgePoint, a lightweight neural network. We design a new detection loss UnfoldSoftmax to improve inference speed. Futhermore, we propose Ortho-Alignment lo...
-
Point cloud registration is essential in computer vision and robotics. Recently, transformer-based methods have achieved advanced point cloud registration performance. However, the standard attention mechanism utilized in these methods considers many low-relevance points, and it has difficulty focusing its attention weights on sparse and meaningful points, leading to limited local structure modeli...
-
Existing grasp prediction approaches are mostly based on offline learning, while, ignoring the exploratory grasp learning during online adaptation to new picking scenarios, i.e., objects that are unseen or out-of-domain (OOD), camera and bin settings, etc. In this paper, we present an uncertainty-based approach for online learning of grasp predictions for robotic bin picking. Specifically, the onl...
-
The prevailing grasp prediction methods predominantly rely on offline learning, overlooking the dynamic grasp learning that occurs during real-time adaptation to novel picking scenarios. These scenarios may involve previously unseen objects, variations in camera perspectives, and bin configurations, among other factors. In this paper, we introduce a novel approach, SSL-ConvSAC, that combines semi-...
-
Visual-based robot pose estimation is a fundamental challenge, involving the determination of the camera’s pose with respect to a robot. Conventional methods for camera-to-robot pose calibration rely on fiducial markers to establish keypoint correspondences. However, these approaches exhibit significant variability in accuracy and robustness, particularly in 2D keypoint detection. In this work, we...
-
This paper introduces a method, Generative Adversarial Networks for Detecting Erroneous Results (GANDER), leveraging Generative Adversarial Networks to provide online error detection in manipulation tasks for autonomous robot systems. GANDER relies on mapping input images of a trained task to a learned manifold that contains only positive task executions and outcomes. When reconstructed through th...
-
Automating the assembly of objects from their parts is a complex problem with innumerable applications in manufacturing, maintenance, and recycling. Unlike existing research, which is limited to target segmentation, pose regression, or using fixed target blueprints, our work presents a holistic multi-level framework for part assembly planning consisting of part assembly sequence inference, part mo...
-
Advancements in robot learning for object manipulation have shown promising results, yet certain everyday objects remain challenging for robots to effectively interact with. This discrepancy arises from the fact that human-designed objects are optimized for human use rather than robot manipulation. To address this gap, we propose a framework to automatically design 3D printable adaptations that ca...
-
This paper primarily focuses on evaluating and benchmarking the robustness of visual representations in the context of object assembly tasks. Specifically, it investigates the alignment and insertion of objects with geometrical extrusions, commonly referred to as a peg-in-hole task. The accuracy required to detect and orient the peg and the hole geometry in SE(3) space for successful assembly pose...
-
Advances in human sensing and machine learning are paving the way for new applications of robotics in sports and fitness, making skill coaching smarter, easier and more accessible. Physical and social human robot interaction in particular has received special attention as a feedback mechanism for human performance augmentation. A core challenge in deploying robots that interact physically with hum...
-
Traditional strength and conditioning training relies on the utilization of free weights, such as weighted implements, to elicit external stimuli. However, this approach poses a significant challenge when attempting to modify or adjust the loads within a single training set. This paper introduces an innovative method for achieving adjustable loads during resistance training by leveraging physical ...
-
Robots have a great potential to help people with movement limitations in activities of daily living, such as dressing. A common problem in almost all dressing tasks is the insertion of a garment’s opening around a part of the human body. The rich contact environment and the deformations of the garment make the task a challenging problem for robots. In this paper, we propose a bi-manual control me...
-
The Rating Scale method has been long deemed the standard for measuring subjective perceptions. However, in the field of physical human-robot collaboration (pHRC), its aptness should be put under scrutiny due to inherent challenges such as response bias, between-subject variations, and the granularity nature.Individual variances can introduce significant bias in the rating scale results. A high gr...
-
Prosthetic legs have been used to restore function in the lower limbs lost due to amputation. Early designs including prosthetic legs with a passive joint or without any joint as well as the Energy Storing and Releasing (ESR) feet have shown deficiency in push-off torque, which results in asymmetric gait pattern, slower walking speed, and higher cost of transportation. Although powered prosthetic ...
-
In Total Knee Replacement Arthroplasty (TKA), surgical robotics can provide image-guided navigation to fit implants with high precision. Its tracking approach highly relies on inserting bone pins into the bones tracked by the optical tracking system. This is normally done by invasive, radiative manners (implantable markers and CT scans), which introduce unnecessary trauma and prolong the preparati...
-
Research in powered prosthesis control has explored the use of impedance-based control algorithms due to their biomimetic capabilities and intuitive structure. Modern impedance controllers feature parameters that smoothly vary over gait phase and task according to a data-driven model. However, these recent efforts only use continuous impedance control during stance and instead utilize discrete tra...
-
Exoskeleton technologies have numerous potential applications, ranging from improving human motor skills to aiding individuals in their daily activities. While exoskeletons are increasingly viewed, for example, as promising tools in industrial ergonomics, the effect of using them on human motor control, particularly on inter-joint coordination, remains relatively uncharted. This paper investigates...
-
The vague interpretation of myoelectrical signals on the residual limb end makes restoring dexterous hand function in amputees still impossible. Understanding motor control between human motion intention and synaptic inputs to motor neurons also remains a significant challenge. The neural decoding methods of surface EMG signals remains challenging, which limit the application of robot hand in real...
-
How easy is it to sneak up on a robot? We examine whether we can detect people using only the incidental sounds they produce as they move, even when they try to be quiet. To do so, we first collect a robotic dataset of high-quality 4-channel audio paired with 360° RGB data of people moving in different indoor settings. Using this dataset, we train models to predict if there is a moving person near...
-
For a service robot, it is crucial to perceive as early as possible that an approaching person intends to interact: in this case, it can proactively enact friendly behaviors that lead to an improved user experience. We solve this perception task with a sequence-to-sequence classifier of a potential user intention to interact, which can be trained in a self-supervised way. Our main contribution is ...
-
When integrating robots into human daily life, persuasive power can be essential. However, there are often group dynamics which can complicate persuasion. This study focuses on how non-verbal cues, specifically gaze and hand gestures, affect the persuasiveness of a social robot. We have designed a protocol to include non-verbal cues in the social robot Vizzy (head and eye gaze, hand gestures) and ...
-
The socially-aware navigation system has evolved to adeptly avoid various obstacles while performing multiple tasks, such as point-to-point navigation, human-following, and -guiding. However, a prominent gap persists: in Human-Robot Interaction (HRI), the procedure of communicating commands to robots demands intricate mathematical formulations. Furthermore, the transition between tasks does not qu...
-
To foster an immersive and natural human-robot interaction (HRI), the implementation of tactile perception and feedback becomes imperative, effectively bridging the conventional sensory gap. In this paper, we propose a dual-modal electronic skin (e-skin) that integrates magnetic tactile sensing and vibration feedback for enhanced HRI. The dual-modal tactile e-skin offers multi-functional tactile s...
-
Interactive hand mesh reconstruction from singleview images poses a significant challenge with the severe occlusion and depth ambiguity inherent in interactive hand gestures. Recent approaches that employ probabilistic models and tokenpruned techniques have shown decent results in multi-view human body reconstruction. Nevertheless, these methods have not fully utilized multi-scale semantic informa...
-
In the context of smart cities, autonomous vehicles, such as unmanned delivery vehicles and taxis are gradually gaining acceptance. However, their application scenarios remain significantly fragmented. Typically, an Autonomous Multi-Functional Vehicle (AMFV) is not engaged in other scenarios when idle in a specific one. Currently, a unified system capable of coordinating and using these resources ...
-
Images captured by robotics under low-light conditions are often plagued by several challenges, including diminished contrast, increased noise, loss of fine details, and unnatural color reproduction. These factors can significantly hinder the performance of computer vision tasks such as object detection and image segmentation. As a result, improving the quality of low-light images is of paramount ...
-
Robotic systems for manipulation in millimeter scale often use a camera with high magnification for visual feedback of the target region. However, the limited field-of-view (FoV) of the microscopic camera necessitates camera motion to capture a broader workspace environment. In this work, we propose an autonomous robotic control method to constrain a robot-held camera within a designated FoV. Furt...
-
To enable the computation of effective randomized patrol routes for single- or multi-robot teams, we present RoSSO, a Python package designed for solving Markov chain optimization problems. We exploit machine-learning techniques such as reverse-mode automatic differentiation and constraint parametrization to achieve superior efficiency compared to general-purpose nonlinear programming solvers. Add...
-
Despite the potential of mobile manipulators and applications where robots require a force-controlled physical interaction with the environment, the majority of robot automation nowadays is still based on fixed manipulators for free-motion tasks (e.g. welding, pick and place, or painting). In this work, we propose a control solution for omnidirectional mobile manipulators in force-tracking tasks, ...
-
Lane changing is a fundamental but challenging operation for moving vehicles. Connected and Automated Vehicles(CAVs) enable autonomous vehicles to cooperate with each other to accomplish the lane changing tasks, profiting from their communication ability. However, dispatching CAVs in mixed traffic remains difficult due to the stochastic behaviors and uncertain intentions of Human-Driven Vehicles(H...
-
Federated learning has been widely applied in autonomous driving since it enables training a learning model among vehicles without sharing users’ data. However, data from autonomous vehicles usually suffer from the non-independent-and-identically-distributed (non-IID) problem, which may cause negative effects on the convergence of the learning process. In this paper, we propose a new contrastive d...
-
Large-scale scenario databases may contain hundreds of thousands of scenarios for the verification and validation (V&V) of autonomous vehicles (AV). Scenarios in the database are often labelled with semantic Operational Design Domain (ODD) tags (e.g., WeatherRainy, RoadTypeHighway and ActorTypeTruck) to be queried via exact tag matching. Such a scenario database design has two major limitations, i...
-
This paper presents a novel approach to modeling human driving behavior, designed for use in evaluating autonomous vehicle control systems in a simulation environments. Our methodology leverages a hierarchical forward-looking, risk-aware estimation framework with learned parameters to generate human-like driving trajectories, accommodating multiple driver levels determined by model parameters. Thi...
-
Predicting temporally consistent road users’ trajectories in a multi-agent setting is a challenging task due to the unknown characteristics of agents and their varying intentions. Besides using semantic map information and modeling interactions, it is important to build an effective mechanism capable of reasoning about behaviors at different levels of granularity.To this end, we propose Dynamic go...
-
The development of connected autonomous vehicles (CAVs) facilitates the enhancement of traffic efficiency in complicated scenarios. Difficulties remain unsolved in developing an effective and efficient coordination strategy for CAVs. In this paper, we formulate the cooperative autonomous driving task of CAVs as an optimal control problem with safety conditions enforced as hard constraints, and pro...
-
Many solutions to address the challenge of robot learning have been devised, namely through exploring novel ways for humans to communicate complex goals and tasks in reinforcement learning (RL) setups. One way that experienced recent research interest directly addresses the problem by considering human feedback as preferences between pairs of trajectories (sequences of state-action pairs). However...
-
Prior robot painting and drawing work, such as FRIDA, has focused on decreasing the sim-to-real gap and expanding input modalities for users, but the interaction with these systems generally exists only in the input stages. To support interactive, human-robot collaborative painting, we introduce the Collaborative FRIDA (CoFRIDA) robot painting framework, which can co-paint by modifying and engagin...
-
People with Visual Impairments (PVI) typically recognize objects through haptic perception. Knowing objects and materials before touching is desired by the target users but under-explored in the field of human-centered robotics. To fill this gap, in this work, a wearable vision-based robotic system, MATERobot, is established for PVI to recognize materials and object categories beforehand. To addre...
-
This paper presents a framework to navigate visually impaired people through unfamiliar environments by means of a mobile manipulator. The Human-Robot system consists of three key components: a mobile base, a robotic arm, and the human subject who gets guided by the robotic arm via physically coupling their hand with the cobot’s end-effector. These components, receiving a goal from the user, trave...
-
Movement primitives (MPs) are compact representations of robot skills that can be learned from demonstrations and combined into complex behaviors. However, merely equipping robots with a fixed set of innate MPs is insufficient to deploy them in dynamic and unpredictable environments. Instead, the full potential of MPs remains to be attained via adaptable, large-scale MP libraries. In this paper, w...
-
This paper proposes the utilization of Supernumerary Robotic Limbs (SuperLimbs) for augmenting astronauts during an Extra-Vehicular Activity (EVA) in a partial-gravity environment. We investigate the effectiveness of SuperLimbs in assisting astronauts to their feet following a fall. Based on preliminary observations from a pilot human study, we categorized post-fall recoveries into a sequence of s...
-
This paper proposes a novel flexible vibrational actuator with a structural anisotropy and its control method to diversify the vibrational behavior. First, the analytical model of the proposed actuator, which comprises a rectangular cross-sectional flexible beam and a rotational-type motor, is introduced. Regarding the structural anisotropy, the rotational axis of the motor is nonparallel to both ...
-
Robot designs can take many inspirations from nature, where there are many examples of highly resilient and fault-tolerant locomotion strategies to navigate complex terrains by using multi-functional appendages. For example, Chukar and Hoatzin birds can repurpose their wings for quadrupedal walking and wing-assisted incline running (WAIR) to climb steep surfaces. We took inspiration from nature an...
-
Soft growing robots with unique navigation (tip extension by eversion) hold great promise in rescue, medical, and industrial applications. Equipping them with grasping capability would enhance their usefulness in constrained environments for various applications. However, in traditional designs, the tip’s eversion naturally conflicts with grasping, and the addition of grippers at the tip would lim...
-
In recent years, bio-syncretic robots actuated by living materials have received widespread attention. Among the common living materials, engineered skeletal muscle tissue (eSKT) has been the focus of researchers due to its high contraction force and good controllability. However, the current performance of eSKT is far from that of natural skeletal muscle tissue. In this paper, an optimized design...
-
Motion planning under sensing uncertainty is critical for robots in unstructured environments, to guarantee safety for both the robot and any nearby humans. Most work on planning under uncertainty does not scale to high-dimensional robots such as manipulators, assumes simplified geometry of the robot or environment, or requires per-object knowledge of noise. Instead, we propose a method that direc...
-
Optimal plans in Constrained Partially Observable Markov Decision Processes (CPOMDPs) maximize reward objectives while satisfying hard cost constraints, generalizing safe planning under state and transition uncertainty. Unfortunately, online CPOMDP planning is extremely difficult in large or continuous problem domains. In many large robotic domains, hierarchical decomposition can simplify planning...
-
Current methods based on Neural Radiance Fields (NeRF) significantly lack the capacity to quantify uncertainty in their predictions, particularly on the unseen space including the occluded and outside scene content. This limitation hinders their extensive applications in robotics, where the reliability of model predictions has to be considered for tasks such as robotic exploration and planning in ...
-
Robotic manipulation relies on analytical or learned models to simulate the system dynamics. These models are often inaccurate and based on offline information, so that the robot planner is unable to cope with mismatches between the expected and the actual behavior of the system (e.g., the presence of an unexpected obstacle). In these situations, the robot should use information gathered online to...
-
Exploration in dynamic and uncertain real-world environments is an open problem in robotics and it constitutes a foundational capability of autonomous systems operating in most of the real-world. While 3D exploration planning has been extensively studied, the environments are assumed static or only reactive collision avoidance is carried out. We propose a novel approach to not only avoid dynamic o...
-
We present a novel learning-based trajectory generation algorithm for outdoor robot navigation. Our goal is to compute collision-free paths that also satisfy the environment-specific traversability constraints. Our approach is designed for global planning using limited onboard robot perception in mapless environments while ensuring comprehensive coverage of all traversable directions. Our formulat...
-
In this work, we propose the Informed Batch Belief Trees (IBBT) algorithm for motion planning under motion and sensing uncertainties. The original stochastic motion planning problem is divided into a deterministic motion planning problem and a graph search problem. First, we solve the deterministic planning problem using Rapidly-exploring Random Graph (RRG) to construct a nominal trajectory graph....
-
Ensuring safe navigation in human-populated environments is crucial for autonomous mobile robots. Although recent advances in machine learning offer promising methods to predict human trajectories in crowded areas, it remains unclear how one can safely incorporate these learned models into a control loop due to the uncertain nature of human motion, which can make predictions of these models imprec...
-
For autonomous mobile robots, uncertainties in the environment and system model can lead to failure in the motion planning pipeline, resulting in potential collisions. In order to achieve a high level of robust autonomy, these robots should be able to proactively predict and recover from such failures. To this end, we propose a Gaussian Process (GP) based model for proactively detecting the risk o...
-
We propose the method to adapt humanoids the ability to change the body structures that modular robots have by using Attach-Lock-Detachable Magnetic Couplings(ALDMag) to give the ability to detach and attach the robot body with an arm-type robot, and the system to manage the connection state of modularized body elements. Robots and we can use the ALDMag to attach and detach mechanical and electric...
-
A human foot with high degrees of freedom (DOF) that has multi-DOF toe joints and a two-DOF ankle provides multiple benefits, such as increased stride length and walking speed, impact mitigation, and enhanced balancing. However, creating such mechanisms for legged robots has been challenging due to increased complexity, heavy weight, and vulnerability to impact. In this paper, a novel leg and toe ...
-
For researchers or administrators of relevant institutions who need to collect hydrological data of a certain water area, using autonomous sailboats to tow floating detection equipment is an energy-saving and convenient scheme for deploying detectors. However, due to the limited pulling force provided by a single autonomous sailboat, this scheme is not suitable for floating equipment with large ma...
-
In this paper, we have developed a protection module for Light Detection and Ranging (LiDAR) sensors used in outdoor unmanned vehicles. Bio-inspired wiping motion was figured to have more efficient and excellent wiping performance than conventional cleaning methods for LiDAR sensors. An water wiping experiment confirmed that the finger wiping motion removed 35% more water than the translational wi...
-
This paper introduces a novel origami-inspired shape-changing robot OSCaR. The objective is to enhance the adaptability of vehicles engaged in ground coverage tasks, such as floor cleaning. The robot exhibits two distinct configurations: it can fold itself for agile navigation through tight spaces, and unfold to cover larger areas efficiently. The folding pattern has a deploy-to-stow ratio of 3 in...
-
In environments like offices, the duration of a robot’s navigation between two locations may vary over time. For instance, reaching a kitchen may take more time during lunchtime since the corridors are crowded with people heading the same way. In this work, we address the problem of routing in such environments with tasks expressed in Metric Interval Temporal Logic (MITL) – a rich robot task speci...
-
As robots become more prevalent, the complexity of robot-robot, robot-human, and robot-environment interactions increases. In these interactions, a robot needs to consider not only the effects of its own actions, but also the effects of other agents’ actions and the possible interactions between agents. Previous works have considered reactive synthesis, where the human/environment is modeled as a ...
-
Reactive planning enables the robots to deal with dynamic events in uncertain environments. However, existing methods heavily rely on the predefined hard-coded robot behaviors, e.g, a pre-coded temporal logic formula that specifies how robot should react. Little attention has been paid for autonomous generation of reactive tasks specifications during the runtime. As a first attempt towards this go...
-
Deploying robots in real-world environments, such as households and manufacturing lines, requires generalization across novel task specifications without violating safety constraints. Linear temporal logic (LTL) is a widely used task specification language with a compositional grammar that naturally induces commonalities among tasks while preserving safety guarantees. However, most prior work on r...
-
This study developed a high-precision paint deposition model that considers the position and direction of a spray-painting gun. Our angle-specific paint deposition model focused on the change in paint deposition due to the change in the painting angle; however, there was a problem with its versatility. We analyzed this problem, and the solution was achieved by separately modeling changes in the fi...
-
A robotic behavior model that can reliably generate behaviors from natural language inputs in real time would substantially expedite the adoption of industrial robots due to enhanced system flexibility. To facilitate these efforts, we construct a framework in which learned behaviors, created by a natural language abstractor, are verifiable by construction. Leveraging recent advancements in motion ...
-
This paper contemplates the possibility of asking robots questions and having them use their ability to go out into the environment and probe it, in combination with what they already know of the world, to provide answers. We describe a method whereby a robot system efficiently answers such questions on the basis of reasoning about observations as they are made, interrelationships between multiple...
-
When agents in a multi-robot team have limited knowledge about their relative performance, their teammates, or the environment, robots must observe individual performance variations and adapt accordingly. We propose robot reputation to assess the historical performance of agents and make future adaptations in a persistent coverage task. We consider a heterogeneous multi-robot team, where robots ar...
-
Recently a line of research has delved into the use of graph neural networks (GNNs) for decentralized control in swarm robotics. However, it has been observed that relying solely on the states of immediate neighbors is insufficient to imitate a centralized control policy. To address this limitation, prior studies proposed incorporating L-hop delayed states into the computation. While this approach...
-
Mutual localization stands as a foundational component within various domains of multi-robot systems. Nevertheless, in relative pose estimation, time synchronization is usually underappreciated and rarely addressed, although it significantly influences estimation accuracy. In this paper, we introduce time synchronization into mutual localization to recover the time offset and relative poses betwee...
-
Multi-robot collaboration in large-scale environments with limited-sized teams and without external infrastructure is challenging, since the software framework required to support complex tasks must be robust to unreliable and intermittent communication links. In this work, we present MOCHA (Multi-robot Opportunistic Communication for Heterogeneous Collaboration), a framework for resilient multi-r...
-
In this paper, we present an approach for coverage path planning for a team of an energy-constrained Unmanned Aerial Vehicle (UAV) and an Unmanned Ground Vehicle (UGV). Both the UAV and the UGV have predefined areas that they have to cover. The goal is to perform complete coverage by both robots while minimizing the coverage time. The UGV can also serve as a mobile recharging station. The UAV and ...
-
Space-filling building blocks of diverse shape permeate nature at all levels of organization, from atoms to honeycombs, and have proven useful in artificial systems, from molecular containers to clay bricks. But, despite the wide variety of space-filling polyhedra known to mathematics, only the cube has been explored in robotics. Thus, here we roboticize a non-cubic space-filling shape: the rhombi...
-
This paper explores the optimal containment control problem for nonlinear and underactuated quadrotors with multiple team leaders governed by nonlinear dynamics, employing the reinforcement learning. A cascade controller is formulated, comprising a position control component to ensure containment achievement and an attitude control component to govern rotational channel. The proposed optimal contr...
-
Planning in learned latent spaces helps to decrease the dimensionality of raw observations. In this work, we propose to leverage the ensemble paradigm to enhance the robustness of latent planning systems. We rely on our Latent Space Roadmap (LSR) framework, which builds a graph in a learned structured latent space to perform planning. Given multiple LSR framework instances, that differ either on t...
-
Event cameras are recent sensors that measure intensity changes in each pixel asynchronously. It is being used due to lower latency and higher temporal resolution compared to traditional frame-based camera. We propose a method of 3D model-based object tracking directly from events captured by event camera. To enable reliable and accurate tracking of objects, we use a new event representation and p...
-
Non-Rigid Structure-from-Motion (NRSfM) reconstructs the time-varying 3D shape of a deforming object from 2D point correspondences in monocular images. Despite promising use-cases such as the grasping of deformable objects and visual navigation in a non-rigid environment, NRSfM has had limited applications in robotics due to a lack of accuracy. To remedy this, we propose a new method which boosts ...
-
Robot navigation within complex environments requires precise state estimation and localization to ensure robust and safe operations. For ambulating mobile robots like robot snakes, traditional methods for sensing require multiple embedded sensors or markers, leading to increased complexity, cost, and increased points of failure. Alternatively, deploying an external camera in the environment is ve...
-
We propose a new visual hierarchical representation paradigm for multi-object tracking. It is more effective to discriminate between objects by attending to objects’ compositional visual regions and contrasting with the background contextual information instead of sticking to only the semantic visual cue such as bounding boxes. This compositional-semantic-contextual hierarchy is flexible to be int...
-
The problem of multi-object tracking (MOT) consists in detecting and tracking all the objects in a video sequence while keeping a unique identifier for each object. It is a challenging and fundamental problem for robotics. In precision agriculture the challenge of achieving a satisfactory solution is amplified by extreme camera motion, sudden illumination changes, and strong occlusions. Most moder...
-
We propose an accurate and robust initialization approach for stereo visual-inertial SLAM systems. Unlike the current state-of-the-art method, which heavily relies on the accuracy of a pure visual SLAM system to estimate inertial variables without updating camera poses, potentially compromising accuracy and robustness, our approach offers a different solution. We realize the crucial impact of prec...
-
Dense, volumetric maps are essential to enable robot navigation and interaction with the environment. To achieve low latency, dense maps are typically computed onboard the robot, often on computationally constrained hardware. Previous works leave a gap between CPU-based systems for robotic mapping which, due to computation constraints, limit map resolution or scale, and GPU-based reconstruction sy...
-
This paper studies the problem of extracting planar regions in uneven terrains from unordered point cloud measurements. Such a problem is critical in various robotic applications such as robotic perceptive locomotion. While existing approaches have shown promising results in effectively extracting planar regions from the environment, they often suffer from issues such as low computational efficien...
-
General scene reconstruction refers to the task of estimating the full 3D geometry and texture of a scene containing previously unseen objects. In many practical applications such as AR/VR, autonomous navigation, and robotics, only a single view of the scene may be available, making the scene reconstruction task challenging. In this paper, we present a method for scene reconstruction by structural...
-
Accurate and dense depth estimation with stereo cameras and LiDAR is an important task for automatic driving and robotic perception. While sparse hints from LiDAR points have improved cost aggregation in stereo matching, their effectiveness is limited by the low density and non-uniform distribution. To address this issue, we propose a novel stereo-LiDAR depth estimation network with Semi-Dense hin...
-
With the rise in consumer depth cameras, a wealth of unlabeled RGB-D data has become available. This prompts the question of how to utilize this data for geometric reasoning of scenes. While many RGB-D registration methods rely on geometric and feature-based similarity, we take a different approach. We use cycle-consistent keypoints as salient points to enforce spatial coherence constraints during...
-
In response to the evolving challenges posed by small unmanned aerial vehicles (UAVs), which possess the potential to transport harmful payloads or independently cause damage, we introduce MMAUD: a comprehensive Multi-Modal Anti-UAV Dataset. MMAUD addresses a critical gap in contemporary threat detection methodologies by focusing on drone detection, UAV-type classification, and trajectory estimati...
-
Segmenting unseen objects from images is a critical perception skill that a robot needs to acquire. In robot manipulation, it can facilitate a robot to grasp and manipulate unseen objects. Mean shift clustering is a widely used method for image segmentation tasks. However, the traditional mean shift clustering algorithm is not differentiable, making it difficult to integrate it into an end-to-end ...
-
We introduce SLCF-Net, a novel approach for the Semantic Scene Completion (SSC) task that sequentially fuses LiDAR and camera data. It jointly estimates missing geometry and semantics in a scene from sequences of RGB images and sparse LiDAR measurements. The images are semantically segmented by a pre-trained 2D U-Net and a dense depth prior is estimated from a depth-conditioned pipeline fueled by ...
-
There is an ever-growing zoo of modern neural network models that can efficiently learn end-to-end control from visual observations. These advanced deep models, ranging from convolutional to Vision Transformers, from small to gigantic networks, have been extensively tested on offline image classification tasks. In this paper, we study these vision models with respect to the open-loop training to c...
-
For a general-purpose robot, it is desirable to imitate human demonstration videos that can effectively solve long-horizon tasks and perform novel ones. Recent advances in skill-based imitation learning have shown that extracting skill embedding from raw human videos is a promising paradigm to enable robots to cope with long-horizon tasks. However, generalization to unseen tasks in a different dom...
-
Offline reinforcement learning (RL) aims to optimize a policy, based on pre-collected data, to maximize the cumulative rewards after performing a sequence of actions. Existing approaches learn a value function from historical data and then guide the updating of the policy parameters by maximizing the value function at a single time. Driven by the gap between maximizing the cumulative rewards of RL...
-
We propose DINOBot, a novel imitation learning framework for robot manipulation, which leverages the image-level and pixel-level capabilities of features extracted from Vision Transformers trained with DINO. When interacting with a novel object, DINOBot first uses these features to retrieve the most visually similar object experienced during human demonstrations, and then uses this object to align...
-
Teaching robots novel skills with demonstrations via human-in-the-loop data collection techniques like kinesthetic teaching or teleoperation puts a heavy burden on human supervisors. In contrast to this paradigm, it is often significantly easier to provide raw, action-free visual data of tasks being performed. Moreover, this data can even be mined from video datasets or the web. Ideally, this data...
-
Preference-based reward learning is a popular technique for teaching robots and autonomous systems how a human user wants them to perform a task. Previous works have shown that actively synthesizing preference queries to maximize information gain about the reward function parameters improves data efficiency. The information gain criterion focuses on precisely identifying all parameters of the rewa...
-
Offline goal-conditioned reinforcement learning (GCRL) aims at solving goal-reaching tasks with sparse rewards from an offline dataset. While prior work has demonstrated various approaches for agents to learn near-optimal policies, these methods encounter limitations when dealing with diverse constraints in complex environments, such as safety constraints. Some of these approaches prioritize goal ...
-
Safe Reinforcement Learning (RL) plays an important role in applying RL algorithms to safety-critical real-world applications, addressing the trade-off between maximizing rewards and adhering to safety constraints. This work introduces a novel approach that combines RL with trajectory optimization to manage this trade-off effectively. Our approach embeds safety constraints within the action space ...
-
Distributional Reinforcement Learning (DRL) not only endeavors to optimize expected returns, but also strives to accurately characterize the full distribution of these returns, a key aspect in enhancing risk-aware decision-making. Previous DRL implementations often inappropriately treat statistical estimations as concrete samples, which undermines the integrity of learning. While several studies h...
-
Ensuring safety in dynamic multi-agent systems is challenging due to limited information about the other agents. Control Barrier Functions (CBFs) are showing promise for safety assurance but current methods make strong assumptions about other agents and often rely on manual tuning to balance safety, feasibility, and performance. In this work, we delve into the problem of adaptive safe learning for...
-
In Reinforcement Learning, the trade-off between exploration and exploitation poses a complex challenge for achieving efficient learning from limited samples. While recent works have been effective in leveraging past experiences for policy updates, they often overlook the potential of reusing past experiences for data collection. Independent of the underlying RL algorithm, we introduce the concept...
-
Safe reinforcement learning (Safe RL) refers to a class of techniques that aim to prevent RL algorithms from violating constraints in the process of decision-making and exploration during trial and error. In this paper, a novel model-free Safe RL algorithm, formulated based on the multi-objective policy optimization framework is introduced where the policy is optimized towards optimality and safet...
-
Perceptive deep reinforcement learning (DRL) has lead to many recent breakthroughs for complex AI systems leveraging image-based input data. Applications of these results range from super-human level video game agents to dexterous, physically intelligent robots. However, training these perceptive DRL-enabled systems remains incredibly compute and memory intensive, often requiring huge training dat...
-
Multi-task reinforcement learning could enable robots to scale across a wide variety of manipulation tasks in homes and workplaces. However, generalizing from one task to another and mitigating negative task interference still remains a challenge. Addressing this challenge by successfully sharing information across tasks will depend on how well the structure underlying the tasks is captured. In th...
-
Monocular 3D lane detection has recently received increasing research attention in autonomous driving due to its application effectiveness and simplicity. However, depending solely on the limited semantic information from a single image makes current monocular detection methods unable to deal with complex scenarios, such as occluded, blurred, and unaligned scenes. In this study, we introduce an en...
-
In the recent progress in embodied navigation and sim-to-robot transfer, modular policies have emerged as a de facto framework. However, there is more to compositionality beyond the decomposition of the learning load into modular components. In this work, we investigate a principled way to syntactically combine these components. Particularly, we propose Exploitation-Guided Exploration (XgX) where ...
-
Robot navigation requires an autonomy pipeline that is robust to environmental changes and effective in varying conditions. Teach and Repeat (T&R) navigation has shown high performance in autonomous repeated tasks under challenging circumstances, but research within T&R has predominantly focused on motion planning as opposed to motion control. In this paper, we propose a novel T&R system based on ...
-
In this paper, we investigate a hybrid scheme that combines nonlinear model predictive control (MPC) and model-based reinforcement learning (RL) for navigation planning of an autonomous model car across offroad, unstructured terrains without relying on predefined maps. Our innovative approach takes inspiration from BADGR, an LSTM-based network that primarily concentrates on environment modeling, b...
-
Metric occupancy maps are widely used in autonomous robot navigation systems. However, when a robot is deployed in an unseen environment, building an accurate metric map is time-consuming. Can an autonomous robot directly navigate in previously unseen environments using coarse maps? In this work, we propose the Coarse Map Navigator (CMN), a navigation framework that can perform robot navigation in...
-
In the field of robotics, tag systems play an important role in various applications, such as object identification and robot control in real-world environments. While typical visual markers use two-dimensional (2D) patterns and RGB cameras for recognizing object IDs and poses, achieving long-distance recognition necessitates increasing marker size and camera magnification to ensure the required r...
-
Extracting motion information from videos with optical flow estimation is vital in multiple practical robot applications. Current optical flow approaches show remarkable accuracy, but top-performing methods have high computational costs and are unsuitable for embedded devices. Although some previous works have focused on developing low-cost optical flow strategies, their estimation quality has a n...
-
Soft manipulators offer the advantages of safety and adaptability. However, due to insufficient stiffness and single motion mode limitations, existing soft manipulators usually exhibit low load capacity and small working space. To address this problem, we propose a novel soft hybrid-driven manipulator with continuous stiffness control capability and multiple motion patterns (omnidirectional bendin...
-
Soft robotic manipulators with many degrees of freedom can carry out complex tasks safely around humans. However, manufacturing of soft robotic hands with several degrees of freedom requires a complex multi-step manual process, which significantly increases their cost. We present a design of a multi-material 15 DoF robotic hand with five fingers including an opposable thumb. Our design has 15 pneu...
-
Hand extension is crucial for stroke survivors with spasticity, where their fingers become rigid and their thumb remains curled within the palm. Due to the underactuated nature of the hand, the dominance of flexor muscles over extensors, and the limited surface area available, developing an extension glove with thumb assistance poses a challenge for researchers. This paper introduces a fully weara...
-
Soft actuators have shown advantages of adaptiveness, large deformation, and safe human-robot interaction, making them suitable for various applications. Herein, a novel soft flat tube twisting actuator (SFTTA) is proposed. The SFTTA is composed of a folded flat tube sandwiched between two silicone rubber laminates. When inflated by compressed air, the folded corners of the flat tube tend to unfol...
-
Robot performance has advanced considerably both in and out of the factory, however in tightly constrained, unknown environments such as inside a jet engine or the human heart, current robots are less adept. In such cases where a borescope or endoscope can’t reach, disassembly or surgery are costly. One promising inspection device inspired by plant growth are "vine robots" that can navigate clutte...
-
Stability and reliable operation under a spectrum of environmental conditions is still an open challenge for soft and continuum style manipulators. The inability to carry sufficient load and effectively reject external disturbances are two drawbacks which limit the scale of continuum designs, preventing widespread adoption of this technology. To tackle these problems, this work details the design ...
-
A robot’s ability to anticipate the 3D action target location of a hand’s movement from egocentric videos can greatly improve safety and efficiency in human-robot interaction (HRI). While previous research predominantly focused on semantic action classification or 2D target region prediction, we argue that predicting the action target’s 3D coordinate could pave the way for more versatile downstrea...
-
Since autonomous driving systems usually face dynamic and ever-changing environments, continual test-time adaptation (CTTA) has been proposed as a strategy for transferring deployed models to continually changing target domains. However, the pursuit of long-term adaptation often introduces catastrophic forgetting and error accumulation problems, which impede the practical implementation of CTTA in...
-
Slip detection plays a pivotal role in the dexterity of robotics, improving the reliability and precision of manipulations but also contributing to safety, efficiency, and adaptability. Deep learning-based slip detection algorithms commonly difficult to concentrate on key features when faced with dense 3D shape data obtained by visuo-tactile sensors. Data from noncontact locations can interfere wi...
-
We propose a novel 3-D human motion and object interaction prediction model that is aware of commonsense knowledge about human–object interaction. We jointly predict human joint motion and human–object interactions. The two prediction results are combined to enforce commonsense knowledge, such as "if the human right hand is predicted to be in contact with an object after 1 second, the distance bet...
-
A robot self-model is a task-agnostic representation of the robot’s physical morphology that can be used for motion planning tasks in the absence of a classical geometric kinematic model. In particular, when the latter is hard to engineer or the robot’s kinematics change unexpectedly, human-free self-modeling is a necessary feature of truly autonomous agents. In this work, we leverage neural field...
-
Affordance detection and pose estimation are of great importance in many robotic applications. Their combination helps the robot gain an enhanced manipulation capability, in which the generated pose can facilitate the corresponding affordance task. Previous methods for affodance-pose joint learning are limited to a predefined set of affordances, thus limiting the adaptability of robots in real-wor...
-
In this paper, we propose a novel method for plane clustering specialized in cluttered scenes using an RGB-D camera and validate its effectiveness through robot grasping experiments. Unlike existing methods, which focus on large- scale indoor structures, our approach—Multi-Object RANSAC emphasizes cluttered environments that contain a wide range of objects with different scales. It enhances plane ...
-
This paper presents a novel strategy to train keypoint detection models for robotics applications. Our goal is to develop methods that can robustly detect and track natural features on robotic manipulators. Such features can be used for vision-based control and pose estimation purposes, when placing artificial markers (e.g. ArUco) on the robot’s body is not possible or practical in runtime. Prior ...
-
While it is generally acknowledged that force feedback is beneficial to robotic control, applications of policy learning to robotic manipulation typically only leverage visual feedback. Recently, symmetric neural models have been used to significantly improve the sample efficiency and performance of policy learning across a variety of robotic manipulation domains. This paper explores an applicatio...
-
Food handling is a challenging task for robotic grippers, as it requires to manipulate highly deformable and fragile items, that can be easily damaged. Moreover, ingredients for the preparation of the different dishes are usually stored in small containers that are often not easily accessible. This paper introduces an innovative soft-rigid, tendon-driven gripper: the Double-Scoop Gripper (DSG). It...
-
A robotic system’s hardware and control policy must be co-optimized to ensure they complement each other to interact robustly with the environment. However, this combined search is extremely high-dimensional and intractable without a suitable underlying representation. This paper uses environmental constraints to structure the co-design space for manipulation. We show that task-relevant constraint...
-
Squirrels exhibit agile leaping between tree branches, often using non-prehensile gripping with compliant and passively adaptive fingers. We aim to test the utility of such gripping in agile robotic maneuvering. In the present study, we first examine the parametric design of a squirrel-inspired underactuated gripper for passive landing on impact. We fix the geometry of the gripper and vary the joi...
-
The seafood processing industry provides fertile ground for robotics to impact the future-of-work from multiple perspectives including productivity, worker safety, and quality of work life. The robotics research challenge in this domain is the realization of flexible and reliable manipulation of soft, deformable, slippery, spiky and scaly objects. In this paper, we propose a novel robot end effect...
-
The detection of unknown traffic obstacles is vital to ensure safe autonomous driving. The standard object-detection methods cannot identify unknown objects that are not included under predefined categories. This is because object-detection methods are trained to assign a background label to pixels corresponding to the presence of unknown objects. To address this problem, the pixel-wise anomaly-de...
-
3D object detectors for point clouds often rely on a pooling-based PointNet [20] to encode sparse points into grid-like voxels or pillars. In this paper, we identify that the common PointNet design introduces an information bottleneck that limits 3D object detection accuracy and scalability. To address this limitation, we propose PVTransformer: a transformer-based point-to-voxel architecture for 3...
-
3D object detection is an essential vision technique for various robotic systems, such as augmented reality and domestic robots. Transformers as versatile network architectures have recently seen great success in 3D point cloud object detection. However, the lack of hierarchy in a plain transformer restrains its ability to learn features at different scales. Such limitation makes transformer detec...
-
In LiDAR-based 3D detection, history point clouds contain rich temporal information helpful for future prediction. In the same way, history detections should contribute to future detections. In this paper, we propose a detection enhancement method, namely FrameFusion, which improves 3D object detection results by fusing history detection frames. In FrameFusion, we "forward" history frames to the c...
-
Developing high-performance, real-time architectures for LiDAR-based 3D object detectors is essential for the successful commercialization of autonomous vehicles. Pillar-based methods stand out as a practical choice for onboard deployment due to their computational efficiency. However, despite their efficiency, these methods can sometimes underperform compared to alternative point encoding techniq...
-
Robots, constrained by limited onboard computing resources, often encounter situations wherein high-resolution and high-bit-rate videos captured by their cameras necessitate compression before further analysis. In this paper, we propose a novel video semantic segmentation paradigm for compressed video. Specifically, our framework draws the inspiration from the principle of Wavelet Transform, and t...
-
We present a new 3D point-based detector model, named Shift-SSD, for precise 3D object detection in autonomous driving. Traditional point-based 3D object detectors often employ architectures that rely on a progressive downsampling of points. While this method effectively reduces computational demands and increases receptive fields, it will compromise the preservation of crucial non-local informati...
-
Panoptic segmentation is a challenging perception task, which can help robots to comprehensively perceive the surrounding environment. In the task, we notice that semantic, instance, and panoptic have rich relations, however, which are rarely explored. In this work, we propose a novel panoptic, instance, and semantic bridged network to delve into the reciprocal relation. To make semantic and insta...
-
Cooking robots can enhance the home experience by reducing the burden of daily chores. However, these robots must perform their tasks dexterously and safely in shared human environments, especially when handling dangerous tools such as kitchen knives. This study focuses on enabling a robot to autonomously and safely learn food-cutting tasks. More specifically, our goal is to enable a collaborative...
-
Object rearrangement is pivotal in robotic-environment interactions, representing a significant capability in embodied AI. In this paper, we present SG-Bot, a novel rearrangement framework that utilizes a coarse-to-fine scheme with a scene graph as the scene representation. Unlike previous methods that rely on either known goal priors or zero-shot large models, SG-Bot exemplifies lightweight, real...
-
Scalable Multi-Robot Collaboration with Large Language Models: Centralized or Decentralized Systems?
A flurry of recent work has demonstrated that pre-trained large language models (LLMs) can be effective task planners for a variety of single-robot tasks. The planning performance of LLMs is significantly improved via prompting techniques, such as in-context learning or re-prompting with state feedback, placing new importance on the token budget for the context window. An under-explored but natura...
-
Humans interpret scenes by recognizing both the identities and positions of objects in their observations. For a robot to perform tasks such as "pick and place", understanding both what the objects are and where they are located is crucial. While the former has been extensively discussed in the literature that uses the large language model to enrich the text descriptions, the latter remains undere...
-
This work stages Foosball as a versatile platform for advancing scientific research, particularly in the realm of robot learning. We present an automated Foosball table along with its corresponding simulated counterpart, showcasing a diverse range of challenges through example tasks within the Foosball environment. Initial findings are shared using a simple baseline approach. Foosball constitutes ...
-
The language-conditioned robotic manipulation aims to transfer natural language instructions into executable actions, from simple "pick-and-place" to tasks requiring intent recognition and visual reasoning. Inspired by the dual-process theory in cognitive science—which suggests two parallel systems of fast and slow thinking in human decision-making—we introduce Robotics with Fast and Slow Thinking...
-
Large Language Models (LLMs) have demonstrated the ability to perform semantic reasoning, planning and write code for robotics tasks. However, most methods rely on pre-existing primitives (i.e. pick, open drawer) or similar examples of robot code alone, which heavily limits their scalability to new scenarios. We present PromptBook, a collection of different prompting paradigms to generate code for...
-
We propose and demonstrate a compositional framework for training and verifying reinforcement learning (RL) systems within a multifidelity sim-to-real pipeline, in order to deploy reliable and adaptable RL policies on physical hardware. By decomposing complex robotic tasks into component subtasks and defining mathematical interfaces between them, the framework allows for the independent training a...
-
Contact-rich manipulation tasks often exhibit a large sim-to-real gap. For instance, industrial assembly tasks frequently involve tight insertions where the clearance is less than 0.1 mm and can even be negative when dealing with a deformable receptacle. This narrow clearance leads to complex contact dynamics that are difficult to model accurately in simulation, making it challenging to transfer s...
-
The automated assembly of complex products requires a system that can automatically plan a physically feasible sequence of actions for assembling many parts together. In this paper, we present ASAP, a physics-based planning approach for automatically generating such a sequence for general-shaped assemblies. ASAP accounts for gravity to design a sequence where each sub-assembly is physically stable...
-
Multi-session mapping serves as the pre-requisite for autonomous robots to fulfill various long-term tasks (e.g., map updating, navigation, collaboration). However, it is challenging to implement multi-session mapping in enclosed or partially enclosed ambiguous environments (e.g., long corridors, industrial warehouses). Existing solutions either depend heavily on the matching of elementary geometr...
-
Graph-patrolling problems in the adversarial domain typically embed models and assumptions about how hostile events, from which an environment must be protected, are generated at a specific time and location. Relying upon such attacker models prevents algorithms from synthesizing strategies that can generalize in different settings, providing good performance under different and uncertain scenario...
-
Graph patrolling algorithms provide effective strategies for coordinating mobile robots in the context of autonomously surveilling valuable assets. Optimizing patrolling strategies often aims to minimize the time between subsequent visits to a vertex, a measure known in the literature as idleness. In the domain of multi-robot patrolling, two approaches have received the most attention so far. The ...
-
Centralized control of a multi-agent system improves upon distributed control especially when multiple agents share a common task e.g., sorting different materials in a recycling facility. Traditionally, each agent in a sorting facility is tuned individually which leads to suboptimal performance if one agent is less efficient than the others. Centralized control overcomes this bottleneck by levera...
-
In safety-critical domains like autonomous driving (AD), errors by the object detector may endanger pedestrians and other vulnerable road users (VRU). As raw evaluation metrics are not an adequate safety indicator, recent works leverage domain knowledge to identify safety-relevant VRU, and to back-annotate the criticality of the interaction to the object detector. However, those approaches do not ...
-
Fast and efficient semantic segmentation of large-scale LiDAR point clouds is a fundamental problem in autonomous driving. To achieve this goal, the existing point-based methods mainly choose to adopt Random Sampling strategy to process large-scale point clouds. However, our quantative and qualitative studies have found that Random Sampling may be less suitable for the autonomous driving scenario,...
-
Tracking objects in three-dimensional space is critical for autonomous driving. To ensure safety while driving, the tracker must be able to reliably track objects across frames and accurately estimate their states such as velocity and acceleration in the present. Existing works frequently focus on the association task while either neglecting the model’s performance on state estimation or deploying...
-
In recent years, autonomous driving has garnered significant attention due to its potential for improving road safety through collaborative perception among connected and autonomous vehicles (CAVs). However, time-varying channel variations in vehicular transmission environments demand dynamic allocation of communication resources. Moreover, in the context of collaborative perception, it is importa...
-
Planning a public transit network is a challenging optimization problem, but essential in order to realize the benefits of autonomous buses. We propose a novel algorithm for planning networks of routes for autonomous buses. We first train a graph neural net model as a policy for constructing route networks, and then use the policy as one of several mutation operators in a evolutionary algorithm. W...
-
The tremendous hype around autonomous driving is eagerly calling for emerging and novel technologies to support advanced mobility use cases. As car manufactures keep developing SAE level 3+ systems to improve the safety and comfort of passengers, traffic authorities need to establish new procedures to manage the transition from human-driven to fully-autonomous vehicles while providing a feedback-l...
-
Mobile autonomy relies on the precise perception of dynamic environments. Robustly tracking moving objects in 3D world thus plays a pivotal role for applications like trajectory prediction, obstacle avoidance, and path planning. While most current methods utilize LiDARs or cameras for Multiple Object Tracking (MOT), the capabilities of 4D imaging radars remain largely unexplored. Recognizing the c...
-
Traffic congestion is a persistent problem in our society. Previous methods for traffic control have proven futile in alleviating current congestion levels leading researchers to explore ideas with robot vehicles given the increased emergence of vehicles with different levels of autonomy on our roads. This gives rise to mixed traffic control, where robot vehicles regulate human-driven vehicles thr...
-
Sit-to-Stand (StS) is a fundamental daily activity that can be challenging for stroke survivors due to strength, motor control, and proprioception deficits in their lower limbs. Existing therapies involve repetitive StS exercises, but these can be physically demanding for therapists while assistive devices may limit patient participation and hinder motor learning. To address these challenges, this...
-
This paper presents an intraoperatively iterative Hough transform (IHT) based in-plane hybrid control of extracorporeal ultrasound (US) guided magnetic catheterization for arterial intervention. One uniqueness lies in that both control and tracking of the arterial robotic ultrasound end-effector have been implemented to improve performance. Firstly, the magnetic catheter model and hybrid visual/fo...
-
Magnetic microrobots can be navigated by an external magnetic field to autonomously move within living organisms with complex and unstructured environments. Potential applications include drug delivery, diagnostics, and therapeutic interventions. Existing techniques commonly impart magnetic properties to the target object, or drive the robot to contact and then manipulate the object, both probably...
-
Design and Implementation of A Robotized Hand-held Dissector for Endoscopic Pulmonary Endarterectomy
Severe chronic pulmonary endarterectomy needs a dissector to delicately remove proliferative intima located in the depth of the pulmonary artery. This work proposed a novel endoscopic robotized steerable dissector for this surgery, enabling easier access to curved deep artery branches. The handheld surgical dissector also provides suction and visualization for surgeons to enhance effectiveness. Th...
-
Retinal surgery is a complex medical procedure that requires high precision dexterity to perform delicate instrument maneuvers with sub-millimeter accuracy. Minimizing the manual tremor and achieving precise and repeatable execution of surgical tasks has motivated the development of robotic platforms to overcome the limitations of manual surgery. However, specific tasks, such as instrument inserti...
-
For robotic transtibial prosthesis control, the global tibia kinematics can be used to monitor gait cycle progression and command smooth and continuous actuation. In this work, these global tibia kinematics define a phase variable impedance controller (PVIC), which is implemented as the nonvolitional base controller within a hybrid volitional control framework (PVI-HVC). The gait progression estim...
-
With increasing numbers of mobile robots arriving in real-world applications, more robots coexist in the same space, interact, and possibly collaborate. Methods to provide such systems with system size scalability are known, for example, from swarm robotics. Example strategies are self-organizing behavior, a strict decentralized approach, and limiting the robot-robot communication. Despite applyin...
-
Coordinating heterogeneous robots is essential for autonomous multi-robot teaming. To execute a set of dependent tasks as quickly as possible, and to complete tasks that cannot be addressed by individual robots, it is necessary to form subteams that can collaboratively finish the tasks. It is also advantageous for robots to wait for teammates and tasks to become available in order to form better s...
-
In this paper we present a fully distributed, asynchronous, and general purpose optimization algorithm for Consensus Simultaneous Localization and Mapping (CSLAM). Multi-robot teams require that agents have timely and accurate solutions to their state as well as the states of the other robots in the team. To optimize this solution we develop a CSLAM back-end based on Consensus ADMM called MESA (Ma...
-
Uncertainty-bounded Active Monitoring of Unknown Dynamic Targets in Road-networks with Minimum Fleet
Fleets of unmanned robots can be beneficial for the long-term monitoring of large areas, e.g., to monitor wild flocks, detect intruders, search and rescue. Monitoring numerous dynamic targets in a collaborative and efficient way is a challenging problem that requires online coordination and information fusion. The majority of existing works either assume a passive all-to-all observation model to m...
-
This paper presents a collaborative quadrotor-quadruped robot system for the manipulation of a cable-towed payload. In particular, we aim to solve the challenge from the unknown dynamics of the cable-towed payload. To this end, we first propose novel dynamic models for both the quadrotor and the quadruped robot, taking into account the nonlinear robot dynamics and the uncertainties associated with...
-
We focus on the problem of long-range dynamic replanning for off-road autonomous vehicles, where a robot plans paths through a previously unobserved environment while continuously receiving noisy local observations. An effective approach for planning under sensing uncertainty is determinization, where one converts a stochastic world into a deterministic one and plans under this simplification. Thi...
-
Accurately assessing the potential value of new sensor observations is a critical aspect of planning for active perception. This task is particularly challenging when reasoning about high-level scene understanding using measurements from vision-based neural networks. Due to appearance-based reasoning, the measurements are susceptible to several environmental effects such as the presence of occlude...
-
Optimal decision-making presents a significant challenge for autonomous systems operating in uncertain, stochastic and time-varying environments. Environmental variability over time can significantly impact the system’s optimal decision making strategy for mission completion. To model such environments, our work combines the previous notion of Time-Varying Markov Decision Processes (TVMDP) with pa...
-
Nearly all state-of-the-art SLAM algorithms are designed to exploit patterns in data from specific sensing modalities, such as time-of-flight and structured light depth sensors, or RGB cameras. This specialization increases localization accuracy in domains where the given modality detects many high-quality features, but comes at the cost of decreasing performance in other, less favorable environme...
-
This paper proposes an informative trajectory planning approach, namely, adaptive particle filter tree with sigma point-based mutual information reward approximation (ASPIRe), for mobile target search and tracking (SAT) in cluttered environments with limited sensing field of view. We develop a novel sigma point-based approximation to accurately estimate mutual information (MI) for general, non-Gau...
-
The human shoulder, with its glenohumeral joint, tendons, ligaments, and muscles, allows for the execution of complex tasks with precision and efficiency. However, current robotic shoulder designs lack the compliance and compactness inherent in their biological counterparts. A major limitation of these designs is their reliance on external sensors like rotary encoders, which restrict mechanical jo...
-
This study presents the design and validation of a variable stiffness actuator incorporating multiple cam mechanisms. The actuator is intended for use in walking assistance, focusing on assisting individuals with diminished ankle function. This study highlights the advantages of variable stiffness actuators over traditional and other modern actuators in mobility assistance. The working principles ...
-
There is a lack of cleaning robots dedicated to the scrubbing of contaminated surfaces. Contaminated surfaces in domestic and industrial settings typically require manual scrubbing which can be costly or hazardous. There is growing demand for automated sanitization systems in hospitals, food-processing plants, and other settings where cleanliness of surfaces is important. To address the opportunit...
-
Cable-driven continuum manipulators have gained considerable attention due to their high dexterity and inherent structural compliance, making them a popular research topic. However, previous studies have overlooked the kinetostatics of these manipulators, which can result in a multi-solution problem. This issue is critical as having multiple equilibrium states can lead to erroneous estimations of ...
-
Some cable-driven parallel robots (CDPRs) can be rapidly deployed on-site. To achieve such deployability, the fixed frame is usually substituted by four masts. However, not having any rigid fixture between the masts reduces the overall stiffness of the CDPR. This paper introduces a CDPR called Rocaspect, that has four compliant masts. The robot behavior and accuracy is evaluated experimentally and...
-
Bowden cables serve as essential components in various mechanical systems, facilitating power transmission from remote actuators to specific destinations. The pretension of Bowden cables profoundly influences system performance, notably in terms of friction. This study investigates the effects of cable pretension and shape on friction and torque efficiency. A custom self-designed testbed, comprisi...
-
We introduce OpenBot-Fleet, a comprehensive open-source cloud robotics system for navigation. OpenBot-Fleet uses smartphones for sensing, local compute and communication, Google Firebase for secure cloud storage and off-board compute, and a robust yet low-cost wheeled robot to act in real-world environments. The robots collect task data and upload it to the cloud where navigation policies can be l...
-
Widely adopted motion forecasting datasets sub-stitute the observed sensory inputs with higher-level abstractions such as 3D boxes and polylines. These sparse shapes are inferred through annotating the original scenes with perception systems’ predictions. Such intermediate representations tie the quality of the motion forecasting models to the performance of computer vision models. Moreover, the h...
-
The use of industrial robots represents a key technology for increasing productivity and efficiency in manufacturing. However, their low absolute position accuracy still denies the broad substitution of machine tools by industrial robots. In this paper, a data-driven method for accuracy enhancement of industrial robots under consideration of kinematic, elastic, and thermal effects is presented. A ...
-
The existing internet-scale image and video datasets cover a wide range of everyday objects and tasks, bringing the potential of learning policies that generalize in diverse scenarios. Prior works have explored visual pre-training with different self-supervised objectives. Still, the generalization capabilities of the learned policies and the advantages over well-tuned baselines remain unclear fro...
-
The grand aim of having a single robot that can manipulate arbitrary objects in diverse settings is at odds with the paucity of robotics datasets. Acquiring and growing such datasets is strenuous due to manual efforts, operational costs, and safety challenges. A path toward such a universal agent requires an efficient framework capable of generalization but within a reasonable data budget. In this...
-
We introduce Dream2Real, a robotics framework which integrates vision-language models (VLMs) trained on 2D data into a 3D object rearrangement pipeline. This is achieved by the robot autonomously constructing a 3D representation of the scene, where objects can be rearranged virtually and an image of the resulting arrangement rendered. These renders are evaluated by a VLM, so that the arrangement w...
-
The pre-train and fine-tune paradigm in machine learning has had dramatic success in a wide range of domains because the use of existing data or pre-trained models on the internet enables quick and easy learning of new tasks. We aim to enable this paradigm in robotic reinforcement learning, allowing a robot to learn a new task with little human effort by leveraging data and models from the Interne...
-
Motion forecasting has become an increasingly critical component of autonomous robotic systems. Onboard compute budgets typically limit the accuracy of real-time systems. In this work we propose methods of improving motion forecasting systems subject to limited compute budgets by combining model ensemble and distillation techniques. The use of ensembles of deep neural networks has been shown to im...
-
Robot software is abundant with variables that represent real-world physical units (e.g., meters, seconds). Operations over different units (e.g., adding meters and seconds) may be incorrect and can lead to dangerous system misbehaviors; manually detecting such mistakes is challenging. Current software analysis techniques identify such mismatches using dimensional analysis rules and ROS-specific a...
-
While individual robots are becoming increasingly capable, the complexity of expected missions increases exponentially in comparison. To cope with this complexity, heterogeneous teams of robots have become a significant research interest in recent years. Making effective use of the robots and their unique skills in a team is challenging. Dynamic runtime conditions often make static task allocation...
-
In this paper, we develop a control framework for the coordination of multiple robots as they navigate through crowded environments. Our framework comprises of a local model predictive control (MPC) for each robot and a social long short-term memory model that forecasts pedestrians’ trajectories. We formulate the local MPC formulation for each individual robot that includes both individual and sha...
-
In this work, we present a coordination strategy tailored for scenarios involving multiple agents and tasks. We devise a range of tasks using signal temporal logic (STL), each earmarked for specific agents. These tasks are then imposed through control barrier function (CBF) constraints to ensure completion. To extend existing methodologies, our framework adeptly manages interactions among multiple...
-
Relative navigation methods are a critical enabling technology for the next generation of autonomous spacecraft conducting close proximity operations. This is especially true for multi-agent inspection operations in which safety including intra-agent or agent-target collisions are a serious concern. Additionally, in an on-orbit servicing operation various failure modes of the target may result in ...
-
Maximizing the utility of limited Earth observing satellite resources is a difficult ongoing problem. Dynamic Targeting is an approach to this challenge that intelligently plans and executes primary sensor observations based on information from a look-ahead sensor. However, current implementations have failed to account for realistic satellite operational constraints and have used static utility f...
-
In the field of resource-constrained robots and the need for effective place recognition in multi-robotic systems, this article introduces RecNet, a novel approach that concurrently addresses both challenges. The core of RecNet’s methodology involves a transformative process: it projects 3D point clouds into range images, compresses them using an encoder-decoder framework, and subsequently reconst...
-
Object permanence, which refers to the concept that objects continue to exist even when they are no longer perceivable through the senses, is a crucial aspect of human cognitive development. In this work, we seek to incorporate this understanding into interactive robots by proposing a set of assumptions and rules to represent object permanence in multi-object, multi-agent interactive scenarios. We...
-
Object tracking is central to robot perception and scene understanding, allowing robots to parse a video stream in terms of moving objects with names. Tracking-by-detection has long been a dominant paradigm for object tracking of specific object categories [1], [2]. Recently, large-scale pre-trained models have shown promising advances in detecting and segmenting objects and parts in 2D static ima...
-
Given the difficulty of manually annotating motion in video, the current best motion estimation methods are trained with synthetic data, and therefore struggle somewhat due to a train/test gap. Self-supervised methods hold the promise of training directly on real video, but typically perform worse. These include methods trained with warp error (i.e., color constancy) combined with smoothness terms...
-
Modern robotic systems are required to operate in dense dynamic environments, requiring highly accurate real-time track identification and estimation. For 3D multi-object tracking, recent approaches process a single measurement frame recursively with greedy association and are prone to errors in ambiguous association decisions. Our method, Sliding Window Tracker (SWTrack), yields more accurate ass...
-
Multi-object tracking (MOT) methods have seen a significant boost in performance recently, due to strong interest from the research community and steadily improving object detection methods. The majority of tracking methods, which follow the tracking-by-detection (TBD) paradigm, blindly trust the incoming detections with no sense of their associated localization uncertainty. This lack of uncertain...
-
The recent advancements in transformer-based visual trackers have led to significant progress, attributed to their strong modeling capabilities. However, as performance improves, running latency correspondingly increases, presenting a challenge for real-time robotics applications, especially on edge devices with computational constraints. In response to this, we introduce LiteTrack, an efficient t...
-
Depth estimation models have shown promising performance on clear scenes but fail to generalize to adverse weather conditions due to illumination variations, weather particles, etc. In this paper, we propose WeatherDepth, a self-supervised robust depth estimation model with curriculum contrastive learning, to tackle performance degradation in complex weather conditions. Concretely, we first presen...
-
Collaborative decision-making is an essential capability for multi-robot systems, such as connected vehicles, to collaboratively control autonomous vehicles in accident-prone scenarios. Under limited communication bandwidth, capturing comprehensive situational awareness by integrating connected agents’ observation is very challenging. In this paper, we propose a novel collaborative decision-making...
-
We investigate a variation of the 3D registration problem, named multi-model 3D registration. In the multi-model registration problem, we are given two point clouds picturing a set of objects at different poses (and possibly including points belonging to the background) and we want to simultaneously reconstruct how all objects moved between the two point clouds. This setup generalizes standard 3D ...
-
Understanding the real world through point cloud video is a crucial aspect of robotics and autonomous driving systems. However, prevailing methods for 4D point cloud recognition have limitations due to sensor resolution, which leads to a lack of detailed information. Recent advances have shown that Vision-Language Models (VLM) pre-trained on web-scale text-image datasets can learn fine-grained vis...
-
For robots to perform a wide variety of tasks, they require a 3D representation of the world that is semantically rich, yet compact and efficient for task-driven perception and planning. Recent approaches have attempted to leverage features from large vision-language models to encode semantics in 3D representations. However, these approaches tend to produce maps with per-point feature vectors, whi...
-
Existing 3D understanding datasets typically provide annotations for a limited number of object classes, with sufficient examples per class. However, real-world object classes are not equally represented in practical settings, leading to poor performance on rarely-occurring categories if the class imbalance is neglected. In this work, we address the challenge of 3D semantic segmentation with a lon...
-
Despite many successful applications of data-driven control in robotics, extracting meaningful diverse behaviors remains a challenge. Typically, task performance needs to be compromised in order to achieve diversity. In many scenarios, task requirements are specified as a multitude of reward terms, each requiring a different trade-off. In this work, we take a constrained optimization viewpoint on ...
-
Along with the advancement of robot skin technology, there has been notable progress in the development of snake robots featuring body-surface tactile perception. In this study, we proposed a locomotion control framework for snake robots that integrates tactile perception to augment their adaptability to various terrains. Our approach embraces a hierarchical reinforcement learning (HRL) architectu...
-
Localization is paramount for autonomous robots. While camera and LiDAR-based approaches have been extensively investigated, they are affected by adverse illumination and weather conditions. Therefore, radar sensors have recently gained attention due to their intrinsic robustness to such conditions. In this paper, we propose RaLF, a novel deep neural network-based approach for localizing radar sca...
-
NeRF can reconstruct incredibly realistic environmental maps in dense simultaneous localization and mapping, providing robots with more comprehensive scene map information. However, NeRF often struggles with geometric distortions in indoor reconstructions. To correct geometric distortions, we develop VPE-SLAM, based on the proposed voxel-permutohedral encoding, which can incrementally reconstruct ...
-
The growing focus on indoor robot navigation utilizing wireless signals has stemmed from the capability of these signals to capture high-resolution angular and temporal measurements. Prior heuristic-based methods, based on radio frequency (RF) propagation, are intuitive and generalizable across simple scenarios, yet fail to navigate in complex environments. On the other hand, end-to-end (e2e) deep...
-
Navigation methods based on deep reinforcement learning (RL) have recently exhibited superior performance, particularly for navigation in dynamic environments. However, most existing methods solely rely on deep neural network feature encoders to extract features from raw LiDAR data, lacking an explicit representation of environmental structure. This limitation hinders effective environmental repre...
-
2D top-down maps are commonly used for the navigation and exploration of mobile robots through unknown areas. Typically, the robot builds the navigation maps incrementally from local observations using onboard sensors. Recent works have shown that predicting the structural patterns in the environment through learning-based approaches can greatly enhance task efficiency. While many such works build...
-
In order to enhance driving convenience and safety, automotive Augmented Reality displays, e.g., head-up displays, have garnered attention and are gradually being deployed. However, when vehicles encounter uneven roads, vertical vibrations lead to mismatches between external physical objects and augmented reality overlay images, adversely affecting the AR display’s visibility. Resolving the proble...
-
Reinforcement learning for control over continuous spaces typically uses high-entropy stochastic policies, such as Gaussian distributions, for local exploration and estimating policy gradient to optimize performance. Many robotic control problems deal with complex unstable dynamics, where applying actions that are off the feasible control manifolds can quickly lead to undesirable divergence. In su...
-
The prominence of embodied Artificial Intelligence (AI), which empowers robots to navigate, perceive, and engage within virtual environments, has attracted significant attention, owing to the remarkable advances in computer vision and large language models. Privacy emerges as a pivotal concern within the realm of embodied AI, as the robot accesses substantial personal information. However, the iss...
-
The sim-to-real gap poses a significant challenge in RL-based multi-agent exploration due to scene quantization and action discretization. Existing platforms suffer from the inefficiency in sampling and the lack of diversity in Multi-Agent Reinforcement Learning (MARL) algorithms across different scenarios, restraining their widespread applications. To fill these gaps, we propose MAexp, a generic ...
-
Offline reinforcement learning (RL) provides a promising approach to avoid costly online interaction with the real environment. However, the performance of offline RL highly depends on the quality of the datasets, which may cause extrapolation error in the learning process. In many robotic applications, an inaccurate simulator is often available. However, the data directly collected from the inacc...
-
In this work, we introduce REFORMA, a novel robust reinforcement learning (RL) approach to design controllers for unmanned aerial vehicles (UAVs) robust to unknown disturbances during flights. These disturbances, typically due to wind turbulence, electromagnetic interference, temperature extremes and many other external physical interference, are highly dynamic and difficult to model. REFORMA can ...
-
Efficiency and performance are significant challenges in applying Machine Learning (ML) to robotics, especially in energy-constrained real-world scenarios. In this context, Hyperdimensional Computing offers an energy-efficient alternative but has been underexplored in robotics. We introduce ReactHD, an HDC-based framework tailored for perception-action-based learning for sensorimotor controls of r...
-
Manipulation of deformable Linear objects (DLOs), including iron wire, rubber, silk, and nylon rope, is ubiquitous in daily life. These objects exhibit diverse physical properties, such as Young’s modulus and bending stiffness. Such diversity poses challenges for developing generalized manipulation policies. However, previous research limited their scope to single-material DLOs and engaged in time...
-
In outdoor environments, Vision-and-Language Navigation (VLN) requires an agent to rely on multi-modal cues from real-world urban environments and natural language instructions. While existing outdoor VLN models predict actions using a combination of panorama and instruction features, this approach ignores objects in the environment and learns data bias to fail navigation. According to our prelimi...
-
Recent results suggest that splitting topological navigation into robot-independent and robot-specific components improves navigation performance by enabling the robot-independent part to be trained with data collected by robots of different types. However, the navigation methods’ performance is still limited by the scarcity of suitable training data and they suffer from poor computational scaling...
-
This paper presents a novel method for state estimation of rigid body attitude system evolving on the manifold S3, which is crucial in robotics and drone applications. We introduce a particle filter with stable embedding that extends the system into Euclidean space while ensuring stability of the manifold. Our particle filter with stable embedding enables accurate state estimation by maintaining e...
-
Aggressive time-optimal control of quadcopters poses a significant challenge in the field of robotics. The state-of-the-art approach leverages reinforcement learning (RL) to train optimal neural policies. However, a critical hurdle is the sim-to-real gap, often addressed by employing a robust inner loop controller —an abstraction that, in theory, constrains the optimality of the trained controller...
-
Quadrotors may crash and cause severe accidents under instantaneously drastic disturbances. To mitigate the effect of such disturbances, these critical issues should be considered: efficient disturbance observation and compensation, full attitude controllability, and instant output power generation of the quadrotor. In this paper, to keep the quadrotor stable even under suddenly drastic disturbanc...
-
Current multi-modal hybrid robots with flight and wheeled modes have fallen into the dilemma that they can only avoid obstacles by re-taking off when encountering obstacles due to the poor performance of wheeled obstacle-crossing. To tackle this problem, this paper presents a novel multi-modal hybrid robot with the ability to actively adjust the wheel’s size, which is inspired by the behavior of t...
-
Existing exploration algorithms mainly generate frontiers using random sampling or motion primitive methods within a specific sensor range or search space. However, frontiers generated within constrained spaces lead to back-and-forth maneuvers in large-scale environments, thereby diminishing exploration efficiency. To address this issue, we propose a method that utilizes a 3D dense map to generate...
-
Inspection of powerlines is a hard problem that requires humans to operate in remote locations and dangerous conditions. This paper proposes a quadcopter unmanned aerial vehicle (UAV) equipped with rolling-capable perching mechanisms and a depth-vision system for the purpose of autonomous power line inspection. The perching mechanism grips onto the power line, allowing the UAV to withstand externa...
-
This paper introduces the open-source framework, GIRA, which implements fundamental robotics algorithms for reconstruction, pose estimation, and occupancy modeling using compact generative models. Compactness enables perception in the large by ensuring that the perceptual models can be communicated through low-bandwidth channels during large-scale mobile robot deployments. The generative property ...
-
Efficiently performing intervention tasks underwater is crucial in various commercial and scientific sectors; however, propeller-driven vehicles face limitations due to their floating nature. In Remotely Operated Vehicles (ROVs) operations, this can be compensated by the ability of the operator, but they come with high operational costs. Instead, Autonomous Underwater Vehicles (AUVs) have shown pr...
-
Time-effective and accurate source localization with mobile robots is crucial in safety-critical scenarios, e.g. leakage detection. This becomes particular challenging in realistic cluttered scenarios, i.e. in the presence of complex current flows or wind. Traditional methods often fall short due to simplifications or limited onboard resources.We propose to combine source localization with a Gauss...
-
This paper presents a strategy for predicting the turning radius of a sailing robot with consideration of aerodynamic and hydrodynamic interferences from the marine environment. The turning radius is initially obtained based on three consecutive designated points during the turning process, which is regarded as the baseline method. Subsequently, on the basis of our constructed turning datasets, a ...
-
This paper tackles a localization problem in large-scale indoor environments with wayfinding maps. A wayfinding map abstractly portrays the environment, and humans can localize themselves based on the map. However, when it comes to using it for robot localization, large geometrical discrepancies between the wayfinding map and the real world make it hard to use conventional localization methods. Ou...
-
Visual localization is an important sub-task in SfM and visual SLAM that involves estimating a 6-DoF camera pose for an input query image relative to a given 3D model of the environment. The most accurate approach is a hierarchical one that splits the task into two stages: image retrieval and camera pose estimation. Each stage requires different image features, with global features compactly encod...
-
As a promising fashion for visual localization, scene coordinate regression (SCR) has seen tremendous progress in the past decade. Most recent methods usually adopt neural networks to learn the mapping from image pixels to 3D scene coordinates, which requires a vast amount of annotated training data. We propose to leverage Neural Radiance Fields (NeRF) to generate training samples for SCR. Despite...
-
State estimation for legged robots is challenging due to their highly dynamic motion and limitations imposed by sensor accuracy. By integrating Kalman filtering, optimization, and learning-based modalities, we propose a hybrid solution that combines proprioception and exteroceptive information for estimating the state of the robot’s trunk. Leveraging joint encoder and IMU measurements, our Kalman ...
-
We propose a topological mapping and localization system able to operate on real human colonoscopies, despite significant shape and illumination changes. The map is a graph where each node codes a colon location by a set of real images, while edges represent traversability between nodes. For close-in-time images, where scene changes are minor, place recognition can be successfully managed with the...
-
Humans are remarkable in their ability to navigate without metric information. We can read abstract 2D maps, such as floor-plans or hand-drawn sketches, and use them to navigate in unseen rich 3D environments, without requiring prior traversals to map out these scenes in detail. We posit that this is enabled by the ability to represent the environment abstractly as interconnected navigational beha...
-
Light Detection and Ranging (LiDAR) technology has proven to be an important part of many robotics systems. Surface normals estimated from LiDAR data are commonly used for a variety of tasks in such systems. As most of the today’s mechanical LiDAR sensors produce sparse data, estimating normals from a single scan in a robust manner poses difficulties.In this paper, we address the problem of estima...
-
A Colored point cloud, as a simple and efficient 3D representation, has many advantages in various fields, including robotic navigation and scene reconstruction. This representation is now commonly used in 3D reconstruction tasks relying on cameras and LiDARs. However, fusing data from these two types of sensors is poorly performed in many existing frameworks, leading to unsatisfactory mapping res...
-
Recently, pre-trained vision models have gained significant attention in motor control, showcasing impressive performance across diverse robotic learning tasks. While previous works predominantly concentrate on the significance of the pre-training phase, the equally important task of extracting more effective representations based on existing pre-trained visual models remains unexplored. To better...
-
This paper introduces a novel approach to improve robotic grasping in dynamic environments by integrating Gaussian Process Distance Fields (GPDF), SE(3) equivariant networks, and Riemannian Mixture Models. The aim is to enable robots to grasp moving objects effectively. Our approach comprises three main components: object shape reconstruction, grasp sampling, and implicit grasp pose selection. GPD...
-
The problem of grasping objects using a multi-finger hand has received significant attention in recent years. However, it remains challenging to handle a large number of unfamiliar objects in real and cluttered environments. In this work, we propose a representation that can be effectively mapped to the multi-finger grasp space. Based on this representation, we develop a simple decision model that...
-
One goal of dexterous robotic grasping is to allow robots to handle objects with the same level of flexibility and adaptability as humans. However, it remains a challenging task to generate an optimal grasping strategy for dexterous hands, especially when it comes to delicate manipulation and accurate adjustment the desired grasping poses for objects of varying shapes and sizes. In this paper, we ...
-
The disposal of waste electrical and electronic equipment (WEEE) presents a sustainability challenge, particularly for waste printed circuit boards (PCBs). PCBs are challenging to sort out from other waste materials in part because traditional industrial end-effectors struggle to reliably grip these irregularly shaped objects with unmodeled surface-mounted components. Vision-based separators, whil...
-
We present a sampling-based approach to reasoning about the caging-based manipulation of rigid and a simplified class of deformable 3D objects subject to energy constraints. Towards this end, we propose the notion of soft fixtures extending earlier work on energy-bounded caging to include a broader set of energy function constraints, such as gravitational and elastic potential energy of 3D deforma...
-
This paper presents a novel in-hand rolling manipulation method in which a ball on a cloth attached to fingertips is controlled using flexible and adaptive deformation of the cloth. First, an analytical model of the ball-on-cloth system is introduced. The shape of the cloth is simplified, and the rolling constraint of the ball on the cloth is defined focusing on the lowest point of the ball. Next,...
-
Simulation to Real-World Transfer allows affordable and fast training of learning-based robots for manipulation tasks using Deep Reinforcement Learning methods. Currently, Asymmetric Actor-Critic approaches are used for Sim2Real to reduce the rich idealized features in simulation to the accessible ones in the real world. However, the feature reduction from the simulation to the real world is condu...
-
Robotics policies are always subjected to complex, second order dynamics that entangle their actions with resulting states. In reinforcement learning (RL) contexts, policies have the burden of deciphering these complicated interactions over massive amounts of experience and complex reward functions to learn how to accomplish tasks. Moreover, policies typically issue actions directly to controllers...
-
We present in-hand manipulation tasks where a robot moves an object in grasp, maintains its external contact mode with the environment, and adjusts its in-hand pose simultaneously. The proposed manipulation task leads to complex contact interactions which can be very susceptible to uncertainties in kinematic and physical parameters. Therefore, we propose a robust in-hand manipulation method, which...
-
Recently, reinforcement learning has led to dexterous manipulation skills of increasing complexity. Nonetheless, learning these skills in simulation still exhibits poor sample-efficiency which stems from the fact these skills are learned from scratch without the benefit of any domain expertise. In this work, we aim to improve the sample efficiency of learning dexterous in-hand manipulation skills ...
-
Executing contact-rich manipulation tasks necessitates the fusion of tactile and visual feedback. However, the distinct nature of these modalities poses significant challenges. In this paper, we introduce a system that leverages visual and tactile sensory inputs to enable dexterous in-hand manipulation. Specifically, we propose Robot Synesthesia, a novel point cloudbased tactile representation ins...
-
This paper presents a novel framework for robust 3D object detection from point clouds via cross-modal hallucination. Our proposed approach is agnostic to either hallucination direction between LiDAR and 4D radar. We introduce multiple alignments on both spatial and feature levels to achieve simultaneous backbone refinement and hallucination generation. Specifically, spatial alignment is proposed ...
-
Fusing the camera and LiDAR information in the unified BEV representation serves as the elegant paradigm for the 3D detection tasks. Current multi-modal fusion methods in BEV can be categorized into LSS-based and Transformer-based in terms of their view transformation. The former leverages inaccurate depth prediction and massive pseudo points for perspective-to-BEV transformation while the latter ...
-
In autonomous driving and robotics, there is a growing interest in utilizing short-term historical data to enhance multi-camera 3D object detection, leveraging the continuous and correlated nature of input video streams. Recent work has focused on spatially aligning BEV-based features over timesteps. However, this is often limited as its gain does not scale well with long-term past observations. T...
-
Small object detection has been a challenging problem in the field of object detection. There has been some works that proposes improvements for this task, such as adding several attention blocks or changing the whole structure of feature fusion networks. However, the computation cost of these models is large, which makes deploying a real-time object detection system unfeasible, while leaving room...
-
The classical human-robot interface in uncalibrated image-based visual servoing (UIBVS) relies on either human annotations or semantic segmentation with categorical labels. Both methods fail to match natural human communication and convey rich semantics in manipulation tasks as effectively as natural language expressions. In this paper, we tackle this problem by using referring expression segmenta...
-
A vision-based drone-to-drone detection system is crucial for various applications like collision avoidance, countering hostile drones, and search-and-rescue operations. However, detecting drones presents unique challenges, including small object sizes, distortion, occlusion, and real-time processing requirements. Current methods integrating multi-scale feature fusion and temporal information have...
-
Accurate 3D object detection is crucial to autonomous driving. Though LiDAR-based detectors have achieved impressive performance, the high cost of LiDAR sensors precludes their widespread adoption in affordable vehicles. Camera-based detectors are cheaper alternatives but often suffer inferior performance compared to their LiDAR-based counterparts due to inherent depth ambiguities in images. In th...
-
Despite outstanding semantic scene segmentation in closed-worlds, deep neural networks segment novel instances poorly, which is required for autonomous agents acting in an open world. To improve out-of-distribution (OOD) detection for segmentation, we introduce a metacognitive approach in the form of a lightweight module that leverages entropy measures, segmentation predictions, and spatial contex...
-
The hierarchy of global and local planners is one of the most commonly utilized system designs in autonomous robot navigation. While the global planner generates a reference path from the current to goal locations based on the pre-built map, the local planner produces a kinodynamic trajectory to follow the reference path while avoiding perceived obstacles. To account for unforeseen or dynamic obst...
-
In visual SLAM (VSLAM) systems, loop closure plays a crucial role in reducing accumulated errors. However, VSLAM systems relying on low-level visual features often suffer from the problem of perceptual confusion in repetitive environments, where scenes in different locations are incorrectly identified as the same. Existing work has attempted to introduce object-level features or artificial landmar...
-
Generalist robot manipulators need to learn a wide variety of manipulation skills across diverse environments. Current robot training pipelines rely on humans to provide kinesthetic demonstrations or to program simulation environments and to code up reward functions for reinforcement learning. Such human involvement is an important bottleneck towards scaling up robot learning across diverse tasks ...
-
Recent planning methods based on Large Language Models typically employ the In-Context Learning paradigm. Complex long-horizon planning tasks require more context(including instructions and demonstrations) to guarantee that the generated plan can be executed correctly. However, in such conditions, LLMs may overlook(unfaithful) the rules in the given context, resulting in the generated plans being ...
-
As autonomous driving technology matures, end-to-end methodologies have emerged as a leading strategy, promising seamless integration from perception to control via deep learning. However, existing systems grapple with challenges such as unexpected open set environments and the complexity of black-box models. At the same time, the evolution of deep learning introduces larger, multimodal foundation...
-
For effective human-robot interaction, robots need to understand, plan, and execute complex, long-horizon tasks described by natural language. Recent advances in large language models (LLMs) have shown promise for translating natural language into robot action sequences for complex tasks. However, existing approaches either translate the natural language directly into robot trajectories or factor ...
-
Artificial cognitive architectures traditionally rely on complex memory models to encode, store, and retrieve information. However, the conventional practice of transferring all data from working memory (WM) to long-term memory (LTM) leads to high data volumes and challenges in efficient information processing and access. Deciding what information to retain or discard within a robot’s LTM is parti...
-
Calibration of fixtures in robotic work cells is essential but also time consuming and error-prone, and poor calibration can easily lead to wasted debugging time in down-stream tasks. Contact-based calibration methods let the user measure points on the fixture’s surface with a tool tip attached to the robot’s end effector. Most such methods require the user to manually annotate correspondences on ...
-
Modern algorithms allow robots to reach a greater level of autonomy and fulfill more challenging tasks. However, on-board limitations regarding computational and battery resources are hindering factors regarding the deployment of such algorithms particularly on mobile robots. Although offloading a majority of the algorithmic components to the edge or even cloud offers an attractive option to lever...
-
The utilization of Large Language Models (LLMs) within the realm of reinforcement learning, particularly as planners, has garnered a significant degree of attention in recent scholarly literature. However, a substantial proportion of existing research predominantly focuses on planning models for robotics that transmute the outputs derived from perception models into linguistic forms, thus adopting...
-
Motion prediction and cost evaluation are vital components in the decision-making system of autonomous vehicles. However, existing methods often ignore the importance of cost learning and treat them as separate modules. In this study, we employ a tree-structured policy planner and propose a differentiable joint training framework for both ego-conditioned prediction and cost models, resulting in a ...
-
Autonomous vehicles require motion forecasting of their surrounding multiagents (pedestrians and vehicles) to make optimal decisions for navigation. The existing methods focus on techniques to utilize the positions and velocities of these agents and fail to capture semantic information from the scene. Moreover, to mitigate the increase in computational complexity associated with the number of agen...
-
As machine learning models become increasingly prevalent in motion forecasting for autonomous vehicles (AVs), it is critical to ensure that model predictions are safe and reliable. In this paper, we examine the robustness of motion forecasting to non-causal perturbations. We construct a new benchmark for evaluating and improving model robustness by applying perturbations to existing data. Specific...
-
Limited visibility and sensor occlusions pose pressing safety challenges for advanced driver-assistance systems (ADAS) and autonomous vehicles (AVs). In this work, our pursuit was to strike a balance: a method that ensures safety in occluded scenarios while preventing overly cautious behavior. We argue that such approaches are crucial for AVs’ future, particularly when navigating alongside human d...
-
Conventional control, such as model-based control, is commonly utilized in autonomous driving due to its efficiency and reliability. However, real-world autonomous driving contends with a multitude of diverse traffic scenarios that are challenging for these planning algorithms. Model-free Deep Reinforcement Learning (DRL) presents a promising avenue in this direction, but learning DRL control poli...
-
In this paper, we focus on the autonomous multiagent taxi routing problem for a large urban environment where the location and number of future ride requests are unknown a-priori, but can be estimated by an empirical distribution. Recent theory has shown that a rollout algorithm with a stable base policy produces a near-optimal stable policy. In the routing setting, a policy is stable if its execu...
-
The safety of autonomous vehicles (AV) has been a long-standing top concern, stemming from the absence of rare and safety-critical scenarios in the long-tail naturalistic driving distribution. To tackle this challenge, a surge of research in scenario-based autonomous driving has emerged, with a focus on generating high-risk driving scenarios and applying them to conduct safety-critical testing of ...
-
Environments for autonomous driving can vary from place to place, leading to challenges in designing a learning model for a new scene. Transfer learning can leverage knowledge from a learned domain to a new domain with limited data. In this work, we focus on end-to-end autonomous driving as the target task, consisting of both perception and control. We first utilize information bottleneck analysis...
-
We pursue the goal of developing robots that can interact zero-shot with generic unseen objects via a diverse repertoire of manipulation skills and show how passive human videos can serve as a rich source of data for learning such generalist robots. Unlike typical robot learning approaches which directly learn how a robot should act from interaction data, we adopt a factorized approach that can le...
-
Although pre-training on a large amount of data is beneficial for robot learning, current paradigms only perform large-scale pretraining for visual representations, whereas representations for other modalities are trained from scratch. In contrast to the abundance of visual data, it is unclear what relevant internet-scale data may be used for pretraining other modalities such as tactile sensing. S...
-
We present Self-Adaptive Robust Attention for Robotics Transformers (SARA-RT): a new paradigm for addressing the emerging challenge of scaling up Robotics Transformers (RT) for on-robot deployment. SARA-RT relies on the new method of fine-tuning proposed by us, called up-training. It converts pre-trained or already fine-tuned Transformer-based robotic policies of quadratic time complexity (includi...
-
Dexterous manipulation, especially of small daily objects, continues to pose complex challenges in robotics. This paper introduces the DenseTact-Mini, an optical tactile sensor with a soft, rounded, smooth gel surface and compact design equipped with a synthetic fingernail. We propose three distinct grasping strategies: tap grasping using adhesion forces such as electrostatic and van der Waals, fi...
-
In order for a bimanual robot to manipulate an object that is held by both hands, it must construct motion plans such that the transformation between its end effectors remains fixed. This amounts to complicated nonlinear equality constraints in the configuration space, which are difficult for trajectory optimizers. In addition, the set of feasible configurations becomes a measure zero set, which p...
-
Deep Evidential Uncertainty Estimation for Semantic Segmentation under Out-Of-Distribution Obstacles
In order to navigate safely and reliably in novel environments, robots must estimate perceptual uncertainty when confronted with out-of-distribution (OOD) obstacles not seen in training data. We present a method to accurately estimate pixel-wise uncertainty in semantic segmentation without requiring real or synthetic OOD examples at training time. From a shared per-pixel latent feature representat...
-
Neural implicit representations have emerged as a promising solution for providing dense geometry in Simultaneous Localization and Mapping (SLAM). However, existing methods in this direction fall short in terms of global consistency and low latency. This paper presents NGEL-SLAM to tackle the above challenges. To ensure global consistency, our system leverages a traditional feature-based tracking ...
-
3D single object tracking (SOT) is an important and challenging task for the autonomous driving and mobile robotics. Most existing methods perform tracking between two consecutive frames while ignoring the motion patterns of the target over a series of frames, which would cause performance degradation in the scenes with sparse points. To break through this limitation, we introduce "Sequence-to-Seq...
-
In this paper, we strongly advocate square-root covariance (instead of information) filtering for Visual-Inertial Navigation Systems (VINS), in particular on resource-constrained edge devices, because of its superior efficiency and numerical stability. Although VINS have made tremendous progress in recent years, they still face resource stringency and numerical instability on embedded systems when...
-
Real-world robotic tasks stretch over extended horizons and encompass multiple stages. Learning long-horizon manipulation tasks, however, is a long-standing challenge, and demands decomposing the overarching task into several manageable subtasks to facilitate policy learning and generalization to unseen tasks. Prior task decomposition methods require task-specific knowledge, are computationally in...
-
Given its wide application in robotics, point cloud registration is a widely researched topic. Conventional methods aim to find a rotation and translation that align two point clouds in 6 degrees of freedom (DoF). However, certain tasks in robotics, such as category-level pose estimation, involve non-uniformly scaled point clouds, requiring a 9DoF transform for accurate alignment. We propose HEGN,...
-
Many applications of path planning in time-varying flow fields, particularly in areas such as marine robotics and ship routing, can be modelled as instances of the time-varying shortest path (TDSP) problem. Although there are no known polynomial-time solutions to TDSP in general, our recent work has identified a tractable case where the flow is modelled as piecewise constant. Extending this method...
-
This paper presents a novel Stochastic Optimal Control (SOC) method based on Model Predictive Path Integral control (MPPI), named Stein Variational Guided MPPI (SVG-MPPI), designed to handle rapidly shifting multimodal optimal action distributions. While MPPI can find a Gaussian-approximated optimal action distribution in closed form, i.e., without iterative solution updates, it struggles with the...
-
This paper introduces a novel, lightweight method to solve the visibility problem for 2D grids. The proposed method evaluates the existence of lines-of-sight from a source point to all other grid cells in a single pass with no preprocessing and independently of the number and shape of obstacles. It has a compute and memory complexity of $\mathcal{O}(n)$, where n = nx ×ny is the size of the grid, a...
-
In this paper, we propose a real-time clothoid tree-based path planning for self-driving robots. Clothoids, curves that exhibit linear curvature profiles, play an important role in road design and path planning due to their appealing properties. Nevertheless, their real-time applications face considerable challenges, primarily stemming from the lack of a closed-form clothoid expression. To address...
-
Path planning for multiple non-holonomic robots in continuous domains constitutes a difficult robotics challenge with many applications. Despite significant recent progress on the topic, computationally efficient and high-quality solutions are lacking, especially in lifelong settings where robots must continuously take on new tasks. In this work, we make it possible to extend key ideas enabling st...
-
Energy-Aware Ergodic Search: Continuous Exploration for Multi-Agent Systems with Battery Constraints
Continuous exploration without interruption is important in scenarios such as search and rescue and precision agriculture, where consistent presence is needed to detect events over large areas. Ergodic search already derives continuous trajectories in these scenarios so that a robot spends more time in areas with high information density. However, existing literature on ergodic search does not con...
-
Series Elastic Actuator-based exoskeleton can offer precise torque control and transparency when interacting with human wearers. Accurate control of SEA-produced torques ensures the wearer’s voluntary motion and supports the implementation of multiple assistive paradigms. In this paper, a novel variable transmission series elastic actuator (VTSEA) is developed to meet torque-speed requirements in ...
-
Ensuring safe interaction between humans and robots is an important challenge in robotics. In recent years, researchers have developed many different soft robots. One possibility to reach this goal is to integrate mechanical springs into their joints. The forthcoming generation of soft robots will be adaptable for joint stiffness to accommodate various tasks. Consequently, the development of varia...
-
Variable stiffness actuator (VSA) can imitate natural muscles in their compliance capbility, which can provide flexible adaptability for robots, improving the safety of robots interacting with the environment or human. This paper presents a new compact serial variable stiffness actuator ((SVSA-III)) with linear stiffness profile based on symmetrical variable lever arm mechanism. The stiffness moto...
-
Recent trends in robotic actuation have highlighted the need for low cost, high performance, and efficient gearing. We present an experimental study comparing pinwheel and non-pinwheel designs of cycloidal gearing. The open source designs are 3D-printable, combined with off-the-shelf components, achieving a high performance-to-cost ratio. Extensive experimental data is presented, that compares two...
-
A pneumatic linear actuator is presented and evaluated. Designed to operate in demanding environments such as MRI, it is developed to be used with two motion control modes: 1) a step-by-step mode with tooth-based gripping to ensure precision, 2) a continuous mode available locally for fine positioning. The actuator can also be disengaged to enable direct handling by an operator, for example for co...
-
In robotics and biomechanics, accurately determining joint parameters and computing the corresponding forward and inverse kinematics are critical yet often challenging tasks, especially when dealing with highly individualized and partly unknown systems. This paper unveils a cutting-edge kinematic optimizer, underpinned by an autoencoder-based architecture, to address these challenges. Utilizing a ...
-
In this paper, we present an iterative method to quickly traverse multi-dimensional paths considering jerk constraints. As a first step, we analyze the traversal of each individual path dimension. We derive a range of feasible target accelerations for each intermediate waypoint of a one-dimensional path using a binary search algorithm. Computing a trajectory from waypoint to waypoint leads to the ...
-
We present an analytic solution to the 3D Dubins path problem for paths composed of an initial circular arc, a straight component, and a final circular arc. These are commonly called CSC paths. By modeling the start and goal configurations of the path as the base frame and final frame of an RRPRR manipulator, we treat this as an inverse kinematics problem. The kinematic features of the 3D Dubins p...
-
Robotic arms are highly common in various automation processes such as manufacturing lines. However, these highly capable robots are usually degraded to simple repetitive tasks such as pick-and-place. On the other hand, designing an optimal robot for one specific task consumes large resources of engineering time and costs. In this paper, we propose a novel concept for optimizing the fitness of a r...
-
Trajectory segmentation refers to dividing a trajectory into meaningful consecutive sub-trajectories. This paper focuses on trajectory segmentation for 3D rigid-body motions. Most segmentation approaches in the literature represent the body’s trajectory as a point trajectory, considering only its translation and neglecting its rotation. We propose a novel trajectory representation for rigid-body m...
-
We propose a shared semantic map architecture to construct and configure Model Predictive Controllers (MPC) dynamically, that solve navigation problems for multiple robotic agents sharing parts of the same environment. The navigation task is represented as a sequence of semantically labeled areas in the map, that must be traversed sequentially, i.e. a route. Each semantic label represents one or m...
-
This paper presents a decentralized cooperative motion planning approach for surface inspection of 3D structures which includes uncertainties like size, number, shape, position, using multi-robot systems (MRS). Given that most of existing works mainly focus on surface inspection of single and fully known 3D structures, our motivation is two-fold: first, 3D structures separately distributed in 3D e...
-
This paper proposes a decentralized trajectory planning framework for the collision avoidance problem of multiple micro aerial vehicles (MAVs) in environments with static and dynamic obstacles. The framework utilizes spatiotemporal occupancy grid maps (SOGM), which forecast the occupancy status of neighboring space in the near future, as the environment representation. Based on this representation...
-
We propose a decentralised multi-robot coordination algorithm that features a rich representation for encoding and communicating each robot’s intent. This representation for “intent messages” enables improved coordination behaviour and communication efficiency in difficult scenarios, such as those where there are unknown points of contention that require negotiation between robots. Each intent mes...
-
This paper presents a method to reduce computations for stochastic dynamic games with game-theoretic belief space planning through partially propagating beliefs. Complex interactions in scenarios such as surveillance, herding, and racing can be modeled using game-theoretic frameworks in the belief space. Stochastic dynamic games can be solved to a local Nash Equilibrium using a game-theoretic beli...
-
Multi-agent multi-target tracking has a wide range of applications, including wildlife patrolling, security surveillance or environment monitoring. Such algorithms often make restrictive assumptions: the number of targets and/or their initial locations may be assumed known, or agents may be pre-assigned to monitor disjoint partitions of the environment, reducing the burden of exploration. This als...
-
We propose an autonomous exploration algorithm designed for decentralized multi-robot teams, which takes into account map and localization uncertainties of range-sensing mobile robots. Virtual landmarks are used to quantify the combined impact of process noise and sensor noise on map uncertainty. Additionally, we employ an iterative expectation-maximization inspired algorithm to assess the potenti...
-
This paper presents a novel approach to optimal multi-robot task allocation in heterogeneous teams of robots. When robots have heterogeneous capabilities and there are diverse objectives and constraints to comply with, computing optimal plans can become especially hard. Moreover, we increase the problem complexity by: 1) considering battery-limited robots that need to schedule recharges; 2) tasks ...
-
Most real-world Multi-Robot Task Allocation (MRTA) problems require fast and efficient decision-making, which is often achieved using heuristics-aided methods such as genetic algorithms, auction-based methods, and bipartite graph matching methods. These methods often assume a form that lends better explainability compared to an end-to-end (learnt) neural network based policy for MRTA. However, der...
-
Dehazing real-world hazy images is challenging due to the complexity of natural haze, varying haze conditions, details preservation, and the risk of overexposure. Existing methods excel in synthetic hazy scenarios but struggle in the real world because they don’t use all available features. Classical dehazing techniques primarily focus on low-level dehazing enhancements, whereas deep learning-base...
-
Fiducial markers, like AprilTag and ArUco, are extensively utilized in robotics applications within industrial environments, encompassing navigation, docking, and object grasping tasks. However, in contrast to controlled laboratory conditions, markers installed in factory grounds or equipment surfaces, often face challenges like damage or contamination. These issues can lead to compromised marker ...
-
A consistent spatial-temporal coordination across multiple agents is fundamental for collaborative perception, which seeks to improve perception abilities through information exchange among agents. To achieve this spatial-temporal alignment, traditional methods depend on external devices to provide localization and clock signals. However, hardware-generated signals could be vulnerable to noise and...
-
When capturing images through the glass during rainy or snowy weather conditions, the resulting images often contain waterdrops adhered on the glass surface, and these waterdrops significantly degrade the image quality and performance of many computer vision algorithms. To tackle these limitations, we propose a method to reconstruct the clear 3D scene implicitly from multi-view images degraded by ...
-
Given the image collection of an object, we aim at building a real-time image-based pose estimation method, which requires neither its CAD model nor hours of object-specific training. Recent NeRF-based methods provide a promising solution by directly optimizing the pose from pixel loss between rendered and target images. However, during inference, they require long converging time, and suffer from...
-
Recognizing human action is one of the most critical factors in the visual perception of robots. Specifically, skeletonbased action recognition has been actively researched to enhance recognition performance at a lower cost. However, action recognition in occlusion situations, where body parts are not visible, is still challenging.We propose an occluded part-aware graph convolutional network (OP-G...
-
Depth perception is crucial for a wide range of robotic applications. Multi-frame self-supervised depth estimation methods have gained research interest due to their ability to leverage large-scale, unlabeled real-world data. However, the self-supervised methods often rely on the assumption of a static scene and their performance tends to degrade in dynamic environments. To address this issue, we ...
-
This paper presents an image-based visual servoing scheme that can control robotic manipulators in 3D space using 2D stereo images without needing to perform stereo reconstruction. We use a stereo camera in an eye-to-hand configuration for controlling the robot to reach target positions by directly mapping image space errors to joint space actuation. We achieve convergence without a-priori knowled...
-
This paper presents a novel method to control an underactuated hand by using only a monocular camera, not using any internal sensors. In food factories, robots are required to handle a wide variety of foods without damaging them. To accomplish this, the use of underactuated hands is effective because they can adapt to various food shapes. However, if internal sensors such as tactile sensors and fo...
-
Learning-based manipulation policies from image inputs often show weak task transfer capabilities. In contrast, visual servoing methods allow efficient task transfer in high-precision scenarios while requiring only a few demonstrations. In this work, we present a framework that formulates the visual servoing task as graph traversal. Our method not only extends the robustness of visual servoing, bu...
-
Existing nighttime unmanned aerial vehicle (UAV) trackers follow an "Enhance-then-Track" architecture - first using a light enhancer to brighten the nighttime video, then employing a daytime tracker to locate the object. This separate enhancement and tracking fails to build an end-to-end trainable vision system. To address this, we propose a novel architecture called Darkness Clue-Prompted Trackin...
-
Visual control enables quadrotors to adaptively navigate using real-time sensory data, bridging perception with action. Yet, challenges persist, including generalization across scenarios, maintaining reliability, and ensuring real-time responsiveness. This paper introduces a perception framework grounded in foundation models for universal object detection and tracking, moving beyond specific train...
-
Multi-object tracking (MOT) on static platforms, such as by surveillance cameras, has achieved significant progress, with various paradigms providing attractive performances. However, the effectiveness of traditional MOT methods is significantly reduced when it comes to dynamic platforms like drones. This decrease is attributed to the distinctive challenges in the MOT-on-drone scenario: (1) object...
-
Deep reinforcement learning (RL) has shown promising results in robot motion planning with first attempts in human-robot collaboration (HRC). However, a fair comparison of RL approaches in HRC under the constraint of guaranteed safety is yet to be made. We, therefore, present human-robot gym, a benchmark suite for safe RL in HRC. We provide challenging, realistic HRC tasks in a modular simulation ...
-
Deploying a safe mobile robot policy in scenarios with human pedestrians is challenging due to their unpredictable movements. Current Reinforcement Learningbased motion planners rely on a single policy to simulate pedestrian movements and could suffer from the over-fitting issue. Alternatively, framing the collision avoidance problem as a multi-agent framework, where agents generate dynamic moveme...
-
Reinforcement Learning (RL) has been widely used to create generalizable autonomous vehicles. However, they rely on fixed reward functions that struggle to balance values like safety and efficiency. How can autonomous vehicles balance different driving objectives and human values in a constantly changing environment? To bridge this gap, we propose an adaptive reward function that utilizes visual a...
-
While reinforcement learning (RL) attracts increasing research attention, maximizing the return while keeping the agent safe at the same time remains an open problem. Motivated to address this challenge, this work proposes a new Fast and Safe Policy Optimization (FSPO) algorithm, which consists of three steps: the first step involves reward improvement update, the second step projects the policy t...
-
Symmetry is a fundamental aspect of many real-world robotic tasks. However, current deep reinforcement learning (DRL) approaches can seldom harness and exploit symmetry effectively. Often, the learned behaviors fail to achieve the desired transformation invariances and suffer from motion artifacts. For instance, a quadruped may exhibit different gaits when commanded to move forward or backward, ev...
-
This work focuses on the dual-arm object rearrangement problem abstracted from a realistic industrial scenario of Cartesian robots. The goal of this problem is to transfer all the objects from sources to targets with the minimum total completion time. To achieve the goal, the core idea is to develop an effective object-to-arm task assignment strategy for minimizing the cumulative task execution ti...
-
Safe Reinforcement Learning (RL) aims to find a policy that achieves high rewards while satisfying cost constraints. When learning from scratch, safe RL agents tend to be overly conservative, which impedes exploration and restrains the overall performance. In many realistic tasks, e.g. autonomous driving, large-scale expert demonstration data are available. We argue that extracting expert policy f...
-
Motivated by the challenge of achieving rapid learning in physical environments, this paper presents the development and training of a robotic system designed to navigate and solve a labyrinth game using model-based reinforcement learning techniques. The method involves extracting low-dimensional observations from camera images, along with a cropped and rectified image patch centered on the curren...
-
Current reinforcement learning algorithms struggle in sparse and complex environments, most notably in long-horizon manipulation tasks entailing a plethora of different sequences. In this work, we propose the Intrinsically Guided Exploration from Large Language Models (IGE-LLMs) framework. By leveraging LLMs as an assistive intrinsic reward, IGE-LLMs guides the exploratory process in reinforcement...
-
Tactile information holds promise for enhancing the manipulation capabilities of multi-fingered robots. In tasks such as in-hand manipulation, where robots frequently switch between contact and non-contact states, it is important to address the partial observability of tactile sensors and to properly consider the history of observations and actions. Previous studies have shown that Recurrent Neura...
-
Multi-task learning of deformable object manipulation is a challenging problem in robot manipulation. Most previous works address this problem in a goal-conditioned way and adapt goal images to specify different tasks, which limits the multi-task learning performance and can not generalize to new tasks. Thus, we adapt language instruction to specify deformable object manipulation tasks and propose...
-
Interactive grasping from clutter, akin to human dexterity, is one of the longest-standing problems in robot learning. Challenges stem from the intricacies of visual perception, the demand for precise motor skills, and the complex interplay between the two. In this work, we present Teacher-Augmented Policy Gradient (TAPG), a novel two-stage learning framework that synergizes reinforcement learning...
-
We propose a new policy class, Composable Interaction Primitives (CIPs), specialized for learning sustained-contact manipulation skills like opening a drawer, pulling a lever, turning a wheel, or shifting gears. CIPs have two primary design goals: to minimize what must be learned by exploiting structure present in the world and the robot, and to support sequential composition by construction, so t...
-
Robotic systems that aspire to operate in uninstrumented real-world environments must perceive the world directly via onboard sensing. Vision-based learning systems aim to eliminate the need for environment instrumentation by building an implicit understanding of the world based on raw pixels, but navigating the contact-rich high-dimensional search space from solely sparse visual reward signals si...
-
Autonomous mobile robots require predictions of human motion to plan a safe trajectory that avoids them. Because human motion cannot be predicted exactly, future trajectories are typically inferred from real-world data via learning-based approximations. These approximations provide useful information on the pedestrian’s behavior, but may deviate from the data, which can lead to collisions during p...
-
This paper proposes a GPU-accelerated optimization framework for collision avoidance problems where the controlled objects and the obstacles can be modeled as the finite union of convex polyhedra. A novel collision avoidance constraint is proposed based on scale-based collision detection and the strong duality of convex optimization. Under this constraint, the high-dimensional non-convex optimizat...
-
The latest robot navigation methods for dynamic environments assume that the states of obstacles, including their geometries and trajectories, are fully observable. While it’s easy to obtain these states accurately in simulations, it’s exceedingly challenging in the real world. Therefore, a viable alternative is to directly map raw sensor observations into robot actions. However, acquiring skills ...
-
Collision Avoidance/Mitigation System (CAMS) for autonomous vehicles is a crucial technology that ensures the safety and reliability of autonomous driving systems. Conventional collision avoidance approaches struggle in complex and various scenarios by avoiding collisions based on rules for specific collision scenarios. This has led to learning-based methods using neural networks for adaptive coll...
-
It is challenging for the mobile robot to achieve autonomous and mapless navigation in the unknown environment with uneven terrain. In this study, we present a layered and systematic pipeline. At the local level, we maintain a tree structure that is dynamically extended with the navigation. This structure unifies the planning with the terrain identification. Besides, it contributes to explicitly i...
-
Learning manipulation from demonstration is a key way for humans to teach complex tasks. However, this domain mainly focuses on kinetic teaching, and does not consider imitation of interaction forces which is essential for more contact rich tasks. We propose a framework that enables robotic imitation of contact from human demonstration using a wearable finger-tip sensor. By developing a multi-moda...
-
Soft continuum robots, fabricated from elastomeric materials, offer unparalleled flexibility and adaptability, making them ideal for applications such as minimally invasive surgery and inspections in constrained environments. With the miniaturization of imaging technologies and the development of novel control algorithms, these devices provide exceptional opportunities to visualize the internal st...
-
Soft robots have shown remarkable distinct capabilities due to their high deformation. Recently increasing attention has been dedicated to developing fully soft robots to exploit their full potential, with a recognition that electronic powering may limit this achievement. Alternative powering sources compatible with soft robots have been identified such as combustion and chemical reactions. A furt...
-
Designing information-rich and space-efficient sensors is a key challenge for soft robotics, and crucial for the development of safe soft robots. Sensing and understanding the environmental interactions with a minimal footprint is especially important in the medical context, where portability and unhindered patient/user movement is a priority, to move towards personalized and decentralized healthc...
-
In our previous work, we designed a human-like white-box and causal generative model of perception NaivPhys4RP, essentially based on cognitive emulation to understand the past, the present and the future of the state of complex worlds from poor observations. In this paper, as recommended in that previous work, we first refine the theoretical model of NaivPhys4RP in terms of integration of variable...
-
Robots require a semantic understanding of their surroundings to operate in an efficient and explainable way in human environments. In the literature, there has been an extensive focus on object labeling and exhaustive scene graph generation; less effort has been focused on the task of purely identifying and mapping large semantic regions. The present work proposes a method for semantic region map...
-
3D visual grounding is a critical skill for household robots, enabling them to navigate, manipulate objects, and answer questions based on their environment. While existing approaches often rely on extensive labeled data or exhibit limitations in handling complex language queries, we propose LLM-Grounder, a novel zero-shot, open-vocabulary, Large Language Model (LLM)-based 3D visual grounding pipe...
-
Huge progress has been made in LiDAR 3D semantic segmentation, but there are two under-explored imbalances on the radial axis: points are unevenly concentrated on the near side, and the distribution of foreground object instances is skewed to the near side. This leads the training of the model to favor semantics at the near side with the majority of points and object instances. Both the cylindrica...
-
Current state-of-the-art methods for panoptic segmentation require an immense amount of annotated training data that is both arduous and expensive to obtain posing a significant challenge for their widespread adoption. Concurrently, recent breakthroughs in visual representation learning have sparked a paradigm shift leading to the advent of large foundation models that can be trained with complete...
-
In the fields of robotic perception and computer vision, achieving accurate semantic segmentation of low-light or nighttime scenes is challenging. This is primarily due to the limited visibility of objects and the reduced texture and color contrasts among them. To address the issue of limited visibility, we propose a hierarchical gated convolution unit, which simultaneously expands the receptive f...
-
The complementarity between camera and LiDAR data makes fusion methods a promising approach to improve 3D semantic segmentation performance. Recent transformer-based methods have also demonstrated superiority in segmentation. However, multimodal solutions incorporating transformers are underexplored and face two key inherent difficulties: over-attention and noise from different modal data. To over...
-
Inventory monitoring in homes, factories, and retail stores relies on maintaining data despite objects being swapped, added, removed, or moved. We introduce Lifelong LERF, a method that allows a mobile robot with minimal compute to jointly optimize a dense language and geometric representation of its surroundings. Lifelong LERF maintains this representation over time by detecting semantic changes ...
-
Robotic manipulation requires accurate perception of the environment, which poses a significant challenge due to its inherent complexity and constantly changing nature. In this context, RGB image and point-cloud observations are two commonly used modalities in visual-based robotic manipulation, but each of these modalities have their own limitations. Commercial point-cloud observations often suffe...
-
We introduce MORPH, a method for co-optimization of hardware design parameters and control policies in simulation using reinforcement learning. Like most co-optimization methods, MORPH relies on a model of the hardware being optimized, usually simulated based on the laws of physics. However, such a model is often difficult to integrate into an effective optimization routine. To address this, we in...
-
Reinforcement learning solely from an agent’s self-generated data is often believed to be infeasible for learning on real robots, due to the amount of data needed. However, if done right, agents learning from real data can be surprisingly efficient through re-using previously collected sub-optimal data. In this paper we demonstrate how the increased understanding of off-policy learning methods and...
-
Robotic affordances, providing information about what actions can be taken in a given situation, can aid robotic manipulation. However, learning about affordances requires expensive large annotated datasets of interactions or demonstrations. In this work, we argue that well-directed interactions with the environment can mitigate this problem and propose an information-based measure to augment the ...
-
We present a novel vision-based, 6-DoF grasping framework based on Deep Reinforcement Learning (DRL) that is capable of directly synthesizing continuous 6-DoF actions in cartesian space. Our proposed approach uses visual observations from an eye-in-hand RGB-D camera, and we mitigate the sim-to-real gap with a combination of domain randomization, image augmentation, and segmentation tools. Our meth...
-
Most existing 6-DoF robot grasping solutions depend on strong supervision on grasp pose to ensure satisfactory performance, which could be laborious and impractical when the robot works in some restricted area. To this end, we propose a self-supervised 6-DoF grasp pose detection framework via an Augmented Reality (AR) teleoperation system that can efficiently learn human demonstrations and provide...
-
Collaboration between humans and robots is becoming increasingly crucial in our daily life. In order to accomplish efficient cooperation, trust recognition is vital, empowering robots to predict human behaviors and make trust-aware decisions. Consequently, there is an urgent need for a generalized approach to recognize human-robot trust. This study addresses this need by introducing an EEG-based m...
-
In Human-Robot Interaction (HRI) scenarios, human factors like trust can greatly impact task performance and interaction quality. Recent research has confirmed that perceived robot proficiency is a major antecedent of trust. By making robots aware of their capabilities, we can allow them to choose when to perform low-confidence actions, thus actively controlling the risk of trust reduction. In thi...
-
This paper proposes an interactive navigation framework by using large language and vision-language models, allowing robots to navigate in environments with traversable obstacles. We utilize the large language model (GPT-3.5) and the open-set Vision-language Model (Grounding DINO) to create an action-aware costmap to perform effective path planning without fine-tuning. With the large models, we ca...
-
In the past decade, there has been significant advancement in designing wearable neural interfaces for controlling neurorobotic systems, particularly bionic limbs. These interfaces function by decoding signals captured noninvasively from the skin’s surface. Portable high-density surface electromyography (HD-sEMG) modules combined with deep learning decoding have attracted interest by achieving exc...
-
In shared autonomy, human-robot handover for object delivery is crucial. Accurate robot predictions of human hand motion and intentions enhance collaboration efficiency. However, low prediction accuracy increases mental and physical demands on the user. In this work, we propose a system for predicting hand motion and intended target during human-robot handover using Inverse Reinforcement Learning ...
-
Theory of Mind (ToM) is a fundamental cognitive architecture that endows humans with the ability to attribute mental states to others. Humans infer the desires, beliefs, and intentions of others by observing their behavior and, in turn, adjust their actions to facilitate better interpersonal communication and team collaboration. In this paper, we investigated trust-aware robot policy with the theo...
-
3D reconstruction based on monocular videos has attracted wide attention, and existing reconstruction methods usually work in a reconstruction-after-scanning manner. However, these methods suffer from insufficient data collection problems due to the lack of effective guidance for users during the scanning process, which affects reconstruction quality. We propose VIDAR, which visually guides users ...
-
Rehabilitative robotics, particularly lower-limb exoskeletons (LLEs), have gained increasing importance in aiding patients regain ambulatory functions. One of the challenges in making these systems effective is the implementation of an assist-as-needed (AAN) control strategy that intervenes only when the patient deviates from the correct movement pattern. Equally crucial is the need for the LLE to...
-
This research contributes to the field of Human-Robot Collaboration (HRC) within dynamic and unstructured environments by extending the previously proposed Fuzzy State-Long Short-Term Memory (FS-LSTM) architecture to handle the uncertainty and irregularity inherent in real-world sensor data. Recognising the challenges posed by low-cost sensors, which are highly susceptible to environmental conditi...
-
In the context of rapid advancements in industrial automation, vision-based robotic grasping plays an increasingly crucial role. In order to enhance visual recognition accuracy, the utilization of large-scale datasets is imperative for training models to acquire implicit knowledge related to the handling of various objects. Creating datasets from scratch is a time and labor-intensive process. More...
-
In a Human-in-the-Loop paradigm, a robotic agent is able to act mostly autonomously in solving a task, but can request help from an external expert when needed. However, knowing when to request such assistance is critical: too few requests can lead to the robot making mistakes, but too many requests can overload the expert. In this paper, we present a Reinforcement Learning based approach to this ...
-
Assessing proficiency in small unmanned aerial vehicles (sUAVs) pilots is complex and not well understood, but increasingly important to employ these vehicles in serious jobs such as wildland firefighting and infrastructure inspection. The limited prior work with UAVs has focused on user training using modalities like simulators and VR and no performance assessments with line-of-sight UAVs. This p...
-
Accurate perception of the surrounding environment stands as a primary objective for robots. Through tactile interaction, vision-based tactile sensors provide the capability to capture high-resolution and multi-modal surface information of objects, thereby facilitating robots in achieving more dexterous manipulations. However, the prevailing GelSight sensors entail intricate calibration procedures...
-
We introduce a novel approach that combines tactile estimation and control for in-hand object manipulation. By integrating measurements from robot kinematics and an image-based tactile sensor, our framework estimates and tracks object pose while simultaneously generating motion plans in a receding horizon fashion to control the pose of a grasped object. This approach consists of a discrete pose es...
-
This work presents a novel shape evaluation and optimization approach for shape sensing, specifically targeting the constrained, irregular, and intricate spatial shapes of flexible bronchoscopes (FB) in human bronchial tree. The proposed evaluation criteria and optimization methods combine clinical significance related to bronchial anatomical structures and address issues related to singular point...
-
It is difficult for robots to retrieve objects in densely cluttered lateral access scenes with movable objects as jamming against adjacent objects and walls can inhibit progress. We propose the use of two action primitives— burrowing and excavating—that can fluidize the scene to unjam obstacles and enable continued progress. Even when these primitives are implemented in an open loop manner at cloc...
-
Iontronic tactile sensors are promising to measure spatial-temporal contact information with high performance. However, no suitable measuring method has been presented, due to issues with crosstalk and non-negligible equivalent resistance. Hence, this study presents an impedance-separating method, which does not require complex analog components. A general Quadri-Terminal Impedance Network (QTIN) ...
-
The rapidly evolving field of robotics necessitates methods that can facilitate the fusion of multiple modalities. Specifically, when it comes to interacting with tangible objects, effectively combining visual and tactile sensory data is key to understanding and navigating the complex dynamics of the physical world, enabling a more nuanced and adaptable response to changing environments. Neverthel...
-
Using tactile signal is a natural way to perceive potential dangers and safeguard robots. One possible method is to use full-body tactile sensors on the robot and perform safety maneuvers when dangerous stimuli are detected. In this work, we proposed a method based on full-body tactile sensors that operates at three different levels of granularity to ensure that robot interacts with the environmen...
-
Roller skating with passive wheels on a quadrupedal robot is more efficient than traditional walking. However, the typical mammalian quadruped robot with 3-DoFs legs can only perform one dynamic roller skating gait and has difficulty achieving turning motion. To address this limitation, we designed a novel quadrupedal robot with each leg having 4-DoFs to enable various roller skating locomotion in...
-
Humanoid robots possess the ability to perform complex tasks in challenging environments. However, they require a model of the surroundings in a representation that is sufficient enough for downstream tasks such as footstep planning. The maps generated by existing mapping algorithms are either sparse, insufficient for footstep planning, memory intensive, or too slow for dynamic humanoid behaviors....
-
In order to perform highly dynamic and agile maneuvers, legged robots typically spend time in underactuated domains (e.g. with feet off the ground) where the system has limited command of its acceleration and a constrained amount of time before transitioning to a new domain (e.g. foot touchdown). Meanwhile, these transitions can instantaneously change the system’s state, possibly causing perturbat...
-
Robot skateboarding is a novel and challenging task for legged robots. Accurately modeling the dynamics of dual floating bases and developing effective planning and control methods present significant complexities in accomplishing skateboarding behavior. This paper focuses on enabling the quadrupedal platform CyberDog2 to achieve dynamic balancing and acceleration on a skateboard. An optimization-...
-
This study focuses on a layered, experience-based, multi-modal contact planning framework for agile quadrupedal locomotion over a constrained rebar environment. To this end, our hierarchical planner incorporates locomotion-specific modules into the high-level contact sequence planner and performs kinodynamically-aware trajectory optimization as the low-level motion planner. Through quantitative an...
-
In recent years, legged and wheeled-legged robots have gained prominence for tasks in environments predominantly created for humans across various domains. One significant challenge faced by many of these robots is their limited capability to navigate stairs, which hampers their functionality in multi-story environments. This study proposes a method aimed at addressing this limitation, employing r...
-
In the natural world, benefited from the advantages of the spine, quadrupeds exhibiting extraordinary flexibility which allowing them to move efficiently on variable terrains. The previous researches have indicated the legged robots which efficiently utilizing their spine can achieve rapid and stable locomotion. However, within the field of legged robot dynamics, the design of the spine and unders...
-
The system observability analysis is of practical importance, for example, due to its ability to identify the unobservable directions of the estimated state which can influence estimation accuracy and help develop consistent and robust estimators. Recent studies focused on analyzing the observability of the state of various multisensor systems with a particular interest in unobservable directions ...
-
Concentric tube continuum robots utilize nested tubes, which are subject to a set of inequalities. Current approaches to account for inequalities rely on branching methods such as if-else statements. It can introduce discontinuities, may result in a complicated decision tree, has a high wall-clock time, and cannot be vectorized. This affects the behavior and result of downstream methods in control...
-
Millimeter-scale magnetic rotating swimmers have multiple potential medical applications. They could, for example, navigate inside the bloodstream of a patient toward an occlusion and remove it. Magnetic rotating swimmers have internal magnets and propeller fins with a helical shape. A rotating magnetic field applies torque on the swimmer and makes it rotate. The shape of the swimmer, combined wit...
-
In recent years, the area of Robot-Assisted Minimally Invasive Surgery (RAMIS) is standing on the the verge of a new wave of innovations. However, autonomy in RAMIS is still in a primitive stage. Therefore, most surgeries still require manual control of the endoscope and the robotic instruments, resulting in surgeons needing to switch attention between performing surgical procedures and moving end...
-
This work proposes a state-of-the-art transformer architecture to detect and segment catheters in axial interventional Ultrasound image sequences. The network architecture was inspired by the Attention in Attention mechanism, temporal tracking networks, and introduced a novel 3D segmentation head that performs 3D deconvolution across time. To train the network, we introduce a new data synthesis pi...
-
This paper presents a 6-DOF hybrid robot for percutaneous needle intervention procedures. The new robot combines the advantages of both serial robots and parallel robots, featuring compactness, high accuracy, and small footprint while overcoming the problems of the high cost of serial robots and the small workspace and singularity issue of parallel robots. Besides, by analyzing the workspace of th...
-
Several robotic systems have emerged in the recent past to enhance the precision of micro-surgeries such as retinal procedures. Significant advancements have recently been achieved to increase the precision of such systems beyond surgeon capabilities. However, little attention has been paid to the impact of non-predicted and sudden movements of the patient and the environment. Therefore, analyzing...
-
A surgeon’s physiological hand tremor can significantly impact the outcome of delicate and precise retinal surgery, such as retinal vein cannulation (RVC) and epiretinal membrane peeling. Robot-assisted eye surgery technology provides ophthalmologists with advanced capabilities such as hand tremor cancellation, hand motion scaling, and safety constraints that enable them to perform these otherwise...
-
Motion scaling is essential to empower users to conduct precise manipulation during teleoperation for robot-assisted microsurgery (RAMS). A constant, small motion scaling ratio can enhance the precision of teleoperation but hinder the operator from quickly reaching distant targets. The concept of self-adaptive motion scaling has been proposed in previous work. However, previous frameworks required...
-
Capsule endoscopes, predominantly serving diagnostic functions, provide lucid internal imagery but are devoid of surgical or therapeutic capabilities. Consequently, despite lesion detection, physicians frequently resort to traditional endoscopic or open surgical procedures for treatment, resulting in more complex, potentially risky interventions. To surmount these limitations, this study introduce...
-
We propose a new metric for robot state estimation based on the recently introduced SE2(3) Lie group definition. Our metric is related to prior metrics for SLAM but explicitly takes into account the linear velocity of the state estimate, improving over current pose-based trajectory analysis. This has the benefit of providing a single, quantitative metric to evaluate state estimation algorithms aga...
-
Offline reinforcement learning has emerged as a promising technology by enhancing its practicality through the use of pre-collected large datasets. Despite its practical benefits, most algorithm development research in offline reinforcement learning still relies on game tasks with synthetic datasets. To address such limitations, this paper provides autonomous driving datasets and benchmarks for of...
-
We present a new reproducible benchmark for evaluating robot manipulation in the real world, specifically focusing on a pick-and-place task. Our benchmark uses the YCB object set, a commonly used dataset in the robotics community, to ensure that our results are comparable to other studies. Additionally, the benchmark is designed to be easily reproducible in the real world, making it accessible to ...
-
Benchmarking is a common method for evaluating trajectory prediction models for autonomous driving. Existing benchmarks rely on datasets, which are biased towards more common scenarios, such as cruising, and distance-based metrics that are computed by averaging over all scenarios. Following such a regiment provides a little insight into the properties of the models both in terms of how well they c...
-
The 3D Average Precision (3DAP) relies on the intersection over union between predictions and ground truth objects. However, camera-only detectors have limited depth accuracy, which may cause otherwise reasonable predictions that suffer from such longitudinal localization errors to be treated as false positives. We therefore propose variants of the 3DAP metric to be more permissive with respect to...
-
We introduce RobotPerf, a vendor-agnostic bench-marking suite designed to evaluate robotics computing performance across a diverse range of hardware platforms using ROS 2 as its common baseline. The suite encompasses ROS 2 packages covering the full robotics pipeline and integrates two distinct benchmarking approaches: black-box testing, which measures performance by eliminating upper layers and r...
-
The field of robotics faces inherent challenges in manipulating deformable objects, particularly in understanding and standardising fabric properties like elasticity, stiffness, and friction. While the significance of these properties is evident in the realm of cloth manipulation, accurately categorising and comprehending them in real-world applications remains elusive. This study sets out to addr...
-
This paper reports a new hierarchical architecture for modeling autonomous multi-robot systems (MRSs): a nonlinear dynamical opinion process is used to model high-level group choice, and multi-objective behavior optimization is used to model individual decisions. Using previously reported theoretical results, we show it is possible to design the behavior of the MRS by the selection of a relatively...
-
Persistent operation of Autonomous Underwater Vehicles (AUVs) without manual interruption for recharging saves time and total cost for offshore monitoring and data collection applications. In order to facilitate AUVs for long mission durations without ship support, they can be equipped with docking capabilities to recharge in situ at Wave Energy Converter (WEC) with dock recharging stations. Howev...
-
Collision avoidance algorithms for Autonomous Surface Vehicles (ASV) that follow the Convention on the International Regulations for Preventing Collisions at Sea (COLREGs) have been proposed in recent years. However, it may be difficult and unsafe to follow COLREGs in congested waters, where multiple ASVs are navigating in the presence of static obstacles and strong currents, due to the complex in...
-
Reliable localization is an essential capability for marine robots navigating in GPS-denied environments. SLAM, commonly used to mitigate dead reckoning errors, still fails in feature-sparse environments or with limited-range sensors. Pose estimation can be improved by incorporating the uncertainty prediction of future poses into the planning process and choosing actions that reduce uncertainty. H...
-
Autonomous Marine Vehicles (AMVs) have been widely used in many critical tasks such as surveillance, patrolling, marine environment monitoring, and hydrographic surveying. However, most typical AMVs cannot meet the diverse demands of different marine tasks. In this article, we design a new type of remote-controlled hydrofoil marine vehicle, named Sea-U-Foil, which is suitable for different marine ...
-
Autonomous underwater vehicles often perform surveys that capture multiple views of targets in order to provide more information for human operators or automatic target recognition algorithms. In this work, we address the problem of choosing the most informative views that minimize survey time while maximizing classifier accuracy. We introduce a novel active perception framework for multi-view ada...
-
We present model predictive selection (MPS), a new method for selecting the stable closed-loop (CL) equilibrium attitude-error quaternion (AEQ) of an uncrewed aerial vehicle (UAV) during the execution of high-speed yaw maneuvers. In this approach, we minimize the cost of yawing measured with a performance figure of merit (PFM) that takes into account both the aerodynamic-torque control input and a...
-
Pushing tasks performed by aerial manipulators can be used for contact-based industrial inspections. Underactuated aerial vehicles are widely employed in aerial manipulation due to their widespread availability and relatively low cost. Industrial infrastructures often consist of diverse oriented work surfaces. When interacting with such surfaces, the coupled gravity compensation and interaction fo...
-
Aerial manipulation has received increasing research interest with wide applications of drones. To perform specific tasks, robotic arms with various mechanical structures will be mounted on the drone. It results in sudden disturbances to the aerial manipulator when switching the robotic arm or interacting with the environment. Hence, it is challenging to design a generic and robust control strateg...
-
Swarms of aerial drones have recently been considered for last-mile deliveries in urban logistics or automated construction. At the same time, collaborative transportation of payloads by multiple drones is another important area of recent research. However, efficient coordination algorithms for collaborative transportation of many payloads by many drones remain to be considered. In this work, we f...
-
The standard quadrotor is one of the most popular and widely used aerial vehicle of recent decades, offering great maneuverability with mechanical simplicity. However, the under-actuation characteristic limits its applications, especially when it comes to generating desired wrench with six degrees of freedom (DOF). Therefore, existing work often compromises between mechanical complexity and the co...
-
In this work, we present a novel actuation strategy for a suspended aerial platform. By utilizing an underactuation approach, we demonstrate the successful oscillation damping of the proposed platform, modeled as a spherical double pendulum. A state estimator is designed in order to obtain the deflection angles of the platform, which uses only onboard IMU measurements. The state estimator is an ex...
-
The problem of autonomous navigation for UAV inspection remains challenging as it requires effectively navigating in close proximity to obstacles, while accounting for dynamic risk factors such as weather conditions, communication reliability, and battery autonomy. This paper introduces the MOAR path planner which addresses the complexities of evolving risks during missions. It offers real-time tr...
-
Solar power is becoming an increasingly popular option for energy production in commercial and private applications. While installing solar panels (photovoltaic cells) in a stationary configuration is simple and inexpensive, such a setup fails to maximise their potential solar energy production. Single- and dual-axis sun trackers automatically adjust the tilt angle of photovoltaic cells so as to d...
-
Automatic inspection of parabolic-trough solar plants is key to preventing failures that can harm the environment and the production of green energy. In this work, we propose a novel methodology to inspect ball joints in parabolic trough collectors, which is a relevant problem that is not adequately covered in the literature. Images collected by an Unmanned Aerial Vehicle are segmented using deep ...
-
Environmental monitoring via UAVs offers unprecedented aerial observation capabilities. However, the limited flight durations of typical multirotors and the demands on human attention in outdoor missions call for more autonomous solutions. Addressing the specific challenges of precise UAV landings – especially amidst wind disturbances, obstacles, and unreliable global localization – we introduce a...
-
Robotic gas distribution mapping improves the understanding of a hazardous gas dispersion while putting the human operator out of danger. Generating an accurate gas distribution map quickly is of utmost importance in situations such as gas leaks and industrial incidents, so that the efficient use of resources in response to incidents can be facilitated. In this paper, to incorporate the operationa...
-
This paper proposes a novel decision-making framework for planning "when" and "where" to deploy robots based on prior data with the goal of persistently monitoring a spatio-temporal phenomenon in an environment. We specifically focus on large lake monitoring, where remote sensors, such as satellites, can provide a snapshot of the target phenomenon at regular cycles. Between these cycles, Autonomou...
-
The creation of precise and high-resolution crop point clouds in agricultural fields has become a key challenge for high-throughput phenotyping applications. This work implements a novel calibration method to calibrate the laser scanning system of an agricultural field robot consisting of two industrial-grade laser scanners used for high-precise 3D crop point cloud creation. The calibration method...
-
Accurate and robust navigation in unstructured environments requires fusing data from multiple sensors. Such fusion ensures that the robot is better aware of its surroundings, including areas of the environment that are not immediately visible but were visible at a different time. To solve this problem, we propose a method for traversability prediction in challenging outdoor environments using a s...
-
Place recognition is a challenging but crucial task in robotics. Current description-based methods may be limited by representation capabilities, while pairwise similarity-based methods require exhaustive searches, which is time-consuming. In this paper, we present a novel coarse-to-fine approach to address these problems, which combines BEV (Bird’s Eye View) feature extraction, coarse-grained mat...
-
Localization using a monocular camera in the pre-built LiDAR point cloud map has drawn increasing attention in the field of autonomous driving and mobile robotics. However, there are still many challenges (e.g. difficulties of map storage, poor localization robustness in large scenes) in accurately and efficiently implementing cross-modal localization. To solve these problems, a novel pipeline ter...
-
Localizing ground penetrating radar (LGPR) has been proven to be a promising technology for robot localization in various dynamic environments. However, the extreme scarcity of underground features introduces false candidate matches and brings unique challenges to this task. In this paper, we propose a sequence-based framework for LGPR to address the aforementioned issues. Specifically, we first i...
-
Vision-based localization for autonomous driving has been of great interest among researchers. When a pre-built 3D map is not available, the techniques of visual simultaneous localization and mapping (SLAM) are typically adopted. Due to error accumulation, visual SLAM (vSLAM) usually suffers from long-term drift. This paper proposes a framework to increase the localization accuracy by fusing the v...
-
Camera relocalization is pivotal in computer vision, with applications in AR, drones, robotics, and autonomous driving. It estimates 3D camera position and orientation (6-DoF) from images. Unlike traditional methods like SLAM, recent strides use deep learning for direct end-to-end pose estimation. We propose EffLoc, a novel efficient Vision Transformer for single-image camera relocalization. EffLo...
-
Robust and accurate pose estimation in unknown environments is an essential part of robotic applications. We focus on LiDAR-based point-to-point ICP combined with effective semantic information. This paper proposes a novel semantic information-assisted ICP method named SAGE-ICP, which leverages semantics in odometry. The semantic information for the whole scan is timely and efficiently extracted b...
-
Absolute Pose Regressors (APRs) directly estimate camera poses from monocular images, but their accuracy is unstable for different queries. Uncertainty-aware APRs provide uncertainty information on the estimated pose, alleviating the impact of these unreliable predictions. However, existing uncertainty modelling techniques are often coupled with a specific APR architecture, resulting in suboptimal...
-
Sparse keypoint matching based on distinct 3D feature representations can improve the efficiency and robustness of point cloud registration. Existing learning-based 3D descriptors and keypoint detectors are either independent or loosely coupled, so they cannot fully adapt to each other. In this work, we propose a tightly coupled keypoint detector and descriptor (TCKDD) based on a multi-task fully ...
-
Significant advances in robotics and machine learning have resulted in many datasets designed to support research into autonomous vehicle technology. However, these datasets are rarely suitable for a wide variety of navigation tasks. For example, datasets that include multiple cameras often have short trajectories without loops that are unsuitable for the evaluation of longer-range SLAM or odometr...
-
This paper introduces a novel sonar odometry system that estimates the relative spatial transformation between two sonar image frames. Considering the unique challenges, such as low resolution and high noise, of sonar imagery for odometry and Simultaneous Localization and Mapping (SLAM), the proposed Direct Imaging Sonar Odometry (DISO) system is designed to estimate the relative transformation be...
-
Maps of LiDAR Simultaneous Localisation and Mapping (SLAM) are often represented as point clouds. They usually take up a huge amount of storage space for large-scale environments, otherwise much structural detail may not be kept. In this paper, a novel paradigm of LiDAR mapping and odometry is designed by leveraging the Continuous and Ultra-compact Representation of LiDAR (CURL) proposed in [1]. T...
-
Enabling autonomous robots to operate robustly in challenging environments is necessary in a future with increased autonomy. For many autonomous systems, estimation and odometry remains a single point of failure, from which it can often be difficult, if not impossible, to recover. As such robust odometry solutions are of key importance. In this work a method for tightly-coupled LiDAR-Radar-Inertia...
-
We introduce Ground-Fusion, a low-cost sensor fusion simultaneous localization and mapping (SLAM) system for ground vehicles. Our system features efficient initialization, effective sensor anomaly detection and handling, real-time dense color mapping, and robust localization in diverse environments. We tightly integrate RGB-D images, inertial measurements, wheel odometer and GNSS signals within a ...
-
Simultaneous Localization and Mapping (SLAM) is a fundamental task in robotics, driving numerous applications such as autonomous driving and virtual reality. Recent progress on neural implicit SLAM has shown encouraging and impressive results. However, the robustness of neural SLAM, particularly in challenging or data-limited situations, remains an unresolved issue. This paper presents HERO-SLAM, ...
-
To achieve longer driving distances, planetary robotics missions require accurate localization to counteract position uncertainty. Freedom and precision in driving allows scientists to reach and study sites of interest. Typically, rover global localization has been performed manually by humans, which is accurate but time-consuming as data is relayed between planets. This paper describes a global l...
-
Legged robots have the potential to traverse complex terrain and access confined spaces beyond the reach of traditional platforms thanks to their ability to carefully select footholds and flexibly adapt their body posture while walking. However, robust deployment in real-world applications is still an open challenge. In this paper, we present a method for legged locomotion control using reinforcem...
-
Robots that operate in agricultural environments need a robust perception system that can deal with occlusions, which are naturally present in agricultural scenarios. In this paper, we address the problem of estimating 3D shapes of fruits when only partial observations are available. Generally speaking, such a shape completion can be realized by exploiting prior knowledge about the geometry of the...
-
In the pursuit of fully autonomous robotic systems capable of taking over tasks traditionally performed by humans, the complexity of open-world environments poses a considerable challenge. Addressing this imperative, this study contributes to the field of Large Language Models (LLMs) applied to task and motion planning for robots. We propose a system architecture that orchestrates a seamless inter...
-
Human-like planning skills and dexterous manipulation have long posed challenges in the fields of robotics and artificial intelligence (AI). The task of reinterpreting calligraphy presents a formidable challenge, as it involves the decomposition of strokes and dexterous utensil control. Previous efforts have primarily focused on supervised learning of a single instrument, limiting the performance ...
-
The design and control of winged aircraft and drones is an iterative process aimed at identifying a compromise of mission-specific costs and constraints. When agility is required, shape-shifting (morphing) drones represent an efficient solution. However, morphing drones require the addition of actuated joints that increase the topology and control coupling, making the design process more complex. ...
-
3D coverage path planning for UAVs is a crucial problem in diverse practical applications. However, existing methods have shown unsatisfactory system simplicity, computation efficiency, and path quality in large and complex scenes. To address these challenges, we propose FC-Planner, a skeleton-guided planning framework that can achieve fast aerial coverage of complex 3D scenes without pre-processi...
-
In drone racing, the time-minimum trajectory is affected by the drone’s capabilities, the layout of the race track, and the configurations of the gates (e.g., their shapes and sizes). However, previous studies neglect the configuration of the gates, simply rendering drone racing a waypoint-passing task. This formulation often leads to a conservative choice of paths through the gates, as the spatia...
-
In this paper, we present a novel trajectory planning method for externally-actuated modular manipulators (EAMMs), consisting of multiple rotor-actuated links with joints that can be either locked or unlocked. This joint-locking feature allows effective balancing of the payload capacity and dexterity of the robot but significantly complicates the planning problem by introducing binary decision var...
-
Aerial robots have the potential to play a crucial role in assisting humans with complex and dangerous tasks. Nevertheless, the future industry demands innovative solutions to streamline the interaction process between humans and drones to enable seamless collaboration and efficient coworking. In this paper, we present a novel tele-immersive framework that promotes cognitive and physical collabora...
-
Drone racing has become a popular international competition and has attained wide attention in recent years. However, the requirements of high-level operation keep the novice pilots away from participating in it. This paper presents a trajectory-based flight assistive system that enables various operators to fly the drone in a racing scene at a high speed. The whole system is structured hierarchic...
-
Sampling-based planning algorithms such as RRT have been proved to be efficient in solving path planning problems for robotic systems. Various improvements to the RRT algorithm have been presented to improve the performance of the extension and convergence of the random trees, such as Informed RRT*. However, with the growth of spatial dimensions, the time consumption of randomly sampling the entir...
-
Continuum robots (CR) offer excellent dexterity and compliance in contrast to rigid-link robots, making them suitable for navigating through, and interacting with, confined environments. However, the study of path planning for CRs while considering external elastic contact is limited. The challenge lies in the fact that CRs can have multiple possible configurations when in contact, rendering the f...
-
We present progress on the problem of reconfiguring a 2D arrangement of building material by a cooperative group of robots. These robots must avoid collisions, deadlocks, and are subjected to the constraint of maintaining connectivity of the structure. We develop two reconfiguration methods, one based on spatio-temporal planning, and one based on target swapping, to increase building efficiency. T...
-
Sampling-based planning algorithms like Rapidly-exploring Random Tree (RRT) are versatile in solving path planning problems. RRT* offers asymptotic optimality but requires growing the tree uniformly over the free space, which leaves room for efficiency improvement. To accelerate convergence, rule-based informed approaches sample states in an admissible ellipsoidal subset of the space determined by...
-
Modern sampling-based motion planning algorithms typically take between hundreds of milliseconds to dozens of seconds to find collision-free motions for high degree-of-freedom problems. This paper presents performance improvements of more than 500x over the state-of-the-art, bringing planning times into the range of microseconds and solution rates into the range of kilohertz, without specialized h...
-
This paper proposes a formulation for the risk-aware path planning problem which utilizes multi-objective optimization to dynamically plan trajectories that satisfy multiple complex mission specifications. In the setting of persistent monitoring, we develop a method for representing environmental information and risk in a way that allows for local sampling to generate Pareto-dominant solutions ove...
-
Reactive trajectory optimization for robotics presents formidable challenges, demanding the rapid generation of purposeful robot motion in complex and swiftly changing dynamic environments. While much existing research predominantly addresses robotic motion planning with predefined objectives, emerging problems in robotic trajectory optimization frequently involve dynamically evolving objectives a...
-
In this paper, a variant of hybrid A* is developed to find the shortest path for a curvature-constrained robot, that is tethered at its start position, such that the tether satisfies user-defined winding angle constraints. A variant of tangent graphs is used as an underlying graph for searching a path using A* in order to reduce the overall computation and define appropriate cost metrics to ensure...
-
This paper introduces a differentiable representation for the optimization of boustrophedon path plans in convex polygons, explores an additional parameter of these path plans that can be optimized, discusses the properties of this representation that can be leveraged during the optimization process and shows that the previously published attempt at optimization of these path plans was too coarse ...
-
In response to the need for sustainable energy solutions, solar panels have gained significant traction. One way to increase the energy capture of solar systems is through solar tracking, a means of reorienting solar panels throughout the day in order to face the sun. The energy consumption increase that comes with solar tracking often far outweighs the amount of energy required to move the panel,...
-
Inspired by the necessity of morphological adaptation in animals, a growing body of work has attempted to expand robot training to encompass physical aspects of a robot’s design. However, reinforcement learning methods capable of optimizing the 3D morphology of a robot have been restricted to reorienting or resizing the limbs of a predetermined and static topological genus. Here we show policy gra...
-
There are many instances of helical mechanisms that are used to efficiently grasp different objects with various shapes and sizes in nature. Inspired by the helical grasping in the nature, we proposed a helical bistable soft gripper with high load capacity and energy saving. An off-the-shelf bistable steel shell (BSS) as the stiff element was inserted into a 3D printing soft helical exoskeleton to...
-
Unlike parallel robots, for which hundreds of different architectures have been proposed, the vast majority of six-degree-of-freedom (DOF) serial robots have one of two simple architectures. In both architectures, the inverse kinematics can be solved in closed form and the singularities described by trivial geometric and algebraic conditions. These conditions can be readily obtained by analyzing t...
-
This paper presents a free-swimming, tetherless, cable-driven modular soft robotic fish. The body comprises a series of 3D-printed wave spring structures that create a flexible biologically inspired shape that is capable of an anguilliform swimming gait. A three-module soft robotic fish was designed, fabricated, and evaluated. The motion of the robot was characterized and different combinations of...
-
This paper introduces DRAGON: Deformable Robot for Agile Guided Observation and Navigation, a free-swimming deformable impeller-powered vectored underwater vehicle (VUV). A 3D-printed wave spring structure directs the water drawn through the center of the robot by an impeller, enabling it to move smoothly in different directions. The robot is designed to have a narrow cylindrical profile to lower ...
-
A kinematically redundant (6+3)-degree-of-freedom (DOF) hybrid parallel robot with an axisymmetric workspace is proposed. By arranging the first revolute joint of each leg such that they have the same rotation axis, this robot can achieve an axisymmetric workspace, resulting in a large reachable workspace. In addition, type II singularities, which critically limit the orientational workspace, can ...
-
Automated storage and retrieval systems (ASRS) are a key component of the modern storage industry, and are used in a wide range of applications, carrying anything from lightweight tape cartridges to entire pallets of goods. Many of these systems are under pressure to maximise the use of space by growing in height and density, but this can create challenges for the the robots that service them. In ...
-
Cable elasticity can significantly impact the accuracy of Cable-Driven Parallel Robots (CDPRs). However, it’s frequently disregarded as negligible in CDPR simulations and designs. In this paper, we propose a numerical approach, referred to as SEECR, which is designed to estimate the behavior of a CDPR featuring elastic cables while ensuring the Static Equilibrium (SE) of the Moving-Platform (MP). ...
-
This paper presents the design, analysis, and performance evaluation of RicMonk, a novel three-link brachiation robot equipped with passive hook-shaped grippers. Brachiation, an agile and energy-efficient mode of locomotion observed in primates, has inspired the development of RicMonk to explore versatile locomotion and maneuvers on ladder-like structures. The robot’s anatomical resemblance to gib...
-
External and internal convertible (EIC) form-based motion control (i.e., EIC-based control) is one of the effective approaches for underactuated balance robots. By sequentially controller design, trajectory tracking of the actuated subsystem and balance of the unactuated subsystem can be achieved simultaneously. However, with certain conditions, there exists uncontrolled robot motion under the EIC...
-
Multiple preferences between robots and tasks have been largely overlooked in previous research on Multi-Robot Task Allocation (MRTA) problems. In this paper, we propose a preference-driven approach based on hedonic game to address the task allocation problem of muti-robot systems in emergency rescue scenarios. We present a distributed framework considering various preferences between robots and t...
-
This paper focuses on coordinating a robot swarm orbiting a convex path without collisions among the individuals. The individual robots lack braking capabilities and can only adjust their courses while maintaining their constant but different speeds. Instead of controlling the spatial relations between the robots, our formation control algorithm aims to deploy a dense robot swarm that mimics the b...
-
Robot swarms can be tasked with a variety of automated sensing and inspection applications in aerial, aquatic, and surface environments. In this paper, we study a simplified two-outcome surface inspection task. We task a group of robots to inspect and collectively classify a 2D surface section based on a binary pattern projected on the surface. We use a decentralized Bayesian decision-making algor...
-
We develop a planner that directs robots to construct a 3D target structure composed of blocks. The robots themselves are cubes of the same size as the blocks, and they may place, carry, or remove one block at a time. When moving, robots are also allowed to climb or descend a block. A construction plan may thus build a staircase-like scaffolding of blocks to reach other blocks at higher levels. Th...
-
Complementary to prevalent LiDAR and camera systems, millimeter-wave (mmWave) radar is robust to adverse weather conditions like fog, rainstorms, and blizzards but offers sparse point clouds. Current techniques enhance the point cloud by the supervision of LiDAR’s data. However, high-performance LiDAR is notably expensive and is not commonly available on vehicles. This paper presents mmEMP, a supe...
-
Cameras and LiDARs are both important sensors for autonomous driving, playing critical roles in 3D object detection. Camera-LiDAR Fusion has been a prevalent solution for robust and accurate driving perception. In contrast to the vast majority of existing arts that focus on how to improve the performance of 3D target detection through cross-modal schemes, deep learning algorithms, and training tri...
-
The ability to detect objects in all lighting (i.e., normal-, over-, and under-exposed) conditions is crucial for real-world applications, such as self-driving. Traditional RGB-based detectors often fail under such varying lighting conditions. Therefore, recent works utilize novel event cameras to supplement or guide the RGB modality; however, these methods typically adopt asymmetric network struc...
-
Enhancing the generalization capacity for semantic segmentation of aerial perception systems for safety-critical applications is vital, especially for environments with low-light and adverse conditions. Multi-spectral fusion techniques aim to maintain the merits of electro-optical (EO) and infrared (IR) images, e.g., retaining low-level features and capturing detailed textures from both modalities...
-
High-definition (HD) semantic map generation of the environment is an essential component of autonomous driving. Existing methods have achieved good performance in this task by fusing different sensor modalities, such as LiDAR and camera. However, current works are based on raw data or network feature-level fusion and only consider short-range HD map generation, limiting their deployment to realis...
-
Visual bird’s eye view (BEV) semantic segmentation helps autonomous vehicles understand the surrounding environment only from front-view (FV) images, including static elements (e.g., roads) and dynamic elements (e.g., vehicles, pedestrians). However, the high cost of annotation procedures of full-supervised methods limits the capability of the visual BEV semantic segmentation, which usually needs ...
-
In the era of big data and large models, automatic annotating functions for multi-modal data are of great significance for real-world AI-driven applications, such as autonomous driving and embodied AI. Unlike traditional closed-set annotation, open-vocabulary annotation is essential to achieve human-level cognition capability. However, there are few open-vocabulary auto-labeling systems for multi-...
-
Semantic segmentation, a fundamental visual task ubiquitously employed in sectors ranging from transportation and robotics to healthcare, has always captivated the research community. In the wake of rapid advancements in large model research, the foundation model for semantic segmentation tasks, termed the Segment Anything Model (SAM), has been introduced. This model substantially addresses the di...
-
Neuromorphic spike data, an upcoming modality with high temporal resolution, has shown promising potential in autonomous driving by mitigating the challenges posed by high-velocity motion blur. However, training the spike depth estimation network holds significant challenges in two aspects: sparse spatial information for pixel-wise tasks and difficulties in achieving paired depth labels for tempor...
-
PillarGen: Enhancing Radar Point Cloud Density and Quality via Pillar-based Point Generation Network
In this paper, we present a novel point generation model, referred to as Pillar-based Point Generation Network (PillarGen), which facilitates the transformation of point clouds from one domain into another. PillarGen can produce synthetic point clouds with enhanced density and quality based on the provided input point clouds. The PillarGen model performs the following three steps: 1) pillar encodi...
-
Scoliosis diagnosis and assessment depend largely on the measurement of the Cobb angle in spine X-ray images. With the emergence of deep learning techniques that employ landmark detection, tilt prediction, and spine segmentation, automated Cobb angle measurement has become increasingly popular. However, these methods encounter difficulties such as high noise sensitivity, intricate computational pr...
-
We present and discuss the Synset Boulevard dataset, designed for the task of surveillance-nature vehicle make and model recognition (VMMR)—to the best of our knowledge the first entirely synthetically generated large-scale VMMR image dataset. Through the simulation of image data rather than the manual annotation of real data, we intend to mitigate common challenges in state-of-the-art VMMR datase...
-
Model-based reinforcement learning (RL) has shown great promise due to its sample efficiency, but still struggles with long-horizon sparse-reward tasks, especially in offline settings where the agent learns from a fixed dataset. We hypothesize that model-based RL agents struggle in these environments due to a lack of long-term planning capabilities, and that planning in a temporally abstract model...
-
Self-supervised skill learning aims to acquire useful behaviors that leverage the underlying dynamics of the environment. Latent variable models, based on mutual information maximization, have been successful in this task but still struggle in the context of robotic manipulation. As it requires impacting a possibly large set of degrees of freedom composing the environment, mutual information maxim...
-
Pre-training robots with a rich set of skills can substantially accelerate the learning of downstream tasks. Prior works have defined pre-training tasks via natural language instructions, but doing so requires tedious human annotation of hundreds of thousands of instructions. Thus, we propose SPRINT, a scalable offline policy pre-training approach which substantially reduces the human effort neede...
-
In reinforcement learning (RL), learning directly from pixels, is commonly known as vision-based RL. Effective state representations are crucial for high performance in vision-based RL. However, in order to learn effective state representations, most current vision-based RL methods based on contrastive unsupervised learning use auxiliary tasks similar to those in computer vision, which does not gu...
-
Learning highly dynamic behaviors for robots has been a longstanding challenge. Traditional approaches have demonstrated robust locomotion, but the exhibited behaviors lack diversity and agility. They employ approximate models, which lead to compromises in performance. Data-driven approaches have been shown to reproduce agile behaviors of animals, but typically have not been able to learn highly d...
-
Model-based RL is a promising approach for real-world robotics due to its improved sample efficiency and generalization capabilities compared to model-free RL. However, effective model-based RL solutions for vision-based real-world applications require bridging the sim-to-real gap for any world model learnt. Due to its significant computational cost, standard domain randomisation does not provide ...
-
Learning strategic robot behavior—like that required in pursuit-evasion interactions—under real-world constraints is extremely challenging. It requires exploiting the dynamics of the interaction, and planning through both physical state and latent intent uncertainty. In this paper, we transform this intractable problem into a supervised learning problem, where a fully-observable robot policy gener...
-
Tracking and catching moving objects is an important ability for robots in a dynamic world. Whilst some objects have highly predictable state evolution e.g., the ballistic trajectory of a tennis ball, reactive targets alter their behavior in response to motion of the manipulator. Reactive applications range from gently capturing living animals such as snakes or fish for biological investigations, ...
-
In contact-rich tasks, the hybrid, multi-modal nature of contact dynamics poses great challenges in model representation, planning, and control. Recent efforts have attempted to address these challenges via data-driven methods, learning dynamical models in combination with model predictive control. Those methods, while effective, rely solely on minimizing forward prediction errors to hope for bett...
-
Multi-task learning has expanded the boundaries of robotic manipulation, enabling the execution of increasingly complex tasks. However, policies learned through reinforcement learning exhibit limited generalization and narrow distributions, which restrict their effectiveness in multi-task training. Addressing the challenge of obtaining policies with generalization and stability represents a non-tr...
-
Furniture assembly remains an unsolved problem in robotic manipulation due to its long task horizon and nongeneralizable operations plan. This paper presents the Tactile Ensemble Skill Transfer (TEST) framework, a pioneering offline reinforcement learning (RL) approach that incorporates tactile feedback in the control loop. TEST’s core design is to learn a skill transition model for high-level pla...
-
Using tactile sensors for manipulation remains one of the most challenging problems in robotics. At the heart of these challenges is generalization: How can we train a tactile-based policy that can manipulate unseen and diverse objects? In this paper, we propose to perform Reinforcement Learning with only visual tactile sensing inputs on diverse objects in a physical simulator. By training with di...
-
Synchronized dual-arm rearrangement is widely studied as a common scenario in industrial applications. It often faces scalability challenges due to the computational complexity of robotic arm rearrangement and the high-dimensional nature of dual-arm planning. To address these challenges, we formulated the problem as cooperative mTSP, a variant of mTSP where agents share cooperative costs, and util...
-
If a robot masters folding a kitchen towel, we would expect it to master folding a large beach towel. However, existing policy learning methods that rely on data augmentation still don’t guarantee such generalization. Our insight is to add equivariance to both the visual object representation and policy architecture. We propose EquivAct which utilizes SIM(3)-equivariant network structures that gua...
-
In this work, we present DiPPeR, a novel and fast 2D path planning framework for quadrupedal locomotion, leveraging diffusion-driven techniques. Our contributions include a scalable dataset generator for map images and corresponding trajectories, an image-conditioned diffusion planner for mobile robots, and a training/inference pipeline employing CNNs. We validate our approach in several mazes, as...
-
Collision-avoiding motion planning for articulated robotic arms is one of the major challenges in robotics. The difficulty of the problem arises from its high dimensionality and the intricate geometry of the feasible space. Our goal is to seek large convex domains in configuration space, which contain no obstacles. In these domains, simple linear trajectories are guaranteed to be collision free, a...
-
PathRL: An End-to-End Path Generation Method for Collision Avoidance via Deep Reinforcement Learning
Robot navigation using deep reinforcement learning (DRL) has shown great potential in improving the performance of mobile robots. Nevertheless, most existing DRL-based navigation methods primarily focus on training a policy that directly commands the robot with low-level controls, like linear and angular velocities, which leads to unstable speeds and unsmooth trajectories of the robot during the l...
-
The past few years have seen immense progress on two fronts that are critical to safe, widespread mobile robot deployment: predicting uncertain motion of multiple agents, and planning robot motion under uncertainty. However, the numerical methods required on each front have resulted in a mismatch of representation for prediction and planning. In prediction, numerical tractability is usually achiev...
-
We present an efficient method for certifying non-collision for piecewise-polynomial motion plans in algebraic reparametrizations of configuration space. Such motion plans include those generated by popular randomized methods including RRTs and PRMs, as well as those generated by many methods in trajectory optimization. Based on Sums-of-Squares optimization, our method provides exact, rigorous cer...
-
Image compression is increasingly important in applications like intelligent driving and smart surveillance systems. This study presents a novel cross view capture distributed image compression network (CVCDIC) to improve the compression quality by using decoder side information. The CVCDIC’s decoder utilizes feature extraction networks to extract features from both the primary image and the side ...
-
Path planning in a changing environment is a challenging task in robotics, as moving objects impose time-dependent constraints. Recent planning methods primarily focus on the spatial aspects, lacking the capability to directly incorporate time constraints. In this paper, we propose a method that leverages a generative model to decompose a complex planning problem into small manageable ones by incr...
-
Model predictive control (MPC) may provide local motion planning for mobile robotic platforms. The challenging aspect is the analytic representation of collision cost for the case when both the obstacle map and robot footprint are arbitrary. We propose a Neural Potential Field: a neural network model that returns a differentiable collision cost based on robot pose, obstacle map, and robot footprin...
-
In this paper, we present a probabilistic and unconstrained model predictive control formulation for robot navigation under uncertainty. We present (1) a closed-form approximation of the probability of collision that naturally models the propagation of uncertainty over the planning horizon and is computationally cheap to evaluate, and (2) a collision-cost formulation which provably preserves forwa...
-
Efficient RRT*-based Safety-Constrained Motion Planning for Continuum Robots in Dynamic Environments
Continuum robots, characterized by their high flexibility and infinite degrees of freedom (DoFs), have gained prominence in applications such as minimally invasive surgery and hazardous environment exploration. However, the intrinsic complexity of continuum robots requires a significant amount of time for their motion planning, posing a hurdle to their practical implementation. To tackle these cha...
-
In-flight high-speed object capturing is crucial in nature to improve survival and adaptation to the environment, such as the predation of frogs, leopards, and eagles. Despite its ubiquitousness in nature, capturing fast-moving objects is extremely challenging in engineering implementations. In this paper, we report an ultrafast gripper based on tunable bistable structures. Different from current ...
-
Deep-sea research represents invaluable opportunities to unravel hidden ecosystems, uncover unknown biodiversity, and provide critical insights into the Earth’s history and the impacts of climate change. Due to the extreme conditions, exploring the deep-sea traditionally requires costly equipment, such as specific diving robots, engineered to withstand the high pressure. Our research aims to reduc...
-
The navigational abilities of tip-everting soft growing robots, known as vine robots, are compromised when tip-mount devices are added to enable carrying of payloads. We present a new method for securing a vine robot to objects or its environment that exploits the unique eversion-based growth mechanism and flexibility of vine robots, while keeping the tip of the vine robot free of encumbrance. Our...
-
This study tackles the representative yet challenging contact-rich peg-in-hole task of robotic assembly, using a soft wrist that can operate more safely and tolerate lower-frequency control signals than a rigid one. Previous studies often use a fully observable formulation, requiring external setups or estimators for the peg-to-hole pose. In contrast, we use a partially observable formulation and ...
-
Soft continuum robots enable new application areas in contrast to standard rigid robots, such as interaction with a varying environment. Due to their compliant continuous structure, they are inherently safe and adaptive to environmental conditions. In this paper, the interaction with the environment is performed at the tool-center-point of a soft continuum manipulator and is realized by a hybrid f...
-
Soft robotic grippers are intrinsically delicate while grasping objects, and can rely on mechanical deformation to adapt to different shapes without explicit control. These characteristics are particularly appealing for agriculture, where items of produce from the same crop can vary significantly in shape and size, and delicate harvesting is among the first concerns for fruit quality. Various soft...
-
Robots performing human-scale manipulation tasks require an extensive amount of knowledge about their surroundings in order to perform their actions competently and human-like. In this work, we investigate the use of virtual reality technology as an implementation for robot environment modeling, and present a technique for translating scene graphs into knowledge bases. To this end, we take advanta...
-
In this article, we introduce a novel strategy for robotic exploration in unknown environments using a semantic topometric map. As it will be presented, the semantic topometric map is generated by segmenting the grid map of the currently explored parts of the environment into regions, such as intersections, pathways, dead-ends, and unexplored frontiers, which constitute the structural semantics of...
-
Precise 3D environmental mapping with semantics is essential in robotics. Existing methods often rely on pre-defined concepts during training or are time-intensive when generating semantic maps. This paper presents Open-Fusion, an approach for real-time open-vocabulary 3D mapping and queryable scene representation using RGB-D data. Open-Fusion harnesses the power of a pretrained vision-language fo...
-
Accurately perceiving and tracking instances over time is essential for the decision-making processes of autonomous agents interacting safely in dynamic environments. With this intention, we propose Mask4Former for the challenging task of 4D panoptic segmentation of LiDAR point clouds. Mask4Former is the first transformer-based approach unifying semantic instance segmentation and tracking of spars...
-
Visual Semantic Navigation (VSN) aims at navigating a robot to a given target object in a previously unseen scene. To tackle this task, the robot must learn a nimble navigation policy by utilizing spatial patterns and semantic co-occurrence relations among objects in the scene. Prevailing approaches extract scene priors from the instant visual observations and solidify them in neural episodic memo...
-
In this article, we propose the novel concept of Belief Scene Graphs, which are utility-driven extensions of partial 3D scene graphs, that enable efficient high-level task planning with partial information. We propose a graph-based learning methodology for the computation of belief (also referred to as expectation) on any given 3D scene graph, which is then used to strategically add new nodes (ref...
-
Visual topic modeling (VTM) provides key insight into data sets based on learned semantic topic models. The Gaussian-Dirichlet Random Field (GDRF), a state-of-the-art VTM technique, models these semantic topics in continuous space as densities. However, ambiguity in learned topics is a disadvantage of such Dirichlet-based VTM algorithms. We propose the Guided Gaussian-Dirichlet Random Field (GGDRF...
-
Point clouds play an important role in 3D analysis, which has broad applications in robotics and autonomous driving. The pre-training fine-tuning paradigm has shown great potential in the point cloud domain. Full fine-tuning is generally effective but leads to a heavy storage and computational burden, which becomes inefficient and unacceptable as the size of pre-trained models scales. Although eff...
-
Multi-modal unsupervised domain adaptation (MM-UDA) for 3D semantic segmentation is a practical solution to embed semantic understanding in autonomous systems without expensive point-wise annotations. While previous MM-UDA methods can achieve overall improvement, they suffer from significant class-imbalanced performance, restricting their adoption in real applications. This imbalanced performance ...
-
Training a robotic policy from scratch using deep reinforcement learning methods can be prohibitively expensive due to sample inefficiency. To address this challenge, transferring policies trained in the source domain to the target domain becomes an attractive paradigm. Previous research has typically focused on domains with similar state and action spaces but differing in other aspects. In this p...
-
This paper presents a parameter-efficient prompt tuning method, named PPT, to adapt a large multi-modal model for 3D point cloud understanding. Existing strategies are quite expensive in computation and storage, and depend on timeconsuming prompt engineering. We address the problems from three aspects. Firstly, a PromptLearner module is devised to replace hand-crafted prompts with learnable contex...
-
Vision-centric bird-eye-view (BEV) perception has shown promising potential in autonomous driving. Recent works mainly focus on improving efficiency or accuracy but neglect the challenges when facing environment changing, resulting in severe degradation of transfer performance. For BEV perception, we figure out the significant domain gaps existing in typical real-world cross-domain scenarios and c...
-
We address the problem of robotic grasping of known and unknown objects using implicit behavior cloning. We train a grasp evaluation model from a small number of demonstrations that outputs higher values for grasp candidates that are more likely to succeed in grasping. This evaluation model serves as an objective function, that we maximize to identify successful grasps. Key to our approach is the ...
-
Training model-free intelligent agents in complex real-world scenarios using reinforcement learning (RL) often necessitates simulation-based environments due to high physical expenses. However, when simulation takes a long time, e.g., in an unsteady 3D fluid simulation with interactions to the controllable solids, existing RL algorithms meet difficulty to accomplish training within a reasonable ti...
-
In this paper, building on recent advances in the fields of gaming AI and social robotics, we present a new approach to facilitate the social robot Haru to imitate game strategies from human players’ demonstrated trajectories and evaluative feedback in a real-time two-player game. Our research shows that Haru is able to learn and imitate human different game strategies from human players in a huma...
-
Meal preparation is an important instrumental activity of daily living (IADL). While existing research has explored robotic assistance in meal preparation tasks such as cutting and cooking, the crucial task of peeling has received less attention. Robot-assisted peeling, conventionally a bimanual task, is challenging to deploy in the homes of care recipients using two wheelchair-mounted robot arms ...
-
In this paper, a method for generating a map from path information described using natural language (textual path) is proposed. In recent years, robotics research mainly focus on vision-and-language navigation (VLN), a navigation task based on images and textual paths. Although VLN is expected to facilitate user instructions to robots, its current implementation requires users to explain the detai...
-
Current methods of deploying robots that operate in dynamic, uncertain environments, such as Uncrewed Aerial Systems in search & rescue missions, require nearly continuous human supervision for vehicle guidance and operation. These methods do not consider high-level mission context resulting in cumbersome manual operation or inefficient exhaustive search patterns. We present a human-centered auton...
-
Routine inspections for critical infrastructures such as bridges are required in most jurisdictions worldwide. Such routine inspections are largely visual in nature, which are qualitative, subjective, and not repeatable. Although robotic infrastructure inspections address such limitations, they cannot replace the superior ability of experts to make decisions in complex situations, thus making huma...
-
Integration of robots into scientific user facilities, such as the National Synchrotron Light Source II, improves their efficiency and capacity. Many such facilities use the opensource Bluesky project for experimental control and orchestration. However, there remains an open challenge in deploying robotic solutions at these facilities that are reconfigurable, extensible, and compatible with pre-ex...
-
Preference-based reinforcement learning (RL) poses as a recent research direction in robot learning, by allowing humans to teach robots through preferences on pairs of desired behaviours. Nonetheless, to obtain realistic robot policies, an arbitrarily large number of queries is required to be answered by humans. In this work, we approach the sample-efficiency challenge by presenting a technique wh...
-
Robots operating alongside humans often encounter unfamiliar environments that make autonomous task completion challenging. Though improving models and increasing dataset size can enhance a robot’s performance in unseen environments, data collection and model refinement may be impractical in every environment. Approaches that utilize human demonstrations through manual operation can aid in refinem...
-
Developing autonomous vehicles that can safely interact with pedestrians requires large amounts of pedestrian and vehicle data in order to learn accurate pedestrian-vehicle interaction models. However, gathering data that include crucial but rare scenarios - such as pedestrians jaywalking into heavy traffic - can be costly and unsafe to collect. We propose a virtual reality human-in-the-loop simul...
-
Shared electric micromobility has surged to a popular model of urban transportation due to its efficiency in short-distance trips and environmentally friendly characteristics compared to traditional automobiles. However, managing thousands of shared electric micromobility vehicles including rebalancing and charging to meet users’ travel demands still has been a challenge. Existing methods generall...
-
It is difficult to guarantee an empty living environment to prevent unexpected contact between the object being manipulated by the robot and unplanned obstacles. In this paper, we propose a planar compliant contact control method for planar manipulation to cope with unexpected contact. We first use sheet gel as a multi-dimensional passive elastic element and combine it with a two-finger gripper to...
-
An emerging trend in the field of human-robot collaboration is the disassembly of end-of-life products. Safety is a crucial requirement of the disassembly process since worn-out or damaged products could break, possibly resulting in dangerous behavior of the robot. To protect the user from such behavior, this work addresses this challenge through the implementation of an energy-aware Cartesian imp...
-
Recently, tactile sensing has attracted great interests in robotics, especially for exploring unstructured objects. Sensor arrays play an important role in the exploration, which generates rich spatio-temporal information. In this work, we propose an efficient tactile recognition model, X-Tacformer. This model pays attention to both spatial and temporal features of tactile sequences from sensor ar...
-
Hand motion tracking is essential in many fields, e.g., immersive virtual reality, teleoperation of robotic hand, and hand rehabilitation of stroke patient, as human hand plays a crucial role in our daily life. The highly under-actuated hand exoskeleton, which can track the 6-DoF motions of each fingertip via a highly under-actuated kinematic chain, exhibits many benefits in wearability and portab...
-
This study presents the Prosthetic Upper-Limb Sensory Enhancement (PULSE), a novel dual feedback device completely integrated into a prosthetic socket. The core of the system includes two compact vibrotactile actuators and two silicone chambers in contact with the user’s skin. These components provide high-frequency tactile cues for initial contact and surface information (e.g. texture) as well as...
-
Surface vibration tactile feedback is capable of conveying various semantic information to humans via handheld electronic devices, such as smartphones, touch panels, and game controllers. However, covering the entire contacting surface of the device with a dense arrangement of actuators can affect its normal use. Determining how to produce desired vibration patterns at any contact point with only ...
-
The integration of high-level assistance algorithms in surgical robotics training curricula may be beneficial in establishing a more comprehensive and robust skillset for aspiring surgeons, improving their clinical performance as a consequence. This work presents the development and validation of a haptic-enhanced Virtual Reality simulator for surgical robotics training, featuring 8 surgical tasks...
-
A miniature haptic stimulation device utilizes focused ultrasound to deliver a tactile haptic sensation to the finger. The 1-3 piezocomposite device has a 1 cm2 footprint, which is an order of magnitude smaller than other ultrasonic haptic devices and is a good candidate for wearable tactile rendering systems. The device focuses energy to a 1 mm3 voxel. The current prototype was validated with a s...
-
Tactile perception is important for robotic systems that interact with the world through touch. Touch is an active sense in which tactile measurements depend on the contact properties of an interaction—e.g., velocity, force, acceleration— as well as properties of the sensor and object under test. These dependencies make training tactile perceptual models challenging. Additionally, the effects of l...
-
Shape-changing displays enable real-time visualization and haptic exploration of 3D surfaces. However, many shape-changing displays are composed of individually actuated rigid bodies, which makes them both mechanically complex and unable to form smooth surfaces. In this work, we build a multi-stable curved line display inspired by physical splines. By using circular splines to initialize a discret...
-
For locomotion, is an arm on a legged robot a liability or an asset for locomotion? Biological systems evolved additional limbs beyond legs that facilitates postural control. This work shows how a manipulator can be an asset for legged locomotion at high speeds or under external perturbations, where the arm serves beyond manipulation. Since the system has 15 degrees of freedom (twelve for the legg...
-
Controlled execution of dynamic motions in quadrupedal robots, especially those with articulated soft bodies, presents a unique set of challenges that traditional methods struggle to address efficiently. In this study, we tackle these issues by relying on a simple yet effective two-stage learning framework to generate dynamic motions for quadrupedal robots. First, a gradient-free evolution strateg...
-
We propose the expert composer policy, a framework to reliably expand the skill repertoire of quadruped agents. The composer policy links pair of experts via transitions to a sampled target state, allowing experts to be composed sequentially. Each expert specializes in a single skill, such as a locomotion gait or a jumping motion. Instead of a hierarchical or mixture-of-experts architecture, we tr...
-
Can a quadrupedal robot perform bipedal motions like humans? Although developing human-like behaviors is more often studied on costly bipedal robot platforms, we present a solution over a lightweight quadrupedal robot that unlocks the agility of the quadruped in an upright standing pose and is capable of a variety of human-like motions. Our framework is with a hierarchical structure. At the low le...
-
We aim to control a robot to physically behave in the real world following any high-level language command like "cartwheel" or "kick". Although human motion datasets exist, this task remains particularly challenging since generative models can produce physically unrealistic motions, which will be more severe for robots due to different body structures and physical properties. Deploying such a moti...
-
Quadrupedal robots have played a crucial role in various environments, from structured environments to complex harsh terrains, thanks to their agile locomotion ability. However, these robots can easily lose their locomotion functionality if damaged by external accidents or internal malfunctions. In this paper, we propose a novel deep reinforcement learning framework to enable a quadrupedal robot t...
-
Electric quadruped robots used in outdoor exploration are susceptible to leg-related electrical or mechanical failures. Unexpected joint power loss and joint locking can immediately pose a falling threat. Typically, controllers lack the capability to actively sense the condition of their own joints and take proactive actions. Maintaining the original motion patterns could lead to disastrous conseq...
-
Navigation in agricultural fields imposes various constraints on manoeuvrability, which can be tackled by using four-wheel steering (4WS) vehicles which are capable of switching between multiple steering mechanisms with distinct kinematic properties. For example, parallel positive steering (PPS) with four wheels in parallel to each other can maintain the vehicle’s heading when moving along a curve...
-
Jerk-constrained trajectories offer a wide range of advantages that collectively improve the performance of robotic systems, including increased energy efficiency, durability, and safety. In this paper, we present a novel approach to jerk-constrained time-optimal trajectory planning (TOTP), which follows a specified path while satisfying up to third-order constraints to ensure safety and smooth mo...
-
There has been a growing interest in parallel strategies for solving trajectory optimization problems. One key step in many algorithmic approaches to trajectory optimization is the solution of moderately-large and sparse linear systems. Iterative methods are particularly well-suited for parallel solves of such systems. However, fast and stable convergence of iterative methods is reliant on the app...
-
Nonlinear Model Predictive Control (NMPC) is a state-of-the-art approach for locomotion and manipulation which leverages trajectory optimization at each control step. While the performance of this approach is computationally bounded, implementations of direct trajectory optimization that use iterative methods to solve the underlying moderately-large and sparse linear systems, are a natural fit for...
-
Trajectory retiming is the task of computing a feasible time parameterization to traverse a path. It is commonly used in the decoupled approach to trajectory optimization whereby a path is first found, then a retiming algorithm computes a speed profile that satisfies kino-dynamic and other constraints. While trajectory retiming is most often formulated with the minimum-time objective (i.e. travers...
-
To enhance robots’ applicability in real-world scenarios, it is essential to establish a complex and multi-tasking behaviour, inspired by human nature. To this purpose, from a hardware perspective, a high number of degrees of freedom is necessary, as is the case for humanoids and collaborative mobile manipulators. From a software standpoint instead, complex hierarchical strategies are often used t...
-
In single-incision laparoscopic surgery (SILS), magnetic anchoring and guidance system (MAGS) is a promising technique to prevent clutter in the surgical workspace and provide a larger vision field. Existing camera designs mainly rely on rigid structure design, resulting in risks of losing magnetic coupling and impacting tissue during the insertion and coupling procedure. In this paper, we propose...
-
Helicobacter pylori, a pervasive bacterial infection associated with gastrointestinal disorders such as gastritis, peptic ulcer disease, and gastric cancer, impacts approximately 50% of the global population. The efficacy of standard clinical eradication therapies is diminishing due to the rise of antibiotic-resistant strains, necessitating alternative treatment strategies. Photodynamic therapy (P...
-
Recently, fiber optic sensors such as fiber Bragg gratings (FBGs) have been widely investigated for shape reconstruction and force estimation of flexible surgical robots. However, most existing approaches need precise model parameters of FBGs inside the fiber and their alignments with the flexible robots for accurate sensing results. Another challenge lies in online acquiring external forces at ar...
-
In this work, we propose a semi-autonomous scheme to synergistically share the complicated task of manipulation and cutting of an unknown deformable tissue (U-DT) between a remote surgeon and a surgical robot. Particularly, utilizing the da Vinci Research Kit (dVRK) platform, we have designed and successfully demonstrated a fully functional shared control scheme for an autonomous tensioning and te...
-
The manipulation of instruments under a microscope suffers from physiological tremor and human errors, which are inevitable in long microsurgery interventions. Robotic systems developed in recent years for microsurgery are expensive and not flexible, as they cannot use standard instruments, and need the surgeon to modify their operative skills and strategies. In this paper, we introduce a modular ...
-
Several robotic frameworks have been recently developed to assist ophthalmic surgeons in performing complex vitreoretinal procedures such as subretinal injection. However, in order to intuitively integrate robots into the surgical workflow, it is crucial to emphasize that an accessibility analysis framework for vitreoretinal surgery must be considered as an essential component. Such a framework, i...
-
Safety is a fundamental requirement of many robotic systems. Control barrier function (CBF)-based approaches have been proposed to guarantee the safety of robotic systems. However, the effectiveness of these approaches highly relies on the choice of CBFs. Inspired by the universal approximation power of neural networks, there is a growing trend toward representing CBFs using neural networks, leadi...
-
Ensuring safety in autonomous systems is essential as they become more integrated with modern society. One way to accomplish this is to identify and maintain a safe operating space. To this end, much effort has been devoted in the field of reachability analysis to obtaining control-invariant sets which ensure that a system inside of these sets can remain in these sets, and are thus essential for g...
-
Cloth manipulation is a category of deformable object manipulation of great interest to the robotics community, from applications of automated laundry-folding and home organizing to textiles and flexible manufacturing. Despite the desire for automated cloth manipulation, the thin-shell dynamics and under-actuation nature of cloth present significant challenges for robots to effectively interact wi...
-
Deploying autonomous systems in safety critical settings necessitates methods to verify their safety properties. This is challenging because real-world systems may be subject to disturbances that affect their performance, but are unknown a priori. This work develops a safety-verification strategy wherein data is collected online and incorporated into a reachability analysis approach to check in re...
-
Dynamic locomotion in legged robots is close to industrial collaboration, but a lack of standardized testing obstructs commercialization. The issues are not merely political, theoretical, or algorithmic but also physical, indicating limited studies and comprehension regarding standard testing infrastructure and equipment. For decades, the approaches we have been testing legged robots were rarely s...
-
Autonomous systems, such as self-driving cars and drones, have made significant strides in recent years by leveraging visual inputs and machine learning for decision-making and control. Despite their impressive performance, these vision-based controllers can make erroneous predictions when faced with novel or out-of-distribution inputs. Such errors can cascade to catastrophic system failures and c...
-
A key source of brittleness for robotic systems is the presence of model uncertainty and external disturbances. Most existing approaches to robust control either seek to bound the worst-case disturbance (which results in conservative behavior), or to learn a deterministic dynamics model (which is unable to capture uncertain dynamics or disturbances). This work proposes a different approach: traini...
-
An open-source software framework is presented that allows real-time underwater mapping with popular marine robotics components, namely a BlueRobotics BlueROV2 with its standard Ping360 Mechanical Scanning Sonar (MSS) and a A50 Doppler Velocity Log (DVL), which are low-cost devices for their respective types - if not even the most affordable ones on the market. The software runs with low computati...
-
Autonomous underwater docking is of the utmost importance for expanding the capabilities of Autonomous Underwater Vehicles (AUVs). Due to a historical focus on underwater docking to only static targets, the research gap in underwater docking to dynamically active targets has been left relatively untouched. We address the state estimation problem that arises when trying to rendezvous a chaser AUV w...
-
Determining the distances from the hull of the own ship to obstacles or land, i.e. water clearance, is a fundamental task in navigation. This is particularly relevant during maneuvering in the harbor or navigating in confined waters. We introduce the concepts of area water clearance and line water clearance. Area water clearance is important especially for path planning and obstacle avoidance. Lin...
-
Autonomous Underwater Vehicles (AUVs) conduct missions underwater without the need for human intervention. A docking station (DS) can extend mission times of an AUV by providing a location for the AUV to recharge its batteries and receive updated mission information. Various methods for locating and tracking a DS exist, but most rely on expensive acoustic sensors, or are vision-based, which is sig...
-
This work presents a camera model for refractive media such as water and its application in underwater visual-inertial odometry. The model is self-calibrating in real-time and is free of known correspondences or calibration targets. It is separable as a distortion model (dependent on refractive index n and radial pixel coordinate) and a virtual pinhole model (as a function of n). We derive the sel...
-
This paper presents an extension to visual inertial odometry (VIO) by introducing tightly-coupled fusion of magnetometer measurements. A sliding window of keyframes is optimized by minimizing re-projection errors, relative inertial errors, and relative magnetometer orientation errors. The results of IMU orientation propagation are used to efficiently transform magnetometer measurements between fra...
-
Lack of information and perceptual ambiguity are key problems in sonar-based mapping applications. We propose a technique for mapping of underwater environments, building on the finite, positive, sonar beamwidth. Our approach models the free-space covered by each emitted acoustic pulse, employing volumetric techniques to create grid-based submaps of the unoccupied water volumes through images coll...
-
Landing safety is a challenge heavily engaging the research community recently, due to the increasing interest in applications availed by aerial vehicles. In this paper, we propose a landing safety pipeline based on state of the art object detectors and OctoMap. First, a point cloud of surface obstacles is generated, which is then inserted in an OctoMap. The unoccupied areas are identified, thus r...
-
Motion planning techniques for quadrotors have advanced significantly over the past decade. Most successful planners have two stages: a front-end that determines a path that incorporates geometric (or kinematic or input) constraints and specifies the homotopy class of the trajectory, and a back-end that optimizes this path to respect dynamics and input constraints. While there are many different c...
-
In this work, we present a nonlinear Model Predictive Control (NMPC) scheme for tracking a ground target using a multirotor with a cable-suspended load. The NMPC framework relies on the dynamic model of the UAV with the suspended load and, hence, an estimate of the load state is obtained by fusing the measurements of a downward-facing camera and a load cell with an Unscented Kalman Filter (UKF). A...
-
This article introduces an experimental emulation of a novel chunk-based flexible multi-DoF aerial 3D printing framework. The experimental demonstration of the overall autonomy focuses on precise motion planning and task allocation for a UAV, traversing through a series of planned space-filling paths involved in the aerial 3D printing process without physically depositing the overlaying material. ...
-
Global navigation satellite systems (GNSS) denied environments/conditions require unmanned aerial vehicles (UAVs) to energy-efficiently and reliably fly. To this end, this study presents perception-and-energy-aware motion planning for UAVs in GNSS-denied environments. The proposed planner solves the trajectory planning problem by optimizing a cost function consisting of two indices: the total ener...
-
Ground testing is of paramount importance to verify and validate space operations and the associated control algorithms before on-orbit deployment. Although state-of-the-art facilities are capable of reproducing zero-G environment with high degree of fidelity, these infrastructures can be complemented with multi-rotors emulating free flying or free floating conditions, exploiting the similarities ...
-
Sub-50g nano-drones are gaining momentum in both academia and industry. Their most compelling applications rely on onboard deep learning models for perception despite severe hardware constraints (i.e., sub-100mW processor). When deployed in unknown environments not represented in the training data, these models often underperform due to domain shift. To cope with this fundamental problem, we propo...
-
Indoor parking lots are the GPS-denied spaces to which vision-based localization approaches have usually been applied to solve localization problems. However, due to the repetitiveness and symmetry of the spaces, visual localization methods commonly confront difficulties in estimating precise 3D poses. In this study, we propose four novel modules that improve localization precision by imposing the...
-
The availability of city-scale Lidar maps enables the potential of city-scale place recognition using mobile cameras. However, the city-scale Lidar maps generally need to be compressed for storage efficiency, which increases the difficulty of direct visual place recognition in compressed Lidar maps. This paper proposes VOLoc, an accurate and efficient visual place recognition method that exploits ...
-
Spiking Neural Networks (SNNs) are at the forefront of neuromorphic computing thanks to their potential energy-efficiency, low latencies, and capacity for continual learning. While these capabilities are well suited for robotics tasks, SNNs have seen limited adaptation in this field thus far. This work introduces a SNN for Visual Place Recognition (VPR) that is both trainable within minutes and qu...
-
17-point algorithm is a popular method in relative pose estimation of multi-cameras. However, the role of overlap in 17-point algorithm remains unexplored. And the relaxed way in solving constrained normal equation leads to sub-optimal results. Both of them influence accuracy of the estimated pose. In this paper, we theoretically analyze the influence of overlap and the solvability of 17-point alg...
-
We present a lightweight ground texture based localization algorithm (L-GROUT) that improves the state of the art in performance and can be run in real-time on single board computers without GPU acceleration. Such computers are ubiquitous on small indoor robots and thus this work enables high-precision, millimeter-level localization without instrumenting, marking, or modifying the environment. The...
-
Achieving efficient and consistent localization with a prior map remains challenging in robotics. Conventional keyframe-based approaches often suffer from sub-optimal viewpoints due to limited field of view (FOV) and/or constrained motion, thus degrading the localization performance. To address this issue, we design a real-time tightly-coupled Neural Radiance Fields (NeRF)-aided visual-inertial na...
-
Vision-aided localization for low-cost mobile robots in diverse environments has attracted widespread attention recently. Although many current systems are applicable in daytime environments, nocturnal visual localization is still an open problem owing to the lack of stable visual information. An insight from most nocturnal scenes is that the static and bright streetlights are reliable visual info...
-
Neural implicit representation has recently achieved significant advancements, especially in the field of SLAM(Simultaneous Localization and Mapping). Previous NeRF-based SLAM methods have difficulties with object-level localization and reconstruction and struggle in dynamic and illumination-varied environments. We propose ONeK-SLAM, a robust object-level SLAM system that effectively combines feat...
-
This paper proposes a new nonlinear factor sparsification paradigm for general feature-based long-term SLAM backend. Given a pose sparsification policy, we aim to scale the SLAM problem with space explored instead of time in a principled way so that the number of time-indexed poses can be limited. At the same time, their influence and the long-lived landmarks are appropriately maintained. To do th...
-
The ability to detect loop closures plays an essential role in any SLAM system. Loop closures allow correcting the drifting pose estimates from a sensor odometry pipeline. In this paper, we address the problem of effectively detecting loop closures in LiDAR SLAM systems in various environments with longer lengths of sequences and agnostic of the scanning pattern of the sensor. While many approache...
-
Odometry and mapping play a pivotal role in the navigation of autonomous vehicles. In this paper, we address the problem of pose estimation and map creation using only radar sensors. We focus on two odometry estimation approaches followed by a mapping step. The first one is a new point-to-point ICP approach that leverages the velocity information provided by 3D radar sensors. The second one is adv...
-
In SLAM (Simultaneous localization and mapping) problems, Pose Graph Optimization (PGO) is a technique to refine an initial estimate of a set of poses (positions and orientations) from a set of pairwise relative measurements. The optimization procedure can be negatively affected even by a single outlier measurement, with possible catastrophic and meaningless results. Although recent works on robus...
-
Correspondence matching plays a crucial role in numerous robotics applications. In comparison to conventional hand-crafted methods and recent data-driven approaches, there is significant interest in plug-and-play algorithms that make full use of pre-trained backbone networks for multi-scale feature extraction and leverage hierarchical refinement strategies to generate matched correspondences. The ...
-
In recent years, object-oriented simultaneous localization and mapping (SLAM) has attracted increasing attention due to its ability to provide high-level semantic information while maintaining computational efficiency. Some researchers have attempted to enhance localization accuracy by integrating the modeled object residuals into bundle adjustment. However, few have demonstrated better results th...
-
Not relying on ground-truth data for training, self-supervised semantic visual odometry (SVO) has recently gained considerable attention. Within self-supervised SVO, feature representation inconsistency between semantic/depth and pose tasks presents a significant challenge, as it may disrupt cross-task feature representations and lead to notable performance degradation. Regrettably, existing self-...
-
Robotic manipulators are essential for future autonomous systems, yet limited trust in their autonomy has confined them to rigid, task-specific systems. The intricate configuration space of manipulators, coupled with the challenges of obstacle avoidance and constraint satisfaction, often makes motion planning the bottleneck for achieving reliable and adaptable autonomy. Recently, a class of consta...
-
We present VAPOR, a novel method for autonomous legged robot navigation in unstructured, densely vegetated outdoor environments using offline Reinforcement Learning (RL). Our method trains a novel RL policy using an actor-critic network and arbitrary data collected in real outdoor vegetation. Our policy uses height and intensity-based cost maps derived from 3D LiDAR point clouds, a goal cost map, ...
-
Classical motion planning for robotic manipulation includes a set of general algorithms that aim to minimize a scene-specific cost of executing a given plan. This approach offers remarkable adaptability, as they can be directly used off-the-shelf for any new scene without needing specific training datasets. However, without a prior understanding of what diverse valid trajectories are and without s...
-
Many computations in robotics can be dramatically accelerated if the robot configuration space is described as a collection of simple sets. For example, recently developed motion planners rely on a convex decomposition of the free space to design collision-free trajectories using fast convex optimization. In this work, we present an efficient method for approximately covering complex configuration...
-
The multi-robot visibility-based pursuit-evasion problem tasks a team of robots with systematically searching an environment to detect (capture) an evader. Previous techniques to generate search strategies for the pursuit team have shown to be either computationally intractable or permit poor solution quality. This paper presents a novel asymptotically optimal algorithm for generating a joint moti...
-
This paper presents a robust image-based visual servoing design for a quad-rotor unmanned aerial vehicle performing a visual target-tracking operation in the presence of turbulent wind. Image information is extracted and processed to control the positioning and heading of the aerial vehicle. A novel adaptive non-singular fast terminal sliding mode strategy is introduced to manage the visual servoi...
-
Model Predictive Control (MPC) has exhibited remarkable capabilities in optimizing objectives and meeting constraints. However, the substantial computational burden associated with solving the Optimal Control Problem (OCP) at each triggering instant introduces significant delays between state sampling and control application. These delays limit the practicality of MPC in resource-constrained syste...
-
We report a novel model-based nullspace adaptive trajectory-tracking control (NS-ATTC) algorithm for fully-actuated 6-degree-of-freedom (DOF) underwater vehicles which estimates unknown plant and actuator model parameters simultaneously. We provide a stability and convergence analysis with proof of asymptotically stable tracking error convergence, as well as a preliminary simulation study demonstr...
-
Autonomous racing is a challenging problem, as the vehicle needs to operate at the friction or handling limits in order to achieve minimum lap times. Autonomous race cars require highly accurate perception, state estimation, planning, and control. Adding to this complexity is the need to accurately identify vehicle model parameters governing lateral tire slip effects, which can evolve over time du...
-
Designing robust algorithms in the face of estimation uncertainty is a challenging task. Indeed, controllers seldom consider estimation uncertainty and only rely on the most likely estimated state. Consequently, sudden changes in the environment or the robot’s dynamics can lead to catastrophic behaviors. Leveraging recent results in risk-sensitive optimal control, this paper presents a risk-sensit...
-
This paper researches a novel global terminal sliding mode control(GTSMC) on a tethered satellites system(TSS) under outer disturbances, and the effect of PI/PD compensation in restraining chattering on sliding surface is appended. By taking advantage of the finite-time convergence of traditional terminal sliding surface, the sliding surface with global and terminal sliding motion is proposed, and...
-
Due to the low motion efficiency and maneuver-ability of underwater robots with six degrees of freedom, it is challenging for them to respond quickly to the attitude requirements during underwater autonomous manipulation. This paper presents a novel autonomous underwater robot with fully vectored propulsion and a model predictive control method to achieve more agile and efficient movements autonom...
-
Morphing quadrotors that can be potentially applied to confined spaces such as warehouses, tanks, and pipelines have flourished in recent years. Most work has focused on the mechanical feasibility of the morphing systems and high-level flight controller design, with limited discussions on low-level control. In this paper, a constrained model predictive control (MPC) is proposed and applied to solv...
-
In this work, we investigate task planning for mobile robots under linear temporal logic (LTL) specifications. This problem is particularly challenging when robots navigate in continuous workspaces due to the high computational complexity involved. Sampling-based methods have emerged as a promising avenue for addressing this challenge by incrementally constructing random trees, thereby sidesteppin...
-
This paper presents a 2D skeleton-based action segmentation method with applications in fine-grained human activity recognition. In contrast with state-of-the-art methods which directly take sequences of 3D skeleton coordinates as inputs and apply Graph Convolutional Networks (GCNs) for spatiotemporal feature learning, our main idea is to use sequences of 2D skeleton heatmaps as inputs and employ ...
-
Tactile sensing is significant for robotics since it can obtain physical contact information during manipulation. To capture multimodal contact information within a compact framework, we designed a novel sensor called ViTacTip, which seamlessly integrates both tactile and visual perception capabilities into a single, integrated sensor unit. ViTacTip features a transparent skin to capture fine feat...
-
Tactile sensing plays a critical role in enabling robots to interact safely with target objects in dynamic and unstructured environments. While various tactile sensors based on different sensing principles or different sensitive materials have been proposed, the development of flexible large-area tactile sensors for robots is still challenging. In this paper, a novel highly sensitive piezoresistiv...
-
Tactile exploration of surfaces is a key component of everyday life, allowing us to make complex inferences about our environments even when vision is occluded. The emergence of biomimetic neuromorphic hardware in recent years has furthered our ability to create biologically plausible sensing solutions. While these platforms continue to improve in regards to latency and power consumption, within r...
-
Tactile sensing has become a popular sensing modality for robot manipulators, due to the promise of providing robots with the ability to measure the rich contact information that gets transmitted through its sense of touch. Among the diverse range of information accessible from tactile sensors, torques transmitted from the grasped object to the fingers through extrinsic environmental contact may b...
-
In this paper, we propose X-lateral teleoperation: a novel hybrid unilateral-bilateral teleoperation framework. Bilateral teleoperation enables kinesthetic coupling between the operator and the remote environment with haptic feedback. However, in free motion, unlike unilateral teleoperators, bilateral teleoperators reflect undesirable operational forces to the operator. The proposed X-lateral tele...
-
Compared to fully-actuated robotic end-effectors, underactuated ones are generally more adaptive, robust, and cost-effective. However, state estimation for underactuated hands is usually more challenging. Vision-based tactile sensors, like Gelsight, can mitigate this issue by providing high-resolution tactile sensing and accurate proprioceptive sensing. As such, we present GelLink, a compact, unde...
-
Camera-based tactile sensors can provide high resolution positional and local geometry information for robotic manipulation. Curved and rounded fingers are often advantageous, but it can be difficult to derive illumination systems that work well within curved geometries. To address this issue, we introduce RainbowSight, a family of curved, compact, camera-based tactile sensors which use addressabl...
-
This study proposes a novel planning framework based on a model predictive control formulation that incorporates signal temporal logic (STL) specifications for task completion guarantees and robustness quantification. This marks the first-ever study to apply STL-guided trajectory optimization for bipedal locomotion push recovery, where the robot experiences unexpected disturbances. Existing recove...
-
Successfully achieving bipedal locomotion remains challenging due to real-world factors such as model uncertainty, random disturbances, and imperfect state estimation. In this work, we propose a novel metric for locomotive robustness – the estimated size of the hybrid forward invariant set associated with the step-to-step dynamics. Here, the forward invariant set can be loosely interpreted as the ...
-
For humans, fast, efficient walking over flat ground represents the vast majority of locomotion that an individual experiences on a daily basis, and for an effective, real-world humanoid robot the same will likely be the case. In this work, we propose a locomotion controller for efficient walking over near-flat ground using a relatively simple, model-based controller that utilizes a novel combinat...
-
This paper presents a novel framework for learning robust bipedal walking by combining a data-driven state representation with a Reinforcement Learning (RL) based locomotion policy. The framework utilizes an autoencoder to learn a low-dimensional latent space that captures the complex dynamics of bipedal locomotion from existing locomotion data. This reduced dimensional state representation is the...
-
High-frequency and accurate state estimation is crucial for biped robots. This paper presents a tightly-coupled LiDAR-Inertial-Kinematic Odometry (LIKO) for biped robot state estimation based on an iterated extended Kalman filter. Beyond state estimation, the foot contact position is also modeled and estimated. This allows for both position and velocity updates from kinematic measurement. Addition...
-
This work focuses on the agile transportation of liquids with robotic manipulators. In contrast to existing methods that are either computationally heavy, system/container specific or dependant on a singularity-prone pendulum model, we present a real-time slosh-free tracking technique. This method solely requires the reference trajectory and the robot’s kinematic constraints to output kinematicall...
-
Collision-free navigation is a critical issue in robotic systems as the environment is often dynamic and uncertain. This paper investigates a data-stream-driven motion control problem for mobile robots to avoid randomly moving obstacles when the probability distribution of the obstacle’s movement is partially observable through data and can be even time-varying. A data-stream-driven ambiguity set ...
-
The presented work tackles the question of quantifying the pose deviations of robots subject to external disturbance forces. While this question may not be central for large robots perfectly rejecting disturbances through high controller gains, it is an important factor when considering collaborative settings where smaller robots may be deviated from their task because of unmodeled physical intera...
-
This paper proposed an iterative learning control (ILC) scheme for deformable open-frame cable-driven parallel robots (D-CDPRs). In contrast to the straightforward inverse kinematics of the rigid frame cable-driven parallel robots (CDPRs), accurate modeling of the deformable frame poses challenges due to errors and uncertainties. To address these issues, the authors propose the use of ILC, a contr...
-
Force-free control (FFC) allows for flexible manipulator motion in response to external forces, making it a vital component of human-robot interaction (HRI). Manual intervention may cause uneven forces on the manipulator or frequencies close to the natural frequency, and mechanical resonance can occur due to the inertia of the manipulator and adjustable equivalent stiffness of the controller. This...
-
The injection of therapeutic agents into the sub-retinal space might allow improved treatment of age-related macular degeneration. Various robotic systems have been developed to achieve the required precision and, in combination with intraoperative Optical Coherence Tomography (iOCT) imaging, methods for autonomous robotic guidance have been proposed. In such systems, the robot’s cognition is ofte...
-
Colon endoscopic robots represent a promising screening modality for the visualization of colon cancers with high sensitivity. However, current colonoscopy robots are often characterized by intricate and bulky mechanical structures, which pose practical challenges when moving through the complex and narrow environment of the colon. Moreover, these robots are typically equipped with a single camera...
-
In the context of telehealth, robotic approaches have proven a valuable solution to in-person visits in remote areas, with decreased costs for patients and infection risks. In particular, in ultrasonography, robots have the potential to reproduce the skills required to acquire high-quality images while reducing the sonographer’s physical efforts. In this paper, we address the control of the intera...
-
Ultrasound (US) imaging is widely used in diagnosing and staging abdominal diseases due to its lack of non-ionizing radiation and prevalent availability. However, significant inter-operator variability and inconsistent image acquisition hinder the widespread adoption of extensive screening programs. Robotic ultrasound systems have emerged as a promising solution, offering standardized acquisition ...
-
In ophthalmic surgeries, such as vitreoretinal operations, surgeons rely on imaging systems, primarily microscopes, for real-time instrument monitoring and motion planning. However, novice surgeons struggle to extract 3D instrument positions from 2D microscope frames, necessitating extensive trial-and-error experience with the background that additional imaging modalities such as iOCT remain inacc...
-
Real-world robot task planning is intractable in part due to partial observability. A common approach to reducing complexity is introducing additional structure into the decision process, such as mixed-observability, factored states, or temporally-extended actions. We propose the locally observable Markov decision process, a novel formulation that models task-level planning where uncertainty perta...
-
Mobile manipulators have recently gained significant attention in the robotics community due to their superior potential in industrial and service applications. However, the high degree of freedom associated with mobile manipulators poses challenges in achieving real-time whole-body motion planning. To bridge the gap, this paper presents a motion planning method capable of generating high-quality,...
-
Mobile robots are nowadays frequently used for interaction tasks in the real world, e.g. for opening doors or for pick-and-place tasks. When used in real-world environments, adapting the robot controllers to uncertain contact dynamics is a significant challenge. Adaptive Model Predictive Control (AMPC) is an approach for controlling robot motions while adapting to uncertain or changing dynamics. H...
-
Mobile manipulation constitutes a fundamental task for robotic assistants and garners significant attention within the robotics community. A critical challenge inherent in mobile manipulation is the effective observation of the target while approaching it for grasping. In this work, we propose a graspability-aware mobile manipulation approach powered by an online grasping pose fusion framework tha...
-
Legged mobile manipulators are receiving much more attention. Mobile platforms can infinitely expand the workspace of robotic arms, providing more possibilities for robot application scenarios. Compared with wheeled mobile manipulators, legged mobile manipulators have higher requirements for cooperative control of legged robots and robotic arms. This work decouples the control of the robotic arm a...
-
Mobile Manipulation (MoMa) systems incorporate the benefits of mobility and dexterity, due to the enlarged space in which they can move and interact with their environment. However, even when equipped with onboard sensors, e.g., an embodied camera, extracting task-relevant visual information in unstructured and cluttered environments, such as households, remains challenging. In this work, we intro...
-
We present a framework for learning visually-guided quadruped locomotion by integrating exteroceptive sensing and central pattern generators (CPGs), i.e. systems of coupled oscillators, into the deep reinforcement learning (DRL) framework. Through both exteroceptive and proprioceptive sensing, the agent learns to coordinate rhythmic behavior among different oscillators to track velocity commands, ...
-
Decimeter scale robots in human environments are small relative to obstacles they encounter, making them prone to flipping over and needing to self-right. We present a multifaceted shell that by its geometry alone enables the hexapedal robot MediumANT to passively self-right without the need for additional sensory feedback. We designed the shell by specifying the cross-sectional geometry in the yz...
-
Legged robots are becoming increasingly agile in exhibiting dynamic behaviors such as running and jumping. Usually, such behaviors are either optimized and engineered offline (i.e. the behavior is designed for before it is needed), either through model-based trajectory optimization, or through deep learning-based methods involving millions of timesteps of simulation interactions. Notably, such off...
-
Scientists have long theorized that the cheetah’s tail contributes to its impressive maneuvrability at high speeds by stabilizing its body. This has inspired the design of several agile robots, including Dima - a wheeled platform that used cheetah-inspired inertial tail swings to better execute rapid acceleration and turning motions. Subsequent research suggests that the effectiveness of the cheet...
-
Animals possessing spinal columns often exhibit exceptional agility for highly dynamic locomotion. The spine grants the trunk with increased degrees of freedom, thereby endowing diverse postures. This paper presents the development of a robot STRAY for quadrupedal locomotion, featuring a four-degree-of-freedom spine design. Using trajectory based reinforcement learning techniques, STRAY is able to...
-
This paper presents the design and analysis of Pegasus, a quadrupedal wheeled robot grounded in biomimicry principles. Pegasus offers two distinct motion modes, including a wheeled motion and a hybrid wheeled-legged motion, enabling adaptability across various tasks and environmental conditions. The robot draws inspiration from the joint structures of quadruped animals and incorporates biomimetic ...
-
In the natural world, insects exhibit remarkable locomotion capabilities through a combination of running and jumping. However, replicating this versatile locomotion in a soft robot poses technical and design complexities. Here, we propose a dynamic soft robot named LeapRun that possesses agile locomotion and the ability to perform continuous jumping. To achieve this, a prototype soft robot (weigh...
-
Subterranean burrowing is inherently difficult for robots because of the high forces experienced as well as the high amount of uncertainty in this domain. Because of the difficulty in modeling forces in granular media, we propose the use of a novel machine-learning control strategy to obtain optimal techniques for vertical self-burrowing. In this paper, we realize a snake-like bio-inspired robot t...
-
Underwater robots are indispensable for aquatic exploration, yet their size and complexity often limit broader application. This research presents a pioneering micro autonomous underwater vehicle (µAUV) design. This robot is distinguished by its utilization of mass-produced drone components, novel jet propulsion mechanisms, and multifunctional spherical shell. Its architecture is modular, appendag...
-
In nature, fish are excellent swimmers due to their flexible and precise control of tail, which allows them to freely transform between the smooth flapping and the motion of rapid response so that they can move with dexterity. Here, inspired by the versatile motion abilities of fish, a novel robotic fish has been developed, featuring the capability of adaptable bistability. Through tuning the bist...
-
The extreme conditions of the deep sea require the use of large and expensive diving robots designed to withstand the high pressure in these depths. In order to reduce the costs for sediment sampling in the deep sea and thus facilitate the explorations of rare deep-sea ecosystems, the goal of this research is to design an alternative manipulator for deep-sea suction sampling. Instead of relying on...
-
Underwater vehicle manipulator systems (UVMS) are increasingly popular platforms for performing subsea operations that require precision manipulation. While there is high demand for fully autonomous or even semi-autonomous systems, most UVMS still require human support teams. Developing new hardware and algorithms for autonomous underwater manipulation is challenging. Simulations do not capture th...
-
This paper presents an aerial robotic platform for rapid remote elevated overhead-perching drill operations for wood health inspection. The platform features an innovative passive prismatic-gripper mechanism affixed to the aerial robot’s top, facilitating overhead drilling. The primary aim is to enhance the safety and efficiency of elevated wood structure inspection using the resistography method,...
-
Modeling the kinematics and dynamics of robotics systems with suspended loads using dual quaternions has not been explored so far. This paper introduces a new innovative control strategy using dual quaternions for UAVs with cable-suspended loads, focusing on the sling load lifting and tracking problems. By utilizing the mathematical efficiency and compactness of dual quaternions, a unified represe...
-
While the field of autonomous Uncrewed Aerial Vehicles (UAVs) has grown rapidly, most applications only focus on passive visual tasks. Aerial interaction aims to execute tasks involving physical interactions, which offers a way to assist humans in high-altitude and high-risk operations. Tactile sensors, being both cost-effective and lightweight, are capable of sensing contact information including...
-
Flapping wing air vehicles(FWAV) or ornithopters are bio-inspired aerial robots that mimic the flying principles of insects and birds. Autonomous take-off is an important capability for FWAV to enhance its performance and extend its working time, which is equipped by almost every kind of bird. As a common method of take-off for birds, jumping take-off has a great ability to adapt to different terr...
-
Aerial unperching of multirotors has received little attention as opposed to perching that has been investigated to elongate operation time. This study presents a new aerial robot capable of both perching and unperching autonomously on/from a ferromagnetic surface during flight, and a switching controller to avoid rotor saturation and mitigate overshoot during transition between free-flight and pe...
-
This paper details the simulation and experimental validation of an autonomous perching and take-off method for a multirotor unmanned aerial vehicle (UAV) using a suction cup perching mechanism on vertical surfaces. The suction cup interaction with different surface types is characterized with experimental tests to accurately model the perching manoeuvre. The resulting model is used to develop a r...
-
Field-VIO: Stereo Visual-Inertial Odometry Based on Quantitative Windows in Agricultural Open Fields
In agricultural open fields, accurate autonomous localization of robots requires long-term data correlation to reduce cumulative error. Our article presents a Stereo Visual-Inertial Odometry (VIO) system based on ORB-SLAM3 to address the malfunction of the Loop Closure Detection (LCD) methods in this environment. In this method, we first propose a concept of quantitative windows to describe the ro...
-
Wheeled mobile robots need the ability to estimate their motion and the effect of their control actions for navigation planning. In this paper, we present ST-VIO, a novel approach which tightly fuses a single-track dynamics model for wheeled ground vehicles with visual-inertial odometry (VIO). Our method calibrates and adapts the dynamics model online to improve the accuracy of forward prediction ...
-
While Global Navigation Satellite System (GNSS) is often used to provide global positioning if available, its intermittency and/or inaccuracy calls for fusion with other sensors. In this paper, we develop a novel GNSS-Visual-Inertial Navigation System (GVINS) that fuses visual, inertial, and raw GNSS measurements within the square-root inverse sliding window filtering (SRI-SWF) framework in a tigh...
-
We propose a real-time visual-inertial dense SLAM system that utilizes the online data streams from back-to-back dual fisheye cameras setup, providing 360◦ coverage of the environment. Firstly, we employ a sliding-window-based front-end to estimate real-time poses from the binocular fisheye images and IMU data. Then, we implement a lightweight panoramic depth completion network based on multi-basi...
-
Focal-Plane Sensor-Processor Arrays (FPSP)s are an emerging technology that can execute vision algorithms directly on the image sensor. Unlike conventional cameras, FPSPs perform computation on the image plane – at individual pixels – enabling high frame rate image processing while consuming low power, making them ideal for mobile robotics. FPSPs, such as the SCAMP-5, use parallel processing and a...
-
In this paper, we introduce JacobiGPU, a technique that uses a GPU to improve the efficiency of loop closure in visual-inertial SLAM systems, particularly when approximating Jacobians using the Finite Difference Method (FDM). Traditional FDM techniques often face computational overhead due to repeated perturbations in pose graphs. We address this overhead with a novel methodology, leveraging strat...
-
We present a novel optimization-based Visual-Inertial SLAM system designed for multiple partially over-lapped camera systems, named MAVIS. Our framework fully exploits the benefits of wide field-of-view from multi-camera systems, and the metric scale measurements provided by an inertial measurement unit (IMU). We introduce an improved IMU pre-integration formulation based on the exponential functi...
-
In urban environments for delivery robots, particularly in areas such as campuses and towns, many custom features defy standard road semantic categorizations. Addressing this challenge, our paper introduces a method leveraging Salient Object Detection (SOD) to extract these unique features, employing them as pivotal factors for enhanced robot loop closure and localization. Traditional geometric fe...
-
Accurate localization is an essential technology for the flexible navigation of robots in large-scale environments. Both SLAM-based and map-based localization will increase the computing load due to the increase in map size, which will affect downstream tasks such as robot navigation and services. To this end, we propose a localization system based on Block Maps (BMs) to reduce the computational l...
-
Robot localization using subsurface features captured by Ground-Penetrating Radar (GPR) complements and improves robustness over existing common sensor modalities, as subsurface features are less sensitive to weather, season and surface scene changes. Here, we propose a novel subsurface feature-based localization method that uses only GPR measurements with a known subsurface map. An efficient feat...
-
State-of-the-art techniques for monocular camera reconstruction predominantly rely on the Structure from Motion (SfM) pipeline. However, such methods often yield reconstruction outcomes that lack crucial scale information, and over time, accumulation of images leads to inevitable drift issues. In contrast, mapping methods based on LiDAR scans are popular in large-scale urban scene reconstruction d...
-
We present COIN-LIO, a LiDAR Inertial Odometry pipeline that tightly couples information from LiDAR intensity with geometry-based point cloud registration. The focus of our work is to improve the robustness of LiDAR-inertial odometry in geometrically degenerate scenarios, like tunnels or flat fields. We project LiDAR intensity returns into an image, and present a novel image processing pipeline th...
-
MegaParticles: Range-based 6-DoF Monte Carlo Localization with GPU-Accelerated Stein Particle Filter
This paper presents a 6-DoF range-based Monte Carlo localization method with a GPU-accelerated Stein particle filter. To update a massive amount of particles, we propose a Gauss-Newton-based Stein variational gradient descent (SVGD) with iterative neighbor particle search. This method uses SVGD to collectively update particle states with gradient and neighborhood information, which provides effici...
-
This paper presents a range inertial localization algorithm for a 3D prior map. The proposed algorithm tightly couples scan-to-scan and scan-to-map point cloud registration factors along with IMU factors on a sliding window factor graph. The tight coupling of the scan-to-scan and scan-to-map registration factors enables a smooth fusion of sensor ego-motion estimation and map-based trajectory corre...
-
Recognizing places from an opposing viewpoint during a return trip is a common experience for human drivers. However, the analogous robotics capability, visual place recognition (VPR) with limited field of view cameras under 180 degree rotations, has proven to be challenging to achieve. To address this problem, this paper presents Same Place Opposing Trajectory (SPOT), a technique for opposing vie...
-
The detection of traversable regions on staircases and the physical modeling constitutes pivotal aspects of the mobility of legged robots. This paper presents an onboard framework tailored to the detection of traversable regions and the modeling of physical attributes of staircases by point cloud data. To mitigate the influence of illumination variations and the overfitting due to the dataset dive...
-
Reliable estimation of terrain traversability is critical for the successful deployment of autonomous systems in wild, outdoor environments. Given the lack of large-scale annotated datasets for off-road navigation, strictly-supervised learning approaches remain limited in their generalization ability. To this end, we introduce a novel, image-based self-supervised learning method for traversability...
-
In this study, we address the off-road traversability estimation problem, that predicts areas where a robot can navigate in off-road environments. An off-road environment is an unstructured environment comprising a combination of traversable and non-traversable spaces, which presents a challenge for estimating traversability. This study highlights three primary factors that affect a robot’s traver...
-
TSCM: A Teacher-Student Model for Vision Place Recognition Using Cross-Metric Knowledge Distillation
Visual place recognition (VPR) plays a pivotal role in autonomous exploration and navigation of mobile robots within complex outdoor environments. While cost-effective and easily deployed, camera sensors are sensitive to lighting and weather changes, and even slight image alterations can greatly affect VPR efficiency and precision. Existing methods overcome this by exploiting powerful yet large ne...
-
This paper presents an accurate and fast 3D global localization method, 3D-BBS, that extends the existing branchand-bound (BnB)-based 2D scan matching (BBS) algorithm. To reduce memory consumption, we utilize a sparse hash table for storing hierarchical 3D voxel maps. To improve the processing cost of BBS in 3D space, we propose an efficient roto-translational space branching. Furthermore, we devi...
-
Dynamic objects diversify the distribution of point cloud in the map, degrading the performance of the robotic downstream tasks. To address this problem, we present a novel real-time dynamic instance-aware static mapping framework called DynaInsRemover, which exploits the geometric discrepancies between instances to efficiently remove dynamic objects and preserve more details of static map. It con...
-
We tackle the problem of learning an implicit scene representation for 3D instance segmentation from a sequence of posed RGB images. Towards this, we introduce 3DIML, a novel framework that efficiently learns a label field that may be rendered from novel viewpoints to produce view-consistent instance segmentation masks. 3DIML significantly improves upon training and inference runtimes of existing ...
-
In robotic applications such as bin-picking or block-stacking, learned predictive models have been developed for manipulation of objects with varying but known dynamic properties (e.g., mass distributions and friction coefficients). When a robot encounters a new object, these properties are often difficult to observe and must be inferred through interaction, which can be expensive in both inferenc...
-
Grasping is one of the most fundamental skills for humans to interact with objects. However, it remains a challenging problem for anthropomorphic hands, due to the lack of object affordance understanding and high-dimensional grasp planning. In this work, we propose an anthropomorphic hand grasping framework to learn realistic and reasonable grasps in cluttered scenes, which tackles the problem in ...
-
We present FuncGrasp, a framework that can infer dense yet reliable grasp configurations for unseen objects using one annotated object and single-view RGB-D observation via categorical priors. Unlike previous works that only transfer a set of grasp poses, FuncGrasp aims to transfer infinite configurations parameterized by an object-centric continuous grasp function across varying instances. To eas...
-
Grasp Quality Networks are important components of grasping-capable autonomous robots, as they allow them to evaluate grasp candidates and select the one with highest chance of success. The widespread use of pick-and-place robots and Grasp Quality Networks raises the question of whether such systems are vulnerable to adversarial attacks, as that could lead to large economic damage. In this paper w...
-
Robotic manipulation is a key enabler for automation in the fulfillment logistics sector. Such robotic systems require perception and manipulation capabilities to handle a wide variety of objects. Existing systems either operate on a closed set of objects or perform object-agnostic manipulation which lacks the capability for deliberate and reliable manipulation at scale. Object identification (ID)...
-
Various sensorized grippers have been developed to handle delicate objects safely. These grippers have sensors mounted on their fingers’ surface that provide direct force measurements. However, multiple sensors are often required on one finger, leading to significant sensor placement and wire routing complexity. Finger-based sensors are limited to sensing external gripping force, and fingers canno...
-
Industrial robots are capable of moving at high speed. Each time they come into contact with their environment, e.g. to pick up an object, they decelerate to a near standstill. A solution involving a compliant pneumatic gripper and adapted trajectory plan is presented to initiate contact at a higher speed while remaining within hardware limits. By adding overload clutches in either the robot arm o...
-
In this paper, we use impact-induced acceleration in conjunction with periodic stick-slip to successfully and quickly transport parts vertically against gravity. We show analytically that vertical vibratory transport is more difficult than its horizontal counterpart, and provide guidelines for achieving optimal vertical vibratory transport of a part. Namely, such a system must be capable of quickl...
-
This paper presents a novel Bionic Soft Finger (BSF) that aims to overcome the limitations of conventional rigid manipulators in terms of adaptability and safety, as well as the challenges faced by soft hands regarding carrying capacity and stability. The BSF design uses a hybrid variable stiffness mechanism combining memory alloy actuators with particle jamming to achieve the desired bending angl...
-
Small-scale robots hold significant promise in the field of minimally invasive surgery (MIS). In this paper, we present a miniature magnetic gripper and develop a data-driven kinematic model. The gripper comprises four fingers, wherein each finger has a maximum size not exceeding 3mm, 4mm and 5.5mm in three dimensions. By integrating permanent magnets and elastic ropes as internal actuation elemen...
-
This paper presents a novel approach to designing, a low-cost gripper that is highly repeatable and functionally integrated. The gripper is optimized to compensate for gripping errors with particular consideration to potential challenges of articulated robots. The primary design goal is to achieve maximum repeatability during the gripping and releasing stages of a pick-and-place process for a chip...
-
While robotic assembly is a well researched topic, recycling and disassembly of products are also becoming ever more important as we transition to a more sustainable economy. In disassembly, we are typically only interested in a subset of product parts, which opens the possibility of using destructive processes such as tearing, cutting, or milling to speed up the disassembly. Currently, such destr...
-
We introduce IFFNeRF to estimate the six degrees-of-freedom (6DoF) camera pose of a given image, building on the Neural Radiance Fields (NeRF) formulation. IFFNeRF is specifically designed to operate in real-time and eliminates the need for an initial pose guess that is proximate to the sought solution. IFFNeRF utilizes the Metropolis-Hasting algorithm to sample surface points from within the NeRF...
-
Combining motion prediction in LiDAR-based 3D object detection is an effective method for improving overall accuracy, especially the downstream autonomous driving tasks. The recent development of low-cost LiDARs (e.g. Livox LiDAR) enables us to explore such 4D perception systems with a lower budget and higher performance. In this paper, we propose a 4D object detector, VeloVox, to establish accura...
-
Environmental perception tasks such as object detection and free space detection based on 3+1D radar severely suffer from the disorder and sparsity of point cloud. To tackle this problem, we propose a novel Multi-Task Radar-based Single Stage Detector, termed MTRadSSD, where we adopt instance-aware sampling strategies to discover multi-class road users and propose an occupancy map tool based on ke...
-
Recent camera-based 3D object detection is limited by the precision of transforming from image to 3D feature spaces, as well as the accuracy of object localization within the 3D space. This paper aims to address such a fundamental problem of camera-based 3D object detection: How to effectively learn depth information for accurate feature lifting and object localization. Different from previous met...
-
Despite the success of deep learning-based object detection methods in recent years, it is still challenging to make the object detector reliable in adverse weather conditions such as rain and snow. For the robust performance of object detectors, unsupervised domain adaptation has been utilized to adapt the detection network trained on clear weather images to adverse weather images. While previous...
-
In the expanding landscape of AI-enabled robotics, robust quantification of predictive uncertainties is of great importance. Three-dimensional (3D) object detection, a critical robotics operation, has seen significant advancements; however, the majority of current works focus only on accuracy and ignore uncertainty quantification. Addressing this gap, our novel study integrates the principles of c...
-
While showing promising results, recent RGB-D camera-based category-level object pose estimation methods have restricted applications due to the heavy reliance on depth sensors. RGB-only methods provide an alternative to this problem yet suffer from inherent scale ambiguity stemming from monocular observations. In this paper, we propose a novel pipeline that decouples the 6D pose and size estimati...
-
Category-level object pose estimation demonstrates robust generalization capabilities that benefit robotics applications. However, exclusive reliance on RGB images without leveraging any 3D information introduces ambiguity in the translation and size of objects, leading to suboptimal performance. In this paper, we propose a framework for category-level pose estimation from a single RGB image in an...
-
Large language models (LLMs) are accelerating the development of language-guided robot planners. Meanwhile, symbolic planners offer the advantage of interpretability. This paper proposes a new task that bridges these two trends, namely, multimodal planning problem specification. The aim is to generate a problem description (PD), a machine-readable file used by the planners to find a plan. By gener...
-
Accurate disturbance estimation is essential for safe robot operations. The recently proposed neural moving horizon estimation (NeuroMHE), which uses a portable neural network to model the MHE’s weightings, has shown promise in further pushing the accuracy and efficiency boundary. Currently, NeuroMHE is trained through gradient descent, with its gradient computed recursively using a Kalman filter....
-
Generalizable articulated object manipulation is essential for home-assistant robots. Recent efforts focus on imitation learning from demonstrations or reinforcement learning in simulation, however, due to the prohibitive costs of real-world data collection and precise object simulation, it still remains challenging for these works to achieve broad adaptability across diverse articulated objects. ...
-
Motivated by the substantial achievements of Large Language Models (LLMs) in the field of natural language processing, recent research has commenced investigations into the application of LLMs for complex, long-horizon sequential task planning challenges in robotics. LLMs are advantageous in offering the potential to enhance the generalizability as task-agnostic planners and facilitate flexible in...
-
Olfaction sensing in autonomous robotics faces challenges in dynamic operations, energy efficiency, and edge processing. It necessitates a machine learning algorithm capable of managing real-world odor interference, ensuring resource efficiency for mobile robotics, and accurately estimating gas features for critical tasks such as odor mapping, localization, and alarm generation. This paper introdu...
-
Scene flow estimation determines a scene’s 3D motion field, by predicting the motion of points in the scene, especially for aiding tasks in autonomous driving. Many networks with large-scale point clouds as input use voxelization to create a pseudo-image for real-time running. However, the voxelization process often results in the loss of point-specific features. This gives rise to a challenge in ...
-
Effective coordination is crucial for motion control with reinforcement learning, especially as the complexity of agents and their motions increases. However, many existing methods struggle to account for the intricate dependencies between joints. We introduce CoordiGraph, a novel architecture that leverages subequivariant principles from physics to enhance coordination of motion control with rein...
-
This paper presents a large-scale climbing robot that employs a parallel mechanism with three translational degrees of freedom as its locomotion method. Using a robot frame having a triangular pyramid shape, the robot provides a good stability during the locomotion and task execution. Three suction cups, called the perimeter cups, are attached to the vertices of the robot’s pyramid base, whereas t...
-
We introduce a Cable Grasping-Convolutional Neural Network (CG-CNN) designed to facilitate robust cable grasping in cluttered environments. Utilizing physics simulations, we generate an extensive dataset that mimics the intricacies of cable grasping, factoring in potential collisions between cables and robotic grippers. We employ the Approximate Convex Decomposition technique to dissect the non-co...
-
In this paper, a numerically efficient flexible control scheme for the absolute accuracy of industrial robots is presented and experimentally validated. A model-based controller that leverages all typically available parameters is combined with an online path iterative learning controller (ILC). The ILC law is employed to compensate for the unknown residual error dynamics caused by elastic and tra...
-
Surface treatment tasks such as grinding, sanding or polishing are a vital step of the value chain in many industries, but are notoriously challenging to automate. We present RoboGrind, an integrated system for the intuitive, interactive automation of surface treatment tasks with industrial robots. It combines a sophisticated 3D perception pipeline for surface scanning and automatic defect identif...
-
Robots need to have a memory of previously observed, but currently occluded objects to work reliably in realistic environments. We investigate the problem of encoding object-oriented memory into a multi-object manipulation reasoning and planning framework. We propose DOOM and LOOM, which leverage transformer relational dynamics to encode the history of trajectories given partial-view point clouds ...
-
Fabric manipulation is a long-standing challenge in robotics due to the enormous state space and complex dynamics. Learning approaches stand out as promising for this domain as they allow us to learn behaviours directly from data. Most prior methods however rely heavily on simulation, which is still limited by the large sim-to-real gap of deformable objects or rely on large datasets. A promising a...
-
Robotic autonomous grasp requires the system to perform multiple functions such as gripper and robot control, making it a task with hybrid output nature. Existing methods based on closed-loop deep reinforcement learning rely on external models for termination evaluation. To achieve more effective grasp for novel objects, we propose a new autonomous grasp control scheme termed HAGrasp that consider...
-
Robotic push-grasping in densely cluttered environments presents significant challenges due to unbalanced synergy and redundancy between both actions, leading to decreased grasp efficiency. In this paper, a novel double-critic deep reinforcement learning framework is introduced to optimize the push-grasping synergy for robotic manipulation in such environments, aiming to significantly reduce pre-g...
-
Shape servoing, a robotic task dedicated to controlling objects to desired goal shapes, is a promising approach to deformable object manipulation. An issue arises, however, with the reliance on the specification of a goal shape. This goal has been obtained either by a laborious domain knowledge engineering process or by manually manipulating the object into the desired shape and capturing the goal...
-
What makes generalization hard for imitation learning in visual robotic manipulation? This question is difficult to approach at face value, but the environment from the perspective of a robot can often be decomposed into enumerable factors of variation, such as the lighting conditions or the placement of the camera. Empirically, generalization to some of these factors have presented a greater obst...
-
In this paper, we propose a transformer-based architecture for predicting contact forces during a physical human-robot interaction. Our Neural Network is composed of two main parts: a Multi-Layer Perceptron called Transducer and a Transformer. The former estimates, based on the kinematic data from a motion capture suit, the current contact forces. The latter predicts – taking as input the same kin...
-
Vision-based human-to-robot handover is an important and challenging task in human-robot interaction. Recent work has attempted to train robot policies by interacting with dynamic virtual humans in simulated environments, where the policies can later be transferred to the real world. However, a major bottleneck is the reliance on human motion capture data, which is expensive to acquire and difficu...
-
Collaborative manipulation task often requires negotiation using explicit or implicit communication. An important example is determining where to move when the goal destination is not uniquely specified, and who should lead the motion. This work is motivated by the ability of humans to communicate the desired destination of motion through back-and-forth force exchanges. Inherent to these exchanges...
-
In this paper, we analyze the possibilities offered by Deep Learning State-of-the-Art architectures such as Transformers and Visual Transformers in generating a prediction of the human’s force in a Human-Robot collaborative object transportation task at a middle distance. We outperform our previous predictor by achieving a success rate of 93.8% in testset and 90.9% in real experiments with 21 volu...
-
Mobile physical human-robot collaboration (pHRC) using collaborative robots (cobots) and mobile robots has attracted much research attention. Many researchers have focused on improving the control performance to comply with human intentions. However, a problem that generally exists with mobile pHRC but often gets neglected is the impact of non-rigid components e.g. deformable tyres, suspension sys...
-
Preserving and encouraging mobility in the elderly and adults with chronic conditions is of paramount importance. However, existing walking aids are either inadequate to provide sufficient support to users’ stability or too bulky and poorly maneuverable to be used outside hospital environments. In addition, they all lack adaptability to individual requirements. To address these challenges, this pa...
-
Ankle exoskeletons can assist the ankle joint and reduce the metabolic cost of walking. However, many existing ankle exoskeletons constrain the natural 3 degrees of freedom (DoF) of the ankle to limit the exoskeleton’s weight and mechanical complexity, thereby compromising comfort and kinematic compatibility with the user.This paper presents a novel ankle exoskeleton frame design that allows for 3...
-
Traditionally, powered exoskeletons have predominantly featured a back-enveloping design due to its simplicity in both implementation and user donning. However, this design results in a backward shift of the center of mass (CoM) in the sagittal plane. This paper identifies the limitations of existing design approaches and determines the optimal anterior-posterior (A/P) CoM position considering fac...
-
Assistive devices, such as exoskeletons and prostheses, have revolutionized the field of rehabilitation and mobility assistance. Efficiently detecting transitions between different activities, such as walking, stair ascending and descending, and sitting, is crucial for ensuring adaptive control and enhancing user experience. We present an approach for real-time transition detection, aimed at optim...
-
Balance loss is a significant challenge in lower-limb exoskeleton applications, as it can lead to potential falls, thereby impacting user safety and confidence. We introduce a control framework for omnidirectional recovery step planning by online optimization of step duration and position in response to external forces. We map the step duration and position to a human-like foot trajectory, which i...
-
Optimized assistance patterns have produced the greatest exoskeleton benefits to energy expenditure of any strategy to date. This strategy may be effective due to the customization of the applied torque profiles to the user as well as the locomotion condition; however, it is currently unclear how sensitive participants are to their unique torque profile. To investigate, we applied previously optim...
-
Powered lower-limb prostheses have shown promise in helping individuals with amputation regain functionality that passive prostheses cannot provide. However, the best method for controlling these devices in coordination with their users is still an open research topic. While powered devices can replicate normative joint kinematics and kinetics, active control also holds the potential to shape syst...
-
Interactive visual grounding in Human-Robot Interaction (HRI) is challenging yet practical due to the inevitable ambiguity in natural languages. It requires robots to disambiguate the user’s input by active information gathering. Previous approaches often rely on predefined templates to ask disambiguation questions, resulting in performance reduction in realistic interactive scenarios. In this pap...
-
Zero-Shot Object Navigation (ZSON) enables agents to navigate towards open-vocabulary objects in unknown environments. The existing works of ZSON mainly focus on following individual instructions to find generic object classes, neglecting the utilization of natural language interaction and the complexities of identifying user-specific objects. To address these limitations, we introduce Zero-shot I...
-
Interactive Object Grasping (IOG) is the task of identifying and grasping the desired object via human-robot natural language interaction. Current IOG systems assume that a human user initially specifies the target object’s category (e.g., bottle). Inspired by pragmatics, where humans often convey their intentions by relying on context to achieve goals, we introduce a new IOG task, Pragmatic-IOG, ...
-
Electrical Impedance Tomography (EIT)-based tactile sensors offer durability, scalability, and cost-effective manufacturing. However, simultaneously reconstructing force and shape from boundary measurements remains challenging due to EIT’s inherent location dependencies and image artifacts. This study presents a model-driven multimodal convolutional neural network (MM-CNN) for joint EIT-based forc...
-
Socially assistive robots are potentially to be integrated with human daily lives in the near future, and expected to be able to improve group dynamics when interacting with groups of people in social settings. In this paper, we developed a system with desktop robot Haru to assist group discussions. The system consists of three modules: a dialogue assistance module which facilitates Haru to speak ...
-
Industrial exoskeletons are a potential solution for reducing work-related musculoskeletal disorders during carrying or lifting tasks. Having sensors, electrical/pneumatic actuators, and control systems, active exoskeletons present a more versatile control system because it is possible to select different assistive strategies based on the performed task. From this perspective, human-machine intera...
-
Ensuring robust tracking of controllers’ movement is critical for human-robot interaction in virtual reality (VR) scenarios. This paper proposes a robust tracking algorithm based on a novel wearable ring-shaped controller equipped with an inertial measurement unit (IMU) and a light-emitting diode (LED). This novel controller design allows users to free up their hands for more immersive experiences...
-
Tactile sensing plays a pivotal role in human perception and manipulation tasks, allowing us to intuitively understand task dynamics and adapt our actions in real time. Transferring such tactile intelligence to robotic systems would help intelligent agents understand task constraints and accurately interpret the dynamics of both the objects they are interacting with and their own operations. While...
-
In this work, we use MEMS microphones as vibration sensors to simultaneously classify texture and estimate contact position and velocity. Vibration sensors are an important facet of both human and robotic tactile sensing, providing fast detection of contact and onset of slip. Microphones are an attractive option for implementing vibration sensing as they offer a fast response and can be sampled qu...
-
Tactile exploration plays a crucial role in understanding object structures for fundamental robotics tasks such as grasping and manipulation. However, efficiently exploring such objects using tactile sensors is challenging, primarily due to the large-scale unknown environments and limited sensing coverage of these sensors. To this end, we present AcTExplore, an active tactile exploration method dr...
-
We present a novel controller design on a robotic locomotor that combines an aerial vehicle with a spring-loaded leg. The main motivation is to enable the terrestrial locomotion capability on aerial vehicles so that they can carry heavy loads: heavy enough that flying is no longer possible, e.g., when the thrust-to-weight ratio (TWR) is small. The robot is designed with a pogo-stick leg and a quad...
-
We present a minimal phase oscillator model for learning quadrupedal locomotion. Each of the four oscillators is coupled only to itself and its corresponding leg through local feedback of the ground reaction force, which can be interpreted as an observer feedback gain. We interpret the oscillator itself as a latent contact state-estimator. Through a systematic ablation study, we show that the comb...
-
Balancing oneself using the spine is a physiological alignment of the body posture in the most efficient manner by the muscular forces for mammals. For this reason, we can see many disabled quadruped animals can still stand or walk even with three limbs. This paper investigates the optimization of dynamic balance during trot gait based on the spatial relationship between the center of mass (CoM) a...
-
The capability of a quadruped robot to negotiate obstacles is tightly connected to its leg workspace and joint torque limits. When facing terrain where the height of obstacles is close to the leg length, the locomotion robustness and safety are reduced since more dynamic motions are required to traverse it. In this paper, we introduce a new mechanism called the Carpal-Claw, which enables quadruped...
-
We present SpaceHopper, a three-legged, small-scale robot designed for future mobile exploration of asteroids and moons. The robot weighs 5.2 kg and has a body size of 245 mm while using space-qualifiable components. Furthermore, SpaceHopper’s design and controls make it well-adapted for investigating dynamic locomotion modes with extended flight-phases. Instead of gyroscopes or fly-wheels, the sy...
-
Learning a locomotion policy for quadruped robots has traditionally been constrained to a specific robot morphology, mass, and size. The learning process must usually be repeated for every new robot, where hyperparameters and reward function weights must be re-tuned to maximize performance for each new system. Alternatively, attempting to train a single policy to accommodate different robot sizes,...
-
This paper presents a safety-critical approach to the coordination of robots in dynamic environments. To this end, we leverage control barrier functions (CBFs) with the forward reachable set to guarantee the safe coordination of the robots while preserving a desired trajectory via a layered controller. The top-level planner generates a safety-ensured trajectory for each agent, accounting for the d...
-
This paper presents a safety-critical control framework tailored for quadruped robots equipped with a roller arm, particularly when performing locomotive tasks such as autonomous robotic inspection in complex, multi-tiered environments. In this study, we consider the problem of operating a quadrupedal robot in distillation columns, locomoting on column trays and transitioning between these trays w...
-
Remote center of motion (RCM) describes a robot with a rod-like end-effector operating through a hole in the interface separating the internal space from the external space. Considering that the control of RCM may be influenced by perturbations (noises) and that the end-effector is frequently replaced to complete different tasks, the structural information related to the robot manipulator and its ...
-
Retractable worm robots possess hyper-flexibility, allowing them to work in confined spaces that are difficult for humans. However, the spatial locomotion control of these robots remains challenging due to the robots’ large degrees of freedom. To address this challenge, we propose a phase synthesis (PS) scheme for retractable worm robots. The scheme combines an undulating gait inspired by caterpil...
-
To achieve high-accuracy manipulation in the presence of unknown disturbances, we propose two novel efficient and robust motion control schemes for high-dimensional robot manipulators. Both controllers incorporate an unknown system dynamics estimator (USDE) to estimate disturbances without requiring acceleration signals and the inverse of inertia matrix. Then, based on the USDE framework, an adapt...
-
To enhance the precision of coarse long-stroke motion axes, complementary short-stroke fine positioning stages are usually introduced. Being mechanically attached, the motion of the combined positioning stages needs to be controlled and synchronized. Therefore, typically suitable model-based controllers of fine stages are designed according to the sophisticated models and identification techniques...
-
Disturbance observer (DOB) is a well-known two-loop control structure that imparts robustness to a controller with a simple implementation. As a nonlinear DOB for the robotic systems, we proposed so-called nonlinear robust internal-loop compensator (NRIC) framework in our previous work. In this paper, we further extend the NRIC in such a way that an optimization scheme can be embedded in the contr...
-
Numerous studies have emphasized the application of autonomous intelligence in human-robot shared control to enhance surgical convenience and efficiency. However, the neglect of human dominance may reduce surgical safety. This paper developed a safety-enhanced human-robot shared control method by intelligently allocating control authority, with the surgeon remaining the leader during the surgical ...
-
Osteotomy holds a pivotal position among the fundamental procedures in craniomaxillofacial (CMF) surgery. However, there are inherent challenges and risks associated with ensuring the recuperation of occlusion, safeguarding the facial nerves and blood vessels, as well as preserving facial aesthetics. In this study, a non-invasive image-to-patient registration method for navigation/robotic CMF surg...
-
Six-axis force/moment (6-A F/M) sensors make surgical robots effectively sense intraoperative force feedback and drilling status information, reducing the operating challenges and psychological burden of doctors, which also improves the quality and safety of surgery. However, it is difficult for current commercial electrical 6-A F/M sensors to adapt the electromagnetic environment in the operating...
-
Prior robot-assisted cranium-milling studies only considered controlling the force in the skull’s vertical direction and neglected the milling cutter’s feed force. Additionally, achieving stable force control in multiple directions is challenging for robots due to the uneven skull surface. Here a hybrid admittance control algorithm incorporating a model-free adaptive nonlinear force control and fu...
-
One issue of robotic microsurgery is that compared to manual surgery, the operation time tends to be longer due to high motion scaling. To address this issue, we developed a new controller that can provide the accuracy required for microsurgery without a high scaling factor by utilizing fingertip and wrist motions. Also, for the better outcome of surgery, the proposed controller has a force feedba...
-
Skull base surgery is a demanding field in which surgeons operate in and around the skull while avoiding critical anatomical structures including nerves and vasculature. While image-guided surgical navigation is the prevailing standard, limitation still exists requiring personalized planning and recognizing the irreplaceable role of a skilled surgeon. This paper presents a collaboratively controll...
-
The increasing awareness of public health issues has highlighted the need for effective disinfection of crowded indoor public areas, leading to the development of automated disinfection robots. However, most of the existing robots spray disinfectant in all areas, and they are still immature to navigate in densely populated environments. Hence, in this paper, we design a new disinfection robotic sy...
-
Lymphadenectomy generally accompanies various oncology surgeries to remove infected cancer cells. However, there are two limitations in robot-assisted lymphadenectomy: 1) lymph nodes are not visible during operation since they are hidden by the superficial fat layer; 2) intra-operative bleeding may occur during lymph node removal caused by collisions between surgical instruments and delicate blood...
-
This paper presents the hardware design and development of Hyblock, a modular hydraulic robot for heavy-duty application such as construction. The robot is equipped with a simple docking mechanism called a C-type expansion dowel and a novel hydraulic circuit MHSB that matches the modular structure. In this paper, we first report on the design of the robot hardware including the dowel and hydraulic...
-
We present a robotic system for the assembly of 3D discrete lattice structures in which the robots are able to self-reproduce, such that the assembly system may scale its own parallelization. Robots and structures are made from a set of compatible building blocks, or voxels, which can be assembled and reassembled into more complex structures. Robotic modules are made by combining actuators with a ...
-
We present LiSTA (LiDAR Spatio-Temporal Analysis), a system to detect probabilistic object-level change over time using multi-mission SLAM. Many applications require such a system, including construction, robotic navigation, long-term autonomy, and environmental monitoring. We focus on the semi-static scenario where objects are added, subtracted, or changed in position over weeks or months. Our sy...
-
We present a scalable combined localization infrastructure deployment and task planning algorithm for underwater assembly. Infrastructure is autonomously modified to suit the needs of manipulation tasks based on an uncertainty model on the infrastructure’s positional accuracy. Our uncertainty model can be combined with the noise characteristics from multiple sensors. For the task planning problem,...
-
Loading multiple different materials with wheel loaders is a challenging task because various materials require different loading techniques. It’s, therefore, difficult to find a single controller capable of handling them all. One solution is to use a base controller and fine-tune it for different materials. Reinforcement Learning (RL) automates this process without the need for collecting additio...
-
In this work, we achieved a self-takeoff of an eagle-scale flapping-wing robot for the first time. Inspired by the takeoff process of Ospreys, we propose a bio-inspired takeoff strategy, then discuss the dynamic model and the requirements for self-takeoff. Based on the requirements of flight strategy, we designed a system with two parts, including a flapping-wing aircraft with a wingspan of 1.8m a...
-
A model-based approach for lateral maneuvering of flapping wing UAVs in closed spaces is presented. Bird-size ornithopters do not have asymmetric actuation in the wing due to mechanical complexity, so they rely upon the tail for lateral maneuvering. The prototype E-Flap can deflect the vertical tail to make maneuvers out of the longitudinal plane. This work defines simplified equations for the ste...
-
An increasing number of underwater robots inspired by Carangidae are developed, which is characterized by high efficiency and flexibility. However, estimating the swimming direction of these robotic fish is challenging due to the constant swinging of the head during movement, which complicates precise control. In this study, we installed two low-cost inertial measurement unit (IMU) sensors separat...
-
Fish schools present high-efficiency group behaviors to collective migration and dynamic escape from the predator through simple individual interactions. The purpose of this research is to infuse swarm robots with "fish-like" intelligence that will enable safe navigation and efficient cooperation, and successful completion of escape tasks in changing environments. In this paper, a novel fish-inspi...
-
Robust underwater motion planning of autonomous underwater vehicles (AUVs) in dynamic cluttered environments is a problem that has yet to be addressed in depth. Due to advances in technology and computational capacity, AUVs are expected to operate safely and autonomously in increasingly challenging environments, necessitating methods that are able to safely navigate robots in real-time. Though, mo...
-
In this work, we address the problem of real-time dense depth estimation from monocular images for mobile underwater vehicles. We formulate a deep learning model that fuses sparse depth measurements from triangulated features to improve the depth predictions and solve the problem of scale ambiguity. To allow prior inputs of arbitrary sparsity, we apply a dense parameterization method. Our model ex...
-
In this paper, we address the challenging problem of data association for underwater SLAM through a novel method for sonar image correspondence using learned features. We introduce SONIC (SONar Image Correspondence), a pose-supervised network designed to yield robust feature correspondence capable of withstanding viewpoint variations. The inherent complexity of the underwater environment stems fro...
-
In this paper, we present CaveSeg - the first visual learning pipeline for semantic segmentation and scene parsing for AUV navigation inside underwater caves. We address the problem of scarce annotated training data by preparing a comprehensive dataset for semantic segmentation of underwater cave scenes. It contains pixel annotations for important navigation markers (e.g. caveline, arrows), obstac...
-
We present a novel system which blends multiple distinct sensing modalities in audio-visual surveys to assist marine biologists in collecting datasets for understanding the ecological relationship of fish and other organisms with their habitats on and around coral reefs. Our system, designed for the CUREE AUV, uses four hydrophones to determine the bearing to biological sound sources through beamf...
-
Point cloud compression is an essential technology for efficient storage and transmission of 3D data. Previous methods usually use hierarchical tree data structures for encoding the spatial sparseness of point clouds. However, the node context within the tree is not fully discovered since the feature space among nodes varies significantly. To address this problem, we innovatively represent the LiD...
-
The ability to fast climb and descend is crucial for Unmanned Aerial Vehicle (UAV) applications in the mountains. The slower descent speed will affect the UAV’s working efficiency in reaching the rescue area. However, during the fast descent of the rotorcraft, a chaotic flow field rampages as the rotorcraft falls into its wake flow. This is known as the vortex ring. Therefore, the safe descent vel...
-
Aerial robots show significant potential for forest canopy research and environmental monitoring by providing data collection capabilities at high spatial and temporal resolutions. However, limited flight endurance hinders their application. Inspired by natural perching behaviours, we propose a multi-modal aerial robot system that integrates tensile perching for energy conservation and a suspended...
-
This work focuses on understanding and identifying the drag forces applied to a rotary-wing Micro Aerial Vehicle (MAV). We propose a lumped drag model that concisely describes the aerodynamical forces the MAV is subject to, with a minimal set of parameters. We only rely on commonly available sensor information onboard a MAV, such as accelerometer data, pose estimate, and throttle commands, which m...
-
This work validates through flight tests a previously developed wide-envelope singularity-free aerodynamic framework, called ϕ-theory, for modeling dual-engine tail-sitting flying-wing vehicles for optimization-based control. The ϕ-theory methodology imposes a specific geometry on aerodynamic coefficients that leads to polynomial differential equations of motion amenable to semidefinite programmin...
-
Electroaerodynamic (EAD) propulsion, where thrust is produced by collisions between electrostatically-accelerated ions and neutral air, is a potentially transformative method for indoor flight owing to its silent and solid-state nature. Like rotors, EAD thrusters exhibit changes in performance based on proximity to surfaces. Unlike rotors, they have no fragile and quickly spinning parts that have ...
-
A risk aware UAS path planning methodology is proposed using monetary value as the sole cost metric. A third party ground risk model is used to generate a non-uniform costmap for a modified A* heuristic search. The Value of a Prevented Fatality provides a basis to convert fatality risk to monetary value terms as a Human Value at Risk (HVaR) measure. Additional operating and UAS Capital Value at Ri...
-
Integrated aerial Platforms (IAPs), comprising multiple aircrafts, are typically fully actuated and hold significant potential for aerial manipulation tasks. Differing from a multiple aerial swarm, the aircrafts within the IAP are interconnected, presenting promising opportunities for enhancing localization. Incorporating the physical constraints of these multiple aircrafts to improve the accuracy...
-
Multi-robot global localization (MR-GL) with unknown initial positions in a large scale environment is a challenging task. The key point is the data association between different robots’ viewpoints. It also makes traditional Appearance-based localization methods unusable. Recently, researchers have utilized the object’s semantic invariance to generate a semantic graph to address this issue. Howeve...
-
Real-time dense reconstruction using Unmanned Aerial Vehicle (UAV) is becoming increasingly popular in large-scale rescue and environmental monitoring tasks. However, due to the energy constraints of a single UAV, the efficiency can be greatly improved through the collaboration of multi-UAVs. Nevertheless, when faced with unknown environments or the loss of Global Navigation Satellite System (GNSS...
-
Collaborative state estimation using different heterogeneous sensors is a fundamental prerequisite for robotic swarms operating in GPS-denied environments, posing a significant research challenge. In this paper, we introduce a centralized system to facilitate collaborative LiDAR-ranging-inertial state estimation, enabling robotic swarms to operate without the need for anchor deployment. The system...
-
Image matching is still challenging in such scenes with large viewpoints or illumination changes or with low textures. In this paper, we propose a Transformer-based pseudo 3D image matching method. It upgrades the 2D features extracted from the source image to 3D features with the help of a reference image and matches to the 2D features extracted from the destination image by the coarse-to-fine 3D...
-
State-of-the-art V-SLAM pipelines utilize robust cost functions and outlier rejection techniques to remove incorrect correspondences. However, these methods are typically fine-tuned to overfit certain benchmarks and struggle to adapt effectively to changes in the application domain or environmental conditions. This renders them impractical for many robotic applications in which robustness in a wid...
-
One of the key challenges of visual Simultaneous Localization and Mapping (SLAM) in large-scale environments is how to effectively use global localization to correct the cumulative errors from long-term tracking. This challenge presents itself in two main aspects: first, the difficulty for robots in revisiting previous locations to perform loop closure, and second, the considerable memory resource...
-
The inherent scale ambiguity issue greatly limits the performance of monocular visual odometry. In recent years, a variety of methods have been proposed for self-supervised learning of ego-motion and depth estimation, incorporating specifically designed scale-consistency constraints that utilize estimated depth as a reference. However, these existing methods neglect the influence of the depth unce...
-
Perceptual aliasing and weak textures pose significant challenges to the task of place recognition, hindering the performance of Simultaneous Localization and Mapping (SLAM) systems. This paper presents a novel model, called UMF (standing for Unifying Local and Global Multimodal Features) that 1) leverages multi-modality by cross-attention blocks between vision and LiDAR features, and 2) includes ...
-
LiDAR-based localization is valuable for applications like mining surveys and underground facility maintenance. However, existing methods can struggle when dealing with uninformative geometric structures in challenging scenarios. This paper presents RELEAD, a LiDAR-centric solution designed to address scan-matching degradation. Our method enables degeneracy-free point cloud registration by solving...
-
Visual Place Recognition (VPR) is critical for navigation and loop closure in autonomous driving tasks, mitigating the impact of shift errors caused by dynamic changes in the environment. Due to the limited ability of backbone networks and extreme environmental changes, current methods fail to capture foundational semantic details that include the distinctive attributes for unique place identifica...
-
We study the problem of aligning a video that captures a local portion of an environment to the 2D LiDAR scan of the entire environment. We introduce a method (VioLA) that starts with building a semantic map of the local scene from the image sequence, then extracts points at a fixed height for registering to the LiDAR map. Due to reconstruction errors or partial coverage of the camera scan, the re...
-
Autonomous driving has traditionally relied heavily on costly and labor-intensive High Definition (HD) maps, hindering scalability. In contrast, Standard Definition (SD) maps are more affordable and have worldwide coverage, offering a scalable alternative. In this work, we systematically explore the effect of SD maps for real-time lane-topology understanding. We propose a novel framework to integr...
-
Neural implicit surface representations are currently receiving a lot of interest as a means to achieve high-fidelity surface reconstruction at a low memory cost, compared to traditional explicit representations. However, state-of-the-art methods still struggle with excessive memory usage and non-smooth surfaces. This is particularly problematic in large-scale applications with sparse inputs, as i...
-
Large-scale dense mapping is vital in robotics, digital twins, and virtual reality. Recently, implicit neural mapping has shown remarkable reconstruction quality. However, incremental large-scale mapping with implicit neural representations remains problematic due to low efficiency, limited video memory, and the catastrophic forgetting phenomenon. To counter these challenges, we introduce the Robo...
-
Camera relocalization is a crucial problem in computer vision and robotics. Recent advancements in neural radiance fields (NeRFs) have shown promise in synthesizing photo-realistic images. Several works have utilized NeRFs for refining camera poses, but they do not account for lighting changes that can affect scene appearance and shadow regions, causing a degraded pose optimization process. In thi...
-
This paper presents a novel framework to learn a concise geometric primitive representation for 3D point clouds. Different from representing each type of primitive individually, we focus on the challenging problem of how to achieve a concise and uniform representation robustly. We employ quadrics to represent diverse primitives with only 10 parameters and propose the first end-to-end learning-base...
-
Mapping plays a crucial role in location and navigation within automatic systems. However, the presence of dynamic objects in 3D point cloud maps generated from scan sensors can introduce map distortion and long traces, thereby posing challenges for accurate mapping and navigation. To address this issue, we propose ERASOR++, an enhanced approach based on the Egocentric Ratio of Pseudo Occupancy fo...
-
3D object-level mapping is a fundamental problem in robotics, which is especially challenging when object CAD models are unavailable during inference. We propose a framework that can reconstruct high-quality object-level maps for unknown objects. Our approach takes multiple RGB-D images as input and outputs dense 3D shapes and 9-DoF poses (including 3 scale parameters) for detected objects. The co...
-
Mapping is crucial for spatial reasoning, planning and robot navigation. Existing approaches range from metric, which require precise geometry-based optimization, to purely topological, where image-as-node based graphs lack explicit object-level reasoning and interconnectivity. In this paper, we propose a novel topological representation of an environment based on , which are semantically meaningf...
-
In multi-object stacking scenarios, exploring the relationships among objects and determining the correct sequence of operations are crucial for robotic manipulation. However, previous algorithms inefficiently combine global and local information, often focusing solely on the local features of objects or the interactions of object features at a global level. This approach leads to imbalanced distr...
-
This paper presents a novel acoustic soft tactile (AST) skin technology operating with sound waves. In this innovative approach, the sound waves generated by a speaker travel in channels embedded in a soft membrane and get modulated due to a deformation of the channel when pressed by an external force and received by a microphone at the end of the channel. The sensor leverages regression and class...
-
Robotic grasping refers to making a robotic system pick an object by applying forces and torques on its surface. Many recent studies use data-driven approaches to address grasping, but the sparse reward nature of this task made the learning process challenging to bootstrap. To avoid constraining the operational space, an increasing number of works propose grasping datasets to learn from. But most ...
-
Grasping with five-fingered humanoid hands is a complex control problem. Throughout the entire grasping motion, all finger joints need to be coordinated to achieve a stable grasp. Grasp synergies provide a simplified, low-dimensional representation of grasp postures and motions, that can be used for the description of human grasps as well as the generation of novel, human-like grasps. However, the...
-
We consider the problem of closed-loop robotic grasping and present a novel planner which uses Visual Feedback and an uncertainty-aware Adaptive Sampling strategy (VFAS) to close the loop. At each iteration, our method VFAS-Grasp builds a set of candidate grasps by generating random perturbations of a seed grasp. The candidates are then scored using a novel metric which combines a learned grasp-qu...
-
This paper and its companion propose a new fractal robotic gripper, drawing inspiration from the centuryold Fractal Vise. The unusual synergistic properties allow it to passively conform to diverse objects using only one actuator. Designed to be easily integrated with prevailing parallel jaw grippers, it alleviates the complexities tied to perception and grasp planning, especially when dealing wit...
-
Accurate grasping is the key to several robotic tasks including assembly and household robotics. Executing a successful grasp in a cluttered environment requires multiple levels of scene understanding: First, the robot needs to analyze the geometric properties of individual objects to find feasible grasps. These grasps need to be compliant with the local object geometry. Second, for each proposed ...
-
Advancing robotic grasping and manipulation requires the ability to test algorithms and/or train learning models on large numbers of grasps. Towards the goal of more advanced grasping, we present the Grasp Reset Mechanism (GRM), a fully automated apparatus for conducting large-scale grasping trials. The GRM automates the process of resetting a grasping environment, repeatably placing an object in ...
-
Robot learning methods have recently made great strides, but generalization and robustness challenges still hinder their widespread deployment. Failing to detect and address potential failures renders state-of-the-art learning systems not combat-ready for high-stakes tasks. Recent advances in interactive imitation learning have presented a promising framework for human-robot teaming, enabling the ...
-
We introduce a novel Fractal Hand robotic gripper. The hand has only 1 actuator, but (2n+1 −1) joints, where a design parameter n defines the depth of the fingers’ tree structures. The hand is synergistic in its operation (because its joint movements are coupled through the hand’s interaction with the grasped object), but it is not anthropomorphic. The basic finger and hand geometry, governing kin...
-
Object-goal navigation is a challenging task that requires guiding an agent to specific objects based on first-person visual observations. The ability of agent to comprehend its surroundings plays a crucial role in achieving successful object finding. However, existing knowledge-graph-based navigators often rely on discrete categorical one-hot vectors and vote counting strategy to construct graph ...
-
In this work, we focus on object search tasks within unexplored environments. We introduce a framework centered around the Probable Object Location (POLo) score. Utilizing a 3D object probability map, the POLo score allows the agent to make data-driven decisions for efficient object search. We further enhance the framework’s practicality by introducing POLoNet, a neural network trained to approxim...
-
Zero-shot object navigation is a challenging task for home-assistance robots. This task emphasizes visual grounding, commonsense inference and locomotion abilities, where the first two are inherent in foundation models. But for the locomotion part, most works still depend on map-based planning approaches. The gap between RGB space and map space makes it difficult to directly transfer the knowledge...
-
Real-time path planning in outdoor environments still challenges modern robotic systems due to differences in terrain traversability, diverse obstacles, and the necessity for fast decision-making. Established approaches have primarily focused on geometric navigation solutions, which work well for structured geometric obstacles but have limitations regarding the semantic interpretation of different...
-
Autonomous navigation in the underwater environment is challenging due to limited visibility, dynamic changes, and the lack of a cost-efficient, accurate localization system. We introduce UIVNAV, a novel end-to-end underwater navigation solution designed to navigate robots over Objects of Interest (OOI) while avoiding obstacles, all without relying on localization. UIVNAV utilizes imitation learni...
-
Unmanned air vehicles (UAVs) have traditionally been considered as "eyes in the sky", that can move in three dimensions and need to avoid any contact with their environment. On the contrary, contact should not be considered as a problem, but as an opportunity to expand the range of UAVs applications. In this paper, we designed, fabricated, and characterized a whisker sensor unit based on MEMS baro...
-
Sensing plays a pivotal role in robotic manipulation, dictating the accuracy and versatility with which objects are handled. Vision-based sensing methods often suffer from fabrication complexity and low durability, while approaches that rely on direct measurements on the gripper often have limited resolution and are difficult to scale. Here, we present a soft robotic gripper made out of two cubic ...
-
Soft robotic gloves can not only provide timely, effective, safe and cheap rehabilitation training for patients with impaired movement function of hand, but also assist in completing daily grasping activities. However, most soft robotic gloves are completely composed of flexible structures. Although they have high flexibility and safety, there are problems such as poor fit and low output force. In...
-
In neurosurgery, soft robots have the potential to introduce significant benefits over traditional metal tools for their ability to safely interact with delicate tissues. In this paper, we introduce a proof-of-concept soft, capacitive origami sensing module (OSM) that can measure forces during neurosurgical retraction. Using origami-inspired design and fabrication principles, the OSM is easily fol...
-
A soft miniaturized continuum robot with 3D shape sensing via functionalized soft optical waveguides
In this paper, we present a fully soft miniaturized continuum robot that integrates 3D optical shape sensing through functionalized tubing used as soft optical waveguides. The sensor is fabricated by laser patterning an off-the-shelf medical tubing, allowing for bidirectional responses to large curvatures in two bending directions, enabling 3D shape sensing and tip tracking of the continuum robot....
-
The emergence of intelligent prostheses has facilitated the life and work of disabled patients. The interaction aspect of prostheses has become a highlight research topic in the field of rehabilitation robotics. However, most of the existing prosthetic interaction methods focus on the use of myoelectricity to classify finite gestures, rather than continuous (infinite) force detection, which greatl...
-
Tremendous variations coupled with large degrees of freedom in UAV-based imaging conditions lead to a significant lack of data in adequately learning UAV-based perception models. Using various synthetic renderers in conjunction with perception models is prevalent to create synthetic data to augment the learning in the ground-based imaging domain. However, severe challenges in the austere UAV-based...
-
Scene transfer for vision-based mobile robotics applications is a highly relevant and challenging problem. The utility of a robot greatly depends on its ability to perform a task in the real world, outside of a well-controlled lab environment. Existing scene transfer end-to-end policy learning approaches often suffer from poor sample efficiency or limited generalization capabilities, making them u...
-
Detecting air flows caused by phenomena such as heat convection is valuable in multiple scenarios, including leak identification and locating thermal updrafts for extending UAV flight duration. Unfortunately, the heat signature of these flows is often too subtle to be seen by a thermal camera. While convection also leads to fluctuations in air density and hence causes so-called schlieren – intensi...
-
Relative drone-to-drone localization is a fundamental building block for any swarm operations. We address this task in the context of miniaturized nano-drones, i.e., ∼10cm in diameter, which show an ever-growing interest due to novel use cases enabled by their reduced form factor. The price for their versatility comes with limited onboard resources, i.e., sensors, processing units, and memory, whi...
-
3D instance segmentation is a fundamental and critical task for enabling robots to operate effectively in unstructured 3D environments. In order to address the challenges posed by the high demand for large-scale annotated data and the limited availability of such data in the context of 3D instance segmentation, we study semi-supervised 3D instance segmentation problem and propose a novel end-to-en...
-
Self-supervised depth estimation has evolved into an image reconstruction task that minimizes a photometric loss. While recent methods have made strides in indoor depth estimation, they often produce inconsistent depth estimation in textureless areas and unsatisfactory depth discrepancies at object boundaries. To address these issues, in this work, we propose GAM-Depth, developed upon two novel co...
-
Estimating robot pose and joint angles is significant in advanced robotics, enabling applications like robot collaboration and online hand-eye calibration. However, the introduction of unknown joint angles makes prediction more complex than simple robot pose estimation, due to its higher dimensionality. Previous methods either regress 3D keypoints directly or utilise a render&compare strategy. The...
-
Advanced Driver-Assistance Systems (ADAS) have successfully integrated learning-based techniques into vehicle perception and decision-making. However, their application in 3D lane detection for effective driving environment perception is hindered by the lack of comprehensive LiDAR datasets. The sparse nature of LiDAR point cloud data prevents an efficient manual annotation process. To solve this p...
-
In this work, we present STOPNet, a framework for 6-DoF object suction detection on production lines, with a focus on but not limited to transparent objects, which is an important and challenging problem in robotic systems and modern industry. Current methods requiring depth input fail on transparent objects due to depth cameras’ deficiency in sensing their geometry, while we proposed a novel fram...
-
For robots to be useful outside labs and specialized factories we need a way to teach them new useful behaviors quickly. Current approaches lack either the generality to onboard new tasks without task-specific engineering, or else lack the data-efficiency to do so in an amount of time that enables practical use. In this work we explore dense tracking as a representational vehicle to allow faster a...
-
Many practically relevant robot grasping problems feature a target object for which all grasps are occluded, e.g., by the environment. Single-shot grasp planning invariably fails in such scenarios. Instead, it is necessary to first manipulate the object into a configuration that affords a grasp. We solve this problem by learning a sequence of actions that utilize the environment to change the obje...
-
We investigate the active manipulation of objects using model-free and long-horizon DRL (Deep Reinforcement Learning) to achieve target shapes. Our proposed approach uses visual observations consisting of segmented images, to mitigate the sim-to-real gap. We address a long-horizon manipulation task requiring a sequence of accurate actions to achieve the target shapes using a robot arm with an RGB-...
-
Articulated objects like cabinets and doors are widespread in daily life. However, directly manipulating 3D articulated objects is challenging because they have diverse geometrical shapes, semantic categories, and kinetic constraints. Prior works mostly focused on recognizing and manipulating articulated objects with specific joint types. They can either estimate the joint parameters or distinguis...
-
Bin picking is an important building block for many robotic systems, in logistics, production or in household use-cases. In recent years, machine learning methods for the prediction of 6-DoF grasps on diverse and unknown objects have shown promising progress. However, existing approaches only consider a single ground truth grasp orientation at a grasp location during training and therefore can onl...
-
Multi-goal robot manipulation tasks with sparse rewards are difficult for reinforcement learning (RL) algorithms due to the inefficiency in collecting successful experiences. Recent algorithms such as Hindsight Experience Replay (HER) expedite learning by taking advantage of failed trajectories and replacing the desired goal with one of the achieved states so that any failed trajectory can be util...
-
In this paper, we tackle the problem of grasping transparent and specular objects. This issue holds importance, yet it remains unsolved within the field of robotics due to failure of recover their accurate geometry by depth cameras. For the first time, we propose ASGrasp, a 6-DoF grasp detection network that uses an RGB-D active stereo camera. ASGrasp utilizes a two-layer learning-based stereo net...
-
Offline learning usually requires a large dataset for training. In this paper, we focus on vision-based robotic manipulation tasks and utilize certain task properties to achieve offline learning with a small dataset. We propose a two-stage agent consisting of a tentative decision stage and a correction stage, where the tentative decision stage determines a tentative action from the original camera...
-
Consider a robot tasked with tidying a desk with a meticulously constructed Lego sports car. A human may recognize that it is not appropriate to disassemble the sports car and put it away as part of the "tidying." How can a robot reach that conclusion? Although large language models (LLMs) have recently been used to enable commonsense reasoning, grounding this reasoning in the real world has been ...
-
In this paper, we investigate how a customer-facing service robot can support decision making in shopping interactions. In this role, a robot needs sometimes to reject a customer’s choice. Thus, we investigate different rejection strategies with the goal of changing customer behavior. The implemented strategies have been developed based on an ethnographic study on assisted shopping and tested in a...
-
The choices made by autonomous robots in social settings bear consequences for humans and their presumptions of robot behavior. Explanations can serve to alleviate detrimental impacts on humans and amplify their comprehension of robot decisions. We model the process of explanation generation for robot navigation as an automated planning problem considering different possible explanation attributes...
-
Safe and efficient navigation in dynamic environments shared with humans remains an open and challenging task for mobile robots. Previous works have shown the efficacy of using reinforcement learning frameworks to train policies for efficient navigation. However, their performance deteriorates when crowd configurations change, i.e. become larger or more complex. Thus, it is crucial to fully unders...
-
This work explores a novel approach to empowering robots with visual perception capabilities using textual descriptions. Our approach involves the integration of GPT-4 with dense captioning, enabling robots to perceive and interpret the visual world through detailed text-based descriptions. To assess both user experience and the technical feasibility of this approach, experiments were conducted wi...
-
The inexorable progress of technology brought forth an era where robots increasingly integrate into human life which necessitates the understanding of human-robot interactions (HRI). This study unravels the details of HRI within interactive storytelling contexts. Through a between-subject experiment with 28 participants, we assessed response latency and utterance lengths to interactive story narra...
-
Social robots for children have focused mainly on conventional education domains such as teaching language, science, and math, while applications focusing on the enhancement of cultural competency are quite scarce. In this paper, we present a prototype of a robot-mediation framework for cross-cultural communication. This framework paves the way for a social robot to act as a mediator between group...
-
Different people have different preferences when it comes to human-robot interaction. Therefore, it is desirable for the robot to adapt its actions to fit users’ preferences. Human feedback is essential to facilitating robot adaptation. However, when the task is complex or the robot action space is large, it requires a large amount of user feedback. ChatGPT is a powerful generative AI tool based o...
-
Robotic exoskeletons can enhance human strength and aid people with physical disabilities. However, designing them to ensure safety and optimal performance presents significant challenges. Developing exoskeletons should incorporate specific optimization algorithms to find the best design. This study investigates the potential of Evolutionary Computation (EC) methods in robotic design optimization,...
-
Community-based locomotor training post-stroke has shown improvements in independent ambulation by increasing dose, intensity, and specificity of walking practice. Robotic ankle exoskeletons hold the potential to facilitate continued rehabilitation at home, but understanding what aspects of the design are most relevant for successful translation to the community presents a challenge. Here, we desi...
-
Recently, the importance of mechanical transparency in human-assistive robots has grown. Traditionally, its primary goal was minimizing interaction forces during assistance. However, under this conventional definition, mechanical transparency was not considered when an interaction force was required during assistance. This research focuses on achieving mechanical transparency within the context of...
-
This paper presents a method to compute smooth ankle trajectories for lower limb exoskeletons with powered ankle joints. The proposed approach defines ankle trajectories using four polynomial functions, each representing one of the four primary phases of gait. These polynomials are computed according to different safety constraints. During the single support phase, ground contact constraints are e...
-
Upper limb paralysis affects the quality of life. Functional Electrical Stimulation (FES) offers a solution to restore lost motor functions. Yet, there remain challenges in controlling FES to induce arbitrary arm movements. Reinforcement learning (RL) emerges as a promising method for controlling arm movement with success in simulation. However, challenges remain in translating the successes into ...
-
The hybridisation of robot-assisted gait training and functional electrical stimulation (FES) can provide numerous physiological benefits to neurological patients. However, the design of an effective hybrid controller poses significant challenges. In this over-actuated system, it is extremely difficult to find the right balance between robotic assistance and FES that will provide personalised assi...
-
Physical therapy (PT) is a key component of many rehabilitation regimens, such as treatments for Parkinson’s disease (PD). However, there are shortages of physical therapists and adherence to self-guided PT is low. Robots have the potential to support physical therapists and increase adherence to self-guided PT, but prior robotic systems have been large and immobile, which can be a barrier to use ...
-
We address the problem of (a) predicting the trajectory of an arm reaching motion, based on a few seconds of the motion’s onset, and (b) leveraging this predictor to facilitate shared-control manipulation tasks, by reducing the operator’s cognitive load through assistance in their anticipated direction of motion. Our novel intent estimator, dubbed the Robot Trajectron (RT), produces a probabilisti...
-
This paper explores the feasibility of employing EEG-based intention detection for real-time robot assistive control. We focus on predicting and distinguishing motor intentions of left/right arm movements by presenting: i) an offline data collection and training pipeline, used to train a classifier for left/right motion intention prediction, and ii) an online real-time prediction pipeline leveragi...
-
Microexpressions are expressions that people inadvertently express, and therefore often represent a person’s true emotion. However, because it has a low intensity and a short duration, it is hard to be recognized correctly. In this paper, we propose a deep learning magnification method to generate macroexpressions from a single microexpression image. In the first stage, we extract the expression i...
-
Anticipating driver intention is an important task when vehicles of mixed and varying levels of human/machine autonomy share roadways. Driver intention can be leveraged to improve road safety, such as warning surrounding vehicles in the event the driver is attempting a dangerous maneuver. In this work, we propose a novel method of utilizing both in-cabin and external camera data to improve state-o...
-
This work explores the capacity of large language models (LLMs) to address problems at the intersection of spatial planning and natural language interfaces for navigation. We focus on following complex instructions that are more akin to natural conversation than traditional explicit procedural directives typically seen in robotics. Unlike most prior work where navigation directives are provided as...
-
In this paper, we present a new paradigm for evaluating egocentric pedestrian trajectory prediction algorithms. Based on various contextual information, we extract driving scenarios for a meaningful and systematic approach to identifying challenges for prediction models. In this regard, we also propose a new metric for more effective ranking within the scenario-based evaluation. We conduct extensi...
-
In human-robot collaboration (HRC) settings, hand motion intention prediction (HMIP) plays a pivotal role in ensuring prompt decision-making, safety, and an intuitive collaboration experience. Precise and robust HMIP with low computational resources remains a challenge due to the stochastic nature of hand motion and the diversity of HRC tasks. This paper proposes a framework that combines hand tra...
-
The outdoor environment has many uneven surfaces that put the robot at risk of sinking or tipping over. Recognizing the type of terrain can help robot avoid risks and choose an appropriate gait. One of the critical problems is how to extract the terrain-related knowledge from sensor data collected as the robot traversed the ground. Many existing vision-based approaches are limited in directly perc...
-
Soft-bubble tactile sensors have the potential to capture dense contact and force information across a large contact surface. However, it is difficult to extract contact forces directly from observing the bubble surface because local contacts change the global surface shape significantly due to membrane mechanics and air pressure. This paper presents a model-based method of reconstructing dense co...
-
Vertical farming, a sustainable key for urban agriculture, has garnered attention for its land use optimization and enhanced food production capabilities. The adoption of automation in vertical farming is a pivotal response to labor shortages, addressing the need for increased efficiency, particularly in labor-intensive tasks like harvesting. Although soft robotic grippers offer a significant prom...
-
Most vision-based tactile sensors use elastomer deformation to infer tactile information, which can not sense some modalities, like temperature. As an important part of human tactile perception, temperature sensing can help robots better interact with the environment. In this work, we propose a novel multi-modal vision-based tactile sensor, SATac, which can simultaneously perceive information on t...
-
Tactile and textile skin technologies have become increasingly important for enhancing human-robot interaction and allowing robots to adapt to different environments. Despite notable advancements, there are ongoing challenges in skin signal processing, particularly in achieving both accuracy and speed in dynamic touch sensing. This paper introduces a new framework that poses the touch sensing prob...
-
Whole-arm tactile feedback is crucial for robots to ensure safe physical interaction with their surroundings. This paper introduces CushSense, a fabric-based soft and stretchable tactile-sensing skin designed for physical human-robot interaction (pHRI) tasks such as robotic caregiving. Using stretchable fabric and hyper-elastic polymer, CushSense identifies contacts by monitoring capacitive change...
-
Simulating tactile perception could potentially leverage the learning capabilities of robotic systems in manipulation tasks. However, the reality gap of simulators for high-resolution tactile sensors remains large. Models trained on simulated data often fail in zero-shot inference and require fine-tuning with real data. In addition, work on high-resolution sensors commonly focus on ones with flat ...
-
Legged robots have the potential to become vital in maintenance, home support, and exploration scenarios. In order to interact with and manipulate their environments, most legged robots are equipped with a dedicated robot arm, which means additional mass and mechanical complexity compared to standard legged robots. In this work, we explore pedipulation - using the legs of a legged robot for manipu...
-
Learning-based methods have improved locomotion skills of quadruped robots through deep reinforcement learning. However, the sim-to-real gap and low sample efficiency still limit the skill transfer. To address this issue, we propose an efficient model-based learning framework that combines a world model with a policy network. We train a differentiable world model to predict future states and use i...
-
Model Predictive Control (MPC) relies heavily on the robot model for its control law. However, a gap always exists between the reduced-order control model with uncertainties and the real robot, which degrades its performance. To address this issue, we propose the controller of integrating a data-driven error model into traditional MPC for quadruped robots. Our approach leverages real-world data fr...
-
This paper addresses the problem of realizing a novel robotic bipedal locomotion called wheel gait, which is achieved by rotating the stance and swing legs in the same direction. First, a model of a planar 3-DOF X-shaped walker with a reaction wheel is introduced, and the mathematical equations are described. Second, the condition for stabilizing zero dynamics is formulated as the time integral va...
-
We propose a limb trajectory planning method that considers both body and limb dynamics in robots, particularly suitable for those with non-trivial limb mass. To simplify the complexity and computation cost of using the full-body dynamics of the limbs, a reduced-order model that can simulate the dynamic characteristics of the original limb is proposed. The performance of the model is experimentall...
-
In the context of interaction with unmodelled systems, it becomes imperative for a robot controller to possess the capability to dynamically adjust its actions in real-time, enhancing its resilience in the face of fluctuating environmental conditions. This adaptation process must be performed in a stability-preserving fashion, and resourcefully exploit the knowledge acquired during the interaction...
-
In recent years, there has been a growing demand for robotic manipulators to perform tasks in various unstructured environments and situations requiring precision and force control. However, traditional robotic arms have limitations in fully leveraging their advantages in such scenarios. To address this demand, we have designed a cable-driven serpentine manipulator (CDSM) that combines force and p...
-
Model-based approaches for planning and control for bipedal locomotion have a long history of success. It can provide stability and safety guarantees while being effective in accomplishing many locomotion tasks. Model-free reinforcement learning, on the other hand, has gained much popularity in recent years due to computational advancements. It can achieve high performance in specific tasks, but i...
-
Nonlinear dynamics bring difficulties to controller design for control-affine systems such as tractor-trailer vehicles, especially when the parameters in the dynamics are unknown. To address this constraint, we propose a derivative-based lifting function construction method, show that the corresponding infinite dimensional Koopman bilinear model over the lifting function is equivalent to the origi...
-
The hybrid nature of multi-contact robotic systems, due to making and breaking contact with the environment, creates significant challenges for high-quality control. Existing model-based methods typically rely on either good prior knowledge of the multi-contact model or require significant offline model tuning effort, thus resulting in low adaptability and robustness. In this paper, we propose a r...
-
An accurate motion model is a fundamental component of most autonomous navigation systems. While much work has been done on improving model formulation, no standard protocol exists for gathering empirical data required to train models. In this work, we address this issue by proposing Data-driven Robot Input Vector Exploration (DRIVE), a protocol that enables characterizing uncrewed ground vehicles...
-
Laparoscope-holding robots significantly enhance the stability and precision of visualization in minimally invasive surgeries. Most existing robots of this kind depend on visual servo systems and struggle with efficient, rapid adjustments in the field-of-view (FOV), especially when identifying organs and needles outside the FOV. This paper presents a laparoscope-holding robot system capable of emp...
-
In the study of minimally invasive surgical robots, a mini parallel continuum robot has shown motion advantage after passing through a long and winding working channel. However, due to the interaction force between the elastic wires of the parallel robots during motion generation processes, the constant curvature assumption has shown modeling errors. This causes the current geometric kinematic mod...
-
Robot-assisted vascular interventional surgery can isolate interventionists and X-ray radiation, and improve surgical accuracy. However, the leader side outside the operating room still has problems such as incomplete collection of operating information and unrealistic tactile feedback. The main objective of this paper is to design a haptic interface that can simultaneously capture the force-posit...
-
In robotic-assisted minimally invasive surgery, the remote center of motion (RCM) achieves precision and safe manipulation of surgical devices through the insertion point into the patient’s body. One of the RCM configurations, one-rotation and one-translation (1R1T) RCM based on a closed-loop design, enables two-degrees-of-freedom transmission from the proximal end of the robotic arm to the distal...
-
This paper proposes a novel form of a three-dimensional coaxial bowtie-shaped mechanical amplifier. The proposed model incorporates a lever mechanism into the Sarrus linkage structure. It allows the target plate to move along one axis with amplified displacement in a parallel manner. The amplifier was assembled after machining the components using a computer numerical control machine. A flexible h...
-
Magnetic continuum robots (MCR) have shown great potential in minimally invasive interventions because they can be actively and remotely navigated through complex in vivo environments. However, the deformation capability of current MCRs is limited by fixed magnetization congurations, preventing them from accessing hard-to-reach areas. This is due to the fact that under a global magnetic field, fix...
-
Traditional rigid endoscopes have challenges in flexibly treating tumors located deep in the brain, and low operability and fixed viewing angles limit its development. This study introduces a novel dual-segment flexible robotic endoscope MicroNeuro, designed to perform biopsies with dexterous surgical manipulation deep in the brain. Taking into account the uncertainty of the control model, an imag...
-
Continuum Dexterous Manipulators (CDMs) are well-suited tools for minimally invasive surgery due to their inherent dexterity and reachability. Nonetheless, their flexible structure and non-linear curvature pose significant challenges for shape-based feedback control. The use of Fiber Bragg Grating (FBG) sensors for shape sensing has shown great potential in estimating the CDM’s tip position and su...
-
The use of autonomous underwater vehicles (AUVs) to accomplish traditionally challenging and dangerous tasks has proliferated thanks to advances in sensing, navigation, manipulation, and on-board computing technologies. Utilizing AUVs in underwater human-robot interaction (UHRI) has witnessed comparatively smaller levels of growth due to limitations in bi-directional communication and significant ...
-
Powerline robotics is slowly becoming key tools for electric utilities. Contrary to drones that are usually limited to inspection tasks, wheeled robots like LineRanger can perform a broader range of applications. In this paper, a suite of mechanical devices is featured, as several new asset management tasks were recently added to LineRanger’s capabilities. While previous applications focused on no...
-
Mission-critical operations, particularly in the context of Search-and-Rescue (SAR) and emergency response situations, demand optimal performance and efficiency from every component involved to maximize the success probability of such operations. In these settings, cellular-enabled collaborative robotic systems have emerged as invaluable assets, assisting first responders in several tasks, ranging...
-
In this article, we propose a novel navigation framework that leverages a two layered graph representation of the environment for efficient large-scale exploration, while it integrates a novel uncertainty awareness scheme to handle dynamic scene changes in previously explored areas. The framework is structured around a novel goal oriented graph representation, that consists of, i) the local sub-gr...
-
Searching for targets in 3D environments can be formulated as submodular maximization problems with routing constraints. However, it involves solving two NP-hard problems: the maximal coverage problem and the traveling salesman problem. Since the time constraint is critical for search problems, this research proposes a Computation-Aware Search for Multiple Objects (CASMO) algorithm to further cons...
-
Efficient task allocation is a challenge for multirobot search. The multi-robot search problem is reformulated as submodular maximization subject to intersection system constraints. The objective function is submodular and consists of a coverage function to cover environments and a balancing function to efficiently dispatch robots. The intersection system is composed of routing and clustering cons...
-
In the paper, we address wireless communication infrastructure building by relay placement based on approaches utilized in wireless network sensors. The problem is motivated by search and inspection missions with mobile robots, where known sensing ranges may be exploited. We investigate the relay placement, establishing network connectivity to support robust food-based communication routing. The p...
-
This systems paper presents the implementation and design of RB5, a wheeled robot for autonomous long-term exploration with fewer and cheaper sensors. Requiring just an RGB-D camera and low-power computing hardware, the system consists of an experimental platform with rocker-bogie suspension. It operates in unknown and GPS-denied environments and on indoor and outdoor terrains. The exploration con...
-
Springtails are tiny arthropods that crawl and jump. They jump by temporarily storing elastic energy in resilin elastic cuticular structures and releasing that energy to accelerate a tail, called a furca, propelling them in the air. This paper presents an autonomous, springtail-inspired microrobot that can crawl and jump. The microrobot has a mass of 980mg and stands 13mm tall, and has on-board se...
-
Several insect species are able to locomote across the air-water interface by leveraging surface tension to remain above the water surface. A subset of these insects, such as the stonefly and waterlily beetle, flap their wings to actively move around the two dimensional surface — a locomotion strategy referred to as interfacial flight. Here, we present an insect-scale robot, the γ-bot, inspired by...
-
This paper presents the VLEIBot* (Very Little Eel-Inspired roBot), a 45-mg/23-mm3 microrobotic swimmer that is propelled by a bioinspired anguilliform propulsor. The propulsor is excited by a single 6-mg high-work-density (HWD) microactuator and undulates periodically due to wave propagation phenomena generated by fluid-structure interaction (FSI) during swimming. The microactuator is composed of ...
-
Insects have long been recognized for their ability to navigate and return home using visual cues from their nest’s environment. However, the precise mechanism underlying this remarkable homing skill remains a subject of ongoing investigation. Drawing inspiration from the learning flights of honey bees and wasps, we propose a robot navigation method that directly learns the home vector direction f...
-
Dragonflies show impressive flying skills by achieving both high efficiency and agility. They can perform distinctive flight maneuvers, such as flying backwards, which has proven to be achieved through "force vectoring" mechanism recently. In this paper, to explore the agile flight ability of dragonflies on man-made flapping wing systems, we designed, optimized and fabricated a dragonfly-inspired ...
-
Flying microrobots have garnered growing research interest owing to their technological intricacies and suitability for various applications leveraging miniaturized size. Electrohydrodynamic (EHD) thrust offers advantages by generating propulsion without moving parts, but real-world use is limited by insufficient thrust generation, manufacturing challenges, fragility, and cost. This work presents ...
-
This paper examines the relevance of using catenary-based curves to model cables in underwater tethered robotic applications in order to take into account the influence of hydrodynamic damping. To this end, an augmented catenary-based model is introduced to deal with the dynamical effects of surge motion, sway motion or a combination of both on a cable. Experimental studies are carried out with ei...
-
This paper reports a numerical method for modeling underwater vehicle (UV) interactions with the free surface using a finite-dimensional dynamical plant model. Although finite-dimensional plant models of fully submerged UV behavior are well-established, they are unable to model the ubiquitous condition of a UV operating at or near the free surface. We report a Monte Carlo-based hybrid model approa...
-
There exists a capability gap in the design of currently available autonomous underwater vehicles (AUV). Most AUVs use a set of thrusters, and optionally control surfaces, to control their depth and pose. AUVs utilizing thrusters can be highly maneuverable, making them well-suited to operate in complex environments such as in close-proximity to coral reefs. However, they are inherently power-ineff...
-
In autonomous robot navigation, the trajectories from path planners are considered to be safe regions, and deviations could endanger vessels. Model Predictive Control (MPC) stands as a popular choice for trajectory tracking problems as it naturally addresses operational constraints, such as dynamics and control constraints. Nevertheless, achieving robustness in changing environments like oceans an...
-
The optical effects that are observed in underwater imagery are more complex than those in-air. This is partially because we enclose most underwater cameras in a watertight enclosure, such as a hemispheric dome window. We then observe optical issues including the distortion effects of the lens, e.g., wide-angle field-of-view (FOV), the refractive effects at the enclosure (water-acrylic and acrylic...
-
Deep learning has shown promising results for multiple 3D point cloud registration datasets. However, in the underwater domain, most registration of multibeam echo-sounder (MBES) point cloud data are still performed using classical methods in the iterative closest point (ICP) family. In this work, we curate and release DotsonEast Dataset, a semi-synthetic MBES registration dataset constructed from...
-
Developing autonomous intervention capabilities for lightweight underwater vehicle manipulator systems (UVMS) has garnered significant attention within recent years because of the opportunity for these systems to reduce intervention operating costs. Developing autonomous UVMS capabilities is challenging, however, because of the lack of available standardized software frameworks and pipelines. Prev...
-
In this paper, the dynamics of an emerging class of rotating nature-inspired micro aerial vehicles known as the Monocopter is proven and shown to be differentially flat. By exploiting this phenomenon, trajectory tracking can now be implemented on Monocopters via feed-forward terms that are computed per the trajectory. To demonstrate this, a Monocopter in the form of a Stable Invertible Coaxial Act...
-
Recently, the utilization of aerial manipulators for performing pushing tasks in non-destructive testing (NDT) applications has seen significant growth. Such operations entail physical interactions between the aerial robotic system and the environment. End-effectors with multiple contact points are often used for placing NDT sensors in contact with a surface to be inspected. Aligning the NDT senso...
-
In this paper, we introduce PADUAV, a novel 5-DOF aerial platform designed to overcome the limitations of traditional tiltrotor vehicles. PADUAV features a unique mechanical design that incorporates two off-the-shelf quadrotors passively articulated to a rigid frame. This innovation enables free pitch rotation without mechanical constraints like cable winding, significantly enhancing its capabilit...
Introduction
Conference ICRA2024 accepted paper complete List. Top ranking conferences for AI and Robotics communities. Total Accepted Paper Count 995