arXiv:2406.08878v4 »Full PDF »Modern approaches to autonomous driving rely heavily on learned components
trained with large amounts of human driving data via imitation learning.
However, these methods require large amounts of expensive data collection and
even then face challenges with safely handling long-tail scenarios and
compounding errors over time. At the same time, pure Reinforcement Learning
(RL) methods can fail to learn performant policies in sparse, constrained, and
challenging-to-define reward settings such as autonomous driving. Both of these
challenges make deploying purely cloned or pure RL policies in safety critical
applications such as autonomous vehicles challenging. In this paper we propose
Combining IMitation and Reinforcement Learning (CIMRL) approach - a safe
reinforcement learning framework that enables training driving policies in
simulation through leveraging imitative motion priors and safety constraints.
CIMRL does not require extensive reward specification and improves on the
closed loop behavior of pure cloning methods. By combining RL and imitation, we
demonstrate that our method achieves state-of-the-art results in closed loop
simulation and real world driving benchmarks.Abstract
MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance
arXiv:2112.03575v1 »Full PDF »Safe exploration is critical for using reinforcement learning (RL) in
risk-sensitive environments. Recent work learns risk measures which measure the
probability of violating constraints, which can then be used to enable safety.
However, learning such risk measures requires significant interaction with the
environment, resulting in excessive constraint violations during learning.
Furthermore, these measures are not easily transferable to new environments. We
cast safe exploration as an offline meta-RL problem, where the objective is to
leverage examples of safe and unsafe behavior across a range of environments to
quickly adapt learned risk measures to a new environment with previously unseen
dynamics. We then propose MEta-learning for Safe Adaptation (MESA), an approach
for meta-learning a risk measure for safe RL. Simulation experiments across 5
continuous control domains suggest that MESA can leverage offline data from a
range of different environments to reduce constraint violations in unseen
environments by up to a factor of 2 while maintaining task performance. See
https://tinyurl.com/safe-meta-rl for code and supplementary material.Abstract
LS3: Latent Space Safe Sets for Long-Horizon Visuomotor Control of
Sparse Reward Iterative Tasks
Conference on Robot Learning (CoRL) 2021. First two authors
contributed equally
Reinforcement learning (RL) has shown impressive success in exploring
high-dimensional environments to learn complex tasks, but can often exhibit
unsafe behaviors and require extensive environment interaction when exploration
is unconstrained. A promising strategy for learning in dynamically uncertain
environments is requiring that the agent can robustly return to learned safe
sets, where task success (and therefore safety) can be guaranteed. While this
approach has been successful in low-dimensions, enforcing this constraint in
environments with visual observations is exceedingly challenging. We present a
novel continuous representation for safe sets by framing it as a binary
classification problem in a learned latent space, which flexibly scales to
image observations. We then present a new algorithm, Latent Space Safe Sets
(LS3), which uses this representation for long-horizon tasks with sparse
rewards. We evaluate LS3 on 4 domains, including a challenging sequential
pushing task in simulation and a physical cable routing task. We find that LS3
can use prior task successes to restrict exploration and learn more efficiently
than prior algorithms while satisfying constraints. See
https://tinyurl.com/latent-ss for code and supplementary material.Abstract
Recovery RL: Safe Reinforcement Learning with Learned Recovery Zones
RA-L and ICRA 2021. First two authors contributed equally
Safety remains a central obstacle preventing widespread use of RL in the real
world: learning new tasks in uncertain environments requires extensive
exploration, but safety requires limiting exploration. We propose Recovery RL,
an algorithm which navigates this tradeoff by (1) leveraging offline data to
learn about constraint violating zones before policy learning and (2)
separating the goals of improving task performance and constraint satisfaction
across two policies: a task policy that only optimizes the task reward and a
recovery policy that guides the agent to safety when constraint violation is
likely. We evaluate Recovery RL on 6 simulation domains, including two
contact-rich manipulation tasks and an image-based navigation task, and an
image-based obstacle avoidance task on a physical robot. We compare Recovery RL
to 5 prior safe RL methods which jointly optimize for task performance and
safety via constrained optimization or reward shaping and find that Recovery RL
outperforms the next best prior method across all domains. Results suggest that
Recovery RL trades off constraint violations and task successes 2 - 20 times
more efficiently in simulation domains and 3 times more efficiently in physical
experiments. See https://tinyurl.com/rl-recovery for videos and supplementary
material.Abstract
Safety Augmented Value Estimation from Demonstrations (SAVED): Safe Deep
Model-Based RL for Sparse Cost Robotic Tasks
Robotics and Automation Letters and International Conference on
Robotics and Automation 2020. Firs...
Reinforcement learning (RL) for robotics is challenging due to the difficulty
in hand-engineering a dense cost function, which can lead to unintended
behavior, and dynamical uncertainty, which makes exploration and constraint
satisfaction challenging. We address these issues with a new model-based
reinforcement learning algorithm, Safety Augmented Value Estimation from
Demonstrations (SAVED), which uses supervision that only identifies task
completion and a modest set of suboptimal demonstrations to constrain
exploration and learn efficiently while handling complex constraints. We then
compare SAVED with 3 state-of-the-art model-based and model-free RL algorithms
on 6 standard simulation benchmarks involving navigation and manipulation and a
physical knot-tying task on the da Vinci surgical robot. Results suggest that
SAVED outperforms prior methods in terms of success rate, constraint
satisfaction, and sample efficiency, making it feasible to safely learn a
control policy directly on a real robot in less than an hour. For tasks on the
robot, baselines succeed less than 5% of the time while SAVED has a success
rate of over 75% in the first 50 training iterations. Code and supplementary
material is available at https://tinyurl.com/saved-rl.Abstract
Rasa: Building Expressive Speech Synthesis Systems for Indian Languages
in Low-resource Settings
Accepted at INTERSPEECH 2024. First two authors listed contributed
equally
We release Rasa, the first multilingual expressive TTS dataset for any Indian
language, which contains 10 hours of neutral speech and 1-3 hours of expressive
speech for each of the 6 Ekman emotions covering 3 languages: Assamese,
Bengali, & Tamil. Our ablation studies reveal that just 1 hour of neutral and
30 minutes of expressive data can yield a Fair system as indicated by MUSHRA
scores. Increasing neutral data to 10 hours, with minimal expressive data,
significantly enhances expressiveness. This offers a practical recipe for
resource-constrained languages, prioritizing easily obtainable neutral data
alongside smaller amounts of expressive data. We show the importance of
syllabically balanced data and pooling emotions to enhance expressiveness. We
also highlight challenges in generating specific emotions, e.g., fear and
surprise.Abstract
arXiv:2407.21783v2 »Full PDF »Modern artificial intelligence (AI) systems are powered by foundation models.
This paper presents a new set of foundation models, called Llama 3. It is a
herd of language models that natively support multilinguality, coding,
reasoning, and tool usage. Our largest model is a dense Transformer with 405B
parameters and a context window of up to 128K tokens. This paper presents an
extensive empirical evaluation of Llama 3. We find that Llama 3 delivers
comparable quality to leading language models such as GPT-4 on a plethora of
tasks. We publicly release Llama 3, including pre-trained and post-trained
versions of the 405B parameter language model and our Llama Guard 3 model for
input and output safety. The paper also presents the results of experiments in
which we integrate image, video, and speech capabilities into Llama 3 via a
compositional approach. We observe this approach performs competitively with
the state-of-the-art on image, video, and speech recognition tasks. The
resulting models are not yet being broadly released as they are still under
development.Abstract
Deep Discriminative to Kernel Density Graph for In- and
Out-of-distribution Calibrated Inference
arXiv:2201.13001v8 »Full PDF »Deep discriminative approaches like random forests and deep neural networks
have recently found applications in many important real-world scenarios.
However, deploying these learning algorithms in safety-critical applications
raises concerns, particularly when it comes to ensuring confidence calibration
for both in-distribution and out-of-distribution data points. Many popular
methods for in-distribution (ID) calibration, such as isotonic and Platt's
sigmoidal regression, exhibit excellent ID calibration performance. However,
these methods are not calibrated for the entire feature space, leading to
overconfidence in the case of out-of-distribution (OOD) samples. On the other
end of the spectrum, existing out-of-distribution (OOD) calibration methods
generally exhibit poor in-distribution (ID) calibration. In this paper, we
address ID and OOD calibration problems jointly. We leveraged the fact that
deep models, including both random forests and deep-nets, learn internal
representations which are unions of polytopes with affine activation functions
to conceptualize them both as partitioning rules of the feature space. We
replace the affine function in each polytope populated by the training data
with a Gaussian kernel. Our experiments on both tabular and vision benchmarks
show that the proposed approaches obtain well-calibrated posteriors while
mostly preserving or improving the classification accuracy of the original
algorithm for ID region, and extrapolate beyond the training data to handle OOD
inputs appropriately.Abstract
Published in Science:
https://www.science.org/doi/10.1126/science.adn0117
Artificial Intelligence (AI) is progressing rapidly, and companies are
shifting their focus to developing generalist AI systems that can autonomously
act and pursue goals. Increases in capabilities and autonomy may soon massively
amplify AI's impact, with risks that include large-scale social harms,
malicious uses, and an irreversible loss of human control over autonomous AI
systems. Although researchers have warned of extreme risks from AI, there is a
lack of consensus about how exactly such risks arise, and how to manage them.
Society's response, despite promising first steps, is incommensurate with the
possibility of rapid, transformative progress that is expected by many experts.
AI safety research is lagging. Present governance initiatives lack the
mechanisms and institutions to prevent misuse and recklessness, and barely
address autonomous systems. In this short consensus paper, we describe extreme
risks from upcoming, advanced AI systems. Drawing on lessons learned from other
safety-critical technologies, we then outline a comprehensive plan combining
technical research and development with proactive, adaptive governance
mechanisms for a more commensurate preparation.Abstract
Deception in Reinforced Autonomous Agents: The Unconventional Rabbit Hat
Trick in Legislation
arXiv:2405.04325v1 »Full PDF »Recent developments in large language models (LLMs), while offering a
powerful foundation for developing natural language agents, raise safety
concerns about them and the autonomous agents built upon them. Deception is one
potential capability of AI agents of particular concern, which we refer to as
an act or statement that misleads, hides the truth, or promotes a belief that
is not true in its entirety or in part. We move away from the conventional
understanding of deception through straight-out lying, making objective selfish
decisions, or giving false information, as seen in previous AI safety research.
We target a specific category of deception achieved through obfuscation and
equivocation. We broadly explain the two types of deception by analogizing them
with the rabbit-out-of-hat magic trick, where (i) the rabbit either comes out
of a hidden trap door or (ii) (our focus) the audience is completely distracted
to see the magician bring out the rabbit right in front of them using sleight
of hand or misdirection. Our novel testbed framework displays intrinsic
deception capabilities of LLM agents in a goal-driven environment when directed
to be deceptive in their natural language generations in a two-agent
adversarial dialogue system built upon the legislative task of "lobbying" for a
bill. Along the lines of a goal-driven environment, we show developing
deceptive capacity through a reinforcement learning setup, building it around
the theories of language philosophy and cognitive psychology. We find that the
lobbyist agent increases its deceptive capabilities by ~ 40% (relative) through
subsequent reinforcement trials of adversarial interactions, and our deception
detection mechanism shows a detection capability of up to 92%. Our results
highlight potential issues in agent-human interaction, with agents potentially
manipulating humans towards its programmed end-goal.Abstract
16737 latest Fairness/Ethics + ML/AI papers
