Reinforcement learning based fine-tuning of large language models (LLMs) on
human preferences has been shown to enhance both their capabilities and safety
behavior. However, in cases related to safety, without precise instructions to
human annotators, the data collected may cause the model to become overly
cautious, or to respond in an undesirable style, such as being judgmental.
Additionally, as model capabilities and usage patterns evolve, there may be a
costly need to add or relabel data to modify safety behavior. We propose a
novel preference modeling approach that utilizes AI feedback and only requires
a small amount of human data. Our method, Rule Based Rewards (RBR), uses a
collection of rules for desired or undesired behaviors (e.g. refusals should
not be judgmental) along with a LLM grader. In contrast to prior methods using
AI feedback, our method uses fine-grained, composable, LLM-graded few-shot
prompts as reward directly in RL training, resulting in greater control,
accuracy and ease of updating. We show that RBRs are an effective training
method, achieving an F1 score of 97.1, compared to a human-feedback baseline of
91.7, resulting in much higher safety-behavior accuracy through better
balancing usefulness and safety.Abstract
arXiv:2410.21276v1 »Full PDF »GPT-4o is an autoregressive omni model that accepts as input any combination
of text, audio, image, and video, and generates any combination of text, audio,
and image outputs. It's trained end-to-end across text, vision, and audio,
meaning all inputs and outputs are processed by the same neural network. GPT-4o
can respond to audio inputs in as little as 232 milliseconds, with an average
of 320 milliseconds, which is similar to human response time in conversation.
It matches GPT-4 Turbo performance on text in English and code, with
significant improvement on text in non-English languages, while also being much
faster and 50\% cheaper in the API. GPT-4o is especially better at vision and
audio understanding compared to existing models. In line with our commitment to
building AI safely and consistent with our voluntary commitments to the White
House, we are sharing the GPT-4o System Card, which includes our Preparedness
Framework evaluations. In this System Card, we provide a detailed look at
GPT-4o's capabilities, limitations, and safety evaluations across multiple
categories, focusing on speech-to-speech while also evaluating text and image
capabilities, and measures we've implemented to ensure the model is safe and
aligned. We also include third-party assessments on dangerous capabilities, as
well as discussion of potential societal impacts of GPT-4o's text and vision
capabilities.Abstract
Learn 2 Rage: Experiencing The Emotional Roller Coaster That Is
Reinforcement Learning
arXiv:2410.18462v1 »Full PDF »This work presents the experiments and solution outline for our teams winning
submission in the Learn To Race Autonomous Racing Virtual Challenge 2022 hosted
by AIcrowd. The objective of the Learn-to-Race competition is to push the
boundary of autonomous technology, with a focus on achieving the safety
benefits of autonomous driving. In the description the competition is framed as
a reinforcement learning (RL) challenge. We focused our initial efforts on
implementation of Soft Actor Critic (SAC) variants. Our goal was to learn
non-trivial control of the race car exclusively from visual and geometric
features, directly mapping pixels to control actions. We made suitable
modifications to the default reward policy aiming to promote smooth steering
and acceleration control. The framework for the competition provided real time
simulation, meaning a single episode (learning experience) is measured in
minutes. Instead of pursuing parallelisation of episodes we opted to explore a
more traditional approach in which the visual perception was processed (via
learned operators) and fed into rule-based controllers. Such a system, while
not as academically "attractive" as a pixels-to-actions approach, results in a
system that requires less training, is more explainable, generalises better and
is easily tuned and ultimately out-performed all other agents in the
competition by a large margin.Abstract
Convex Markov Games: A Framework for Fairness, Imitation, and Creativity
in Multi-Agent Learning
arXiv:2410.16600v1 »Full PDF »Expert imitation, behavioral diversity, and fairness preferences give rise to
preferences in sequential decision making domains that do not decompose
additively across time. We introduce the class of convex Markov games that
allow general convex preferences over occupancy measures. Despite infinite time
horizon and strictly higher generality than Markov games, pure strategy Nash
equilibria exist under strict convexity. Furthermore, equilibria can be
approximated efficiently by performing gradient descent on an upper bound of
exploitability. Our experiments imitate human choices in ultimatum games,
reveal novel solutions to the repeated prisoner's dilemma, and find fair
solutions in a repeated asymmetric coordination game. In the prisoner's
dilemma, our algorithm finds a policy profile that deviates from observed human
play only slightly, yet achieves higher per-player utility while also being
three orders of magnitude less exploitable.Abstract
WorldMedQA-V: a multilingual, multimodal medical examination dataset for
multimodal language models evaluation
Multimodal/vision language models (VLMs) are increasingly being deployed in
healthcare settings worldwide, necessitating robust benchmarks to ensure their
safety, efficacy, and fairness. Multiple-choice question and answer (QA)
datasets derived from national medical examinations have long served as
valuable evaluation tools, but existing datasets are largely text-only and
available in a limited subset of languages and countries. To address these
challenges, we present WorldMedQA-V, an updated multilingual, multimodal
benchmarking dataset designed to evaluate VLMs in healthcare. WorldMedQA-V
includes 568 labeled multiple-choice QAs paired with 568 medical images from
four countries (Brazil, Israel, Japan, and Spain), covering original languages
and validated English translations by native clinicians, respectively. Baseline
performance for common open- and closed-source models are provided in the local
language and English translations, and with and without images provided to the
model. The WorldMedQA-V benchmark aims to better match AI systems to the
diverse healthcare environments in which they are deployed, fostering more
equitable, effective, and representative applications.Abstract
Stacked Universal Successor Feature Approximators for Safety in
Reinforcement Learning
Real-world problems often involve complex objective structures that resist
distillation into reinforcement learning environments with a single objective.
Operation costs must be balanced with multi-dimensional task performance and
end-states' effects on future availability, all while ensuring safety for other
agents in the environment and the reinforcement learning agent itself. System
redundancy through secondary backup controllers has proven to be an effective
method to ensure safety in real-world applications where the risk of violating
constraints is extremely high. In this work, we investigate the utility of a
stacked, continuous-control variation of universal successor feature
approximation (USFA) adapted for soft actor-critic (SAC) and coupled with a
suite of secondary safety controllers, which we call stacked USFA for safety
(SUSFAS). Our method improves performance on secondary objectives compared to
SAC baselines using an intervening secondary controller such as a runtime
assurance (RTA) controller.Abstract
FUTURE-AI: International consensus guideline for trustworthy and
deployable artificial intelligence in healthcare
arXiv:2309.12325v3 »Full PDF »Despite major advances in artificial intelligence (AI) for medicine and
healthcare, the deployment and adoption of AI technologies remain limited in
real-world clinical practice. In recent years, concerns have been raised about
the technical, clinical, ethical and legal risks associated with medical AI. To
increase real world adoption, it is essential that medical AI tools are trusted
and accepted by patients, clinicians, health organisations and authorities.
This work describes the FUTURE-AI guideline as the first international
consensus framework for guiding the development and deployment of trustworthy
AI tools in healthcare. The FUTURE-AI consortium was founded in 2021 and
currently comprises 118 inter-disciplinary experts from 51 countries
representing all continents, including AI scientists, clinicians, ethicists,
and social scientists. Over a two-year period, the consortium defined guiding
principles and best practices for trustworthy AI through an iterative process
comprising an in-depth literature review, a modified Delphi survey, and online
consensus meetings. The FUTURE-AI framework was established based on 6 guiding
principles for trustworthy AI in healthcare, i.e. Fairness, Universality,
Traceability, Usability, Robustness and Explainability. Through consensus, a
set of 28 best practices were defined, addressing technical, clinical, legal
and socio-ethical dimensions. The recommendations cover the entire lifecycle of
medical AI, from design, development and validation to regulation, deployment,
and monitoring. FUTURE-AI is a risk-informed, assumption-free guideline which
provides a structured approach for constructing medical AI tools that will be
trusted, deployed and adopted in real-world practice. Researchers are
encouraged to take the recommendations into account in proof-of-concept stages
to facilitate future translation towards clinical practice of medical AI.Abstract
Introducing v0.5 of the AI Safety Benchmark from MLCommons
arXiv:2404.12241v2 »Full PDF »This paper introduces v0.5 of the AI Safety Benchmark, which has been created
by the MLCommons AI Safety Working Group. The AI Safety Benchmark has been
designed to assess the safety risks of AI systems that use chat-tuned language
models. We introduce a principled approach to specifying and constructing the
benchmark, which for v0.5 covers only a single use case (an adult chatting to a
general-purpose assistant in English), and a limited set of personas (i.e.,
typical users, malicious users, and vulnerable users). We created a new
taxonomy of 13 hazard categories, of which 7 have tests in the v0.5 benchmark.
We plan to release version 1.0 of the AI Safety Benchmark by the end of 2024.
The v1.0 benchmark will provide meaningful insights into the safety of AI
systems. However, the v0.5 benchmark should not be used to assess the safety of
AI systems. We have sought to fully document the limitations, flaws, and
challenges of v0.5. This release of v0.5 of the AI Safety Benchmark includes
(1) a principled approach to specifying and constructing the benchmark, which
comprises use cases, types of systems under test (SUTs), language and context,
personas, tests, and test items; (2) a taxonomy of 13 hazard categories with
definitions and subcategories; (3) tests for seven of the hazard categories,
each comprising a unique set of test items, i.e., prompts. There are 43,090
test items in total, which we created with templates; (4) a grading system for
AI systems against the benchmark; (5) an openly available platform, and
downloadable tool, called ModelBench that can be used to evaluate the safety of
AI systems on the benchmark; (6) an example evaluation report which benchmarks
the performance of over a dozen openly available chat-tuned language models;
(7) a test specification for the benchmark.Abstract
Gemma: Open Models Based on Gemini Research and Technology
arXiv:2403.08295v4 »Full PDF »This work introduces Gemma, a family of lightweight, state-of-the art open
models built from the research and technology used to create Gemini models.
Gemma models demonstrate strong performance across academic benchmarks for
language understanding, reasoning, and safety. We release two sizes of models
(2 billion and 7 billion parameters), and provide both pretrained and
fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out
of 18 text-based tasks, and we present comprehensive evaluations of safety and
responsibility aspects of the models, alongside a detailed description of model
development. We believe the responsible release of LLMs is critical for
improving the safety of frontier models, and for enabling the next wave of LLM
innovations.Abstract
Enhancing Functional Safety in Automotive AMS Circuits through
Unsupervised Machine Learning
Given the widespread use of safety-critical applications in the automotive
field, it is crucial to ensure the Functional Safety (FuSa) of circuits and
components within automotive systems. The Analog and Mixed-Signal (AMS)
circuits prevalent in these systems are more vulnerable to faults induced by
parametric perturbations, noise, environmental stress, and other factors, in
comparison to their digital counterparts. However, their continuous signal
characteristics present an opportunity for early anomaly detection, enabling
the implementation of safety mechanisms to prevent system failure. To address
this need, we propose a novel framework based on unsupervised machine learning
for early anomaly detection in AMS circuits. The proposed approach involves
injecting anomalies at various circuit locations and individual components to
create a diverse and comprehensive anomaly dataset, followed by the extraction
of features from the observed circuit signals. Subsequently, we employ
clustering algorithms to facilitate anomaly detection. Finally, we propose a
time series framework to enhance and expedite anomaly detection performance.
Our approach encompasses a systematic analysis of anomaly abstraction at
multiple levels pertaining to the automotive domain, from hardware- to
block-level, where anomalies are injected to create diverse fault scenarios. By
monitoring the system behavior under these anomalous conditions, we capture the
propagation of anomalies and their effects at different abstraction levels,
thereby potentially paving the way for the implementation of reliable safety
mechanisms to ensure the FuSa of automotive SoCs. Our experimental findings
indicate that our approach achieves 100% anomaly detection accuracy and
significantly optimizes the associated latency by 5X, underscoring the
effectiveness of our devised solution.Abstract
16737 latest Fairness/Ethics + ML/AI papers
