arXiv:2402.01787v3 »Full PDF »Text-to-image (T2I) models have emerged as a significant advancement in
generative AI; however, there exist safety concerns regarding their potential
to produce harmful image outputs even when users input seemingly safe prompts.
This phenomenon, where T2I models generate harmful representations that were
not explicit in the input prompt, poses a potentially greater risk than
adversarial prompts, leaving users unintentionally exposed to harms. Our paper
addresses this issue by formalizing a definition for this phenomenon which we
term harm amplification. We further contribute to the field by developing a
framework of methodologies to quantify harm amplification in which we consider
the harm of the model output in the context of user input. We then empirically
examine how to apply these different methodologies to simulate real-world
deployment scenarios including a quantification of disparate impacts across
genders resulting from harm amplification. Together, our work aims to offer
researchers tools to comprehensively address safety challenges in T2I systems
and contribute to the responsible deployment of generative AI models.Abstract
arXiv:2408.07009v1 »Full PDF »We introduce Imagen 3, a latent diffusion model that generates high quality
images from text prompts. We describe our quality and responsibility
evaluations. Imagen 3 is preferred over other state-of-the-art (SOTA) models at
the time of evaluation. In addition, we discuss issues around safety and
representation, as well as methods we used to minimize the potential harm of
our models.Abstract
Building a Domain-specific Guardrail Model in Production
arXiv:2408.01452v1 »Full PDF »Generative AI holds the promise of enabling a range of sought-after
capabilities and revolutionizing workflows in various consumer and enterprise
verticals. However, putting a model in production involves much more than just
generating an output. It involves ensuring the model is reliable, safe,
performant and also adheres to the policy of operation in a particular domain.
Guardrails as a necessity for models has evolved around the need to enforce
appropriate behavior of models, especially when they are in production. In this
paper, we use education as a use case, given its stringent requirements of the
appropriateness of content in the domain, to demonstrate how a guardrail model
can be trained and deployed in production. Specifically, we describe our
experience in building a production-grade guardrail model for a K-12
educational platform. We begin by formulating the requirements for deployment
to this sensitive domain. We then describe the training and benchmarking of our
domain-specific guardrail model, which outperforms competing open- and closed-
instruction-tuned models of similar and larger size, on proprietary
education-related benchmarks and public benchmarks related to general aspects
of safety. Finally, we detail the choices we made on architecture and the
optimizations for deploying this service in production; these range across the
stack from the hardware infrastructure to the serving layer to language model
inference optimizations. We hope this paper will be instructive to other
practitioners looking to create production-grade domain-specific services based
on generative AI and large language models.Abstract
Act as You Learn: Adaptive Decision-Making in Non-Stationary Markov
Decision Processes
Accepted for publication at the International Conference on
Autonomous Agents and MultiAgent Syste...
A fundamental (and largely open) challenge in sequential decision-making is
dealing with non-stationary environments, where exogenous environmental
conditions change over time. Such problems are traditionally modeled as
non-stationary Markov decision processes (NSMDP). However, existing approaches
for decision-making in NSMDPs have two major shortcomings: first, they assume
that the updated environmental dynamics at the current time are known (although
future dynamics can change); and second, planning is largely pessimistic, i.e.,
the agent acts ``safely'' to account for the non-stationary evolution of the
environment. We argue that both these assumptions are invalid in practice --
updated environmental conditions are rarely known, and as the agent interacts
with the environment, it can learn about the updated dynamics and avoid being
pessimistic, at least in states whose dynamics it is confident about. We
present a heuristic search algorithm called \textit{Adaptive Monte Carlo Tree
Search (ADA-MCTS)} that addresses these challenges. We show that the agent can
learn the updated dynamics of the environment over time and then act as it
learns, i.e., if the agent is in a region of the state space about which it has
updated knowledge, it can avoid being pessimistic. To quantify ``updated
knowledge,'' we disintegrate the aleatoric and epistemic uncertainty in the
agent's updated belief and show how the agent can use these estimates for
decision-making. We compare the proposed approach with the multiple
state-of-the-art approaches in decision-making across multiple well-established
open-source problems and empirically show that our approach is faster and
highly adaptive without sacrificing safety.Abstract
Detection of Unknown-Unknowns in Human-in-Plant Human-in-Loop Systems
Using Physics Guided Process Models
arXiv:2309.02603v2 »Full PDF »Unknown-unknowns are operational scenarios in systems that are not accounted
for in the design and test phase. In such scenarios, the operational behavior
of the Human-in-loop (HIL) Human-in-Plant (HIP) systems is not guaranteed to
meet requirements such as safety and efficacy. We propose a novel framework for
analyzing the operational output characteristics of safety-critical HIL-HIP
systems that can discover unknown-unknown scenarios and evaluate potential
safety hazards. We propose dynamics-induced hybrid recurrent neural networks
(DiH-RNN) to mine a physics-guided surrogate model (PGSM) that checks for
deviation of the cyber-physical system (CPS) from safety-certified operational
characteristics. The PGSM enables early detection of unknown-unknowns based on
the physical laws governing the system. We demonstrate the detection of
operational changes in an Artificial Pancreas(AP) due to unknown insulin
cartridge errors.Abstract
Fairguard: Harness Logic-based Fairness Rules in Smart Cities
arXiv:2302.11137v7 »Full PDF »Smart cities operate on computational predictive frameworks that collect,
aggregate, and utilize data from large-scale sensor networks. However, these
frameworks are prone to multiple sources of data and algorithmic bias, which
often lead to unfair prediction results. In this work, we first demonstrate
that bias persists at a micro-level both temporally and spatially by studying
real city data from Chattanooga, TN. To alleviate the issue of such bias, we
introduce Fairguard, a micro-level temporal logic-based approach for fair smart
city policy adjustment and generation in complex temporal-spatial domains. The
Fairguard framework consists of two phases: first, we develop a static
generator that is able to reduce data bias based on temporal logic conditions
by minimizing correlations between selected attributes. Then, to ensure
fairness in predictive algorithms, we design a dynamic component to regulate
prediction results and generate future fair predictions by harnessing logic
rules. Evaluations show that logic-enabled static Fairguard can effectively
reduce the biased correlations while dynamic Fairguard can guarantee fairness
on protected groups at run-time with minimal impact on overall performance.Abstract
This is a pre-print for a book chapter to appear in Vorobeychik,
Yevgeniy., and Mukhopadhyay, Ayan...
Emergency response management (ERM) is a challenge faced by communities
across the globe. First responders must respond to various incidents, such as
fires, traffic accidents, and medical emergencies. They must respond quickly to
incidents to minimize the risk to human life. Consequently, considerable
attention has been devoted to studying emergency incidents and response in the
last several decades. In particular, data-driven models help reduce human and
financial loss and improve design codes, traffic regulations, and safety
measures. This tutorial paper explores four sub-problems within emergency
response: incident prediction, incident detection, resource allocation, and
resource dispatch. We aim to present mathematical formulations for these
problems and broad frameworks for each problem. We also share open-source
(synthetic) data from a large metropolitan area in the USA for future work on
data-driven emergency response.Abstract
Dynamic Simplex: Balancing Safety and Performance in Autonomous Cyber
Physical Systems
arXiv:2302.09750v1 »Full PDF »Learning Enabled Components (LEC) have greatly assisted cyber-physical
systems in achieving higher levels of autonomy. However, LEC's susceptibility
to dynamic and uncertain operating conditions is a critical challenge for the
safety of these systems. Redundant controller architectures have been widely
adopted for safety assurance in such contexts. These architectures augment LEC
"performant" controllers that are difficult to verify with "safety" controllers
and the decision logic to switch between them. While these architectures ensure
safety, we point out two limitations. First, they are trained offline to learn
a conservative policy of always selecting a controller that maintains the
system's safety, which limits the system's adaptability to dynamic and
non-stationary environments. Second, they do not support reverse switching from
the safety controller to the performant controller, even when the threat to
safety is no longer present. To address these limitations, we propose a dynamic
simplex strategy with an online controller switching logic that allows two-way
switching. We consider switching as a sequential decision-making problem and
model it as a semi-Markov decision process. We leverage a combination of a
myopic selector using surrogate models (for the forward switch) and a
non-myopic planner (for the reverse switch) to balance safety and performance.
We evaluate this approach using an autonomous vehicle case study in the CARLA
simulator using different driving conditions, locations, and component
failures. We show that the proposed approach results in fewer collisions and
higher performance than state-of-the-art alternatives.Abstract
No Language Left Behind: Scaling Human-Centered Machine Translation
Driven by the goal of eradicating language barriers on a global scale,
machine translation has solidified itself as a key focus of artificial
intelligence research today. However, such efforts have coalesced around a
small subset of languages, leaving behind the vast majority of mostly
low-resource languages. What does it take to break the 200 language barrier
while ensuring safe, high quality results, all while keeping ethical
considerations in mind? In No Language Left Behind, we took on this challenge
by first contextualizing the need for low-resource language translation support
through exploratory interviews with native speakers. Then, we created datasets
and models aimed at narrowing the performance gap between low and high-resource
languages. More specifically, we developed a conditional compute model based on
Sparsely Gated Mixture of Experts that is trained on data obtained with novel
and effective data mining techniques tailored for low-resource languages. We
propose multiple architectural and training improvements to counteract
overfitting while training on thousands of tasks. Critically, we evaluated the
performance of over 40,000 different translation directions using a
human-translated benchmark, Flores-200, and combined human evaluation with a
novel toxicity benchmark covering all languages in Flores-200 to assess
translation safety. Our model achieves an improvement of 44% BLEU relative to
the previous state-of-the-art, laying important groundwork towards realizing a
universal translation system. Finally, we open source all contributions
described in this work, accessible at
https://github.com/facebookresearch/fairseq/tree/nllb.Abstract
Don't Throw it Away! The Utility of Unlabeled Data in Fair Decision
Making
arXiv:2205.04790v3 »Full PDF »Decision making algorithms, in practice, are often trained on data that
exhibits a variety of biases. Decision-makers often aim to take decisions based
on some ground-truth target that is assumed or expected to be unbiased, i.e.,
equally distributed across socially salient groups. In many practical settings,
the ground-truth cannot be directly observed, and instead, we have to rely on a
biased proxy measure of the ground-truth, i.e., biased labels, in the data. In
addition, data is often selectively labeled, i.e., even the biased labels are
only observed for a small fraction of the data that received a positive
decision. To overcome label and selection biases, recent work proposes to learn
stochastic, exploring decision policies via i) online training of new policies
at each time-step and ii) enforcing fairness as a constraint on performance.
However, the existing approach uses only labeled data, disregarding a large
amount of unlabeled data, and thereby suffers from high instability and
variance in the learned decision policies at different times. In this paper, we
propose a novel method based on a variational autoencoder for practical fair
decision-making. Our method learns an unbiased data representation leveraging
both labeled and unlabeled data and uses the representations to learn a policy
in an online process. Using synthetic data, we empirically validate that our
method converges to the optimal (fair) policy according to the ground-truth
with low variance. In real-world experiments, we further show that our training
approach not only offers a more stable learning process but also yields
policies with higher fairness as well as utility than previous approaches.Abstract
16737 latest Fairness/Ethics + ML/AI papers
