arXiv:2410.12999v1 »Full PDF »Balancing safety and usefulness in large language models has become a
critical challenge in recent years. Models often exhibit unsafe behavior or
adopt an overly cautious approach, leading to frequent overrefusal of benign
prompts, which reduces their usefulness. Addressing these issues requires
methods that maintain safety while avoiding overrefusal. In this work, we
examine how the overgeneration of training data using advanced teacher models
(e.g., GPT-4o), including responses to both general-purpose and toxic prompts,
influences the safety and overrefusal balance of instruction-following language
models. Additionally, we present POROver, a strategy to use preference
optimization methods in order to reduce overrefusal, via employing a superior
teacher model's completions. Our results show that overgenerating completions
for general-purpose prompts significantly improves the balance between safety
and usefulness. Specifically, the F1 score calculated between safety and
usefulness increases from 70.8% to 88.3%. Moreover, overgeneration for toxic
prompts substantially reduces overrefusal, decreasing it from 94.4% to 45.2%.
Furthermore, preference optimization algorithms, when applied with carefully
curated preference data, can effectively reduce a model's overrefusal from
45.2% to 15.0% while maintaining comparable safety levels. Our code and data
are available at https://github.com/batuhankmkaraman/POROver.Abstract
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your
Phone
We introduce phi-3-mini, a 3.8 billion parameter language model trained on
3.3 trillion tokens, whose overall performance, as measured by both academic
benchmarks and internal testing, rivals that of models such as Mixtral 8x7B and
GPT-3.5 (e.g., phi-3-mini achieves 69% on MMLU and 8.38 on MT-bench), despite
being small enough to be deployed on a phone. Our training dataset is a
scaled-up version of the one used for phi-2, composed of heavily filtered
publicly available web data and synthetic data. The model is also further
aligned for robustness, safety, and chat format. We also provide
parameter-scaling results with a 7B, 14B models trained for 4.8T tokens, called
phi-3-small, phi-3-medium, both significantly more capable than phi-3-mini
(e.g., respectively 75%, 78% on MMLU, and 8.7, 8.9 on MT-bench). To enhance
multilingual, multimodal, and long-context capabilities, we introduce three
models in the phi-3.5 series: phi-3.5-mini, phi-3.5-MoE, and phi-3.5-Vision.
The phi-3.5-MoE, a 16 x 3.8B MoE model with 6.6 billion active parameters,
achieves superior performance in language reasoning, math, and code tasks
compared to other open-source models of similar scale, such as Llama 3.1 and
the Mixtral series, and on par with Gemini-1.5-Flash and GPT-4o-mini.
Meanwhile, phi-3.5-Vision, a 4.2 billion parameter model derived from
phi-3.5-mini, excels in reasoning tasks and is adept at handling both
single-image and text prompts, as well as multi-image and text prompts.Abstract
Authored by the Center for Research on Foundation Models (CRFM) at
the Stanford Institute for Huma...
Language models (LMs) are becoming the foundation for almost all major
language technologies, but their capabilities, limitations, and risks are not
well understood. We present Holistic Evaluation of Language Models (HELM) to
improve the transparency of language models. First, we taxonomize the vast
space of potential scenarios (i.e. use cases) and metrics (i.e. desiderata)
that are of interest for LMs. Then we select a broad subset based on coverage
and feasibility, noting what's missing or underrepresented (e.g. question
answering for neglected English dialects, metrics for trustworthiness). Second,
we adopt a multi-metric approach: We measure 7 metrics (accuracy, calibration,
robustness, fairness, bias, toxicity, and efficiency) for each of 16 core
scenarios when possible (87.5% of the time). This ensures metrics beyond
accuracy don't fall to the wayside, and that trade-offs are clearly exposed. We
also perform 7 targeted evaluations, based on 26 targeted scenarios, to analyze
specific aspects (e.g. reasoning, disinformation). Third, we conduct a
large-scale evaluation of 30 prominent language models (spanning open,
limited-access, and closed models) on all 42 scenarios, 21 of which were not
previously used in mainstream LM evaluation. Prior to HELM, models on average
were evaluated on just 17.9% of the core HELM scenarios, with some prominent
models not sharing a single scenario in common. We improve this to 96.0%: now
all 30 models have been densely benchmarked on the same core scenarios and
metrics under standardized conditions. Our evaluation surfaces 25 top-level
findings. For full transparency, we release all raw model prompts and
completions publicly for further analysis, as well as a general modular
toolkit. We intend for HELM to be a living benchmark for the community,
continuously updated with new scenarios, metrics, and models.Abstract
SMoA: Improving Multi-agent Large Language Models with Sparse
Mixture-of-Agents
While multi-agent systems have been shown to significantly enhance the
performance of Large Language Models (LLMs) across various tasks and
applications, the dense interaction between scaling agents potentially hampers
their efficiency and diversity. To address these challenges, we draw
inspiration from the sparse mixture-of-agents (SMoE) and propose a sparse
mixture-of-agents (SMoA) framework to improve the efficiency and diversity of
multi-agent LLMs. Unlike completely connected structures, SMoA introduces novel
Response Selection and Early Stopping mechanisms to sparsify information flows
among individual LLM agents, striking a balance between performance and
efficiency. Additionally, inspired by the expert diversity principle in SMoE
frameworks for workload balance between experts, we assign distinct role
descriptions to each LLM agent, fostering diverse and divergent thinking.
Extensive experiments on reasoning, alignment, and fairness benchmarks
demonstrate that SMoA achieves performance comparable to traditional
mixture-of-agents approaches but with significantly lower computational costs.
Further analysis reveals that SMoA is more stable, has a greater capacity to
scale, and offers considerable potential through hyper-parameter optimization.
Code and data will be available at: https://github.com/David-Li0406/SMoA.Abstract
ICPR 2024 Competition on Safe Segmentation of Drive Scenes in
Unstructured Traffic and Adverse Weather Conditions
The ICPR 2024 Competition on Safe Segmentation of Drive Scenes in
Unstructured Traffic and Adverse Weather Conditions served as a rigorous
platform to evaluate and benchmark state-of-the-art semantic segmentation
models under challenging conditions for autonomous driving. Over several
months, participants were provided with the IDD-AW dataset, consisting of 5000
high-quality RGB-NIR image pairs, each annotated at the pixel level and
captured under adverse weather conditions such as rain, fog, low light, and
snow. A key aspect of the competition was the use and improvement of the Safe
mean Intersection over Union (Safe mIoU) metric, designed to penalize unsafe
incorrect predictions that could be overlooked by traditional mIoU. This
innovative metric emphasized the importance of safety in developing autonomous
driving systems. The competition showed significant advancements in the field,
with participants demonstrating models that excelled in semantic segmentation
and prioritized safety and robustness in unstructured and adverse conditions.
The results of the competition set new benchmarks in the domain, highlighting
the critical role of safety in deploying autonomous vehicles in real-world
scenarios. The contributions from this competition are expected to drive
further innovation in autonomous driving technology, addressing the critical
challenges of operating in diverse and unpredictable environments.Abstract
Bypassing the Safety Training of Open-Source LLMs with Priming Attacks
With the recent surge in popularity of LLMs has come an ever-increasing need
for LLM safety training. In this paper, we investigate the fragility of SOTA
open-source LLMs under simple, optimization-free attacks we refer to as
priming attacks, which are easy to execute and effectively bypass
alignment from safety training. Our proposed attack improves the Attack Success
Rate on Harmful Behaviors, as measured by Llama Guard, by up to 3.3×
compared to baselines. Source code and data are available at
https://github.com/uiuc-focal-lab/llm-priming-attacks.Abstract
Practical Bias Mitigation through Proxy Sensitive Attribute Label
Generation
Modelling Uncertainty in the Financial World (MUFin) Workshop in
AAAI2023
Addressing bias in the trained machine learning system often requires access
to sensitive attributes. In practice, these attributes are not available either
due to legal and policy regulations or data unavailability for a given
demographic. Existing bias mitigation algorithms are limited in their
applicability to real-world scenarios as they require access to sensitive
attributes to achieve fairness. In this research work, we aim to address this
bottleneck through our proposed unsupervised proxy-sensitive attribute label
generation technique. Towards this end, we propose a two-stage approach of
unsupervised embedding generation followed by clustering to obtain
proxy-sensitive labels. The efficacy of our work relies on the assumption that
bias propagates through non-sensitive attributes that are correlated to the
sensitive attributes and, when mapped to the high dimensional latent space,
produces clusters of different demographic groups that exist in the data.
Experimental results demonstrate that bias mitigation using existing algorithms
such as Fair Mixup and Adversarial Debiasing yields comparable results on
derived proxy labels when compared against using true sensitive attributes.Abstract
PDB-Struct: A Comprehensive Benchmark for Structure-based Protein Design
Structure-based protein design has attracted increasing interest, with
numerous methods being introduced in recent years. However, a universally
accepted method for evaluation has not been established, since the wet-lab
validation can be overly time-consuming for the development of new algorithms,
and the in silico validation with recovery and perplexity metrics is
efficient but may not precisely reflect true foldability. To address this gap,
we introduce two novel metrics: refoldability-based metric, which leverages
high-accuracy protein structure prediction models as a proxy for wet lab
experiments, and stability-based metric, which assesses whether models can
assign high likelihoods to experimentally stable proteins. We curate datasets
from high-quality CATH protein data, high-throughput de novo
designed proteins, and mega-scale experimental mutagenesis experiments, and in
doing so, present the PDB-Struct benchmark that evaluates both
recent and previously uncompared protein design methods. Experimental results
indicate that ByProt, ProteinMPNN, and ESM-IF perform exceptionally well on our
benchmark, while ESM-Design and AF-Design fall short on the refoldability
metric. We also show that while some methods exhibit high sequence recovery,
they do not perform as well on our new benchmark. Our proposed benchmark paves
the way for a fair and comprehensive evaluation of protein design methods in
the future. Code is available at https://github.com/WANG-CR/PDB-Struct.Abstract
IDD-AW: A Benchmark for Safe and Robust Segmentation of Drive Scenes in
Unstructured Traffic and Adverse Weather
8 pages excluding references. Accepted in WACV 2024
Large-scale deployment of fully autonomous vehicles requires a very high
degree of robustness to unstructured traffic, and weather conditions, and
should prevent unsafe mispredictions. While there are several datasets and
benchmarks focusing on segmentation for drive scenes, they are not specifically
focused on safety and robustness issues. We introduce the IDD-AW dataset, which
provides 5000 pairs of high-quality images with pixel-level annotations,
captured under rain, fog, low light, and snow in unstructured driving
conditions. As compared to other adverse weather datasets, we provide i.) more
annotated images, ii.) paired Near-Infrared (NIR) image for each frame, iii.)
larger label set with a 4-level label hierarchy to capture unstructured traffic
conditions. We benchmark state-of-the-art models for semantic segmentation in
IDD-AW. We also propose a new metric called ''Safe mean Intersection over Union
(Safe mIoU)'' for hierarchical datasets which penalizes dangerous
mispredictions that are not captured in the traditional definition of mean
Intersection over Union (mIoU). The results show that IDD-AW is one of the most
challenging datasets to date for these tasks. The dataset and code will be
available here: http://iddaw.github.io.Abstract
arXiv:2310.14164v1 »Full PDF »Contextual bandit algorithms are at the core of many applications, including
recommender systems, clinical trials, and optimal portfolio selection. One of
the most popular problems studied in the contextual bandit literature is to
maximize the sum of the rewards in each round by ensuring a sublinear regret
against the best-fixed context-dependent policy. However, in many applications,
the cumulative reward is not the right objective - the bandit algorithm must be
fair in order to avoid the echo-chamber effect and comply with the regulatory
requirements. In this paper, we consider the α-Fair Contextual Bandits
problem, where the objective is to maximize the global α-fair utility
function - a non-decreasing concave function of the cumulative rewards in the
adversarial setting. The problem is challenging due to the non-separability of
the objective across rounds. We design an efficient algorithm that guarantees
an approximately sublinear regret in the full-information and bandit feedback
settings.Abstract
16737 latest Fairness/Ethics + ML/AI papers
