arXiv:2406.00515v2 »Full PDF »Large Language Models (LLMs) have garnered remarkable advancements across
diverse code-related tasks, known as Code LLMs, particularly in code generation
that generates source code with LLM from natural language descriptions. This
burgeoning field has captured significant interest from both academic
researchers and industry professionals due to its practical significance in
software development, e.g., GitHub Copilot. Despite the active exploration of
LLMs for a variety of code tasks, either from the perspective of natural
language processing (NLP) or software engineering (SE) or both, there is a
noticeable absence of a comprehensive and up-to-date literature review
dedicated to LLM for code generation. In this survey, we aim to bridge this gap
by providing a systematic literature review that serves as a valuable reference
for researchers investigating the cutting-edge progress in LLMs for code
generation. We introduce a taxonomy to categorize and discuss the recent
developments in LLMs for code generation, covering aspects such as data
curation, latest advances, performance evaluation, ethical implications,
environmental impact, and real-world applications. In addition, we present a
historical overview of the evolution of LLMs for code generation and offer an
empirical comparison using the HumanEval, MBPP, and BigCodeBench benchmarks
across various levels of difficulty and types of programming tasks to highlight
the progressive enhancements in LLM capabilities for code generation. We
identify critical challenges and promising opportunities regarding the gap
between academia and practical development. Furthermore, we have established a
dedicated resource GitHub page (https://github.com/juyongjiang/CodeLLMSurvey)
to continuously document and disseminate the most recent advances in the field.Abstract
44 pages; updated authors list and fixed author names
We introduce HyperCLOVA X, a family of large language models (LLMs) tailored
to the Korean language and culture, along with competitive capabilities in
English, math, and coding. HyperCLOVA X was trained on a balanced mix of
Korean, English, and code data, followed by instruction-tuning with
high-quality human-annotated datasets while abiding by strict safety guidelines
reflecting our commitment to responsible AI. The model is evaluated across
various benchmarks, including comprehensive reasoning, knowledge, commonsense,
factuality, coding, math, chatting, instruction-following, and harmlessness, in
both Korean and English. HyperCLOVA X exhibits strong reasoning capabilities in
Korean backed by a deep understanding of the language and cultural nuances.
Further analysis of the inherent bilingual nature and its extension to
multilingualism highlights the model's cross-lingual proficiency and strong
generalization ability to untargeted languages, including machine translation
between several language pairs and cross-lingual inference tasks. We believe
that HyperCLOVA X can provide helpful guidance for regions or countries in
developing their sovereign LLMs.Abstract
Fairness Risks for Group-conditionally Missing Demographics
arXiv:2402.13393v2 »Full PDF »Fairness-aware classification models have gained increasing attention in
recent years as concerns grow on discrimination against some demographic
groups. Most existing models require full knowledge of the sensitive features,
which can be impractical due to privacy, legal issues, and an individual's fear
of discrimination. The key challenge we will address is the group dependency of
the unavailability, e.g., people of some age range may be more reluctant to
reveal their age. Our solution augments general fairness risks with
probabilistic imputations of the sensitive features, while jointly learning the
group-conditionally missing probabilities in a variational auto-encoder. Our
model is demonstrated effective on both image and tabular datasets, achieving
an improved balance between accuracy and fairness.Abstract
Narrative Feature or Structured Feature? A Study of Large Language
Models to Identify Cancer Patients at Risk of Heart Failure
Cancer treatments are known to introduce cardiotoxicity, negatively impacting
outcomes and survivorship. Identifying cancer patients at risk of heart failure
(HF) is critical to improving cancer treatment outcomes and safety. This study
examined machine learning (ML) models to identify cancer patients at risk of HF
using electronic health records (EHRs), including traditional ML, Time-Aware
long short-term memory (T-LSTM), and large language models (LLMs) using novel
narrative features derived from the structured medical codes. We identified a
cancer cohort of 12,806 patients from the University of Florida Health,
diagnosed with lung, breast, and colorectal cancers, among which 1,602
individuals developed HF after cancer. The LLM, GatorTron-3.9B, achieved the
best F1 scores, outperforming the traditional support vector machines by 39%,
the T-LSTM deep learning model by 7%, and a widely used transformer model,
BERT, by 5.6%. The analysis shows that the proposed narrative features
remarkably increased feature density and improved performance.Abstract
FEED: Fairness-Enhanced Meta-Learning for Domain Generalization
Generalizing to out-of-distribution data while being aware of model fairness
is a significant and challenging problem in meta-learning. The goal of this
problem is to find a set of fairness-aware invariant parameters of classifier
that is trained using data drawn from a family of related training domains with
distribution shift on non-sensitive features as well as different levels of
dependence between model predictions and sensitive features so that the
classifier can achieve good generalization performance on unknown but distinct
test domains. To tackle this challenge, existing state-of-the-art methods
either address the domain generalization problem but completely ignore learning
with fairness or solely specify shifted domains with various fairness levels.
This paper introduces an approach to fairness-aware meta-learning that
significantly enhances domain generalization capabilities. Our framework,
Fairness-Enhanced Meta-Learning for Domain Generalization (FEED), disentangles
latent data representations into content, style, and sensitive vectors. This
disentanglement facilitates the robust generalization of machine learning
models across diverse domains while adhering to fairness constraints. Unlike
traditional methods that focus primarily on domain invariance or sensitivity to
shifts, our model integrates a fairness-aware invariance criterion directly
into the meta-learning process. This integration ensures that the learned
parameters uphold fairness consistently, even when domain characteristics vary
widely. We validate our approach through extensive experiments across multiple
benchmarks, demonstrating not only superior performance in maintaining high
accuracy and fairness but also significant improvements over existing
state-of-the-art methods in domain generalization tasks.Abstract
Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench
Generative models such as Large Language Models (LLM) and Multimodal Large
Language models (MLLMs) trained on massive web corpora can memorize and
disclose individuals' confidential and private data, raising legal and ethical
concerns. While many previous works have addressed this issue in LLM via
machine unlearning, it remains largely unexplored for MLLMs. To tackle this
challenge, we introduce Multimodal Large Language Model Unlearning Benchmark
(MLLMU-Bench), a novel benchmark aimed at advancing the understanding of
multimodal machine unlearning. MLLMU-Bench consists of 500 fictitious profiles
and 153 profiles for public celebrities, each profile feature over 14
customized question-answer pairs, evaluated from both multimodal (image+text)
and unimodal (text) perspectives. The benchmark is divided into four sets to
assess unlearning algorithms in terms of efficacy, generalizability, and model
utility. Finally, we provide baseline results using existing generative model
unlearning algorithms. Surprisingly, our experiments show that unimodal
unlearning algorithms excel in generation and cloze tasks, while multimodal
unlearning approaches perform better in classification tasks with multimodal
inputs.Abstract
GLBench: A Comprehensive Benchmark for Graph with Large Language Models
arXiv:2407.07457v4 »Full PDF »The emergence of large language models (LLMs) has revolutionized the way we
interact with graphs, leading to a new paradigm called GraphLLM. Despite the
rapid development of GraphLLM methods in recent years, the progress and
understanding of this field remain unclear due to the lack of a benchmark with
consistent experimental protocols. To bridge this gap, we introduce GLBench,
the first comprehensive benchmark for evaluating GraphLLM methods in both
supervised and zero-shot scenarios. GLBench provides a fair and thorough
evaluation of different categories of GraphLLM methods, along with traditional
baselines such as graph neural networks. Through extensive experiments on a
collection of real-world datasets with consistent data processing and splitting
strategies, we have uncovered several key findings. Firstly, GraphLLM methods
outperform traditional baselines in supervised settings, with LLM-as-enhancers
showing the most robust performance. However, using LLMs as predictors is less
effective and often leads to uncontrollable output issues. We also notice that
no clear scaling laws exist for current GraphLLM methods. In addition, both
structures and semantics are crucial for effective zero-shot transfer, and our
proposed simple baseline can even outperform several models tailored for
zero-shot scenarios. The data and code of the benchmark can be found at
https://github.com/NineAbyss/GLBench.Abstract
arXiv:2411.00827v1 »Full PDF »As large Vision-Language Models (VLMs) continue to gain prominence, ensuring
their safety deployment in real-world applications has become a critical
concern. Recently, significant research efforts have focused on evaluating the
robustness of VLMs against jailbreak attacks. Due to challenges in obtaining
multi-modal data, current studies often assess VLM robustness by generating
adversarial or query-relevant images based on harmful text datasets. However,
the jailbreak images generated this way exhibit certain limitations.
Adversarial images require white-box access to the target VLM and are
relatively easy to defend against, while query-relevant images must be linked
to the target harmful content, limiting their diversity and effectiveness. In
this paper, we propose a novel jailbreak method named IDEATOR, which
autonomously generates malicious image-text pairs for black-box jailbreak
attacks. IDEATOR is a VLM-based approach inspired by our conjecture that a VLM
itself might be a powerful red team model for generating jailbreak prompts.
Specifically, IDEATOR employs a VLM to generate jailbreak texts while
leveraging a state-of-the-art diffusion model to create corresponding jailbreak
images. Extensive experiments demonstrate the high effectiveness and
transferability of IDEATOR. It successfully jailbreaks MiniGPT-4 with a 94%
success rate and transfers seamlessly to LLaVA and InstructBLIP, achieving high
success rates of 82% and 88%, respectively. IDEATOR uncovers previously
unrecognized vulnerabilities in VLMs, calling for advanced safety mechanisms.Abstract
Reliable and Efficient Concept Erasure of Text-to-Image Diffusion Models
Text-to-image models encounter safety issues, including concerns related to
copyright and Not-Safe-For-Work (NSFW) content. Despite several methods have
been proposed for erasing inappropriate concepts from diffusion models, they
often exhibit incomplete erasure, consume a lot of computing resources, and
inadvertently damage generation ability. In this work, we introduce Reliable
and Efficient Concept Erasure (RECE), a novel approach that modifies the model
in 3 seconds without necessitating additional fine-tuning. Specifically, RECE
efficiently leverages a closed-form solution to derive new target embeddings,
which are capable of regenerating erased concepts within the unlearned model.
To mitigate inappropriate content potentially represented by derived
embeddings, RECE further aligns them with harmless concepts in cross-attention
layers. The derivation and erasure of new representation embeddings are
conducted iteratively to achieve a thorough erasure of inappropriate concepts.
Besides, to preserve the model's generation ability, RECE introduces an
additional regularization term during the derivation process, resulting in
minimizing the impact on unrelated concepts during the erasure process. All the
processes above are in closed-form, guaranteeing extremely efficient erasure in
only 3 seconds. Benchmarking against previous approaches, our method achieves
more efficient and thorough erasure with minor damage to original generation
ability and demonstrates enhanced robustness against red-teaming tools. Code is
available at \url{https://github.com/CharlesGong12/RECE}.Abstract
BlueSuffix: Reinforced Blue Teaming for Vision-Language Models Against
Jailbreak Attacks
arXiv:2410.20971v1 »Full PDF »Despite their superb multimodal capabilities, Vision-Language Models (VLMs)
have been shown to be vulnerable to jailbreak attacks, which are inference-time
attacks that induce the model to output harmful responses with tricky prompts.
It is thus essential to defend VLMs against potential jailbreaks for their
trustworthy deployment in real-world applications. In this work, we focus on
black-box defense for VLMs against jailbreak attacks. Existing black-box
defense methods are either unimodal or bimodal. Unimodal methods enhance either
the vision or language module of the VLM, while bimodal methods robustify the
model through text-image representation realignment. However, these methods
suffer from two limitations: 1) they fail to fully exploit the cross-modal
information, or 2) they degrade the model performance on benign inputs. To
address these limitations, we propose a novel blue-team method BlueSuffix that
defends the black-box target VLM against jailbreak attacks without compromising
its performance. BlueSuffix includes three key components: 1) a visual purifier
against jailbreak images, 2) a textual purifier against jailbreak texts, and 3)
a blue-team suffix generator fine-tuned via reinforcement learning for
enhancing cross-modal robustness. We empirically show on three VLMs (LLaVA,
MiniGPT-4, and Gemini) and two safety benchmarks (MM-SafetyBench and
RedTeam-2K) that BlueSuffix outperforms the baseline defenses by a significant
margin. Our BlueSuffix opens up a promising direction for defending VLMs
against jailbreak attacks.Abstract
16737 latest Fairness/Ethics + ML/AI papers
