arXiv:2311.05792v1 »Full PDF »Despite the promises of ML in education, its adoption in the classroom has
surfaced numerous issues regarding fairness, accountability, and transparency,
as well as concerns about data privacy and student consent. A root cause of
these issues is the lack of understanding of the complex dynamics of education,
including teacher-student interactions, collaborative learning, and classroom
environment. To overcome these challenges and fully utilize the potential of ML
in education, software practitioners need to work closely with educators and
students to fully understand the context of the data (the backbone of ML
applications) and collaboratively define the ML data specifications. To gain a
deeper understanding of such a collaborative process, we conduct ten co-design
sessions with ML software practitioners, educators, and students. In the
sessions, teachers and students work with ML engineers, UX designers, and legal
practitioners to define dataset characteristics for a given ML application. We
find that stakeholders contextualize data based on their domain and procedural
knowledge, proactively design data requirements to mitigate downstream harms
and data reliability concerns, and exhibit role-based collaborative strategies
and contribution patterns. Further, we find that beyond a seat at the table,
meaningful stakeholder participation in ML requires structured supports:
defined processes for continuous iteration and co-evaluation, shared contextual
data quality standards, and information scaffolds for both technical and
non-technical stakeholders to traverse expertise boundaries.Abstract
Assessing the Fairness of AI Systems: AI Practitioners' Processes,
Challenges, and Needs for Support
Camera-ready preprint of paper accepted to the CSCW conference
Various tools and practices have been developed to support practitioners in
identifying, assessing, and mitigating fairness-related harms caused by AI
systems. However, prior research has highlighted gaps between the intended
design of these tools and practices and their use within particular contexts,
including gaps caused by the role that organizational factors play in shaping
fairness work. In this paper, we investigate these gaps for one such practice:
disaggregated evaluations of AI systems, intended to uncover performance
disparities between demographic groups. By conducting semi-structured
interviews and structured workshops with thirty-three AI practitioners from ten
teams at three technology companies, we identify practitioners' processes,
challenges, and needs for support when designing disaggregated evaluations. We
find that practitioners face challenges when choosing performance metrics,
identifying the most relevant direct stakeholders and demographic groups on
which to focus, and collecting datasets with which to conduct disaggregated
evaluations. More generally, we identify impacts on fairness work stemming from
a lack of engagement with direct stakeholders or domain experts, business
imperatives that prioritize customers over marginalized groups, and the drive
to deploy AI systems at scale.Abstract
A Conceptual Framework for Ethical Evaluation of Machine Learning
Systems
arXiv:2408.10239v1 »Full PDF »Research in Responsible AI has developed a range of principles and practices
to ensure that machine learning systems are used in a manner that is ethical
and aligned with human values. However, a critical yet often neglected aspect
of ethical ML is the ethical implications that appear when designing
evaluations of ML systems. For instance, teams may have to balance a trade-off
between highly informative tests to ensure downstream product safety, with
potential fairness harms inherent to the implemented testing procedures. We
conceptualize ethics-related concerns in standard ML evaluation techniques.
Specifically, we present a utility framework, characterizing the key trade-off
in ethical evaluation as balancing information gain against potential ethical
harms. The framework is then a tool for characterizing challenges teams face,
and systematically disentangling competing considerations that teams seek to
balance. Differentiating between different types of issues encountered in
evaluation allows us to highlight best practices from analogous domains, such
as clinical trials and automotive crash testing, which navigate these issues in
ways that can offer inspiration to improve evaluation processes in ML. Our
analysis underscores the critical need for development teams to deliberately
assess and manage ethical complexities that arise during the evaluation of ML
systems, and for the industry to move towards designing institutional policies
to support ethical evaluations.Abstract
Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs
AI assistants such as ChatGPT are trained to respond to users by saying, "I
am a large language model". This raises questions. Do such models know that
they are LLMs and reliably act on this knowledge? Are they aware of their
current circumstances, such as being deployed to the public? We refer to a
model's knowledge of itself and its circumstances as situational awareness. To
quantify situational awareness in LLMs, we introduce a range of behavioral
tests, based on question answering and instruction following. These tests form
the Situational Awareness Dataset (SAD)Situational Awareness Dataset (SAD), a benchmark comprising 7
task categories and over 13,000 questions. The benchmark tests numerous
abilities, including the capacity of LLMs to (i) recognize their own generated
text, (ii) predict their own behavior, (iii) determine whether a prompt is from
internal evaluation or real-world deployment, and (iv) follow instructions that
depend on self-knowledge.
We evaluate 16 LLMs on SAD, including both base (pretrained) and chat models.
While all models perform better than chance, even the highest-scoring model
(Claude 3 Opus) is far from a human baseline on certain tasks. We also observe
that performance on SAD is only partially predicted by metrics of general
knowledge (e.g. MMLU). Chat models, which are finetuned to serve as AI
assistants, outperform their corresponding base models on SAD but not on
general knowledge tasks. The purpose of SAD is to facilitate scientific
understanding of situational awareness in LLMs by breaking it down into
quantitative abilities. Situational awareness is important because it enhances
a model's capacity for autonomous planning and action. While this has potential
benefits for automation, it also introduces novel risks related to AI safety
and control. Code and latest results available at
https://situational-awareness-dataset.org .Abstract
Unsupervised Domain Adaptation for Self-Driving from Past Traversal
Features
arXiv:2309.12140v1 »Full PDF »The rapid development of 3D object detection systems for self-driving cars
has significantly improved accuracy. However, these systems struggle to
generalize across diverse driving environments, which can lead to
safety-critical failures in detecting traffic participants. To address this, we
propose a method that utilizes unlabeled repeated traversals of multiple
locations to adapt object detectors to new driving environments. By
incorporating statistics computed from repeated LiDAR scans, we guide the
adaptation process effectively. Our approach enhances LiDAR-based detection
models using spatial quantized historical features and introduces a lightweight
regression head to leverage the statistics for feature regularization.
Additionally, we leverage the statistics for a novel self-training process to
stabilize the training. The framework is detector model-agnostic and
experiments on real-world datasets demonstrate significant improvements,
achieving up to a 20-point performance gain, especially in detecting
pedestrians and distant objects. Code is available at
https://github.com/zhangtravis/Hist-DA.Abstract
The AI Revolution: Opportunities and Challenges for the Finance Sector
arXiv:2308.16538v1 »Full PDF »This report examines Artificial Intelligence (AI) in the financial sector,
outlining its potential to revolutionise the industry and identify its
challenges. It underscores the criticality of a well-rounded understanding of
AI, its capabilities, and its implications to effectively leverage its
potential while mitigating associated risks. The potential of AI potential
extends from augmenting existing operations to paving the way for novel
applications in the finance sector. The application of AI in the financial
sector is transforming the industry. Its use spans areas from customer service
enhancements, fraud detection, and risk management to credit assessments and
high-frequency trading. However, along with these benefits, AI also presents
several challenges. These include issues related to transparency,
interpretability, fairness, accountability, and trustworthiness. The use of AI
in the financial sector further raises critical questions about data privacy
and security. A further issue identified in this report is the systemic risk
that AI can introduce to the financial sector. Being prone to errors, AI can
exacerbate existing systemic risks, potentially leading to financial crises.
Regulation is crucial to harnessing the benefits of AI while mitigating its
potential risks. Despite the global recognition of this need, there remains a
lack of clear guidelines or legislation for AI use in finance. This report
discusses key principles that could guide the formation of effective AI
regulation in the financial sector, including the need for a risk-based
approach, the inclusion of ethical considerations, and the importance of
maintaining a balance between innovation and consumer protection. The report
provides recommendations for academia, the finance industry, and regulators.Abstract
Diagnostics for Deep Neural Networks with Automated Copy/Paste Attacks
Best paper award at the NeurIPS 2022 ML Safety Workshop --
https://neurips2022.mlsafety.org/
This paper considers the problem of helping humans exercise scalable
oversight over deep neural networks (DNNs). Adversarial examples can be useful
by helping to reveal weaknesses in DNNs, but they can be difficult to interpret
or draw actionable conclusions from. Some previous works have proposed using
human-interpretable adversarial attacks including copy/paste attacks in which
one natural image pasted into another causes an unexpected misclassification.
We build on these with two contributions. First, we introduce Search for
Natural Adversarial Features Using Embeddings (SNAFUE) which offers a fully
automated method for finding copy/paste attacks. Second, we use SNAFUE to red
team an ImageNet classifier. We reproduce copy/paste attacks from previous
works and find hundreds of other easily-describable vulnerabilities, all
without a human in the loop. Code is available at
https://github.com/thestephencasper/snafueAbstract
Ithaca365: Dataset and Driving Perception under Repeated and Challenging
Weather Conditions
Advances in perception for self-driving cars have accelerated in recent years
due to the availability of large-scale datasets, typically collected at
specific locations and under nice weather conditions. Yet, to achieve the high
safety requirement, these perceptual systems must operate robustly under a wide
variety of weather conditions including snow and rain. In this paper, we
present a new dataset to enable robust autonomous driving via a novel data
collection process - data is repeatedly recorded along a 15 km route under
diverse scene (urban, highway, rural, campus), weather (snow, rain, sun), time
(day/night), and traffic conditions (pedestrians, cyclists and cars). The
dataset includes images and point clouds from cameras and LiDAR sensors, along
with high-precision GPS/INS to establish correspondence across routes. The
dataset includes road and object annotations using amodal masks to capture
partial occlusions and 3D bounding boxes. We demonstrate the uniqueness of this
dataset by analyzing the performance of baselines in amodal segmentation of
road and objects, depth estimation, and 3D object detection. The repeated
routes opens new research directions in object discovery, continual learning,
and anomaly detection. Link to Ithaca365: https://ithaca365.mae.cornell.edu/Abstract
Real-world Mapping of Gaze Fixations Using Instance Segmentation for
Road Construction Safety Applications
Research studies have shown that a large proportion of hazards remain
unrecognized, which expose construction workers to unanticipated safety risks.
Recent studies have also found that a strong correlation exists between viewing
patterns of workers, captured using eye-tracking devices, and their hazard
recognition performance. Therefore, it is important to analyze the viewing
patterns of workers to gain a better understanding of their hazard recognition
performance. This paper proposes a method that can automatically map the gaze
fixations collected using a wearable eye-tracker to the predefined areas of
interests. The proposed method detects these areas or objects (i.e., hazards)
of interests through a computer vision-based segmentation technique and
transfer learning. The mapped fixation data is then used to analyze the viewing
behaviors of workers and compute their attention distribution. The proposed
method is implemented on an under construction road as a case study to evaluate
the performance of the proposed method.Abstract
16737 latest Fairness/Ethics + ML/AI papers
