Skip to yearly menu bar Skip to main content

Workshop Fri, Jul 18, 2025 • 9:00 AM – 5:00 PM PDT

2nd Workshop on Models of Human Feedback for AI Alignment (MoFA)

Belen Martin Urcelay · Micah Carroll · Maria Teresa Parreira · Thomas Kleine Buening · Andreas Krause · Anca Dragan

Project Page [ OpenReview]

Abstract

Our workshop brings together experts in machine learning, cognitive science, behavioral psychology, and economics to explore human-AI alignment by examining human (and AI) feedback mechanisms, their mathematical models, and practical implications. By fostering collaboration between technical and behavioral science communities, we aim to develop more realistic models of human feedback that can better inform the development of aligned AI systems.

Video

Chat is not available.

Schedule

Timezone: America/Los_Angeles

9:00 AM

Opening Remarks 'Intro'

Video

9:05 AM

Explainable Decision Support and Justification for Model Alignment in Human-Robot Teams Invited Talk

Matthew Luebbers

Video

9:40 AM

Aligned Textual Scoring Rule Oral

Yuxuan Lu · Yifan Wu · Jason Hartline · Michael Curry

Video

9:55 AM

Copilot Arena: A Platform for Code LLM Evaluation in the Wild Oral

Wayne Chi · Valerie Chen · Anastasios Angelopoulos · Wei-Lin Chiang · Aditya Mittal · Naman Jain · Tianjun Zhang · Ion Stoica · Chris Donahue · Ameet Talwalkar

Video

10:10 AM

Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes Oral

Katarzyna Kobalczyk · Claudio Fanconi · Hao Sun · Mihaela van der Schaar

Video

10:25 AM

Alignment is social: lessons from human alignment for AI Invited Talk

Gillian Hadfield

Video

11:00 AM

Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset Oral

Lily Zhang · Smitha Milli · Karen Jusko · Jonathan Smith · Brandon Amos · Wassim Bouaziz · Jack Kussman · Manon Revel · Lisa Titus · Bhaktipriya Radharapu · Jane Dwivedi-Yu · Vidya Sarma · Kristopher Rose · Maximilian Nickel

Video

11:15 AM

Deep Context-Dependent Choice Model Oral

Shuhan Zhang · Zhi Wang · Rui Gao · Shuang Li

Video

11:30 AM

The Limits of Preferences: Navigating Human-AI Feedback Tradeoffs in Alignment Invited Talk

Valentina Pyatkin

Video

12:05 PM

Lunch and Poster Session I Poster

1:30 PM

Personalization and pluralistic alignment of LLMs via reinforcement learning fine-tuning Invited Talk

Natasha Jaques

Video

2:05 PM

Panel Panel

Stephen Casper · Yilun Zhou · Goran Radanovic · Natasha Jaques

Video

3:00 PM

Coffee Break 'Break'

3:30 PM

Doctor Approved: Generating Medically Accurate Skin Disease Images through AI–Expert Feedback Oral

Janet Wang · Yunbei Zhang · Zhengming Ding · Jihun Hamm

Video

3:45 PM

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings Oral

Jenny Huang · Yunyi Shen · Dennis Wei · Tamara Broderick

Video

4:00 PM

Poster Session II Poster

4:55 PM

Closing Remarks Close

→ Learning interpretable descriptions of human preferences Poster

Rajiv Movva · Emma Pierson

→ Vertical Moral Growth: A Novel Developmental Framework for Human Feedback Quality in AI Alignment Poster

Taichiro Endo

→ Improvement-Guided Iterative DPO for Diffusion Models Poster

Ying Fan · Fei Deng · Yang Zhao · Sahil Singla · Rahul Jain · Tingbo Hou · Kangwook Lee · Feng Yang · Deepak Ramachandran · Qifei Wang

→ Empirical Studies on the Limitations of Direct Preference Optimization, and a Possible Quick Fix Poster

Jiarui Yao · Yong LIN · Tong Zhang

→ Inference-Time Reward Hacking in Large Language Models Poster

Hadi Khalaf · Claudio Mayrink Verdun · Alex Oesterling · Himabindu Lakkaraju · Flavio Calmon

→ FSPO: Few-Shot Preference Optimization of Synthetic Preference Data Elicits LLM Personalization to Real Users Poster

Anikait Singh · Sheryl Hsu · Kyle Hsu · Eric Mitchell · Stefano Ermon · Tatsunori Hashimoto · Archit Sharma · Chelsea Finn

→ BiasLab: Toward Explainable Political Bias Detection with Dual-Axis Human Annotations and Rationale Indicators Poster

KMA SOLAIMAN

→ Implicit User Feedback in Human-LLM Dialogues: Informative to Understand Users yet Noisy as a Learning Signal Poster

Yuhan Liu · Michael Zhang · Eunsol Choi

→ Configurable Preference Tuning with Rubric-Guided Synthetic Data Poster

Victor Gallego

→ Enhancing Personalized Multi-Turn Dialogue with Curiosity Reward Poster

Yanming Wan · Jiaxing Wu · Marwa Abdulhai · Lior Shani · Natasha Jaques

→ Tracing Human-like Traits in LLMs: Origins, Real-World Manifestation, and Controllability Poster

Pengrui Han · Rafal Kocielnik · Peiyang Song · Ramit Debnath · Dean Mobbs · Anima Anandkumar · R. Michael Alvarez

→ ReDit: Reward Dithering for Improved LLM Policy Optimization Poster

Chenxing Wei · Jiarui Yu · Ying He · Hande Dong · Yao Shu · Fei Yu

→ Mitigating Fine-tuning Risks in LLMs via Safety-Aware Probing Optimization Poster

Chengcan Wu · Zhixin Zhang · Zeming Wei · Yihao Zhang · Meng Sun

→ Do Language Models Understand Discrimination? Testing Alignment with Human Legal Reasoning under the ECHR Poster

Tatiana Botskina

→ Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization Poster

Matteo Gallici · Haitz Sáez de Ocáriz Borde

→ Playing the Data: Video Games as a Tool to Annotate and Train Models on Large Datasets Poster

Parham Ghasemloo Gheidari · Kai-Hsiang Chang · Roman Sarrazin-Gendron · Renata Mutalova · Alexander Butyaev · Attila Szantner · Jérôme Waldispühl

→ In-Context Personalized Alignment with Feedback History under Counterfactual Evaluation Poster

Xisen Jin · Zheng Li · Zhenwei DAI · Hui Liu · Xianfeng Tang · Chen Luo · Rahul Goutam · Xiang Ren · Qi He

→ Towards a Sharp Analysis of Offline Policy Learning for $f$-Divergence-Regularized Contextual Bandits Poster

Qingyue Zhao · Kaixuan Ji · Heyang Zhao · Tong Zhang · Quanquan Gu

→ Multi-Task Reward Learning from Human Ratings Poster

Mingkang Wu · Devin White · Evelyn Rose · Vernon Lawhern · Nicholas Waytowich · Yongcan Cao

→ Doubly Robust Alignment for Large Language Models Poster

Erhan Xu · Kai Ye · Hongyi Zhou · Luhan Zhu · Francesco Quinzan · Chengchun Shi

→ KL-Regularised Q-Learning: A Token-level Action-Value perspective on Online RLHF Poster

Lennie Wells · Edward J. Young · Jason Brown · Sergio Bacallado

CUDA: Capturing Uncertainty and Diversity in Preference Feedback Augmentation Oral

Sehyeok Kang · Jaewook Jeong · Se-Young Yun

→ Entropy Controllable Direct Preference Optimization Poster

Motoki Omura · Yasuhiro Fujita · Toshiki Kataoka

→ The Strong, weak and benign Goodhart’s law. An independence-free and paradigm-agnostic formalisation Poster

Adrien Majka · El-Mahdi El-Mhamdi

→ Unanchoring the Mind: DAE-Guided Counterfactual Reasoning for Rare Disease Diagnosis Poster

Yuting Yan · Yinghao Fu · Wendi Ren · Shuang Li

→ Understanding Likelihood Over-optimisation in Direct Alignment Algorithms Poster

Zhengyan Shi · Sander Land · Acyr Locatelli · Matthieu Geist · Max Bartolo

→ Advancing LLM Safe Alignment with Safety Representation Ranking Poster

Tianqi Du · Zeming Wei · Quan Chen · Chenheng Zhang · Yisen Wang

→ LoRe: Personalizing LLMs via Low-Rank Reward Modeling Poster

Avinandan Bose · Zhihan Xiong · Yuejie Chi · Simon Du · Lin Xiao · Maryam Fazel

→ Online Learning and Equilibrium Computation with Ranking Feedback Poster

Mingyang Liu · Yongshan Chen · Zhiyuan Fan · Gabriele Farina · Asuman Ozdaglar · Kaiqing Zhang

→ Efficient Generative Models Personalization via Optimal Experimental Design Poster

Guy Schacht · Mojmir Mutny · Riccardo De Santi · Ziyad Sheebaelhamd · Andreas Krause

→ Aligning Neural Style Representations for Style-based Clustering Poster

Abhishek Dangeti · Pavan Gajula · Vikram Jamwal · Vivek Srivastava

→ Aggregated Individual Reporting for Post-Deployment Evaluation Poster

Jessica Dai · Inioluwa Raji · Benjamin Recht · Irene Y. Chen

→ Full-Stack Alignment: Co-Aligning AI and Institutions with Thicker Models of Value Poster

Ryan Lowe · Joe Edelman · Tan Zhi-Xuan · Oliver Klingefjord · Ellie Hain · Vincent Wang-Maścianica · Atrisha Sarkar · Michiel Bakker · Fazl Barez · Matija Franklin · Andreas Haupt · Jobstq Heitzig · Wesley H. Holliday · Julian Jara-Ettinger · Atoosa Kasirzadeh · Ryan Kearns · James Kirkpatrick · Andrew Koh · Joel Lehman · Sydney Levine · Manon Revel · Ivan Vendrov

→ Self-Concordant Preference Learning from Noisy Labels Poster

Shiv Shankar · Madalina Fiterau

→ Selective Preference Aggregation Poster

Shreyas Kadekodi · Hayden McTavish · Berk Ustun

→ Auto-Guideline Alignment: Probing and Simulating Human Ideological Preferences in LLMs via Prompt Engineering Poster

Chien-Hua Chen · Chang Chih Meng · Li-Ni Fu · Hen-Hsen Huang · I-Chen Wu

→ Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval Poster

Taiye Chen · Zeming Wei · Ang Li · Yisen Wang

→ EconEvals: Benchmarks and Litmus Tests for LLM Agents in Unknown Environments Poster

Sara Fish · Julia Shephard · Minkai Li · Ran Shorrer · Yannai A. Gonczarowski

→ Human Feedback Guided Reinforcement Learning for Unknown Temporal Tasks via Weighted Finite Automata Poster

Nathaniel Smith · Yu Wang

→ Alignment as Distribution Learning: Your Preference Model is Explicitly a Language Model Poster

Jihun Yun · Juno Kim · Jongho Park · Junhyuck Kim · Jongha (Jon) Ryu · Jaewoong Cho · Kwang-Sung Jun

→ Rewrite-to-Rank: Optimizing Ad Visibility via Retrieval-Aware Text Rewriting Poster

Chloe Ho · Ishneet Singh · Diya Sharma · Tanvi Anumandla · Michael Lu · Vasu Sharma · Kevin Zhu

→ Mimicking Human Intuition: Cognitive Belief-Driven Reinforcement Learning Poster

Xingrui Gu · Guanren Qiao · Chuyi Jiang

→ Robust Multi-Objective Controlled Decoding of Large Language Models Poster

Seongho Son · William Bankes · Sangwoong Yoon · Shyam Sundhar Ramesh · Xiaohang Tang · Ilija Bogunovic

→ CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics Poster

Perampalli Shravan Nayak · Mehar Bhatia · Xiaofeng Zhang · Verena Rieser · Lisa Hendricks · Sjoerd van Steenkiste · Yash Goyal · Karolina Stanczak · Aishwarya Agrawal

→ In-Context Alignment at Scale: When More is Less Poster

Neelabh Madan · Lakshmi Subramanian

→ Dynamic Guardian Models: Realtime Content Moderation With User-Defined Policies Poster

Monte Hoover · Vatsal Baherwani · Neel Jain · Khalid Saifullah · Joseph Vincent · Chirag Jain · Melissa Rad · C. Bayan Bruss · Ashwinee Panda · Tom Goldstein

→ On the strength of goodhart's law Poster

Adrien Majka · Wassim Bouaziz · El-Mahdi El-Mhamdi

→ Reasoning Isn't Enough: Examining Truth-Bias and Sycophancy in LLMs Poster

Emilio Barkett · Olivia Long · Madhavendra Thakur

→ Angular Steering: Behavior Control via Rotation in Activation Space Poster

Hieu M. Vu · Tan Nguyen

→ A Unified Perspective on Reward Distillation Through Ratio Matching Poster

Kenan Hasanaliyev · Schwinn Saereesitthipitak · Rohan Sanda

→ Geometry-Aware Preference Learning for 3D Texture Generation Poster

AmirHossein Zamani · Tianhao Xie · Amir Aghdam · Tiberiu Popa · Eugene Belilovsky

→ What Matters when Modeling Human Behavior using Imitation Learning? Poster

Aneri Muni · Esther Derman · Vincent Taboga · Pierre-Luc Bacon · Erick Delage

→ Expected Reward Prediction, with Applications to Model Routing Poster

Kenan Hasanaliyev · Silas Alberti · Jenny Hamer · Dheeraj Rajagopal · Kevin Robinson · Jasper Snoek · Victor Veitch · Alexander D'Amour

→ Language Model Personalization via Reward Factorization Poster

Idan Shenfeld · Felix Faltings · Pulkit Agrawal · Aldo Pacchiano

→ Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning Poster

Kai Ye · Hongyi Zhou · Jin Zhu · Francesco Quinzan · Chengchun Shi

→ Theoretical Analysis of KL-regularized RLHF with Multiple Reference Models Poster

Gholamali Aminian · Amir R. Asadi · Idan Shenfeld · Youssef Mroueh

→ Alignment of Large Language Models with Constrained Learning Poster

Botong Zhang · Shuo Li · Ignacio Hounie · Osbert Bastani · Dongsheng Ding · Alejandro Ribeiro

→ Robust Reward Modeling via Causal Rubrics Poster

Pragya Srivastava · Harman Singh · Rahul Madhavan · Gandharv Patil · Sravanti Addepalli · Arun Sai Suggala · Rengarajan Aravamudhan · Soumya Sharma · Anirban Laha · Aravindan Raghuveer · Karthikeyan Shanmugam · Doina Precup

→ ActiveDPO: Active Direct Preference Optimization for Sample-Efficient Alignment Poster

Xiaoqiang Lin · Arun Verma · Zhongxiang Dai · Daniela Rus · See-Kiong Ng · Bryan Kian Hsiang Low

→ Mechanism Design for Alignment via Human Feedback Poster

Julian Manyika · Michael Wooldridge · Jiarui Gan

→ Composition and Alignment of Diffusion Models using Constrained Learning Poster

Shervin Khalafi · Ignacio Hounie · Dongsheng Ding · Alejandro Ribeiro