Skip to yearly menu bar Skip to main content


Workshop

Models of Human Feedback for AI Alignment

Thomas Kleine Buening ⋅ Harshit Sikchi ⋅ Christos Dimitrakakis ⋅ Scott Niekum ⋅ Constantin Rothkopf ⋅ Aadirupa Saha ⋅ Lirong Xia
Project Page

Abstract

Video

Chat is not available.

Schedule

Time zone: Conference local time zone
9:00 AM
Opening Remarks
9:50 AM
AI Alignment with Changing and Influenceable Reward Functions
Micah Carroll ⋅ Davis Foote ⋅ Anand Siththaranjan ⋅ Stuart Russell ⋅ Anca Dragan
10:00 AM
RLHF and IIA: Perverse Incentives
Wanqiao Xu ⋅ Shi Dong ⋅ Xiuyuan Lu ⋅ Grace Lam ⋅ Zheng Wen ⋅ Benjamin Van Roy
11:00 AM
MaxMin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences
Souradip Chakraborty ⋅ Jiahao Qiu ⋅ Hui Yuan ⋅ Alec Koppel ⋅ Furong Huang ⋅ Dinesh Manocha ⋅ Amrit Singh Bedi ⋅ Mengdi Wang
11:10 AM
Modeling the Plurality of Human Preferences via Ideal Points
Daiwei Chen ⋅ Yi Chen ⋅ Aniket Rege ⋅ Ramya Vinayak
11:20 AM
Prompt Optimization with Human Feedback
Xiaoqiang Lin ⋅ Zhongxiang Dai ⋅ Arun Verma ⋅ See-Kiong Ng ⋅ Patrick Jaillet ⋅ Bryan Kian Hsiang Low
1:45 PM
Preference Learning Algorithms Do Not Learn Preference Rankings
Angelica Chen ⋅ Sadhika Malladi ⋅ Lily Zhang ⋅ Xinyi Chen ⋅ Richard Zhang ⋅ Rajesh Ranganath ⋅ Kyunghyun Cho
1:55 PM
Scalable Oversight by Accounting for Unreliable Feedback
Shivam Singhal ⋅ Cassidy Laidlaw ⋅ Anca Dragan
5:00 PM
Learning the eye of the beholder: Statistical modeling and estimation for personalized color perception
Xuanzhou Chen ⋅ Austin Xu ⋅ Jingyan Wang ⋅ Ashwin Pananjady
5:00 PM
Scalably Solving Assistance Games
Cassidy Laidlaw ⋅ Eli Bronstein ⋅ Timothy Guo ⋅ Dylan Feng ⋅ Lukas Berglund ⋅ Justin Svegliato ⋅ Stuart Russell ⋅ Anca Dragan
5:00 PM
Off-Policy Evaluation from Logged Human Feedback
Aniruddha Bhargava ⋅ Lalit Jain ⋅ Branislav Kveton ⋅ Ge Liu ⋅ Subhojyoti Mukherjee
5:00 PM
Preference Elicitation for Offline Reinforcement Learning
Alizée Pace ⋅ Bernhard Schölkopf ⋅ Gunnar Ratsch ⋅ Giorgia Ramponi
5:00 PM
Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation
Katie Collins ⋅ Najoung Kim ⋅ Yonatan Bitton ⋅ Verena Rieser ⋅ Shayegan Omidshafiei ⋅ Yushi Hu ⋅ Sherol Chen ⋅ Senjuti Dutta ⋅ Minsuk Chang ⋅ Kimin Lee ⋅ Youwei Liang ⋅ Georgina Evans ⋅ Sahil Singla ⋅ Gang Li ⋅ Adrian Weller ⋅ Junfeng He ⋅ Deepak Ramachandran ⋅ Krishnamurthy Dvijotham
5:00 PM
AI Alignment with Changing and Influenceable Reward Functions
Micah Carroll ⋅ Davis Foote ⋅ Anand Siththaranjan ⋅ Stuart Russell ⋅ Anca Dragan
5:00 PM
Concept-Based Interpretable Reinforcement Learning with Limited to No Human Labels
Zhuorui Ye ⋅ Stephanie Milani ⋅ Fei Fang ⋅ Geoff Gordon
5:00 PM
Learning to Assist Humans without Inferring Rewards
Vivek Myers ⋅ Evan Ellis ⋅ Benjamin Eysenbach ⋅ Sergey Levine ⋅ Anca Dragan
5:00 PM
Uncertainty-aware Preference Alignment in Reinforcement Learning from Human Feedback
Sheng Xu ⋅ Bo Yue ⋅ Hongyuan Zha ⋅ Guiliang Liu
5:00 PM
Reinforcement Learning from Human Text Feedback: Learning a Reward Model from Human Text Input
Belen Martin Urcelay ⋅ Andreas Krause ⋅ Giorgia Ramponi
5:00 PM
Language Alignment via Nash-learning and Adaptive feedback
Ari Azarafrooz ⋅ Farshid Faal
5:00 PM
Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
Haoyu Wang ⋅ Guozheng Ma ⋅ Ziqiao Meng ⋅ Zeyu Qin ⋅ Li Shen ⋅ Zhong Zhang ⋅ Bingzhe Wu ⋅ Liu Liu ⋅ Yatao Bian ⋅ Tingyang Xu ⋅ Xueqian Wang ⋅ Peilin Zhao
5:00 PM
Multi-Agent Imitation Learning: Value is Easy, Regret is Hard
Jingwu Tang ⋅ Gokul Swamy ⋅ Fei Fang ⋅ Steven Wu
5:00 PM
Efficient Inverse Reinforcement Learning without Compounding Errors
Nicolas Espinosa Dice ⋅ Gokul Swamy ⋅ Sanjiban Choudhury ⋅ Wen Sun
5:00 PM
Revisiting Successor Features for Inverse Reinforcement Learning
Arnav Kumar Jain ⋅ Harley Wiltzer ⋅ Jesse Farebrother ⋅ Irina Rish ⋅ Glen Berseth ⋅ Sanjiban Choudhury
5:00 PM
DPO Meets PPO: Reinforced Token Optimization for RLHF
Han Zhong ⋅ Guhao Feng ⋅ Wei Xiong ⋅ Xinle Cheng ⋅ Li Zhao ⋅ Di He ⋅ Jiang Bian ⋅ Liwei Wang
5:00 PM
Models That Prove Their Own Correctness
Noga Amit ⋅ Shafi Goldwasser ⋅ Orr Paradise ⋅ Guy Rothblum
5:00 PM
Regularized Best-of-N Sampling to Mitigate Reward Hacking for Language Model Alignment
Yuu Jinnai ⋅ Tetsuro Morimura ⋅ Kaito Ariu ⋅ Kenshi Abe
5:00 PM
PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling
Utsav Singh ⋅ Wesley A. Suttle ⋅ Brian Sadler ⋅ Vinay Namboodiri ⋅ Amrit Singh Bedi
5:00 PM
AMBER: An Entropy Maximizing Environment Design Algorithm for Inverse Reinforcement Learning
Paul Nitschke ⋅ Lars L. Ankile ⋅ Eura Nofshin ⋅ Siddharth Swaroop ⋅ Finale Doshi-Velez ⋅ Weiwei Pan
5:00 PM
Modeling the Plurality of Human Preferences via Ideal Points
Daiwei Chen ⋅ Yi Chen ⋅ Aniket Rege ⋅ Ramya Vinayak
5:00 PM
Stochastic Concept Bottleneck Models
Moritz Vandenhirtz ⋅ Sonia Laguna ⋅ Ričards Marcinkevičs ⋅ Julia Vogt
5:00 PM
Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
Rafael Rafailov ⋅ Yaswanth Chittepu ⋅ Ryan Park ⋅ Harshit Sikchi ⋅ Joey Hejna ⋅ William Knox ⋅ Chelsea Finn ⋅ Scott Niekum
5:00 PM
Informed Meta-Learning
Katarzyna Kobalczyk ⋅ M van der Schaar
5:00 PM
DPM: Dual Preferences-based Multi-Agent Reinforcement Learning
Sehyeok Kang ⋅ Yongsik Lee ⋅ Se-Young Yun
5:00 PM
Hummer: Towards Limited Competitive Preference Dataset
Li Jiang ⋅ Yusen Wu ⋅ Junwu Xiong ⋅ Jingqing Ruan ⋅ Yichuan Ding ⋅ Qingpei Guo ⋅ zujie wen ⋅ JUN ZHOU ⋅ Xiaotie Deng
5:00 PM
Comparing Few to Rank Many: Optimal Design for Learning Preferences
Kiran Thekumparampil ⋅ Gaurush Hiranandani ⋅ Kousha Kalantari ⋅ Shoham Sabach ⋅ Branislav Kveton
5:00 PM
MultiScale Policy Learning for Alignment with Long Term Objectives
Richa Rastogi ⋅ Yuta Saito ⋅ Thorsten Joachims
5:00 PM
Is poisoning a real threat to LLM alignment? Maybe more so than you think
Pankayaraj Pathmanathan ⋅ Souradip Chakraborty ⋅ Xiangyu Liu ⋅ Yongyuan Liang ⋅ Furong Huang
5:00 PM
Towards Aligning Language Models with Textual Feedback
Saüc Abadal ⋅ Shehzaad Dhuliawala ⋅ Keerthiram Murugesan ⋅ Mrinmaya Sachan
5:00 PM
Bootstrapping Language Models with DPO Implicit Rewards
Changyu Chen ⋅ Zichen Liu ⋅ Chao Du ⋅ Tianyu Pang ⋅ Qian Liu ⋅ Arunesh Sinha ⋅ Pradeep Varakantham ⋅ Min Lin
5:00 PM
Distributional Preference Alignment of LLMs via Optimal Transport
Igor Melnyk ⋅ Youssef Mroueh ⋅ Brian Belgodere ⋅ Mattia Rigotti ⋅ Apoorva Nitsure ⋅ Mikhail Yurochkin ⋅ Kristjan Greenewald ⋅ Jiri Navratil ⋅ Jarret Ross
5:00 PM
Scalable Oversight by Accounting for Unreliable Feedback
Shivam Singhal ⋅ Cassidy Laidlaw ⋅ Anca Dragan
5:00 PM
MaxMin-RLHF: Towards Equitable Alignment of Large Language Models with Diverse Human Preferences
Souradip Chakraborty ⋅ Jiahao Qiu ⋅ Hui Yuan ⋅ Alec Koppel ⋅ Furong Huang ⋅ Dinesh Manocha ⋅ Amrit Singh Bedi ⋅ Mengdi Wang
5:00 PM
Relatively Rational: Learning Utilities and Rationalities Jointly from Pairwise Preferences
Taku Yamagata ⋅ Tobias Oberkofler ⋅ Timo Kaufmann ⋅ Viktor Bengs ⋅ Eyke Hüllermeier ⋅ Raul Santos-Rodriguez
5:00 PM
Filtered Direct Preference Optimization
Tetsuro Morimura ⋅ Mitsuki Sakamoto ⋅ Yuu Jinnai ⋅ Kenshi Abe ⋅ Kaito Ariu
5:00 PM
Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment
Zhaofeng Wu ⋅ Ananth Balashankar ⋅ Yoon Kim ⋅ Jacob Eisenstein ⋅ Ahmad Beirami
5:00 PM
Optimal Design for Human Feedback
Subhojyoti Mukherjee ⋅ Anusha Lalitha ⋅ Kousha Kalantari ⋅ Aniket Anand Deshmukh ⋅ Ge Liu ⋅ Yifei Ma ⋅ Branislav Kveton
5:00 PM
Aligning Crowd Feedback via Distributional Preference Reward Modeling
Dexun Li ⋅ Cong Zhang ⋅ Kuicai Dong ⋅ Derrick Goh Xin Deik ⋅ Ruiming Tang ⋅ Yong Liu
5:00 PM
Preference Learning Algorithms Do Not Learn Preference Rankings
Angelica Chen ⋅ Sadhika Malladi ⋅ Lily Zhang ⋅ Xinyi Chen ⋅ Richard Zhang ⋅ Rajesh Ranganath ⋅ Kyunghyun Cho
5:00 PM
New Desiderata for Direct Preference Optimization
Xiangkun Hu ⋅ Tong He ⋅ David Wipf
5:00 PM
Accelerating Best-of-N via Speculative Rejection
Ruiqi Zhang ⋅ Momin Haider ⋅ Ming Yin ⋅ Jiahao Qiu ⋅ Mengdi Wang ⋅ Peter Bartlett ⋅ Andrea Zanette
5:00 PM
A Theoretical Framework for Partially Observed Reward-States in RLHF
Chinmaya Kausik ⋅ Mirco Mutti ⋅ Aldo Pacchiano ⋅ Ambuj Tewari
5:00 PM
Weak-to-Strong Extrapolation Expedites Alignment
Chujie Zheng ⋅ Ziqi Wang ⋅ Heng Ji ⋅ Minlie Huang ⋅ Nanyun Peng
5:00 PM
Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization
Hritik Bansal ⋅ Ashima Suvarna ⋅ Gantavya Bhatt ⋅ Nanyun Peng ⋅ Kai-Wei Chang ⋅ Aditya Grover
5:00 PM
RLHF and IIA: Perverse Incentives
Wanqiao Xu ⋅ Shi Dong ⋅ Xiuyuan Lu ⋅ Grace Lam ⋅ Zheng Wen ⋅ Benjamin Van Roy
5:00 PM
Aligning Large Language Models with Representation Editing: A Control Perspective
Lingkai Kong ⋅ Haorui Wang ⋅ Wenhao Mu ⋅ Yuanqi Du ⋅ Yuchen Zhuang ⋅ Yifei Zhou ⋅ Yue Song ⋅ Rongzhi Zhang ⋅ Kai Wang ⋅ Chao Zhang
5:00 PM
Cross-Domain Knowledge Transfer for RL via Preference Consistency
Ting-Hsuan Huang ⋅ Ping-Chun Hsieh
5:00 PM
Is a Good Description Worth a Thousand Pictures? Reducing Multimodal Alignment to Text-Based, Unimodal Alignment
Amin Memarian ⋅ Touraj Laleh ⋅ Irina Rish ⋅ Ardavan S. Nobandegani
5:00 PM
Generalizing Offline Alignment Theoretical Paradigm with Diverse Divergence Constraints
Haoyuan Sun ⋅ Yuxin Zheng ⋅ Yifei Zhao ⋅ Yongzhe Chang ⋅ Xueqian Wang
5:00 PM
Adversarial Multi-dueling Bandits
Pratik Gajane
5:00 PM
Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Xuening Feng ⋅ Zhaohui Jiang ⋅ Timo Kaufmann ⋅ Eyke Hüllermeier ⋅ Paul Weng ⋅ Yifei Zhu
5:00 PM
REBEL: Reinforcement Learning via Regressing Relative Rewards
Zhaolin Gao ⋅ Jonathan Chang ⋅ Wenhao Zhan ⋅ Owen Oertell ⋅ Gokul Swamy ⋅ Kianté Brantley ⋅ Thorsten Joachims ⋅ Drew Bagnell ⋅ Jason Lee ⋅ Wen Sun
5:00 PM
Free-Energy Equilibria: Toward a Theory of Interactions Between Boundedly-Rational Agents
David Hyland ⋅ Tomáš Gavenčiak ⋅ Lancelot Da Costa ⋅ Conor Heins ⋅ Vojtech Kovarik ⋅ Julian Gutierrez ⋅ Michael Wooldridge ⋅ Jan Kulveit
5:00 PM
Towards Safe Large Language Models for Medicine
Tessa Han ⋅ Aounon Kumar ⋅ Chirag Agarwal ⋅ Himabindu Lakkaraju
5:00 PM
Prompt Optimization with Human Feedback
Arunesh Sinha ⋅ See-Kiong Ng ⋅ Patrick Jaillet ⋅ Bryan Kian Hsiang Low ⋅ Xiaoqiang Lin ⋅ Zhongxiang Dai
Hummer: Towards Limited Competitive Preference Dataset
Li Jiang ⋅ Yusen Wu ⋅ Junwu Xiong ⋅ Jingqing Ruan ⋅ Yichuan Ding ⋅ Qingpei Guo ⋅ zujie wen ⋅ JUN ZHOU ⋅ Xiaotie Deng