Skip to yearly menu bar Skip to main content


Workshop

Tokenization Workshop (TokShop)

Tomasz Limisiewicz ⋅ Valentin Hofmann ⋅ Sachin Kumar ⋅ Farhan Samir ⋅ Jindřich Libovický ⋅ Jindřich Helcl ⋅ Orevaoghene Ahia ⋅ Elizabeth Salesky
Project Page

Abstract

Video

Chat is not available.

Schedule

Timezone: Asia/Seoul
12:45 AM
Opening
1:00 AM
Coffee Break
2:00 AM
Insights from Pixel Language Modeling
Desmond Elliott
→ Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
Brian Zheng ⋅ Alisa Liu ⋅ Orevaoghene M Ahia ⋅ Jonathan Hayase ⋅ Yejin Choi ⋅ Noah Smith
→ Continuous Chain of Thought Enables Parallel Exploration and Reasoning
Halil Alperen Gozeten ⋅ Muhammed Emrullah Ildiz ⋅ Xuechen Zhang ⋅ Hrayr Harutyunyan ⋅ Ankit Singh Rawat ⋅ Samet Oymak
→ Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8
Preston Firestone ⋅ Shubham Ugare ⋅ Gagandeep Singh ⋅ Sasa Misailovic
→ Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives
Ander Artola Velasco ⋅ Stratis Tsirtsis ⋅ Nastaran Okati ⋅ Manuel Gomez-Rodriguez
→ Evaluating Morphological Alignment of Tokenizers in 70 Languages
Catherine Arnett ⋅ Marisa Hudspeth ⋅ Brendan O'Connor
→ Byte Latent Transformer: Patches Scale Better Than Tokens
Artidoro Pagnoni ⋅ Ramakanth Pasunuru ⋅ Pedro Rodriguez ⋅ John Nguyen ⋅ Benjamin Muller ⋅ Margaret Li ⋅ Chunting Zhou ⋅ LILI YU ⋅ JASON WESTON ⋅ Luke Zettlemoyer ⋅ Gargi Ghosh ⋅ Mike Lewis ⋅ Ari Holtzman ⋅ Srinivasan Iyer
→ Contextual morphologically-guided tokenization for pretrained Latin BERT models
Marisa Hudspeth ⋅ Patrick J. Burns ⋅ Brendan O'Connor
→ SuperBPE: Space Travel for Language Models
Alisa Liu ⋅ Jonathan Hayase ⋅ Valentin Hofmann ⋅ Sewoong Oh ⋅ Noah Smith ⋅ Yejin Choi
→ zip2zip: Inference-Time Adaptive Vocabularies for Language Models via Token Compression
Saibo Geng ⋅ Nathan Thomas Elian Ranchin ⋅ Yunzhen Yao ⋅ Maxime Peyrard ⋅ Chris Wendler ⋅ Michael Gastpar ⋅ Robert West
→ How Much is Enough? The Diminishing Returns of Tokenization Training Data
Varshini Reddy ⋅ Craig Schmidt ⋅ Yuval Pinter ⋅ Chris Tanner
→ FLEXITOKENS: Flexible Tokenization for Evolving Language Models
Abraham Owodunni ⋅ Orevaoghene M Ahia ⋅ Sachin Kumar
→ Sampling from Your Language Model One Byte at a Time
Jonathan Hayase ⋅ Alisa Liu ⋅ Noah Smith ⋅ Sewoong Oh
→ ByteSpan: Information-Driven Subword Tokenisation
Zebulon Goriely ⋅ Suchir Salhan ⋅ Pietro Lesci ⋅ Julius Cheng ⋅ Paula Buttery
2:50 AM
MorphTok: Morphologically Grounded Tokenization for Indic languages
Maharaj Brahma ⋅ N J Karthika ⋅ Atul Singh ⋅ Devaraja Adiga ⋅ Smruti Bhate ⋅ Ganesh Ramakrishnan ⋅ Rohit Saluja ⋅ Maunendra Sankar Desarkar
2:50 AM
Causal Estimation of Tokenisation Bias
Pietro Lesci ⋅ Clara Meister ⋅ Thomas Hofmann ⋅ Andreas Vlachos ⋅ Tiago Pimentel
2:50 AM
Adversarial Tokenization
Renato Geh ⋅ Zilei Shao ⋅ Guy Van den Broeck
4:00 AM
Lunch Break
→ Canonical Autoregressive Generation
Ivi Chatzi ⋅ Nina Corvelo Benz ⋅ Stratis Tsirtsis ⋅ Manuel Gomez-Rodriguez
→ You Only Train Once: Efficient Tokenizer Selection for Arithmetic in Language Models
Mucong Ding ⋅ Sean McLeish ⋅ Kazem Meidani ⋅ Igor Melnyk ⋅ Nam Nguyen ⋅ C. Bayan Bruss ⋅ Furong Huang
→ Conditional Unigram Tokenization with Parallel Data
Gianluca Vico ⋅ Jindřich Libovický
→ One-D-Piece: Image Tokenizer Meets Quality-Controllable Compression
Keita Miwa ⋅ Kento Sasaki ⋅ Hidehisa Arai ⋅ Tsubasa Takahashi ⋅ Yu Yamaguchi
→ Tokenizing Nonverbal Communication in Salsa Dance
Bermet Burkanova ⋅ Payam Jome Yazdian ⋅ Chuxuan Zhang ⋅ Trinity Evans ⋅ Paige Tuttösí ⋅ Angelica Lim
→ Watermarking Autoregressive Image Generation
Nikola Jovanović ⋅ Ismail Labiad ⋅ Tomas Soucek ⋅ Martin Vechev ⋅ Pierre Fernandez
→ Overcoming Vocabulary Constraints with Pixel-level Fallback
Jonas F. Lotz ⋅ Hendra Setiawan ⋅ Stephan Peitz ⋅ Yova Kementchedjhieva
→ Continuous Autoregressive Generation with Mixture of Gaussians
Alex Quach ⋅ Johnson Tsun-Hsuan Wang ⋅ Ramin Hasani ⋅ Mathias Lechner ⋅ Alexander Amini
→ Motion-Focused Tokenization for Source-Free Video Domain Adaptation
Tzu Ling Liu ⋅ Ian Stavness ⋅ Mrigank Rochan
→ Discrete JEPA: Learning Discrete Token Representations without Reconstruction
Junyeob Baek ⋅ Hosung Lee ⋅ Christopher Hoang ⋅ Mengye Ren ⋅ Sungjin Ahn
→ CAT: Content-Adaptive Image Tokenization
Junhong Shen ⋅ Kushal Tirumala ⋅ Michihiro Yasunaga ⋅ Ishan Misra ⋅ Luke Zettlemoyer ⋅ LILI YU ⋅ Chunting Zhou
→ Entropy-Driven Pre-tokenization for Byte Pair Encoding
Yifan Hu ⋅ Ningyue Liang ⋅ Dachuan Zhao ⋅ Jonathan Geuter ⋅ Varshini Reddy ⋅ Craig Schmidt ⋅ Chris Tanner
5:50 AM
Tokenisation is NP-Complete
Philip Whittington ⋅ Gregor Bachmann ⋅ Tiago Pimentel
5:50 AM
HH-Codec: High Compression High-fidelity Discrete Neural Codec for Spoken Language Modeling
rongkun xue ⋅ Yazhe Niu ⋅ Shuai Hu ⋅ Zixin Yin ⋅ Yongqiang Yao ⋅ Jing Yang
5:50 AM
GeneticBPE: Motif-Preserving Tokenization for Robust miRNA Modeling
Prabhav Sanga ⋅ Jaskaran Singh ⋅ ARUN DUBEY
5:50 AM
Pitfalls, Subtleties, and Techniques in Automata-Based Subword-Level Constrained Generation
Marco Cognetta ⋅ David Pohl ⋅ Junyoung Lee ⋅ Naoaki Okazaki
7:00 AM
Coffee Break
9:00 AM
Closing Remarks