On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance

Large Language Models (LLMs) are increasingly used for zero-shot annotation and LLM-as-a-judge tasks, yet their reliability hinges on how model-internalized priors interact with user-provided instructions. We investigate three dimensions of this interaction: (1) how an LLM's familiarity with data and task definitions affects performance, (2) the extent to which additional information in prompts can correct zero-shot errors (``decision stickiness''), and (3) model susceptibility to misaligned task definitions. Through experiments on toxicity detection across diverse datasets (spanning social media, gaming, news, and forums) using both dense and mixture-of-experts models, we find that nearly two-thirds of zero-shot errors are resistant to correction, with an overall rescue rate (fraction of initial errors corrected by prompting) of only 34.8\%. High-confidence errors prove especially resistant to correction. When given misaligned definitions, LLMs follow them while maintaining confidence levels unchanged from the aligned condition. Crucially, we introduce Definition-Specific Familiarity (DSF), which measures alignment between a model's internal concept and the task definition. After controlling for dataset-level confounds, DSF shows a positive association with model performance (partial $r=+0.41$), while three distinct memorization metrics (ROUGE-L, BERTScore, and embedding cosine similarity) all fail to show a positive association. These findings show the limitations of prompt-based correction in annotation tasks, highlighting the importance of definition alignment over text-level memorization.

Large language models (LLMs) are increasingly used to label data, judge model outputs, and make decisions about subjective concepts such as whether online text is "toxic." But these models are not blank slates: they already have internal ideas about what concepts like toxicity mean, learned from their training data and later tuning. This matters because a researcher's or platform's definition of toxicity may differ from the model's built-in understanding. In this paper, we study how well LLMs follow user-provided definitions when performing annotation tasks. We test nine models on five toxicity-related datasets from social media, gaming, news, and online forums. We find that models perform better when their internal understanding of a concept matches the task definition, not simply when they may have seen similar text before. We also find that prompting has limited power to fix mistakes: most initial errors remain wrong even after models are given better definitions or examples. Most concerningly, models can confidently apply definitions that do not fit the kind of data being labeled, making confidence scores a poor warning signal. These findings suggest that LLM annotation systems should not rely only on larger models, better prompts, or confidence scores. Instead, practitioners should carefully test whether a model's internal understanding matches the specific definition they want to apply.