Lookahead Automated Feature Engineering for Tabular Prediction via Kaggle-Guided Knowledge Transfer

Feature Engineering (FE) can substantially boost accuracy in tabular prediction, yet identifying effective transformations remains challenging: the space of possible operations is vast, and current LLM-based approaches often default to generic ''common-sense'' features and make myopic, step-by-step choices that overlook beneficial feature combinations. We propose Forge (Feature Optimization with Retrieved knowledge Guidance and lookahead Exploration), a retrieval-augmented lookahead framework that makes LLM-driven feature engineering both more informed and less myopic. Forge improves the quality of proposed transformations by transferring practical feature-engineering expertise: it grounds generation in high-performing Kaggle solutions and retrieves task-relevant patterns that guide the LLM beyond ad hoc transformations. It further improves the search process by enabling lookahead selection over multiple candidate feature programs, increasing the chance of discovering feature sets that work well together rather than optimizing each step in isolation. Across real-world tabular tasks with semantically meaningful metadata, FORGE consistently outperforms LLM-based automated feature engineering baselines and serves as a plug-in feature generator whose engineered features generalize across diverse downstream tabular learners.