Contrastive Symbolic Regression: Aligned Representations, Adaptive Prediction, and Diverse Ensembles

Existing symbolic regression approaches primarily focus on learning explicit input-output mappings, often neglecting relational structures among data instances. This paper introduces Contrastive Symbolic Regression (CSR), a feature-construction-based symbolic regression approach that integrates evolutionary feature construction with contrastive learning to shape a representation space where geometric proximity reflects similarity in the target space. CSR employs a contrastive objective that optimizes a linear transformation of constructed features, with a closed-form solution for aligning the feature space with the target space. The constructed features are applied to K-nearest neighbor regression, where we propose an efficient leave-one-out cross-validation (LOOCV) method that addresses standard LOOCV's computational expense and adaptively selects the neighborhood size, along with a linear-rank weighted K-nearest neighbor variant for faithful assessment of representation quality during evolution. A determinantal point process-based ensemble selection mechanism further enhances robustness by jointly considering model quality and diversity. Extensive experiments on 58 real-world regression datasets demonstrate that CSR consistently surpasses both traditional symbolic regression and modern machine learning counterparts, highlighting CSR as a promising direction for interpretable and effective regression modeling.

Symbolic regression automatically discovers interpretable mathematical formulas from data, but existing methods focus on finding a single global equation and miss complex local patterns. Our method, Contrastive Symbolic Regression (CSR), instead evolves mathematical features that reorganize data so that points with similar outputs end up close together. Predictions are then made by finding the nearest neighbors in this learned space — similar to how a doctor consults similar past cases. An efficient ensemble mechanism further improves robustness. Tested on 58 real-world datasets, CSR consistently outperforms both traditional symbolic regression and modern machine learning methods while remaining interpretable.