LassoFlexNet: a Flexible Neural Architecture for Tabular Data

Despite their dominance in vision and language, deep neural networks often underperform relative to tree-based models on tabular data. To bridge this gap, we incorporate five key inductive biases into deep tabular learning: robustness to irrelevant features, axis alignment, localized irregularities, feature heterogeneity, and training stability. We propose *LassoFlexNet*, an architecture that evaluates the linear and nonlinear marginal contribution of each input via Per-Feature Embeddings, and sparsely selects relevant variables using a Tied Group Lasso mechanism. Because these components introduce optimization challenges that destabilize standard proximal methods, we analyze stochastic hierarchical proximal dynamics and develop a *Sequential Hierarchical Proximal Adaptive Gradient optimizer with exponential moving averages (EMA)* designed to stabilize training in practice. Across 52 datasets from three benchmarks, LassoFlexNet matches or outperforms leading tree-based models, achieving up to a $10$\% relative gain, while maintaining Lasso-like interpretability. We substantiate these empirical results with ablation studies and theoretical proofs confirming the architecture's enhanced expressivity and structural breaking of undesired rotational invariance.

When a bank decides whether to approve a loan, a hospital scores a patient's risk, or a retailer forecasts demand, the model behind the decision is almost never deep learning — older tree-based methods such as XGBoost still tend to win on these tables of numbers and categories. We introduce LassoFlexNet, a neural network designed specifically for spreadsheet-like data: it gives every column its own learned re-encoding and uses a built-in switch, inspired by classical Lasso regression, that turns off columns the model does not need. To keep this switch from making training unstable, we developed a matching training procedure that prunes columns gradually rather than all at once. Across 52 datasets from three independent benchmarks, LassoFlexNet matches or beats the leading tree-based models — with gains of up to 10% on some tasks — while still letting practitioners see, and explain to a customer, regulator, or patient, exactly which columns drove each prediction.