I’m working on a forecasting / predictive modeling problem where I’m comparing four groups of models:
Baseline: Naive / Seasonal Naive
Time Series: ARIMA, SARIMA, ETS, Prophet, etc.
Statistical: Linear Regression, Ridge, GAM, etc.
Machine Learning: Random Forest, Gradient Boosting, XGBoost, etc.
One issue I’m struggling with is feature selection.
Should I create one universal feature subset before modeling and give the same predictors to the Statistical and ML models, or should feature selection be model-specific?
For example, a linear model may benefit from correlation filtering, VIF, or LASSO, while tree-based models may select a different set of variables because they can capture nonlinearities and interactions.
I’m also unsure about the correct modeling pipeline. Where exactly should feature selection happen relative to the train/test split, cross-validation, preprocessing, and hyperparameter tuning?
Some questions I’m trying to resolve:
Should all comparable models use the same feature set for a fair model comparison?
If model-specific feature selection is appropriate, how should it be implemented without introducing data leakage?
Should feature selection be performed separately within each model’s training process?
If different models select different predictors, how should that variation be explained to stakeholders?
Which feature selection or interpretation methods are actually useful for identifying the underlying drivers of the dependent variable, rather than simply variables that improve predictive accuracy?
How should I distinguish between predictive importance and explanatory importance when interpreting the results?
Ultimately, I’m trying to build a workflow that balances forecasting accuracy, fair model comparison, and interpretability.
What would be a statistically sound end-to-end approach for handling feature selection across these different model types?
