Model Comparison

Why the selected model strategy was chosen — not by accuracy alone

The selected model is not chosen by accuracy alone. The task is imbalanced (base rate ~12.75%), so F1, precision, recall, PR-AUC, ROC-AUC, and alert rate are more informative metrics than raw accuracy.

Selected Strategy

Model

Ensemble LGBM(0.5) + XGB(0.5)

Threshold

Per-Category

Test F1

0.3800

Test ROC-AUC

0.7661

Model Metrics Comparison (Test Set)

Validation vs Test Metrics

Rank	Method	Threshold	Val F1	Val Prec	Val Rec	Val PR-AUC	Val ROC-AUC	Test F1	Test Prec	Test Rec	Test PR-AUC	Test ROC-AUC
1	Ensemble LGBM(0.50) + XGB(0.50)	Per-Category	0.3756	0.2901	0.5326	0.3137	0.7563	0.3800	0.3012	0.5146	0.3294	0.7661
2	Tuned LightGBM	Per-Category	0.3744	0.2897	0.5290	0.3089	0.7547	0.3789	0.3000	0.5140	0.3270	0.7643
3	Ensemble LGBM(0.50) + XGB(0.50)	Global	0.3720	0.2838	0.5396	0.3137	0.7563	0.3797	0.2967	0.5269	0.3294	0.7661
4	Tuned LightGBM	Global	0.3713	0.2853	0.5313	0.3089	0.7547	0.3781	0.2972	0.5197	0.3270	0.7643
5	Tuned XGBoost	Per-Category	0.3704	0.2785	0.5529	0.3119	0.7539	0.3761	0.2901	0.5347	0.3247	0.7637
6	Tuned XGBoost	Global	0.3683	0.2720	0.5699	0.3119	0.7539	0.3777	0.2860	0.5559	0.3247	0.7637

Baseline Models (Logistic Regression)

Baseline models using logistic regression with different feature sets

Best Hyperparameters

Model	Param ID	Threshold	Val F1	Test F1	Test ROC-AUC	Fit Time (s)
Tuned LightGBM	lgbm_deeper_depth10_leaf127_lr003	0.1823	0.3786	0.3832	0.7686	361.8
Tuned XGBoost	xgb_current_depth8_lr005	0.1790	0.3780	0.3832	0.7698	31.1