Central question
Can recent activity and driving behaviour identify users at risk of churn - and is the model strong enough to guide retention decisions?
Project overview
This project follows a product-analytics question from exploration to model recommendation. The dataset contains 14,999 navigation-app users and behavioural measures such as sessions, drives, activity days, driving days and distance travelled. Seven hundred rows had no churn label, leaving 14,299 labelled observations for modelling.
The analysis was framed around retention decisions, so the cost of missing a likely churner mattered more than reporting a high overall accuracy on an imbalanced target.
After excluding missing targets
Minority class
Champion XGBoost model
Main deployment limitation
Evidence summary
The behavioural story and the model result
The analysis moves from behavioural patterns to model performance, error analysis and feature importance.01 · Behavioural exploration
Frequent use was the clearest retention signal
Churn fell as driving frequency increased, while users covering unusually long daily distances were more likely to leave.What it showsUsers with no driving days had a 40% churn rate, compared with 0% among users active on all 30 driving days. Professional-driver profiles also churned less often than other users.
Why it mattersRecent engagement appears more useful for retention targeting than device type. A low-activity segment is a more defensible starting point for intervention design.
LimitationThese are observational associations. The dataset does not establish why low-activity or long-distance users disengaged.
02 · Model comparison
Accuracy concealed weak churn detection
The target was imbalanced: 82.3% of labelled users were retained and only 17.7% churned. Recall was therefore the critical metric.| Model / split | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|
| Random forest · CV | 46.6% | 12.1% | 19.2% | 82.0% |
| XGBoost · CV | 43.0% | 15.8% | 23.1% | 81.3% |
| XGBoost · validation | 39.6% | 15.0% | 21.7% | 80.9% |
| XGBoost · test | 45.1% | 18.3% | 26.1% | 81.6% |
| XGBoost · threshold 0.194 | 31.4% | 51.3% | 39.0% | 71.5% |
What it showsXGBoost was selected over random forest, yet its test recall was only 18.3%. Moving the decision threshold to 0.194 raised recall to 51.3% at the cost of lower precision and accuracy.
Why it mattersA threshold is a product decision, not merely a modelling default. The preferred setting depends on the relative cost of missed churners and unnecessary outreach.
LimitationEven after threshold adjustment, the model's precision was 31.4%. Richer behavioural history is needed before operational use.
03 · Error analysis
The final model missed 414 of 507 churners
The confusion matrix makes the main deployment risk easier to see than accuracy alone.What it showsAt the default threshold, the model correctly identified 93 churners and missed 414. It also flagged 113 retained users incorrectly.
Why it mattersMost users who actually churned would receive no intervention. This is why the 81.6% headline accuracy should not drive the decision.
LimitationAverage precision was 0.34, confirming limited ranking power across probability thresholds.
04 · Feature importance
Intensity and recency dominated the signal
The leading XGBoost features describe how intensely and how recently a user engaged with the app.Relative XGBoost feature-importance scores.
What it showsKilometres per hour, time since onboarding, sessions per day and the share of recent sessions ranked above device type and professional-driver status.
Why it mattersFuture data collection should prioritise longitudinal activity and recency rather than demographic proxies.
LimitationTree importance indicates contribution to splits, not the direction of an effect or a causal relationship.
Exploration and statistical evidence
The overall churn rate was approximately 17.7% and remained broadly consistent across iPhone and Android users. Engagement variables told a stronger story: churn was more closely associated with how often users were active and how frequently they drove than with device type.
The project combined distribution checks, outlier review, visual analysis and hypothesis testing before any predictive model was accepted.
Feature engineering and model comparison
Behavioural ratios were engineered to capture intensity and habit, including kilometres per drive, drives per day and the share of drives to favourite locations. Logistic regression provided an interpretable baseline, while random forest and XGBoost captured nonlinear interactions.
The final selection used separate training, validation and test sets. Recall was prioritised during tuning since a false negative means a user likely to churn would receive no retention intervention.
Result and recommendation
The champion XGBoost model achieved 81.6% accuracy, 45.1% precision, 18.3% recall and an F1 score of 26.1% on the test set. Its accuracy looks strong at first glance, yet the low recall means most actual churners were still missed.
The model is therefore better suited to exploratory segmentation and future feature development than consequential retention decisions. That recommendation is more valuable than overstating a model that lacks enough predictive signal.
What I would improve next
The next iteration would test richer behavioural sequences, recency features and threshold calibration against a clearly defined intervention cost. Performance should then be assessed through a controlled retention experiment rather than model metrics alone.
- Collect more recent and longitudinal user behaviour.
- Measure the cost of false negatives against the cost of unnecessary outreach.
- Test probability thresholds aligned with the retention strategy.
- Monitor model drift and intervention outcomes after deployment.
People behind the project