Central question
What operational and predictive insights can be extracted from NYC taxi trips while keeping model assumptions and limitations visible?
Project overview
This project consolidates a sequence of analytics tasks into one portfolio case study. Using 22,699 sampled yellow-taxi trips, it covers data inspection, exploratory visualisation, statistical testing, multiple linear regression and tree-based machine learning.
The result is a compact demonstration of how business questions evolve: first understand the system, then test a focused relationship, and only then build and evaluate predictive models.
2017 sample
See leakage caveat
Fare model
Fare model
Evidence summary
From trip patterns to predictive performance
The report presents the strongest findings alongside the model-governance issues identified during the analysis.01 · Statistical comparison
Card trips had higher average fares than cash trips
A Welch two-sample t-test compared fare amounts for the two dominant payment methods.What it showsAverage fare was $13.43 for credit-card trips and $12.21 for cash trips. The difference was statistically significant (t = 6.867; p = 6.80 × 10⁻¹²).
Why it mattersPayment method is associated with fare level in this sample and may help describe trip mix or customer behaviour.
LimitationThe test does not show that payment method causes a higher fare. Trip distance, route, timing and passenger behaviour may explain the association.
02 · Fare regression
Test performance was strong, with a clear caveat
The regression used route-level mean distance and duration alongside trip characteristics to estimate fare amount.| Split | R² | MAE | RMSE | Mean residual |
|---|---|---|---|---|
| Training | 0.840 | $2.19 | $4.23 | - |
| Test | 0.868 | $2.13 | $3.79 | −$0.015 |
What it showsThe test model explained 86.8% of fare variance, with an average absolute error of $2.13. Train and test results were close, while the mean test residual was −$0.015.
Why it mattersThe model captured the dominant fare structure and produced errors that were broadly centred around zero.
LimitationRoute averages were computed before the train/test split. This leakage inflated test R² by roughly 0.03, so the reported metrics must be treated as provisional.
03 · Tipping classification
Random forest led the model comparison
The classification extension predicted whether a credit-card customer would leave a tip of at least 20%.| Model / split | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|
| Random forest · CV | 67.5% | 75.7% | 71.4% | 68.0% |
| Random forest · test | 67.5% | 77.9% | 72.3% | 68.7% |
| XGBoost · CV | 66.9% | 72.3% | 69.5% | 66.6% |
| XGBoost · test | 67.7% | 74.5% | 71.0% | 67.9% |
What it showsRandom forest achieved the stronger test F1 score (72.3%) and recall (77.9%), outperforming XGBoost on the selected objective.
Why it mattersThe result is useful for comparing algorithms and interpreting features; it should not be used to restrict access to taxi service or disadvantage customers.
LimitationThe target captures observed tipping in one historical sample and should not be treated as a stable customer characteristic.
04 · Error and feature analysis
Recall improved at the cost of many false positives
The test confusion matrix and feature ranking explain what sits behind the random-forest metrics.Leading random-forest features
What it showsThe model correctly identified 1,252 generous tips and missed 355, while producing 602 false positives. Vendor, predicted fare, route duration and route distance were the leading features.
Why it mattersThe matrix makes the trade-off concrete: strong recall does not guarantee precise targeting. The feature ranking also highlights a vendor effect that deserves separate investigation.
LimitationImpurity-based feature importance does not reveal direction, fairness or causality. Vendor should be audited before any operational interpretation.
Exploration and business insight
Most trips were shorter than two miles and trip frequency fell rapidly as distance increased. Fare, distance and duration contained extreme values that required contextual review rather than automatic deletion. The project also transformed pickup and drop-off timestamps into trip-duration features and compared demand across time periods.
A two-sample test found a statistically significant difference in average fare between credit-card and cash trips (p < 0.001). The result supports an association in the observed data; it does not establish that payment method causes higher fares.
Fare prediction
The multiple linear regression model reported a test R² of 0.868, MAE of $2.13 and RMSE of $3.79. Distance and route-level duration information carried much of the predictive signal, and diagnostic checks were used to assess model assumptions and residual behaviour.
The model's training and test scores were close, which initially suggested stable generalisation. A later governance review identified an important limitation in how route averages had been created.
Model-governance caveat
Route-level mean distance and duration were calculated using the full dataset before the split. That lets information from test observations influence the features and introduces data leakage. The notebook estimated that this increased R² by roughly 0.03.
The correct next version would estimate route means on training data only, define a principled fallback for routes unseen during training and then recalculate every test metric. The caveat is displayed here deliberately: a portfolio should show analytical judgement as clearly as technical output.
Classification extension
The project also compared random forest and XGBoost classifiers for identifying generous tippers. The random forest was the stronger test model, reaching 77.9% recall and an F1 score of 72.3%, while accuracy was 68.7%.
- Used stratified train/test splitting for the binary target.
- Tuned tree-based models with cross-validation.
- Compared precision, recall, F1 and accuracy rather than relying on one score.
- Translated statistical and model results into bounded business recommendations.
People behind the project