Telco Churn MLOps

Retaining an existing customer costs significantly less than acquiring a new one — but only if retention efforts target the right customers, with enough margin to justify the campaign's cost. The project starts from a real telecom dataset (Telco Customer Churn, 7,043 customers, 50 columns: demographics, contract details, subscribed services and billing) with a dual objective:

1. Estimate each customer's churn probability with enough confidence to prioritize intervention.
2. Sustain that confidence over time, detecting when the model degrades against new data and deciding objectively when to replace it — not a model trained once and abandoned.

The system was designed as layers with a single-direction dependency (Data → Features → Training → Registry/Serving → Monitoring → Presentation), each decoupled from the next so that, for example, serving predictions never depends on monitoring being up. Every non-obvious architecture or tooling decision was documented as an ADR before implementation (14 ADRs across 8 phases) — the project's explicit goal is to demonstrate the full MLOps lifecycle with the same practices a real team would use, not just ship a classifier with a good metric.

The EDA identified, among other findings, four columns with data leakage and low signal-value columns, explicitly documented and excluded in ADR 0006:

• Leakage: Churn Score, Customer Status, Churn Category, Churn Reason — all derived from churn or only available after it already happened.
• Low signal value: Latitude, Longitude, Zip Code, CLTV.

Before reaching the model, every data batch passes through a Pandera schema contract (types, nullability, valid categorical values) verified with automated tests — the goal is for a data-quality issue to fail loudly in the pipeline, not silently in a production prediction.

Of the original 50 columns, 37 raw features remain after excluding leakage, IDs and low-value columns, plus 2 derived variables (is_new_customer, num_extra_services) designed from EDA findings rather than arbitrarily. Encoding and scaling deliberately live in the training layer, not in features — features produces clean, human-readable data; training decides how to encode it based on what each model family needs, a separation of concerns documented in ADR 0007.

4 model families were compared under the same evaluation protocol — Logistic Regression (baseline), XGBoost, LightGBM and CatBoost — with explicit focus on how each handles the dataset's high-cardinality categoricals (ADR 0009). LightGBM won with F1=0.929 in the initial comparison; that result carried into a final pipeline with Bayesian hyperparameter tuning via Optuna (50 trials, Stratified K-Fold to avoid overfitting the split selection), which raised the final champion's F1 to 0.931 with a ROC-AUC of 0.992.

Every experiment (parameters, metrics, artifacts) is logged to MLflow, with the Model Registry managing which version serves in production via aliases (champion), not the now-deprecated stages system. Code is versioned with Git and data/models with DVC against a DagsHub remote — the combination makes it possible to reproduce any past result knowing exactly which code, data and configuration produced it, without relying on a local notebook staying intact.

The system simulates incoming data with perturbations of increasing intensity and evaluates them with Evidently AI to quantitatively detect drift (dataset_drift_share). A retraining threshold (10%, calibrated against the project's own results, not an arbitrary number) decides whether retraining is warranted; the resulting challenger is compared against the current champion under an explicit business criterion — not just aggregate F1. To be promoted, the challenger must:

• Win by a minimum margin of 1 point.
• Not regress by more than a defined tolerance in any relevant business segment: contract type, payment method, internet type.

Retraining and promotion are deliberately manual steps, with a human in the loop, not an automatic cron job.

The model is served via a REST API (FastAPI, request validation with Pydantic) with two business endpoints: prediction and SHAP explanation (TreeExplainer, exact values for tree-based models, no background dataset required). An interactive Streamlit dashboard consumes that API and adds a business-value layer: for each prediction, it estimates how much revenue is at risk if that specific customer churns (the remaining value of their current contract) and compares that number against the cost of a retention campaign to decide, with an explicit criterion, whether intervening is worth it — the part of the project that connects the technical prediction to the real business decision.

149 automated tests (unit, data-validation and integration) run on every push and pull request against main via GitHub Actions, alongside lint (Ruff), formatting and type-checking (MyPy) as a mandatory gate before merging. The integration tests go beyond mocking external dependencies: they exercise the real production pipeline (no mocks) and spin up the full stack with Docker Compose to catch, in CI, the class of bug a unit test can't see by design — a container that builds but doesn't start, or an artifact that never makes it into the final image.

The API and dashboard are packaged as independent Docker images (multi-stage builds, using uv to install exact dependencies via lockfile) and can be orchestrated locally with Docker Compose. The final hosting destination was deliberately left open during development: two free-tier platforms (Render, then Hugging Face Spaces) changed their pricing policy between when the deployment was designed and when it was attempted — a real infrastructure decision and its trade-off, not a technical limitation of the project.

The project documented and solved three classes of real, not hypothetical, problems:

1. An MLflow artifact-storage bug on DagsHub that blocked downloading the pipeline from the Registry — diagnosed with evidence (reproduced, isolated from causes like quota or credentials) and resolved by redesigning to rely less on that download, versioning the champion with DVC instead of MLflow.
2. A Docker image that built without error but crash-looped in production due to a missing OS-level library (libgomp1) lost between stages of a multi-stage build.
3. Two free hosting platforms that changed their pricing policy midway through implementation.

Each episode was documented with the full diagnosis, not just the fix, as part of the project's engineering standard.

The result isn't just a classifier with 0.99 ROC-AUC and 0.93 F1 on the production champion — it's a system where every component (what data went in, which model was trained with which hyperparameters, why a new version was or wasn't promoted, how reliable it remains against changing data) is traceable and auditable. The project shows that the hardest part of a production ML system is rarely the model — it's everything around it: validation, reproducibility, version governance, continuous monitoring, and the final translation of a probability into a business decision with a real cost.