Mobily AI/ML Modernization.
Cairo, Egypt
Source: resume + original portfolio + LinkedIn
An end-to-end modernization effort migrating 15 mission-critical telecom use cases from legacy SAS environments into Dataiku-centered ML and data pipelines.
The Customer Value Management (CVM) and B2B business divisions operated a complex legacy analytics ecosystem consisting of proprietary SAS modules (Viya, Model Studio, Job Executor, CAS actions) and extensive Teradata stored procedures. This infrastructure incurred substantial operational overhead, presented high maintenance complexity with hundreds of custom macros, and created computational bottlenecks during large-scale customer network simulations.
The core challenge lay in re-architecting and migrating 13 critical analytical pipelines into a modern governed platform (Dataiku DSS) without interrupting live business operations. The target state required ensuring strict numeric parity with legacy baselines while offloading heavy transformations onto distributed compute clusters and optimized Teradata SQL database pushdown engines.
My approach focused on reverse engineering monolithic legacy SAS models and scripts, and restructuring them into modular, governed workflows containerized on Kubernetes and orchestrated via Dataiku DSS. To eliminate compute bottlenecks, we systematically offloaded resource-intensive feature preparation and training workloads to distributed clusters on Cloudera, while routing graph-theory and specialized modeling tasks to dedicated high-RAM Kubernetes pods. Standardized Git-backed project versioning and an MLflow-driven Model Registry ensured complete traceability from development to production.
Moving beyond simple model migration, we established a rigorous promotion pipeline. Models are developed on Design nodes, packaged into versioned, immutable bundles, and promoted to UAT Automation nodes where they run against non-production staging tables to perform strict regression checks and assert parity against legacy baselines. Once approved, CI/CD scripts deploy these bundles to Production Automation nodes. Daily and monthly production scoring routines run inside isolated Kubernetes containers, staging outputs in temporary schemas before executing atomic Teradata merges to protect database locks and ensure zero-downtime campaign updates.
Pipeline Deep Dives
// 01 — Social Network Analysis (SNA) & Graph Mining
Constructs undirected relationship graphs from millions of monthly call details on local high-RAM compute nodes. Implements high-performance modularity optimization graph algorithms for community detection and centrality metrics (Out-Degree and weighted out-degree) to identify high-influence subscribers for targeted campaigns.
// 02 — Price Elasticity & Demand Simulation Engine
Calculates demand elasticity coefficients using a bottom-up hierarchical rollup cascade (Package → L2 → L1 → L0) to resolve data sparsity. Applies Interquartile Range (IQR) filtering and runs parallel log-log regressions across 30 pricing scenarios to map products into strategic performance quadrants.
// 03 — Dual Collaborative Filtering Recommendation Systems
Generates cross-sell and up-sell guidance by running matrix operations over product purchase history. Leverages User-User similarity matching for standard accounts and Item-Item matrices for maximum-holding accounts. Resolves sparse data via segment augmentation and prioritizes upgrades by monthly and non-recurring revenue profiles.
// 04 — Value & Sophistication Microsegmentation
Classifies accounts across financial value (revenue ranks + growth potential) and tech sophistication (complexity indices). Employs linear interpolation binning and optimizes scoring runs using delta-processing paths that evaluate only new, maturing, or recovering accounts to prevent full-table recalculations.
// 05 — Churn, Propensity & Consumer Risk Pipelines
Predicts voluntary/involuntary churn and consumer credit defaults. Applies Weight of Evidence (WoE) mapping and target encoding to sparse inputs. Trains autotuned ensemble gradient boosting algorithms using segment-specific activity thresholds to ensure high-precision retention scoring.
Strict Schema Validation Contracts
Automated scenario pre-checks match incoming datasets against strict structural definitions (column names, types, and modeling roles: inputs, rejects, features, targets) before scheduling model runs, ensuring garbage data never pollutes downstream scoring.
Adversarial Data Drift Monitoring
To monitor input data drift, I implemented non-blocking flow branches that train a binary classifier to distinguish between reference baseline data and live scoring batches. A high ROC-AUC score triggers notifications indicating significant feature distribution shifts.
Automated Drift Tracking
Evaluation steps automatically calculate performance metrics (AUC, Kolmogorov-Smirnov stats, lift curves) by comparing historical model predictions against subsequent actual ground-truth labels, exposing concept drift before it impacts campaigns.
Atomic Staging & Error Recovery
Connection retry loops handle database timeouts, while strict step-validators abort scoring runs if upstream validation fails. Teardown rules truncate target tables on script exceptions to maintain strict atomic data integrity inside Teradata.
- →Scale vs. Complexity: Migrating and re-architecting 13 critical enterprise telecom analytics pipelines under strict legacy parity constraints.
- →Massive Footprint: Serving a commercial subscriber footprint of over 13 million active lines across the network.
- →Distributed Compute: Offloading calculations to distributed containers on Cloudera, Teradata SQL pushdowns, and high-RAM local graph nodes to solve compute bottlenecks.
- →Enterprise MLOps: Implementing strict MLOps controls, including input schema contracts, adversarial data drift tracking, and automated performance evaluation.
I engineered a resilient, Kubernetes-native MLOps architecture that bridges legacy SAS deconstruction with a modern, governed promotion lifecycle. Source SAS macros and Teradata logic are reverse-engineered into strict data contracts and developed as Git-versioned Dataiku flows with MLflow experiment tracking. To enforce zero-regression deployments, I designed a UAT parity gate that automatically asserts numerical equivalence between the new container-based pipelines and legacy baselines before CI/CD automation promotes the immutable bundles to production. The runtime layer leverages containerized high-RAM inference, fortified by adversarial drift detection and atomic staging swaps to guarantee high-availability scoring.
- →Achieved full numerical output parity against historical SAS baselines, validating migration accuracy.
- →Optimized pipeline execution window durations, meeting all SLA targets (e.g. B2B recommendations within 4 hours).
- →Unlocked high-performance graph processing utilizing NetworKit on K8s clusters.