Databricks Details Real-Time Recommendation Architecture

Databricks has published a comprehensive reference architecture designed to help e-commerce platforms build multi-stage recommendation and ranking systems. Based on a real-world deployment for an Asian fashion retailer serving over one million monthly active users across a catalog of more than 100,000 SKUs, the system demonstrates how a single platform can replace fragmented machine learning pipelines. The architecture leverages Databricks tools to ingest approximately 1,000 events per second, including clicks, searches, and cart additions, directly into Unity Catalog Delta tables.

The system utilizes Lakeflow Connect's Zerobus Ingest for data collection and offers two distinct serving paths to achieve low double-digit millisecond latencies. Path A handles pre-computed batch recommendations for predictable surfaces like homepage carousels. Path B manages real-time, session-aware scoring. For this real-time path, the system uses Databricks AI Search to retrieve 200 to 500 candidate items. It then performs point lookups against Lakebase online tables to assemble user and item features, feeding them into a LightGBM scoring model to predict conversion probabilities before applying final business rules to output 10 to 20 items.

For machine learning practitioners, this unified approach solves the notorious cold-start problem for new users and products while eliminating the need for complex glue code. By managing data engineering, feature storage, model training with MLflow, and low-latency serving under Unity Catalog, teams can maintain strict data lineage and training-serving consistency. Ultimately, this streamlined workflow allows engineering teams to rapidly iterate on models and deploy updates in days rather than months, helping businesses target industry benchmarks of 10% to 30% lifts in conversion rates.