Databricks Speeds Up Feature Store to 200ms Latency

Databricks has introduced a real-time pipeline architecture for its Feature Store, reducing the delay between raw event generation and online feature availability to a 200-millisecond p99 latency. Previously, data scientists relying on traditional Spark microbatch pipelines faced lag times ranging from minutes to hours. This latency reduction is crucial for time-sensitive applications like fraud detection, where a model must evaluate a transaction by comparing a user's 30-day historical baseline against their transaction volume over the last 10 minutes.

At the core of this update is Spark Real-Time Mode (RTM), a new execution engine running on serverless Lakeflow Spark Delta Pipelines. Unlike traditional microbatch modes that process data in discrete intervals, RTM processes individual rows continuously. It calculates rolling window aggregations on the fly using a local RocksDB state store. To maintain fault tolerance without bottlenecking performance, RTM amortizes checkpointing costs over five-minute intervals. If a failure occurs, the system replays at most five minutes of data from Kafka, maintaining exactly-once processing guarantees while slashing steady-state latency.

To handle the high volume of small, frequent updates generated by rolling windows, the platform utilizes Lakebase for online storage. Standard databases like Postgres often suffer from write-ahead log amplification because they write full 8KB page images after checkpoints. Lakebase avoids this bottleneck by writing compact change records to a quorum of distributed safekeeper nodes, allowing the system to scale to tens of thousands of reads per second with tens of milliseconds of latency. Finally, Databricks Model Serving retrieves these features automatically, scaling independently to support more than 100,000 queries per second on CPU endpoints.

For machine learning practitioners, this integration eliminates the need to build and maintain custom, complex streaming infrastructure to serve fresh features. By defining a feature once, developers can deploy it across both offline batch training and online real-time serving. The system automatically manages the underlying infrastructure, tracks lineage within Unity Catalog, and logs feature dependencies using MLflow, simplifying the entire lifecycle of real-time AI applications.