Update propagation to a data warehouses with specified latency

TLDR: In this article, a method and system for performing real-time transformations of dynamically increasing databases is described, in which a session, identified as a real time session, is initialized.

Abstract: A method and system thereof for performing real time transformations of dynamically increasing databases is described. A session, identified as a real time session, is initialized. The real time session repeatedly executes a persistent (e.g., continually running) data transport pipeline of the analytic application. The data transport pipeline extracts data from a changing database, transforms the data, and writes the transformed data to storage (e.g., a data warehouse or data mart). The data transport pipeline is executed at the end of each time interval in a plurality of contiguous time intervals occurring during the real time session. The data transport pipeline remains running after it is executed, until the real time session is completed. Accordingly, new data are transformed in a timely manner, and processing resources are not consumed by having to repeatedly re-establish (re-initialize) the data transport pipeline.