flâneur

Leveraging Database Streaming for aggregating data across microservice databases

hieuhani.github.io · 1,152 words · saved by 1 readers

I’m working on an asset management product that is based on multiple independent underlying financial assets, such as cash, stocks, and certificates of deposit. The product requires an overview of all assets, which are aggregated from the independent databases of each product microservice. Initially, the MVP version was implemented on the client side, but as the dataset grew, this became inefficient, particularly when handling pagination. We also have an ETL pipeline for batch processing, which works well for analytical purposes but is not suitable when customers need to see updated asset information immediately after each transaction. We also handle cases where the accrual value increases due to special events, such as receiving bond coupons or the start of a new savings period. This means the data is sourced not only from the microservice databases but also from external events, which are captured and processed through a materialized view. This view aggregates data from multiple Post

Leveraging Database Streaming for aggregating data across microservice databases Leveraging Database Streaming for aggregating data across microservice databases Nov 8, 2024 I’m working on an asset management product that is based on multiple independent underlying financial assets, such as cash, stocks, and certificates of deposit. The product requires an overview of all assets, which are aggregated from the independent databases of each product microservice. The MVP version was implemented on the client side, but as the dataset grew, this became inefficient, particularly when handling pagina

related reading