The Stellar (XLM) Development Foundation has announced a significant advancement in data processing speeds within its Horizon API, leveraging Captive Data Processing (CDP) components. According to Stellar, this integration allows for data reingestion that is up to nine times faster than previous methods.

Enhanced Reingestion with CDP

Horizon, the HTTP API for accessing data on the Stellar network, has traditionally faced challenges with data reingestion speed. However, the introduction of CDP technology has transformed this process, enabling faster and more efficient data handling. The improvement is particularly noticeable in scenarios involving large-scale data reingestion.

Technical Benchmarking

To evaluate the performance gains, Stellar conducted tests using a specific hardware configuration. The setup included an EC2 instance (m5.4xlarge) with 16 virtual CPUs, 64 GB of RAM, and a PostgreSQL 12 database running on an RDS instance (db.r5.4xlarge) with 128 GB of RAM. These tests compared the performance of Captive Core and CDP in reingesting data from 10,000 ledgers.

CDP demonstrated superior parallelization capabilities, achieving optimal results with 16 workers, whereas Captive Core’s performance was limited by disk I/O, showing diminishing returns beyond four workers.

Parallel Reingestion Capabilities

One of the standout features of the CDP-enhanced Horizon is its ability to support parallel reingestion. This method divides the reingestion range into subranges that are processed simultaneously, significantly cutting down the time required for data ingestion. The performance gains were quantified by reingesting 10,000 ledgers, with CDP outperforming Captive Core in terms of speed and efficiency.

Full History Reingestion

Stellar also analyzed the time required to reingest its entire transaction history. Using the optimal parallel setup, they projected that reingesting 10,000 ledgers with Captive Core would take approximately 66 days. In contrast, the CDP approach, which utilizes precomputed ledger metadata, could accomplish the same task in about seven days.

Implications for Future Development

This enhancement in Horizon’s reingestion capabilities is expected to have a significant impact on the Stellar network’s performance and scalability. By reducing the time and resources required for data processing, developers can focus on building more robust applications on the Stellar blockchain.

For more detailed information on the technical implementation of these enhancements, you can visit the Stellar website.

Image source: Shutterstock