Parallel chunk merging in Manticore Search
Starting from Manticore Search 24.4.0, RT table compaction has a more capable execution model. Instead of merging chunk pairs one-by-one in a serial flow, optimization now supports two important im...

Source: DEV Community
Starting from Manticore Search 24.4.0, RT table compaction has a more capable execution model. Instead of merging chunk pairs one-by-one in a serial flow, optimization now supports two important improvements: disk chunk merges can run in parallel each merge job can merge more than two chunks at once parallel_chunk_merges: how many RT disk chunk merge jobs may run at the same time merge_chunks_per_job: how many RT disk chunks a single job can merge in one pass The compaction docs were also updated to describe optimization as an N-way merge handled by a background worker pool rather than a single serial merge thread. Why this matters For RT workloads, the interesting number is often not just how fast you can insert documents, but how long it takes until compaction catches up and the table returns to its target chunk count. That is especially noticeable when: you ingest data at a sustained rate optimize_cutoff is low enough that merges kick in early you wait for compaction to finish befor