GuideBuild performance
Memory and parallelism
How many units Douze runs at once, how memory grows with parallelism, and how to build in a container with little memory.
Douze runs the steps of a workflow at the same time, and the units of each step in parallel: as many as the machine
has cores, by default. That makes a build fast, and it also sets how much memory the build holds. This page explains
what uses memory, how to trade speed for memory with -p, and what to set in a container.
ParallelismLink to this heading
-p (--parallelism) is the most units that run at once, across all steps. Its default is the number of
processors. site.parallelism in douze.yml sets it for a site, and -p wins over it:
site:
parallelism: 8
douze -p 4
Steps do not wait for each other to finish: a page goes on to markdown as soon as render is done with it, while
other pages are still rendering. Filter command steps run up to -p processes at once. The hover steps have their own
limit, processes (4 language servers by default).
What uses memoryLink to this heading
Each unit that runs holds its working set: the page it renders and its transient strings, the regexes of the code
grammars it uses, the image it encodes. At -p 14, about 14 of them are in memory at once. Whatever the parallelism,
a build also holds:
- the site's data and the data of every page;
- the pages that wait at a barrier, a step that needs every page before it can run (
collect,localize, a batch command step). They wait as UTF-8 bytes.
So lowering -p lowers the part that grows with parallelism, and the build takes longer. It does not lower what the
site's data and barriers need.
FiguresLink to this heading
The benchmarks measure a documentation site of 70,000 files with 78 MB of JSON data, on 14 cores, with highlighting and TypeScript hovers. Footprint is the dirty memory that macOS reports at the peak.
-p |
Time | Peak footprint |
|---|---|---|
| 14 | 32 s | 5.3 GB |
| 4 | 46 s | 4.5 GB |
The repository's Sample site, 21 files, peaks at about 45 MB.
In a containerLink to this heading
When memory is scarce, lower -p first:
RUN douze -s site -o /out -p 2
- A build that runs out of memory fails with an error, and keeps the cache of the previous build.
- Lowering
-pis the way to use less memory: a heap limit set in the environment (DOTNET_GCHeapHardLimit) does not capdouze. --no-cachesaves the cache's disk space, not memory.