Tags: zerebubuth/planet-dump-ng
Tags
Try to avoid pblock overflow by tracking estimated size Planet dump would fail intermittently complaining of block overflow (#25). The root cause of this was failing to encode [relation 6677259](https://www.openstreetmap.org/relation/6677259) in the history PBF. PBF files are sequences of individual blocks and every block must be smaller than 16MiB. To try and satisfy this, the PBF encoder checks the size of the encoded pblock on a regular basis, every `N` elements. The assumption is that the average size of the element multiplied by `N` would be smaller than 16MiB. Unfortunately with relation 6677259 that wasn't the case, as versions in its history have more than 25,000 members and were taking up about 100kiB each (so only 164 versions are needed to overflow, relation 6677259 has 440-ish). This change implements an _approximate_ size counter for relations, which should help to make sure that the pgroup is flushed when it gets too large, even if it hasn't reached `N` elements yet. This hasn't seemed necessary yet for ways (they are limited to a fixed number of nodes each) or nodes, but extending to handle those - if necessary - should be straightforward.
Recheck relation size limit more often and with a smaller margin. This rechecks the relation block size every 200 elements instead of every 500, so should be more likely to catch overruns. It also only permits a 12.5% overrun of the "ideal" block length before flushing, which should limit the scope for reaching the error condition (100% overrun).
PreviousNext