Indeed the cross-chunk recombination is quite something. We're used to clean cut boundaries and here he is going away from this rule to bring new opportunities. But I wonder how hard it is to think about, design and test such assoc. combiners ..
Yes, it may be quit hard to design such a combiner due to constraints
both on the input (as the need for cross-chunk recombination),
and on the output (as too early finalization preventing further combinations);
with the consequence that we often have to imagine a genuine intermediate
data representation of the problem.
But this complexity is mitigated by the fact that these associative combiners
can be built incrementally using a reduced set
of patterns, combiner compositions and transformations.
Have a look to this remarkably well written post [1]
on incremental regular expressions (it appears that incremental computation is deeply related with parallel computation).
It shows well, on a non-trivial example, how to build such an intermediate data
representation with its associative combiner.
