If I pre-chunk input into groups of N chunks where N >= number of logical CPU cores, this seems to make sense and it simplifies the data deps a lot to run filtering for all N chunks before deflating, rather than trying to confirm another thread finished part of its work...
Still trying to grok how to work this into something like Rayon for rust; running over lists of chunks sounds like it would work.
ok just one more. ;) I think this works more cleanly for streaming and inconsistent load, and still feels conceptually clean. Has a model of enqueuing things to a work queue, and as long as they get grabbed in order the data dependencies stay correct. But then there's more state stuff that I have to grok the rust-y way to do it.
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!