If I pre-chunk input into groups of N chunks where N >= number of logical CPU cores, this seems to make sense and it simplifies the data deps a lot to run filtering for all N chunks before deflating, rather than trying to confirm another thread finished part of its work...
Still trying to grok how to work this into something like Rayon for rust; running over lists of chunks sounds like it would work.
ok just one more. ;) I think this works more cleanly for streaming and inconsistent load, and still feels conceptually clean. Has a model of enqueuing things to a work queue, and as long as they get grabbed in order the data dependencies stay correct. But then there's more state stuff that I have to grok the rust-y way to do it.
This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either!