I’m currently writing some ETL code. Typically, I have consecutive daily files that are parsed, data is extracted and processed with a chain of transducers. But sometimes a file is missing. This information needs to be preserved and processed differently. My question is: how to best do that?
In more abstract terms: I could need some input for error handling in transducers: Since transducers are inherently sequential (they are just composed functions) how do you distinguish between valid values that should be further processed and errors?
Is there a better way than either aborting the whole computation or making each downstream transducer aware of success/failure values?
Maybe a “routing” transducer would be useful: Something that can dispatch values (like a multimethod) but with different downstream transducers per value type. Has anyone gone down this path and can share his/her opinion?