National R&D Project Analysis Pipeline
Researchers had to file a request to the development team every time they wanted to change anything in the pipeline, from data preprocessing to model inference. Given that this was a government research project where research direction changed often, researchers needed to be able to adjust the configuration themselves, without going through the development team.
Key technical decision
Broke everything from preprocessing to inference into individual nodes, connected together
Preprocessing steps and model combinations needed differed by researcher, so a single fixed pipeline wasn't enough. Making each step an independent node that researchers could pick and connect as needed meant they could try new experiment configurations immediately, without a development team request.
