National R&D Project Analysis Pipeline
Links preprocessing through inference as nodes so researchers can assemble the pipeline themselves.
- Timeline
- About 6 months
- Team size
- 4 people
- Platforms
- Web · Server
- Configuration style
- Node-based pipeline
Background
Researchers had to file a request to the development team every time they wanted to change anything in the pipeline, from data preprocessing to model inference. Given that this was a government research project where research direction changed often, researchers needed to be able to adjust the configuration themselves, without going through the development team.
Scope
- Data preprocessing node configuration
- Inference node connection and execution
- Pipeline save and reuse
- Execution result lookup
- Document-search-based auxiliary node
Technical decisions
What we chose, and why it fitBroke everything from preprocessing to inference into individual nodes, connected together
Preprocessing steps and model combinations needed differed by researcher, so a single fixed pipeline wasn't enough. Making each step an independent node that researchers could pick and connect as needed meant they could try new experiment configurations immediately, without a development team request.
Built a document search node as a separate RAG structure
Researchers frequently needed to look up reference papers or past experiment records during their work. We made this a separate search node that could be freely connected to other inference nodes.
Engineering challenges
Problem
Because nodes could be connected freely, researchers often connected them in the wrong order, causing executions to fail partway through.
Approach
We defined input and output data formats for each node, and flagged mismatched connections on screen before execution rather than after.
Result
Problems could now be caught at the connection stage instead of discovered as a failure after execution.
Problem
When PyTorch-based inference nodes used heavy models, execution took longer and delayed other researchers' pipeline runs on the same server.
Approach
We managed inference node execution through a queue and let each researcher check the running pipeline's status on their own screen.
Result
Even when server resources overlapped, researchers could see which execution was being processed first.
Handover and operations
We provided a node configuration guide for researchers and documented how to save and reuse pipelines. We handed over the source code and execution environment configuration, and supported new node type additions separately as the project progressed.