Pipeline management software for clusters.
Agent Success
Agent success rate when using this tile
67%
Improvement
Agent success rate improvement when using this tile compared to baseline
1.05x
Baseline
Agent success rate without this tile
64%
{
"context": "This criteria evaluates how well the engineer uses Toil's API to implement conditional workflow execution. The focus is on proper use of Job classes, the promise/return value system for passing data between jobs, and conditional job graph construction.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Job class definition",
"description": "All processing components (QualityCheckJob, FullProcessingJob, PartialProcessingJob, ErrorReportJob, DataWorkflow) are defined as classes that inherit from toil.job.Job",
"max_score": 15
},
{
"name": "Job.run() implementation",
"description": "Each job class properly overrides the run(self, fileStore) method to implement its logic",
"max_score": 10
},
{
"name": "Return value usage",
"description": "QualityCheckJob returns a quality score value from its run() method that can be accessed by successor jobs",
"max_score": 15
},
{
"name": "Promise system",
"description": "Uses job.rv() or promise mechanism to pass the quality score from QualityCheckJob to DataWorkflow for conditional branching decisions",
"max_score": 20
},
{
"name": "Conditional job addition",
"description": "DataWorkflow conditionally adds child jobs (addChild() or addFollowOn()) based on the quality score value: FullProcessingJob for score >= 80, PartialProcessingJob for 50-79, ErrorReportJob for < 50",
"max_score": 25
},
{
"name": "Workflow execution",
"description": "Uses Toil context manager (with Toil(options) as toil:) and toil.start() method to execute the workflow with the root job",
"max_score": 10
},
{
"name": "Correct test outcomes",
"description": "The workflow produces correct outputs for all three test cases: returns appropriate result string based on which job path was executed",
"max_score": 5
}
]
}tessl i tessl/pypi-toil@9.0.0docs
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10