Pipeline management software for clusters.
Agent Success
Agent success rate when using this tile
67%
Improvement
Agent success rate improvement when using this tile compared to baseline
1.05x
Baseline
Agent success rate without this tile
64%
{
"context": "This criteria evaluates how well the engineer uses Toil's job dependency management features (addChild and addFollowOn) to create a multi-stage data processing pipeline with correct execution order and parallelization.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Job class definition",
"description": "Defines job classes that inherit from toil.job.Job for the different pipeline stages (e.g., DownloadJob, ProcessJob, AggregateJob, CleanupJob or similar)",
"max_score": 15
},
{
"name": "addChild usage",
"description": "Uses Job.addChild() method to add parallel jobs (download jobs and/or processing jobs) that should execute concurrently",
"max_score": 25
},
{
"name": "addFollowOn usage",
"description": "Uses Job.addFollowOn() method to create sequential stages that execute after children complete (e.g., processing coordinator after downloads, aggregation after processing, cleanup after aggregation)",
"max_score": 25
},
{
"name": "Correct execution order",
"description": "Implements the job hierarchy such that stages execute in the correct order: downloads in parallel, then processing in parallel, then aggregation, then cleanup",
"max_score": 20
},
{
"name": "Workflow initialization",
"description": "Uses toil.common.Toil class or similar to initialize and start the workflow with the root job and job store",
"max_score": 10
},
{
"name": "Root job coordination",
"description": "Creates a root/coordinator job that properly sets up the job dependency chain using addChild and addFollowOn to coordinate the entire workflow",
"max_score": 5
}
]
}tessl i tessl/pypi-toil@9.0.0docs
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10