Pipeline management software for clusters.
Agent Success
Agent success rate when using this tile
67%
Improvement
Agent success rate improvement when using this tile compared to baseline
1.05x
Baseline
Agent success rate without this tile
64%
{
"context": "This criteria evaluates how well the engineer uses Toil's job store persistence API to initialize, configure, and perform file operations across FileJobStore, AWSJobStore, and GoogleJobStore backends. The focus is on correct usage of Toil's Config, Toil context manager, and job store file management methods.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Toil Initialization",
"description": "Uses the Toil class constructor or Toil() to initialize job stores with appropriate locator strings for different backends (file paths, aws:region:bucket, google:project:bucket format)",
"max_score": 20
},
{
"name": "Config Usage",
"description": "Creates and properly configures Config objects with appropriate settings, including workflow ID when needed",
"max_score": 15
},
{
"name": "Context Manager Pattern",
"description": "Uses Toil as a context manager (with statement) or properly calls start() and stop() methods for job store lifecycle management",
"max_score": 15
},
{
"name": "File Import/Write",
"description": "Uses appropriate job store methods for writing files (importFile, writeGlobalFile, or similar) to persist files in the job store",
"max_score": 15
},
{
"name": "File Read/Export",
"description": "Uses appropriate job store methods for reading files (exportFile, readGlobalFile, or similar) from the job store",
"max_score": 15
},
{
"name": "File Deletion",
"description": "Uses deleteGlobalFile or equivalent method to remove files from the job store",
"max_score": 10
},
{
"name": "Job Store Cleanup",
"description": "Properly destroys or cleans up job stores using destroyJobStore, Toil.destroy(), or equivalent cleanup methods",
"max_score": 10
}
]
}tessl i tessl/pypi-toil@9.0.0docs
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10