Pipeline management software for clusters.
Agent Success
Agent success rate when using this tile
67%
Improvement
Agent success rate improvement when using this tile compared to baseline
1.05x
Baseline
Agent success rate without this tile
64%
{
"context": "This criteria evaluates how well the engineer uses Toil's Service job abstraction to manage long-running processes like databases. The focus is on correct usage of the Job.Service class, lifecycle methods, service attachment, and connection information passing.",
"type": "weighted_checklist",
"checklist": [
{
"name": "Service class definition",
"description": "Defines a service by creating a class that inherits from Job.Service",
"max_score": 20
},
{
"name": "Lifecycle methods implementation",
"description": "Implements the required lifecycle methods: start(), stop(), and check() to manage the database service",
"max_score": 25
},
{
"name": "Service attachment",
"description": "Uses addService() to attach the service to a job so it starts after the job completes and stops after children finish",
"max_score": 20
},
{
"name": "Connection information passing",
"description": "Returns connection information (host/port) from start() method to make it available to dependent jobs via promises",
"max_score": 20
},
{
"name": "Job hierarchy setup",
"description": "Creates proper job hierarchy with addChild() or addFollowOn() to establish service dependencies and enable parallel execution",
"max_score": 15
}
]
}tessl i tessl/pypi-toil@9.0.0docs
evals
scenario-1
scenario-2
scenario-3
scenario-4
scenario-5
scenario-6
scenario-7
scenario-8
scenario-9
scenario-10