CtrlK
CommunityDocumentationLog inGet started
Tessl Logo

tessl/pypi-toil

Pipeline management software for clusters.

Agent Success

Agent success rate when using this tile

67%

Improvement

Agent success rate improvement when using this tile compared to baseline

1.05x

Baseline

Agent success rate without this tile

64%

Overview
Eval results
Files

rubric.jsonevals/scenario-8/

{
  "context": "This criteria evaluates how well the engineer uses Toil's Service job abstraction to manage long-running processes like databases. The focus is on correct usage of the Job.Service class, lifecycle methods, service attachment, and connection information passing.",
  "type": "weighted_checklist",
  "checklist": [
    {
      "name": "Service class definition",
      "description": "Defines a service by creating a class that inherits from Job.Service",
      "max_score": 20
    },
    {
      "name": "Lifecycle methods implementation",
      "description": "Implements the required lifecycle methods: start(), stop(), and check() to manage the database service",
      "max_score": 25
    },
    {
      "name": "Service attachment",
      "description": "Uses addService() to attach the service to a job so it starts after the job completes and stops after children finish",
      "max_score": 20
    },
    {
      "name": "Connection information passing",
      "description": "Returns connection information (host/port) from start() method to make it available to dependent jobs via promises",
      "max_score": 20
    },
    {
      "name": "Job hierarchy setup",
      "description": "Creates proper job hierarchy with addChild() or addFollowOn() to establish service dependencies and enable parallel execution",
      "max_score": 15
    }
  ]
}
tessl i tessl/pypi-toil@9.0.0

tile.json