Repository tasks with runnable environments and verifiable outcomes.

A paid batch spanning agreed change types in approved repositories. Optional state-conditioned tasks require operational facts from approved tools; paired states test whether those facts affect the correct implementation.

Proposed specification: Accepted changes shall satisfy disclosed behavior and regression requirements, with reference success, rejected invalid implementations and reproducible initial state.

Use this task type as the starting point for the call.

• SWE-rebench V2 (opens in a new tab)
• SWE-PolyBench (opens in a new tab)

A small optimization batch in one repository on one hardware class, with multiple workload sizes and measured timing variance.

Proposed specification: Performance credit shall require preserved correctness and improvement beyond the agreed noise band and threshold under the declared measurement protocol.

Use this task type as the starting point for the call.

• SWE-fficiency (opens in a new tab)

A paid batch across approved repositories covering root-cause tracing, change impact and supported review findings, with reference and incorrect-answer controls.

Proposed specification: Accepted tasks shall distinguish correct, grounded findings from plausible unsupported answers without requiring one predetermined exploration sequence.

Use this task type as the starting point for the call.

• Turing's codebase-exploration methodology (opens in a new tab)

A controlled service catalog and representative task batch covering retrieval, composition and one applicable recovery path; validate the selected API or MCP interface.

Proposed specification: Task credit shall depend on supported results and required state changes, with declared interfaces, recoverable errors and valid alternative workflows.

Use this task type as the starting point for the call.

• MCP-Atlas (opens in a new tab)
• MCP-Universe (opens in a new tab)

Paired benign/adversarial cases for one coding workflow, with authorized references and prohibited-action controls across the selected execution paths.

Proposed specification: Acceptance shall demonstrate permitted-task solvability and independently record attempted and realized policy violations, including configured approval-order constraints.

Use this task type as the starting point for the call.

• Anthropic's coding sandbox methodology (opens in a new tab)
• Execution-path containment analysis (opens in a new tab)

One coherent project with linked milestones, a reference solution, incorrect-result controls and repeated clean execution.

Proposed specification: The project shall contain verified dependencies between milestones and final integration checks, with explicit persistence, restoration and completion rules.

Use this task type as the starting point for the call.

• SWE-EVO (opens in a new tab)
• METR's duration measurement (opens in a new tab)

A paid batch covering build repair, artifact production and service configuration, with reference runs, shortcut controls and measured reset/resource costs.

Proposed specification: Accepted tasks shall verify the substantive artifact or service outcome outside agent control and reproduce the declared initial state without cross-run contamination.

Use this task type as the starting point for the call.

• Terminal-Bench 2.0 and Harbor methodology (opens in a new tab)

One small service stack with configuration, dependency or resource faults; include clean and distracting-symptom cases where relevant.

Proposed specification: Diagnosis and remediation shall be scored separately; accepted recovery shall preserve declared invariants and remain healthy under the specified workload.

Use this task type as the starting point for the call.

• SREGym (opens in a new tab)
• AIOpsLab (opens in a new tab)