When will SWE-bench be solved? | Manifold

When will SWE-bench be solved?

Basic

4

Ṁ20

2030

8%

2024

30%

2025

18%

2026

8%

2027

8%

2028

8%

2029

19%

later

Resolves to the first year in which https://www.swebench.com posts a 90% resolved score in the “verified” category. The current score is 38.80% by Amazon Q Developer Agent.

This question is managed and resolved by Manifold.

#Technical AI Timelines

Get

1,000

and

3.00

Sort by:

bought Ṁ5 later YES

Relatively significant chance of never (which resolves at later). Frontier LLMs are just going to get more and more contaminated with this benchmark's test data - at some point authors might just stop tracking

Related questions

80% on SWE-Bench Verified by Jan 1 2025

AI resolves at least X% on SWE-bench assistance, by 2025?

Will an AI SWE model score higher than 50% on SWE-bench in 2024?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2025?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2027?

What will be the best score on the SWE-Bench (unassisted) benchmark before 2025?

AI resolves at least X% on SWE-bench WITH assistance, by 2028?

AI resolves at least X% on SWE-bench without any assistance, by 2028?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2026?

Related questions

80% on SWE-Bench Verified by Jan 1 2025

What will be the best score on the SWE-Bench (unassisted) benchmark before 2025?

AI resolves at least X% on SWE-bench assistance, by 2025?

AI resolves at least X% on SWE-bench WITH assistance, by 2028?

Will an AI SWE model score higher than 50% on SWE-bench in 2024?

AI resolves at least X% on SWE-bench without any assistance, by 2028?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2025?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2026?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2027?

© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules