AI resolves at least X% on SWE-bench assistance, by 2025? | Manifold

AI resolves at least X% on SWE-bench assistance, by 2025?

Plus

21

Ṁ3961

Dec 31

98.7%

X=5

98.2%

X=10

93%

X=20

64%

X=40

12%

X=80

The SWE-bench is a benchmark developed to evaluate if language models can resolve real-world GitHub issues. The leaderboard showcases various models and their performances in terms of the percentage of SWE-bench instances they resolved. Each instance in the SWE-bench represents a GitHub issue. The leaderboard is categorized into two main sections: Unassisted and Assisted.

Assisted: In this category, models have the advantage of the "oracle" retrieval setting where the correct files to edit are directly given to them.

This question is only about the Assisted category of this benchmark.

http://www.swebench.com/#
Current SOTA is 4.8

The prediction market will resolve based on the SWE-bench leaderboard standings as of 31 December 2024.

Multiple answers can be correct.

This question is managed and resolved by Manifold.

#Technical AI Timelines

#Programming Automation

Get

1,000

and

3.00

Sort by:

Do all the of the options bellow the threshold resolve to true?

bought Ṁ50 X=20 YES

reposted

bump

Related questions

AI resolves at least X% on SWE-bench WITH assistance, by 2028?

Will any model get above human level (92%) on the Simple Bench benchmark before September 1st, 2025.

-6% 1d36% chance

Will an AI SWE model score higher than 50% on SWE-bench in 2024?

What will be the best score on the SWE-Bench (unassisted) benchmark before 2025?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2027?

AI resolves at least X% on SWE-bench without any assistance, by 2028?

80% on SWE-Bench Verified by Jan 1 2025

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2025?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2026?

Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?

Related questions

AI resolves at least X% on SWE-bench WITH assistance, by 2028?

AI resolves at least X% on SWE-bench without any assistance, by 2028?

Will any model get above human level (92%) on the Simple Bench benchmark before September 1st, 2025.

80% on SWE-Bench Verified by Jan 1 2025

Will an AI SWE model score higher than 50% on SWE-bench in 2024?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2025?

What will be the best score on the SWE-Bench (unassisted) benchmark before 2025?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2026?

Will an autonomous agent resolve 90% of tasks on SWE-bench by 2027?

Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?

© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules