Will an autonomous agent resolve 90% of tasks on SWE-bench by 2025?
Basic
10
Ṁ225Dec 31
19%
chance
1D
1W
1M
ALL
Resolves "Yes" if, at time of closure, there is an entry on the SWE-bench leaderboard (https://www.swebench.com/) with score greater or equal to 90%.
Linked Questions:
Get
1,000
and1.00
Sort by:
@DavidFWatson That's an excellent question. Let's explore possibilities:
This could be included in the question, i.e. what matters is only the number on the benchmark, regardless of whether it was gamed
I could wait a certain amount of time to check if no controversy emerges. Feels like one month would be safe. The question then resolves yes if one month after the deadline, I judge that there is no consensus that the number was gamed. This makes the question more informative.
Related questions
Related questions
Will an autonomous agent resolve 90% of tasks on SWE-bench by 2026?
70% chance
Will an autonomous agent resolve 90% of tasks on SWE-bench by 2027?
72% chance
AI resolves at least X% on SWE-bench WITH assistance, by 2028?
AI resolves at least X% on SWE-bench assistance, by 2025?
Will an AI agent system be able to score at least 40% on level 3 tasks in the GAIA benchmark before 2025.
49% chance
Will >50% of the tasks in the WebArena benchmark be solved by EOY 2024?
62% chance
Will a smart agent pass our Turing test by the end of 2025?
59% chance
AI resolves at least X% on SWE-bench without any assistance, by 2028?
Will self-improving AI agents crush SOTA in a complex environment (e.g. AAA game, tool use, science) in next 12 months?
27% chance
SoAI 23 3/10: Will Self-improving Al agents crush SOTA in a complex environment (e.g. AAA game, tool use, science)?
42% chance