By what factor will the cost for SotA SWE-agents drop from 2024 to 2025?
Standard
9
Ṁ8502025
1D
1W
1M
ALL
5%
<2x
7%
<10x
13%
<50x
20%
<250x
54%
>=250x
Algorithmic progress can be measured by reduction in cost to achieve equivalent performance. SWE-bench-lite is a popular benchmark for measuring scaffolded-LLM SWE capabilities.
By what factor will the cost of SWE-bench-lite SoTA drop between mid 2024-2025? Mid-2024 SotA is 43% costing $2,700 (per the devs), so this question will resolve Yes on the answer which most tightly bounds the reduction in cost to achieve 43% on July 1, 2025.
E.g. if in June 2025, 43% on SWE-lite costs $500 then that'd be a 5.4x reduction and the question would resolve (2) "<10x".
Get
1,000
and1.00
Related questions
Related questions
Will self-improving AI agents crush SOTA in a complex environment (e.g. AAA game, tool use, science) in next 12 months?
27% chance
Will we reach "weak AGI" by the end of 2025?
28% chance
SoAI 23 3/10: Will Self-improving Al agents crush SOTA in a complex environment (e.g. AAA game, tool use, science)?
42% chance
Will some U.S. software engineers be negatively affected financially due to AI by end of 2025?
71% chance
Will open-source AI win (through 2025)?
33% chance
Will AI be Recursively Self Improving by mid 2026?
25% chance
AI resolves at least X% on SWE-bench assistance, by 2025?
Will AI resolve P vs NP by 2050?
26% chance
Will the "OpenAI hint at or claim to have AGI before 2025 end" market go below 10% before 2024 ends?
41% chance
AI resolves at least X% on SWE-bench WITH assistance, by 2028?