When will a single model first achieve 10@k solve rate >= 90% on the CodeContests dataset?
Basic
7
Ṁ3062032
1D
1W
1M
ALL
4%
5%
Before 2023
11%
2023-2023
17%
2024-2025
25%
2025-2026
18%
2026-2027
17%
2027-2032
1.6%
2032-never
10@k - unlimited samples may be generated, but at most 10 code samples may be submitted per problem.
I will post my own responses and pick whichever one of those turns out to be correct.
CodeContests dataset: https://github.com/deepmind/code_contests
Close date updated to 2050-01-01 11:59 pm
Apr 8, 4:16pm: Options are written as [a, b), so 2024-2025 means occurring after the start of 2024 and before the start of 2025.
Close date updated to 2032-01-01 11:59 pm
Get
1,000
and1.00
Sort by:
@ValeryCherepanov Yeah the before 2023 option seems like free money to me, I think the market is just old and people don't have it on their radar.
Related questions
Related questions
Will any model get above human level (92%) on the Simple Bench benchmark before September 1st, 2025.
68% chance
When will an AI model be better than me at competitive programming?
Will the winner of the ARC Kaggle competition solve more than 50% of the tasks
74% chance
By when will SAT solvers be 10x faster (than 2023 on same hardware)?
Will an AI outcompete the best humans on any one programming contest of IOI, ICPC, or CodeForces before 2025?
36% chance
When will AI be able to solve 50% of official Jane Street puzzles?
Will an AI achieve a perfect score on the Miklós Schweitzer Competition before 2035?
57% chance
Will any AI be able to formalize >=90% of IMO problems by the start of 2025?
20% chance
Benchmark Gap #3: Once a model achieves superhuman performance on a competitive programming benchmark, will it be less than 2 years before there are "entry level" AI programmers in industry use?
68% chance
When will no-calculator SOTA on the MATH dataset pass 90%?