Will an AI achieve >85% performance on the FrontierMath benchmark before 2027?
➕
Plus
59
Ṁ21k
2027
62%
chance

From a recent arXiv preprint,

We introduce FrontierMath, a benchmark of hundreds of original, exceptionally challenging mathematics problems crafted and vetted by expert mathematicians. The questions cover most major branches of modern mathematics -- from computationally intensive problems in number theory and real analysis to abstract questions in algebraic geometry and category theory. Solving a typical problem requires multiple hours of effort from a researcher in the relevant branch of mathematics, and for the upper end questions, multiple days. FrontierMath uses new, unpublished problems and automated verification to reliably evaluate models while minimizing risk of data contamination. Current state-of-the-art AI models solve under 2% of problems, revealing a vast gap between AI capabilities and the prowess of the mathematical community. As AI systems advance toward expert-level mathematical abilities, FrontierMath offers a rigorous testbed that quantifies their progress.

This question resolves to YES if the state-of-the-art average accuracy score on the FrontierMath benchmark, as reported prior to midnight, January 1st 2027 Pacific Time, is above 85.0% for any AI model. Credible reports include but are not limited to blog posts, arXiv preprints, and papers. Otherwise, this question resolves to NO.

I will use my discretion in determining whether a result should be considered valid. Obvious cheating, such as including the test set in the training data, does not count.

This market was duplicated and modified from this excellent market by Matthew Barnett: /MatthewBarnett/will-an-ai-achieve-85-performance-o

See also:

/Bayesian/will-an-ai-achieve-30-performance-o

/Bayesian/what-will-true-of-the-sota-ai-on-th-ROldIhZZgt

  • Update 2024-21-12 (PST): Only the original FrontierMath benchmark will be considered for resolution, even if new tiers or problems are added to FrontierMath in the future. (AI summary of creator comment)

Get
Ṁ1,000
and
S3.00
Sort by:

[Asking this on all the FrontierMath markets.]

If FrontierMath changes (e.g. if Tier 4 is added to what's considered to be the official FrontierMath benchmark), how does that affect the resolution of this question?

It seems to me like the fair way to do it is to go based on the original FrontierMath benchmark (modulo small tweaks/corrections), but I'm not totally sure that in the future we will have benchmark scores that are separated out by original vs. new problems.

@EricNeyman Only the original FrontierMath benchmark counts for the purpose of this question

According to Epoch.ai the human+peer-review error rate in the benchmark is around 5%, so of the 95% of problems with correct labels the ai would need to get around 90% accuracy in them this is basically expert level

opened a Ṁ25,000 YES at 30% order

YES limit order of 25,000M at 30%

© Manifold Markets, Inc.Terms + Mana-only TermsPrivacyRules