MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?
➕
Plus
12
Ṁ1165
2027
12%
chance
Get
Ṁ1,000
and
S3.00
Sort by:

I think it will resolve as NO because:

  1. People will lose interest in MMLU after around 95%, so it's possible that GPT-x or whatever SOTA there will be will not even be evaluated on it.

  2. There are definitely at least some ambiguous or simply incorrect questions in MMLU, I am not sure how much but I will not be surprised if it's close to 1%.

  3. Reaching 99% is usually much harder than 98%.

  4. Metaculus predicts 93% in mid-2025, and I personally predict 91.5%. Will still be a long way to go.

© Manifold Markets, Inc.Terms + Mana-only TermsPrivacyRules