When will SOTA on MMLU STEM be >= 90%? | Manifold

When will SOTA on MMLU STEM be >= 90%?

Basic

8

Ṁ584

resolved Oct 4

1D

1W

1M

ALL

100%85%

2024-2025

0.1%

Before 2023

1.4%

2023-2024

9%

2025-2026

3%

2026-2027

0.2%

2027-2032

0.8%

2032-never

MMLU: https://arxiv.org/abs/2009.03300v3.

#Technical AI Timelines

Get

1,000

and

1.00

Sort by:

https://openai.com/index/openai-o1-mini-advancing-cost-efficient-reasoning/

@4atj Yeah this resolves it

GPT-4 is 82.7% https://arxiv.org/pdf/2303.12712.pdf#page=36

Metaculus says 91 in mid-2025, STEM is one of the hardest. I think it's unlikely to happen before 2025.

What do these answers mean? If it happens in 2025, will 2024-2025 and 2025-2026 both be chosen?

@NcyRocks Sorry: python style ranges, so X - Y means after the start of X and before the start of Y. If I can I'll edit the options to make that clearer

Related questions

MMLU 99% #5: Will SOTA for MMLU (average) pass 99% by the start of 2028?

MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?

By 2026, will it be standard practice to sandbox SOTA LLMs?

Will there be a period of 12 contiguous months during which no new compute-SOTA LM is released, by Jan 1, 2033?

Will the transformer architecture be replaced in SOTA LLMs by 2028?

MMLU 99% #3: Will SOTA for MMLU (average) pass 99% by the start of 2026?

MMLU 99% #2: Will SOTA for MMLU (average) pass 99% by the start of 2025?

When will no-calculator SOTA on the MATH dataset pass 90%?

BIG-bench accuracy 75% #2: Will SOTA for a single model on BIG-bench pass 75% by the start of 2025?

When will SOTA for Atari 100k pass human median and mean score on all 57 games?

Related questions

MMLU 99% #5: Will SOTA for MMLU (average) pass 99% by the start of 2028?

MMLU 99% #3: Will SOTA for MMLU (average) pass 99% by the start of 2026?

MMLU 99% #4: Will SOTA for MMLU (average) pass 99% by the start of 2027?

MMLU 99% #2: Will SOTA for MMLU (average) pass 99% by the start of 2025?

By 2026, will it be standard practice to sandbox SOTA LLMs?

When will no-calculator SOTA on the MATH dataset pass 90%?

Will there be a period of 12 contiguous months during which no new compute-SOTA LM is released, by Jan 1, 2033?

BIG-bench accuracy 75% #2: Will SOTA for a single model on BIG-bench pass 75% by the start of 2025?

Will the transformer architecture be replaced in SOTA LLMs by 2028?

When will SOTA for Atari 100k pass human median and mean score on all 57 games?

Terms & Conditions•Privacy Policy•Sweepstakes Rules