Will Transformer based architectures still be SOTA for language modelling by 2026? | Manifold

Will Transformer based architectures still be SOTA for language modelling by 2026?

Plus

55

Ṁ3876

2026

70%

chance

1D

1W

1M

ALL

The intent is to capture whether there is a major paradigm shift, on the same order of magnitude of RNNs to Transformers. The market resolves based on the architecture of the SOTA language model. If the architecture is still recognizable as a transformer with modifications, this market resolves as Yes. If the architecture is at least as different from Transformers as Transformers are from RNNs then it resolves No. If the evaluation numbers are ambiguous which architecture is the SOTA due to incomparable evaluations or non Pareto improvement over all evaluations, but one is obviously better overall, then I will use my judgement. If I deem it too close to call, I will resolve this market as Yes, because it indicates that no architecture has clearly surpassed Transformers.

This question is managed and resolved by Manifold.

#Technical AI Timelines

Get

1,000

and

3.00

Sort by:

I bet yes because I measure SOTA by frontier models, and I would be surprised if the labs were to invest the resources required for a large scale training run on an architecture which was new.

I like this question, added some liquidity

Is Mamba a transformer?

Would you count a model like Stanford's Monarch Mixer as a Transformer?

Is an ARDM (auto-regressive diffusion model) a transformer?

predictedYES

@ampdot Probably not; for a more definite answer you would have to elaborate more on what you have in mind.

Yes.

Plus memory and retrieval.

And hierarchical/nested/capsules (already used in video and high-end competition for long prompts)

And mixture of experts, and flash-attention and other sparsity techniques.

Will Transformer based architectures still be SOTA for language modelling by 2026?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition

Related questions

By the end of 2026, will we have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006?

When will a non-Transformer model become the top open source LLM?

Will transformers still be the dominant DL architecture in 2026?

Will we see most new language models shifting to addition-only architectures like BitNet/BitNet 1.58b in 2024?

Will a Kolmogorov-Arnold Network based model achieve SOTA on some significant machine learning tasks in 2024?

Will 'jailbreaks' in large language models be solved in principle by the end of 2024?

By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?

Will the most capable, public multimodal model at the end of 2027 in my judgement use a transformer-like architecture?

Will the transformer architecture be replaced in SOTA LLMs by 2028?

By 2025 will there be a competitive large language model with >50% of the total training data generated from a large language model?

Related questions

By the end of 2026, will we have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006?

Will 'jailbreaks' in large language models be solved in principle by the end of 2024?

When will a non-Transformer model become the top open source LLM?

By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?

Will transformers still be the dominant DL architecture in 2026?

Will the most capable, public multimodal model at the end of 2027 in my judgement use a transformer-like architecture?

Will we see most new language models shifting to addition-only architectures like BitNet/BitNet 1.58b in 2024?

Will the transformer architecture be replaced in SOTA LLMs by 2028?

Will a Kolmogorov-Arnold Network based model achieve SOTA on some significant machine learning tasks in 2024?

By 2025 will there be a competitive large language model with >50% of the total training data generated from a large language model?

© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules