Will Transformer based architectures still be SOTA for language modelling by 2026? | Manifold

Will Transformer based architectures still be SOTA for language modelling by 2026?

Plus

55

Ṁ3798

2026

68%

chance

1D

1W

1M

ALL

The intent is to capture whether there is a major paradigm shift, on the same order of magnitude of RNNs to Transformers. The market resolves based on the architecture of the SOTA language model. If the architecture is still recognizable as a transformer with modifications, this market resolves as Yes. If the architecture is at least as different from Transformers as Transformers are from RNNs then it resolves No. If the evaluation numbers are ambiguous which architecture is the SOTA due to incomparable evaluations or non Pareto improvement over all evaluations, but one is obviously better overall, then I will use my judgement. If I deem it too close to call, I will resolve this market as Yes, because it indicates that no architecture has clearly surpassed Transformers.

This question is managed and resolved by Manifold.

#Technical AI Timelines

Get

1,000

and

3.00

Sort by:

I bet yes because I measure SOTA by frontier models, and I would be surprised if the labs were to invest the resources required for a large scale training run on an architecture which was new.

I like this question, added some liquidity

Is Mamba a transformer?

Would you count a model like Stanford's Monarch Mixer as a Transformer?

Is an ARDM (auto-regressive diffusion model) a transformer?

predicts YES

@ampdot Probably not; for a more definite answer you would have to elaborate more on what you have in mind.

Yes.

Plus memory and retrieval.

And hierarchical/nested/capsules (already used in video and high-end competition for long prompts)

And mixture of experts, and flash-attention and other sparsity techniques.

Will Transformer based architectures still be SOTA for language modelling by 2026?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition

Related questions

By the end of 2026, will we have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006?

Will transformers still be the dominant DL architecture in 2026?

Will the transformer architecture be replaced in SOTA LLMs by 2028?

On January 1, 2027, a Transformer-like model will continue to hold the state-of-the-art position in most benchmark

Will a language model that runs locally on a consumer cellphone beat GPT4 by EOY 2026?

By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?

Will the most capable, public multimodal model at the end of 2027 in my judgement use a transformer-like architecture?

When will a non-Transformer model become the top open source LLM?

Will we see most new language models shifting to addition-only architectures like BitNet/BitNet 1.58b in 2024?

Will any open-source Transformers LLM model that function as a dense mixture of experts be released by end of 2024?

Related questions

By the end of 2026, will we have transparency into any useful internal pattern within a Large Language Model whose semantics would have been unfamiliar to AI and cognitive science in 2006?

By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?

Will transformers still be the dominant DL architecture in 2026?

Will the most capable, public multimodal model at the end of 2027 in my judgement use a transformer-like architecture?

Will the transformer architecture be replaced in SOTA LLMs by 2028?

When will a non-Transformer model become the top open source LLM?

On January 1, 2027, a Transformer-like model will continue to hold the state-of-the-art position in most benchmark

Will we see most new language models shifting to addition-only architectures like BitNet/BitNet 1.58b in 2024?

Will a language model that runs locally on a consumer cellphone beat GPT4 by EOY 2026?

Will any open-source Transformers LLM model that function as a dense mixture of experts be released by end of 2024?

© Manifold Markets, Inc.•Terms + Mana-only Terms•Privacy•Rules