By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?
Basic
6
Ṁ135
2025
68%
chance

My job is related to NLP, and I expect I will be able to judge the position on transformers in the field. No specific criteria, sorry, the question is just about my opinion.

Get
Ṁ1,000
and
S3.00
Sort by:

Mamba has lots of ongoing work, but didn't look like there's conclusive evidence it beats transformers yet

My guess is that we will build on transformers, perhaps significantly, but still consider the result to be under the category of 'transformer'. It will have the basic components like attention heads and mlp layers and a residual stream. It will just also have new stuff.

@NathanHelmBurger I wonder if we can pin down specific predictions about what we believe will or won't change, then

predicts YES

@wadimiusz yeah, unclear. RWKV and Hyena seem like promising advances. As the large labs start getting more cagey about their exact techniques we might not even know which techniques they are using.

https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning

By the start of 2026, will I still think that transformers are the main architecture for tasks related to natural language processing?, 8k, beautiful, illustration, trending on art station, picture of the day, epic composition

© Manifold Markets, Inc.Terms + Mana-only TermsPrivacyRules