Proposition

What about hybrid models, like Jamba? They might be the best of both worlds.

Yes given that an architecture qualifies that levers a combination of transformer models and supporting infra components that wouldn’t be considered breakthrough technologies on their own (e.g. RAG).

So do mixtures of experts count? The linked page this not contain any actual details.

@EchoNolan I talked to Sasha, and his response is basically that as long as the E in the MoE is Transformer, its a transformer.

See: https://manifold.markets/HanchiSun/is-attention-all-you-need

@Tasty_Y and https://manifold.markets/ICRainbow/on-january-1-2027-a-transformerlike?r=SUNSYWluYm93

@_T_Grass

i have strong principled reasons this should stay at 50% for the next 24 hours

@jacksonpolack such as?

subsidy phasein

@jacksonpolack Hm, I will add in subsidy at a later point wherever the market stabilizes to maintain that