The open-source model Deepseek is currently the clear leader in cost-efficiency: (Screenshot from https://artificialanalysis.ai/)

https://github.com/cfahlgren1/natural-sql

https://twitter.com/rohanpaul_ai/status/1736827830971867312

GPT-3.5 class model with longer context size is offered at 1/3 of gpt-3.5-turbo price

@WieDan How is Gemini "open-source AI"?

@osmarks it's obviously not, but Gemini Ultra being barely better than GPT-4 arguably shows that having enormous resources and data doesn't really give you that much of an edge.

@BairAiushin My reading of that is just that Google is bizarrely incompetent some of the time.

@BairAiushin yeah i think we're deep in the s curve

@osmarks Google had an extra 9 months over GPT-4 + some open source knowledge, and still failed to make the gains we expected, even with delaying Ultra. This definitely is good news for YES betters here. Plus, open source is getting closer to gpt-4. I'm updating a bit towards open source seeing that it is progressing faster.

@ShadowyZephyr I haven't seen any open source models get remotely close to GPT-4, the best ones are just barely starting to get close to 3.5 Turbo.

@SemioticRivalry this is true but if the other models are going to slow down they have 2 years to catch up. In the scenario where everyone is blasting full speed ahead them catching up would not be nearly enough.

@SemioticRivalry Mixtral is reportedly better than GPT-3.5 on benchmarks.

The executive order threatens large proprietary models, but leaves open source models alone? Is that what I heard correctly just now? Theres a way that could give open source models an advantage over proprietary ones; government regulation.

@VAPOR Not really correct, no. Threshold is based on compute, so in theory a hypothetical Llama-3 would have the reporting requirements even if then released as open source.

(I don't think those reporting requirements are a threat.)

@DaveK It could be trained abroad though?

@mariopasquato US regulators tend not to look kindly on, “One Weird Trick to avoid US regulation” gambits. Most entities training such models want to do business in America

Related: https://manifold.markets/ChrisCanal/will-an-opensource-llm-on-huggingfa

Will an opensource LLM on huggingface beat an average human at the most common LLM benchmarks by July 1, 2024?

79% chance. Open source models are measured against ARC, HellaSwag, MMLU, and TruthfulQA on https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard . I've added the following plot to this huggingface space so we can see the progress of open source models over time: [image]I'm wondering when the average human baseline will be passed. A linear trend indicates July 2024 with a 0.89 pearson coef: [image]But this trend might not be linear. This questions will resolve Yes if the average human baseline on Open LLM Leaderboard on huggingface is surpassed before July 1, 2024. [link preview]

More cost efficient to whom?

For certain applications today, the terms of use of closed-source alternatives are a non-starter so OSLLM's are already more cost efficient than, "not available." Drinking one's own piss to stay alive comes to mind as an analogy.

@PatrickDelaney I am very curious as to the kind of life you have lived such that "Drinking one's own piss to stay alive" was the first analogy that came to mind here.

Betting NO because if the market resolves YES then we're more likely to go extinct.

@MartinRandall Even in this non-conditional (non policy based) market, one should also signal their true beliefs to politicians! You're making them less likely to act based on not even profitable, but wishful mindset!

The close date of the market is 2025-01-02T07:59:00Z, but in the description there is:

> Resolves YES if, at the end of 2025

Is it expected that the close date is about 1 year earlier than the planned resolution? (Probably this is expected, asking to be sure that there is no error is market close date or in the description)

@bessarabov Oops, fixed.