Varun, good article! I’m wondering if we can translate your blog into Chinese and post it on the Chinese community. We will highlight your name and keep the original link on the top of the translated version. Thank you!
Sep 23, 2023·edited Sep 23, 2023Liked by Varun Shenoy
I believe in open source and use Linux myself, but if it’s the general public who has to decide on what to interact with, then we’ll be in a similar situation as we are in right now when it comes to smartphones. This will come down consumers choosing to stick to Microsoft, as it currently stands. Who knows what Apple will come up with, but they are not famous for throwing dice when it comes to (user) data processing.
On the consumer side, I totally agree. ChatGPT was the first entrant to the market and has significant market share over Claude, etc.
I'm not so sure this will hold true on the business side, where people are building on top of the LLMs. As long as their end customer experience is unaffected, an enterprise's decision around the specific model to use will be more dependent on speed, accuracy, throughput, and sufficient capability.
You’re right, the business market may be up for grabs for open-source, depending how easy the integration will be, along with the costs for maintenance.
The majority of phones are a closed-source duopoly and are increasingly essential to everyday life, yet open source phones remain hobby projects despite being "better" because you can control them yourself. There is too much power at stake with AI for the entrenched parties to let victory slip through their fingers.
I also think there's too much to lose for non-OpenAI/Google players if there is no comparable open-source alternative, even if they are more expensive or only good at a narrowly defined task.
Thank you for an insightful article! Could you please explain how does knowing a probability distribution of tokens (logits) help to guarantee the generation of JSON?
At each decoding step, you can mask out valid tokens according to some kind of grammar that defines JSON, for example, and then pick the most probable token from the remaining logits.
Similar logic is in Outlines [1] and Llama.cpp [2].
Agree with the majority of the points outlined in this piece. Many LLM-powered applications, including summarization, do not necessarily require reasoning capabilities, and numerous smaller open-source LLMs are sufficiently adept.
The main hurdle at this point is access - currently, it is still easier to fire a request to OpenAI's API rather than a Llama 2 endpoint. Also, enterprises often find it riskier to rely on open-source models due to the inherent uncertainties associated with them.
I agree on the point re: access, but I do think it's going to get substantially easier. A lot of open source servers [1] are adopting OpenAI's query/response style so you can just drop in, say Llama 2, for GPT-4. I also think open source LLMs are generally less risky and more transparent than closed source models. Lots of tools to interpret model logits, etc [2].
> but I do think it's going to get substantially easier
Agree. AWS Bedrock is one of those catalysts, as many companies already have the contracts and agreements in place with AWS--unlike a small Llama-hosting startup.
Varun, good article! I’m wondering if we can translate your blog into Chinese and post it on the Chinese community. We will highlight your name and keep the original link on the top of the translated version. Thank you!
Absolutely, thank you for asking!
I believe in open source and use Linux myself, but if it’s the general public who has to decide on what to interact with, then we’ll be in a similar situation as we are in right now when it comes to smartphones. This will come down consumers choosing to stick to Microsoft, as it currently stands. Who knows what Apple will come up with, but they are not famous for throwing dice when it comes to (user) data processing.
I recently addressed this topic in one of my own blog articles, if you’re interested: https://open.substack.com/pub/marknuyens/p/challenging-openai?r=2oo9qe&utm_medium=ios&utm_campaign=post
On the consumer side, I totally agree. ChatGPT was the first entrant to the market and has significant market share over Claude, etc.
I'm not so sure this will hold true on the business side, where people are building on top of the LLMs. As long as their end customer experience is unaffected, an enterprise's decision around the specific model to use will be more dependent on speed, accuracy, throughput, and sufficient capability.
You’re right, the business market may be up for grabs for open-source, depending how easy the integration will be, along with the costs for maintenance.
The majority of phones are a closed-source duopoly and are increasingly essential to everyday life, yet open source phones remain hobby projects despite being "better" because you can control them yourself. There is too much power at stake with AI for the entrenched parties to let victory slip through their fingers.
I agree.
I also think there's too much to lose for non-OpenAI/Google players if there is no comparable open-source alternative, even if they are more expensive or only good at a narrowly defined task.
Thank you for an insightful article! Could you please explain how does knowing a probability distribution of tokens (logits) help to guarantee the generation of JSON?
At each decoding step, you can mask out valid tokens according to some kind of grammar that defines JSON, for example, and then pick the most probable token from the remaining logits.
Similar logic is in Outlines [1] and Llama.cpp [2].
[1] https://github.com/outlines-dev/outlines/
[2] https://github.com/ggerganov/llama.cpp/pull/1773
Many thanks for your prompt reply, Varun! I will investigate the resources you shared.
Agree with the majority of the points outlined in this piece. Many LLM-powered applications, including summarization, do not necessarily require reasoning capabilities, and numerous smaller open-source LLMs are sufficiently adept.
The main hurdle at this point is access - currently, it is still easier to fire a request to OpenAI's API rather than a Llama 2 endpoint. Also, enterprises often find it riskier to rely on open-source models due to the inherent uncertainties associated with them.
Thanks for sharing, Varun!
I agree on the point re: access, but I do think it's going to get substantially easier. A lot of open source servers [1] are adopting OpenAI's query/response style so you can just drop in, say Llama 2, for GPT-4. I also think open source LLMs are generally less risky and more transparent than closed source models. Lots of tools to interpret model logits, etc [2].
[1] https://vllm.readthedocs.io/en/latest/getting_started/quickstart.html
[2] https://github.com/outlines-dev/outlines/
> but I do think it's going to get substantially easier
Agree. AWS Bedrock is one of those catalysts, as many companies already have the contracts and agreements in place with AWS--unlike a small Llama-hosting startup.
This was a great read Varun!
Thanks Kenny!