LLaMA 3 has climbed all the way to the top of the leaderboard.

The GPT-4 is above.

Claude 3, including Opus, the large model, is in the rearview mirror.

OpenAI's massive advantage is gone.

We now have a top open source contender.

There are so many things here that are legitimately stunning, shocking, whatever you want to call it.

Number one is that LLaMA, Meta's model, is now just about the same level as GPT-4.

That Facebook/Meta is the closest competitor.

Number two is that it's a 70 billion model, compared to the 1.7 trillion or whatever that number is for GPT-4.

And finally, it's open source?

How is this happening?

If it stopped there, then that would be by itself strange enough.

But wait, there's more, because there's another model that's in the works.

The LLaMA 3 400 plus billion parameters.

But it's not done training yet, so the results will improve.

Dr. Jim Fan from NVIDIA predicts GPT-5 will be announced before LLaMA 3, the 400 billion model, before that gets released.

Most people at this point probably noticed that OpenAI tends to respond to other announcements with their own announcements, trying to always be kind of at the forefront of the AI race.

This is Chris Paxton at Hello Robot, formerly AI at Meta.

He's ex-Meta AI, ex-NVIDIA AI.

It's genuinely hard to believe a 70 billion model is up there with the 1.8 trillion model, GPT-4.

I guess training data really is everything, which is something that Mark Zuckerberg talked about in that interview yesterday.

They have a lot of training data that is very curated, very high quality.

A model that's tiny in comparison to GPT-4 is now performing at a level very similar to GPT-4.

And here's why that's important.

Why that's kind of very important.

Because we have now GPT-4 at home, or at least very close.

We're basically at a point where you can create these fairly inexpensive rigs, these machine learning rigs, capable of running very sophisticated AI systems that you can run from home, powering your very own agents for your business, for work, for personal use.

Something like DevOn supposedly runs on GPT-4.

Imagine running something like DevOn, something that's able to code, create code, fix code, or maybe eventually something a little bit even more generalized that can build websites, run e-commerce stores, etc., that can run on a machine like this that costs $3,000 to build, maybe more, maybe less, depending on what kind of chips, what kind of GPUs you choose to go with, whether it's top-of-the-line or something that's maybe slightly outdated.

As I was recording, people started posting their benchmarks, their screenshots of running LLaMA 3, 70 billion, locally via OLLaMA.

Here they're running it at around 14 tokens per second on an M.2 Ultra 76 GPU, which can be found for about $7,000 to $8,000.

It's a little bit slower than you would expect from GPT-4, from ChatGPT, because Groq is, remember, 300 tokens per second, GPT-4 is probably, I don't know, 20 tokens per second, GPT-3.5 Turbo is probably 100 tokens per second.

Just a rough estimate, I might be off there, but here they're running LLaMA 3, $70 billion on an M.3 Max on this thing, really, okay, for $3,200, but they're only getting about 8 tokens per second, so very slow, but this is a laptop they're running on, wow.

Here they're comparing the speed, looks like the big chunky $8,000 computer runs it about 45% faster, but as you can see, we're running it on hardware that costs between $3,000 to $8,000, $9,000, and it's, I mean, it's usable, it's a little bit on the slow side.

Of course, if you're using the 8 billion model, the smaller model, it's much faster.

Matt Shumer from HyperWrite AI says, we now have an open source model that is beating Cloud 3 Opus, being served at nearly 300 tokens per second on the Groq computer chip.

The applications built off of this tech will be nothing short of revolutionary.

Take a look at how quickly this text generates here.

We've seen examples of Groq being used in real-time conversation agents, where an AI, for example, calls you on the phone and is able to carry a real-time conversation with you.

For example, a sales call, booking an appointment, customer service, etc.

We just crossed into territory where a GPT-4 level model can basically run a call center talking in real-time to customers, calling people to book appointments 24 hours a day in real-time for pennies.

Groq, of course, is that new chip that's kind of making a lot of waves.

It's different because it's not a GPU, it's not a graphical processing unit.

It's an LPU, a language processing unit.

This guy apparently invested in it, Shamoff Palapatiya.

How's he into everything?

Whatever, I'm not jealous.

Everybody's talking about Groq, you're on Cloud 9, it seems.

I've played this before from the channel AI Jason.

He built a little sales agent, a telemarketing agent that calls you to set up appointments.

If you submit information, it calls you to sell you on it.

And then schedule your appointment.

Take a listen.

Hi Jason, I'm reaching out from Crazy Fitness regarding your interest in our gym membership.

I'm here to provide you with all the information you need and answer any questions you might have.

That's AI.

Thanks for calling.

Running on Groq.

What specific fitness goals are you looking to achieve?

I just want to lose some weight.

I haven't done any exercise at all for the past three or six months since COVID.

I totally get that.

Our personalized training and nutrition programs can help you reach your weight loss goals.

Let's discuss the membership options that are best for you.

Sounds good.

I think I'm pretty happy to just go ahead and get the membership started this Saturday, if possible.

Great to hear that.

To complete the process, I just need your credit card number to secure your membership.

You might be thinking that's probably edited.

There's probably long pauses in between when he speaks and when the AI phone agent answers.

The guy on the right is the founder of Groq, ex-Google, and he's the real deal.

We'll talk about him in just a second, but take a listen.

Just ask Groq.

All right, let's do that.

We're going to turn it on.

Let's do it.

I don't have the answers to this, Groq, but what makes you different?

Well first... To another...

Hey, I'm Groq.

You can talk to me like I'm a person.

All right, I will do.

What makes you different from the Large Language Model that I may have used in the past?

Sure, I'm a bit different from those big language models you've used before.

I'm a language user interface, or LUI, designed to have more natural...

Got it!

How similar are you to a human brain?

Well, I'm not a human brain, but I'm designed to understand and respond to like ones.

I can process and generate human language in a natural way...

Got it!

Tell me something most people don't know.

Here's something interesting.

Did you know that octopuses have three hearts?

And people are noticing Groq.

Again, this is Groq's CEO, Jonathan Ross.

Typically tweet out some sort of developer metric.

Where are you as of this morning and why are developers so important?

We are at 75,000 developers.

That is slightly over 30 days from launching our development console.

Our comparison took Vidya seven years to get to 1,000 developers.

And we're at 75,000 in about 30-ish days.

For those of you that are not aware, this is Shamov Palapitiya.

Whenever I mention him, there's always a number of people that really do not like him.

Allegedly, it seems that he created these SPACs, these sort of investment vehicles, pumped them really hard on the various media platforms.

People lost money in the process.

I haven't followed all that, so I do leave that here as an asterisk.

But he wasn't born wealthy, he was born in Sri Lanka, was an early senior executive at Facebook.

And then went on to form his own investment company.

And somewhere during that whole thing, he became a multi-billionaire.

He started from the bottom and now he's here talking to the CEO of Groq.

But the reason I kind of took that long and winding road to get here is that Shamov tends to be very accurate in his predictions in terms of AI, in terms of tech.

He will often say something that at that time doesn't seem obvious.

But whenever I look back on some of his takes, they are eerily accurate.

I'm sure that doesn't happen all the time.

I'm sure there's plenty of times where he's wrong.

When he has a tech prediction, I tend to listen.

That if you build something innovative and you launch it, it's going to be four years before anyone can even copy it, let alone pull ahead of it.

That just felt like a much better approach and it's atoms.

You can monetize that more easily. So right around that time...

Here he's talking about building hardware instead of software, monetizing atoms instead of bits.

At the time the TPU paper came out, my name was in it, people started asking about it.

TPU is Google's kind of proprietary chip, the Tensor Processing Unit.

NVIDIA is GPUs, Google is TPUs, and Groq is LPUs, Language Processing Unit.

It would do differently.

Well, I was investing in public markets as well at the time, little dalliance in the public markets.

And Sundar goes on in a press release and starts talking about TPU.

And I was so shocked.

I thought there is no conceivable world in which Google should be building their own hardware.

They must know something that the rest of us don't know.

We need to know that so that we can go and commercialize that for the rest of the world.

And I probably met you a few weeks afterwards and that was probably the fastest investment I'd ever made.

I remember the key moment is you did not have a company.

We had to incorporate the company after the check was written, which is always either a sign of complete stupidity or in 15 or 20 years you'll look like a genius.

But the odds of the latter are quite small.

And yes, he does have a very high opinion of himself.

Tell us about the design decisions you were making in Groq at the time, knowing what you knew then.

Because at the time is very different from what it is now.

Well, again, when we started fundraising, we actually weren't even sure that we were going to do something in hardware.

But it was something that I think you asked, Shamath, which is what would you do differently?

And my answer was the software.

Because the big problem we had was we could build these chips in Google, programming them.

Every single team at Google had a dedicated person who was hand optimizing the models.

This is absolutely crazy.

Right around then, we had started hiring some people from NVIDIA and they're like, no, no, you don't understand.

This is just how it works.

This is how we do it too.

We've got these things called kernels, CUDA kernels, and we hand optimize them.

We just make it look like we're not doing that.

All of you understand algorithms and big O complexity.

That's linear complexity.

For every application, you need an engineer.

NVIDIA now has 50,000 people in their ecosystem.

These are really low level kernel writing, assembly writing hackers who understand GPUs and ML and everything.

Not going to scale.

Why is it that LLMs prefer Groq?

What was the design decision or what happened in the design of LLMs?

Some part of it is skill, obviously, but some part of it was a little bit of luck.

What exactly happened that makes you so much faster than NVIDIA and why there's all of these developments?

What is the crux of it?

We didn't know that it was going to be language, but the inspiration, the last thing that I worked on was getting the AlphaGo software, the Go playing software at DeepMind working on TPU.

Having watched that, it was very clear that inference was going to be a scaled problem.

Everyone else had been looking at inference as you take one chip, you run a model on it, it runs whatever.

But what happened with AlphaGo was we ported the software over.

And even though we had 170 GPUs versus 48 TPUs, the 48 TPUs won 99 out of 100 games with the exact same software.

What that meant was compute was going to result in better performance.

The insight was let's build scaled inference.

We built in the interconnect, we built it for scale.

And that's what we do now when we're running one of these models, we have hundreds or thousands of chips contributing just like we did with AlphaGo, but it built for this to cobble together.

But here's where we link it all back to LLaMA 3.

Take a listen to what they think about where this AI arms race in chatbots, aka LLMs, where it's going.

And keep in mind, this was likely shot maybe a few days before LLaMA 3 got released, maybe a little bit more.

Take a listen.

If you're building, if you're making, if you're starting a company today, you clearly want to have the ability to swap from LLaMA to Mistral to Anthropic back as often as possible.

Whatever's latest.

And just as a, as somebody who sees these models run, do you have any comment on the quality of these models and where you think some of these companies are going or what you see some doing well versus others?

They're all starting to catch up with each other, starting to see some leapfrogging.

It started off with GPT-4 pulling ahead and it had a lead for about a year over everyone else.

Of course, Anthropic has caught up.

We're seeing some great stuff from Mistral.

But across the board, they're all starting to bunch up in quality.

One of the interesting things, Mistral in particular, have been able to get closer to quality with smaller, less expensive models to run, which I think gives them a huge advantage.

I think Coheer has an interesting take on a sort of rag-optimized model, so people are finding niches.

And there's going to be a couple that are going to be the best across the board at the highest end.

But what we're seeing is a lot of complaints about the cost to run these models.

They're just astronomical and they're not, you're not going to be able to scale up applications for users.

OpenAI has published or has disclosed, as has Meta, as has Tesla and a couple of others, just the total quantum of GPU capacity that they're buying.

And you can kind of work backwards to Figure out how big the inference market can be, because it's really only supported by them as you guys scale up.

Can you give people a sense of the scale of what folks are fighting for?

I think Facebook announced that by the end of this year, they're going to have the equivalent of 650,000 H100s.

By the end of this year, Groq will have deployed 100,000 of our LPUs, which do outperform the H100s on a throughput and on a latency basis.

We will probably get pretty close to the equivalent of Meta ourselves.

By the end of next year, we're going to deploy 1.5 million LPUs.

For comparison, last year, NVIDIA deployed a total of 500,000 H100s.

1.5 million means that Groq will probably have more inference, generative AI capacity than all of the hyperscalers and cloud service providers combined.

Probably about 50% of the inference compute in the world.

That's just great.

He's trying to calculate if he can become a cruller in their next cards, the much sought after NVIDIA GPUs that all these big tech AI focused labs are trying to get their hands on.

You've probably seen this beautiful picture.

That's one of them right there.

This is the NVIDIA H100 GPU shipments by customer.

This was the estimate for 2023.

As you can see, Meta and Microsoft are leading the race with 150,000 H100s.

Interestingly enough, Mark Zuckerberg actually was kind of fortunate to have all those H100s.

And it was not because they were preparing to train Large Language Models.

Apparently that was not on the horizon yet.

You can tell me when.

We were like, stock price is getting hammered.

People are like, what's happening with all this capex?

People aren't buying the Metaverse.

And presumably you're spending that capex to get these H100s.

Back then, how did you know to get the H100s?

How did you know we'll need the GPUs?

I think it was because we were working on reels.

We got into this situation where we always want to have enough capacity to build something that we can't quite see that we're on the horizon yet.

And we got into this position with reels where we needed more GPUs to train.

It was largely a content play.

They wanted to be able to recommend better content, but it wasn't a large business.

They wanted better content, but it wasn't a Large Language Model play.

ChatGPT, GPT-4, all that hasn't made the appearance yet.

The hype cycle didn't start yet.

And Andrej Karpathy, the ex-OpenAI, ex-Tesla person that's now, I believe he's teaching more.

He's teaching more about AI through YouTube and various other platforms, talking a little bit more about the architecture of the model.

But there's two things that jump out at me.

One is specifically they're talking about training data.

He's saying that the 15 trillion that they've trained this model on is a very, very large data set to train for a model as small as 8 billion parameters.

And this is not normally done, but is new and very welcome.

This is training 75x more than other models.

He's saying I think it's extremely welcome because we all get a very capable model that is very small, easy to work with.

And Meta mentions that even at this point, the model doesn't seem to be converging.

In other words, the LLMs we work with, we work with all the time, are significantly undertrained by a factor of maybe 100 to 1000 times or more, meaning that we can be releasing more long trained smaller models.

We're not even hitting a wall here.

There's more to go.

But the other thing that jumped out at me that I wasn't even really thinking about is this.

Meta talks about determining sort of the strength of the model by looking at flops.

And the reason that this number of flops is important is if you had to reduce the strength of a model to a single number, this is how you try to do it.

Because it combines the size of the model with the length of training into a single strength of how many total flops went into it.

Why is this important?

Because we actually have a limit to how many flops, sort of a threshold for what that number could be.

For example, LLaMA 370 billion, that number would be expressed as 9 times 10 to the 24th if the 400 billion model, so that's the next big one that potentially is coming out, hopefully is coming out and will be open source.

If it's safe to open source it, that would be 4 times 10 to the 25th.

He's saying this starts to really get up there.

The Biden executive order had the reporting requirements set at 1 times 10 to the 26th.

This would be 2x below that.

The point is we're kind of approaching the limits, or will be once the new, the big LLaMA 3 model is released, of how strong these models could be before I guess there's a reporting requirement set.

EU, if I'm reading this correctly, I believe this is the ec.europa.eu.

It says 10 to the 25th as the threshold for systemic risk.

EU limits it at 10 to the 25th power, LLaMA 370 billion, so the one that's out now is 10 to the 24th power.

The US reporting requirement is set at 10 to the 26th power.

And the next big LLaMA 3 model will be 10 to the 25th power.

It will certainly be interesting to see what actions, if any, the various governments will do once these models start kind of encroaching into that territory.

Which brings us back to OpenAI.

Everybody is expecting OpenAI to counter with something big, GPT-5.

And having that GPT-5 be the next generation, the next big milestone that everyone will try to shoot for.

But of course a lot of people are concerned that these fast, cheap, powerful models will destroy a lot of businesses, will disrupt them.

OpenAI itself is kind of notorious for crushing the various startups that are building using the OpenAI ecosystem.

However, that question is posed to Sam Altman.

How should people think about what to build in this fast moving world to take advantage of the process, to not get crushed by the next wave of innovation?

This was very recently, this was April 15th, that he's talking about this.

Take a listen.

You mentioned you see your time investing and you know, I see you engage with so many large enterprises around the world today.

For me as an investor, I see so many AI companies and I'm not investing in any application layer AI companies.

Just respectfully we've seen OpenAI come out with products and it's like, well that killed the whole industry.

You know, I think fundamentally there are two strategies to build on AI right now or startups doing with AI.

There's one strategy which is assume the model is not going to get better.

And then you kind of like build all these little things on top of it.

And then there's another strategy which is build assuming that OpenAI is going to stay on the same rate of trajectory and models are going to keep getting better at the same pace.

It would seem to me that 95% of the world should be betting on the latter category, but a lot of the startups have been built in a former category.

And then when we just do our fundamental job, which is make the model and its tooling better with every crank, then you get the OpenAI killed my startup meme.

If you're building something on GPT-4 that a reasonable observer would say if GPT-5 is as much better as GPT-4 over GPT-3 was, not because we don't like you, but just because we like have a mission, we're going to steamroll you.

There's a giant set of startups where you benefit from GPT-5 being way better.

And if you build those and AI progress keeps going the way that we think it's going to go, I think on the most part, you'll be really happy.

For the most part, you'll be really happy.

As an investor looking for an investment thesis that might actually last, what are those that will not be steamrolled that I can invest in, Sam, versus those that could be?

Ask the company whether 100x improvement in the model is something they're excited about.

Actually we can tell pretty well because we know the companies that come to us saying we want the next model, when is it coming out, when is it coming out, I want to be the first to try it.

It's going to be the best thing for my company.

And then there's a lot of companies that we don't hear from in that regard.

And I think that's a pretty good delineation is if there's a clear path to how better underlying intelligence accelerates that product in that company, most companies can tell that story really clearly.

Like Klana would be an example of that?

Klana is a good example.

Because for Klana, I mean, the numbers are astonishing.

And think how much better that gets if the next model is as good as we hope it's going to be.

I just this morning to an AI, like medical advisor, I guess they would call it.

And they're like here's the place that the model is underperforming.

It's still pretty useful for like these kinds of things.

But if the model could just get like this much better on these metrics, we'd have all these other businesses. So like, can you all do that faster?

And then we can have like this like thing that'll save all these lives and give people who have not had access to medical care, like some access.

And you know, how soon can we get that in here's how many people are dying every day you delay.

It was effective pitch, actually.

Klana is the first of the customer success stories that they featured recently on the OpenAI website, which is an AI for personal shopping, customer service and employee productivity.

The better that the AI model gets as we move to GPT-5, LLaMA 4, Cloud 4, etc.

Companies like this just keep improving.

They will be riding that improvement wave as AI gets better.

They get better.

With that said, make sure you're subscribed.

We're going to be talking about how to start running some of the stuff locally on your machine as well as how to set it up to run in the cloud.

I mean, now that it's getting less and less expensive, I mean, my best guess is in the future, it will be just like hosting.

No one actually self hosts anymore.

I mean, a few people do, I'm sure, but for the most part, you just kind of outsource it to the cloud because it's easier, more secure, it's pretty cheap.

But the breakthroughs and how inexpensive it is to run these models, how smaller models, how effective they can be, how they're faster, cheaper, better, able to run locally.

This is going to unlock incredible new capabilities.

We've covered the more agents is all you need paper recently.

The more agents you run, the more sort of sample you take, or if you have multiple agents working together, the results get better, the outputs become better, the agents get smarter.

Now that we can run something like LLaMA 3 with Groq at 800 tokens per second, that one it looks like is the 8 billion parameter.

This is going to open up a lot of capabilities.

Make sure you're subscribed.

A lot more good stuff coming soon.

My name is Wes Roth and thank you for watching.