I did not see this coming.

Grok 1.5 Vision preview.

Xai drops this new announcement.

4.7 million views in a matter of hours, and it's shockingly good.

In a head-to-head comparison between Grok 1.5 Vision, GPT-4 Vision, Claude 3 Opus, and Gemini Pro 1.5, the latest release, Grok holds its own.

It holds its own against the Titans.

I'm going to say it. I'm shocked.

This seems big enough forward to cover this.

April 12th, 2024, Grok 1.5 Vision preview connecting the digital and physical worlds with our first multimodal model.

In addition to its strong text capabilities, groi now processes a wide variety of visual information like documents, diagrams, charts, screenshots, and photographs.

It will be available soon to our early testers and existing Grok users.

Grok 1.5 is competitive with existing Frontier multimodal models in a number of domains.

We are particularly excited about Grok's capabilities in understanding our physical world.

Grok was built as an AI that was truth-seeking.

I forget the exact terminology Elon used, but it was supposed to understand the truth of the universe.

It's interesting that they're pointing this out here.

They're excited about Grok's capabilities and understanding the physical world.

Grok outperforms its peers in our new real-world Q&A Benchmark that measures real-world spatial understanding.

For all data sets below, we evaluate Grok in a zero-shot setting without Chain of Thought prompting.

This is interesting because everybody kind of has their own little tricks, their own little shenanigans for showcasing the best of their model.

For example, yesterday when OpenAI showcased their new and improved GPT-4 model, they also open-sourced a sort of set of benchmarks to evaluate the different models.

They're calling it a lightweight library for evaluating language models, and they are emphasizing zero-shot Chain of Thought setting, meaning they don't give it examples of How to solve the problems, and they're asking it to think through the problem step by step before solving it.

It's interesting here that Grok is saying for all data sets below, we evaluate Grok in a zero-shot setting.

Again, same as OpenAI, but they're saying without Chain of Thought prompting.

And here are the benchmarks.

First of all, GPT-4 with vision.

This was the reigning king for a long, long time.

CLA 3 Opus, when it came out, kind of rocked the rankings quite a bit.

It became the number one model on the LLM Arena, and we've tested it on this channel quite a bit.

It's good.

It's eerily good.

There's definitely stuff happening there that's very interesting.

It was definitely a step forward.

Gemini Pro 1.5 is also extremely good.

It's a big leap over Gemini 1.0.

We believe that this is where they introduced mixture of experts.

They introduced the 1 million token context window.

And then the claw 3 Sonet, which is a smaller CLA three model.

It's interesting that they put this in here, but the point is these three: the GPT-4 Vision, CLA 3 Opus, and Gemini Pro 1.5.

I mean, these are the reigning Champions, these are the really good models, all with their own separate strengths.

The fact that Grok caught up this quickly is really interesting.

And let's come back here in just a second to go through exactly what it's doing good at because this is, I mean, this seems like kind of a big deal, doesn't it?

Here they're giving a few examples of what it's good at.

For example, writing code from a diagram.

On the left here we have a flowchart: start, create a ROM number, read it.

I guess you're trying to guess a random number that is generated.

The user is asking, can you translate this into python code?

They drew up a little diagram on the board, took a picture of it, and say, hey, make this into a python code into an actual software program.

Grok answers, certainly, the flowchart you're provided describes a simple guessing game where the computer generates a random number and the user has to guess it.

Here's the python code that represents the logic in the flowcharts.

It generates that and as far as you can tell, near perfectly gets all the writing in like U1 and the wrong guest, try again.

It creates software from these little diagrams that you doodle on the Whiteboard.

Next, we have calculating calories from the back of the Cardon nutrition facts.

How many calories are in five slices of this kind of a tough question since a serving size is three slices.

You got to do some math there.

Grok calculates it that five slices will have 100 calories.

I did the math on this, it checks out, but real fast, this is kind of a hard thing to do.

A lot of other Vision mods we tested, they easily get confused with stuff like this if there's a lot of lines and you have the three slices and then you have the parentheses And 18 Gams and then this tends to trip them up.

This is, this is impressive.

They were saying my son Drew this, can you tell me a short bedtime story based on his drawing?

And sure enough, Grok provides a story, very, very cool.

Explaining a meme, so Grok better be good at this.

If Elon, the meme poster on X on Twitter, if he can't produce an AI that is excellent at explaining memes, then what is even the point?

It has to be a true meme scholar.

Startups, you have this image and big companies, you have this image.

The user says, I don't get it, please explain.

He describes that startup, everyone's actively participating, big companies only one person is actually digging the hole.

And the humorous image comes from the exaggeration of the differences between startups and big companies.

And then explains further pretty much Nails it.

Then we have a picture of what potentially could be a wiki Wikipedia, and the user saying, please convert this table to a CSV, right?

You comma separate values like an Excel spreadsheet or something similar.

The table and the image can be translated to that format as follows, and it does it including the um blanking and what this is called, the headers, the column headers.

This is also something that I've seen other models struggle with a little bit.

Next, they're showing an image of some damage around a wooden plank with a nail, and the user asking, these holes are appearing around the screws in my deck, is it rotten?

Should I get my boards replaced?

And Grok says that yes, it appears that your deck has started to rot.

This is indicated by the presence of holes around the screws, which is a common sign of wood Decay, and they give you some more advice about that.

This is one thing where I was very impressed with GPT-4 with vision because for like quality assurance purposes, you would be able to show pictures of things that were Broken, like bolts that were stripped or cars with Nicks and some damage on them scratches, and you would just be able to say what's wrong in this image, and it would kind Of tell you, it would assess the damage.

Here the fact that it's able to Figure out if there's rot, that there's wood Decay, that's certainly extremely extremely useful.

And then next we have a solving a coding problem, we're asking, can you write python code to solve this?

And it appears like a a semi complicated problem, not one that's easily understood at first glance, but Grok is able to write the code to solve it.

If this is representative of what we will receive when we get Grok of our results, then this would be very very impressive real world understanding.

In order to develop useful real world AI assistance, it is crucial to advance the model's understanding of the physical world.

And of course, some of the same people behind all these companies are some of the same people behind Tesla.

Tesla likely has the world's largest collection of various footage of cars being driven through various conditions, various roads, etc.

Certainly this idea of real world understanding is important.

They're saying towards this goal, we are introducing a new Benchmark real world QA.

We'll come back to this, but looks like they've introduced their very own real world QA, real world understanding Benchmark.

And as you may have expected, they are whooping everybody else's neural Nets at that Benchmark.

And this Benchmark is designed to evaluate basic real world spatial understanding capabilities of multimodal models.

While many of the examples in the current Benchmark are relatively easy for humans, they often pose a challenge for Frontier models.

For example, in this image, which object is larger, the pizza cutter or the scissors?

And it selects that se, they're about the same size.

It's somewhat difficult because the scissors, it's obstructed right, it's hidden behind multiple objects.

Next we have a traffic situation, where can we go from the current Lane?

Well, turn left.

I mean, really the only hint is the sign, right?

You do have the arrow here, but just visually it almost seems like you could go forward.

I guess it seems like the only hint as to what you can do is the sign, and it correctly picks up on that.

And also indeed understands that it is in the leftmost lane because you really really can't see the road that well given this front camera view from our sedan.

Do we have enough space to drive around the gray car in front of us?

And answer is yes.

Given the picture, in which cardinal direction is the dinosaur facing?

And let me see if I could do this, it's hard to see these okay, I see it.

East, the dinosaur is facing, yeah, roughly East, and so Grok chooses East.

I do feel like other models would definitely have a hard time with this one.

The initial release of the real world Q&A consists of over 700 images with a question and easily verifiable answer for each image.

The data set consists of anonymous images taken from vehicles in addition to other real world images, and they are releasing it under the Creative Commons.

You can download it here.

This is kind of a flex by Elon because I mean they do have a lot of data on the various driving around cars and freeways and such.

Certainly we can expect them to handsomely beat the other models at those particular tasks.

And they continue into the future advancing both our multimodal understanding and generation capabilities are important steps in building beneficial AGI that can understand The universe.

In the coming months, we anticipate to make significant improvements in both capabilities across various modalities such as images, audio, and video, and they are hiring.

Here's by the way the collection of images from that Creative Commons data set, the 700 and some images that they've used in that benchmark Mark.

I mean here are some of those images, there's a lot of cars and May dog, but yeah, they're probably like 90 95% images from cars driving around streets, a lot of it is San Francisco or the bay area as well as other sort of random images.

Grok takes the number one spot for that, the real world QA with Gemini Pro 1.5 a close second, then GPT-4 Vision followed by the Claude models Claude 3.

Next we have the mmm U that is a benchmark that's designed to measure perception knowledge and reasoning in Large Language Models.

A lot of the questions looks like they have a l picture and they're supposed to Figure out what the graph is showing or select some data from the graph music notes Etc.

Clot 3 did the best followed by Gemini Pro 1.5 then by GPT.

And then Gro, but there's not an ocean of difference.

They're not that far apart.

Next, we have math Vista.

These are visual and mathematical reasoning tests, puzzle tests, etc.

If I'm reading this correctly, the leaderboards on math Vista on their website show a human performance is around 60, and then Gro has 52.8 and is higher than every single Other one.

Next, the AI 2D, I believe, is testing their ability to understand diagrams at 88.3.

Grok is at the top, the only thing better is clot 3 Sonet.

Strangely enough, a smaller model, but GPT-4V is significantly less.

CLA 3 Opus is just a tiny bit less, and Gemini Pro 1.5 is a little bit less.

I'm not going to go through every single one of these, but text vqa, it's the winner.

Chart Q&A, it's lower than the rest but similar to GPT for vision.

CLA and Gemini are higher in the 80 and 81%. Doc vqa is at 85% with the highest model clocking in at 89%. So, to sum it all up, it's good, really good.

What lessons can we draw from this?

At this point, I got to say, don't bet against Elon Musk.

Of course, there's still a number of things to get straightened out.

Just getting the score and the valal is not necessarily the end all be all for how powerful the model is, but it's a start.

The new search function on X that has been, from what I understand, utilizing Gro to kind of find the news that's more like relevant to you.

I mean, for the first time ever, I'm actually looking at it.

It was so horrible before, it had no relevant news to me whatsoever, but here they're giving me news that I care about.

Mostly AI X is quickly becoming one of the biggest global destinations for news, tons of traffic, tons of people using it.

They've recently did a bot purge, getting rid of a lot of the automated traffic.

It's their own private data source that's real-time.

Elon had the money, he had the AI talent, he has the data, he has the users to test this thing out, he's got the distribution.

The only thing he didn't have was the model.

We should reserve judgment until we actually have the model in our hot little hands, until we actually test it and see if it does indeed do all the things that it claims to Do.

But I got to say, from where I'm sitting, some was beginning to look like about a decade ago, Elon was a little concerned that Google would develop a GI in isolation by Itself.

It would have this amazing new technology and no one else would.

It's possible, by the way, that Demi saabi, the man that's now running Google deep mine, that maybe he alerted Elon to what was happening.

Elon goes to Sam Altman and they form OpenAI.

I believe Anthropic, the people behind Cloud 3, they split off from OpenAI at some point, becoming their own company.

And keep in mind that the point here was to have to provide a counterbalance to Google.

That was at least the stated goal of Elon Musk back in 2014 or 2015, whenever that whole thing was just brewing.

As we're approaching April 20th, 2024, there's Grok.

Elon wanted a counterbalance to Google, and now there is.

Let's count them, not one, not two, but three major competitors.

And we're not even counting the open source Mistral, the newcomer Command R Plus, as well as all the other open source competitors.

But yeah, at this point, I got to say, I got to give credit where credit is due, Elon Musk.

You don't want to be betting against this guy.

But let me know what you think in the comments.

I know that this is a controversial Figure.

People have a range of emotions about him.

If we are indeed approaching AI, whatever that means to you, whether you think that's next year or 10 years from now, would you trust him with a powerful technology like That?

Would you trust him more than Sam Altman, more than Google?

Meanwhile, it does seem like Demis Hassabis isn't too thrilled how things are going over there at Google deep mine.

A year ago, Google hastily merged the two labs into one, that's Google Brain and Google Deep Mine.

They put it under Hassabis, but the tensions between them have lingered.

The dynamic has left Hassabis deeply frustrated.

Whatever the case is, the fact that we can watch some of this stuff unfolding live through perhaps various tweets or leaks from inside the companies or even cord documents, The fact that we can kind of watch this play out, I got to say, is quite amazing.

I hope you enjoy that.

Subscribe, you will want to pay attention for what's coming next because this is one field that is heating up.

With that said, my name is Wes Rth, and thank you for watching.