Around two days ago, there was a rather interesting development.

There was something in China that I think you should all be aware of because it changes the dynamic of the AI race, which leads me to believe now that currently China is hot on the heels of the rest of the world in terms of AI developments.

It seems that the US has been leading the charge in terms of state-of-the-art models, but recently this is the most interesting thing we've seen in recent times.

Let's take a look at exactly what China has been doing and what this announcement is.

Essentially what we have here is we have a situation where China has potentially taken the lead with their new model.

Two days ago, SenseTime launched SenseNova 5.0, which according to this report actually beats GPT-4 on nearly all benchmarks, and there are actually a variety of different things that are truly surprising about this.

It isn't just some hyped up GPT-4 thing.

If the claims are true, and we're going to dive into them, this means that things are truly, truly ramping up, and it seems that across the board from different nations and different countries, we're definitely going to be seeing increased competition in terms of what AI systems are capable of.

We can see here that they've essentially compared this to GPT-4 Turbo, but there are a lot of things that this presentation dives into, and bear in mind that the majority of this is in Chinese, but I've done my best to translate most of this so that you can completely understand exactly what's going on.

You can see here that their SenseNova 5.0 actually looks pretty decent.

On the left, you can see it states that it is hybrid used, in the middle, they state that it is trained on over 10 billion tokens.

The inference actually supports up to 200,000 tokens, which is rather interesting because we've seen a longer and longer context windows, so a 200k context window is something that is I wouldn't say surprising, but it does show that of course things are going well, and of course the main main statement from this is that performance exceeds GPT-4 Turbo.

The reason, as you all know, why GPT-4 Turbo is highly touted as the state-of-the-art model at that time, and that's why people constantly, constantly, constantly try to one-up that in terms of the benchmarks.

This was the start of the presentation, but things do get a lot more interesting, and I'm going to show you all exactly what that is.

There was one screenshot here where I'm not entirely sure if they actually had these AI systems hooked up to a game, but the text states that a live demonstration was conducted on the comparison of multiple functions of their model and GPT-4, including creative writing, logical reasoning, diagrams, image understanding, and calculations of food calories based on pictures, and this is what the text actually states here, but we will get into the benchmarks, but I just wanted to include this for full transparency.

In order to show off the muscles of the larger models, Sense Time also played the King of Fighters, and at first, the green player GPT-4 had the slight upper hand, but was quickly overwhelmed by the red player Sense Chat's light various combos.

I'm not sure if they're actually talking about this as if it's some kind of metaphor for their system beating GPT-4, or if they actually hooked it up to a Street Fighter style game and Sense Chat Lite was able to best GPT-4.

If you are someone who can translate this more effectively because I did honestly put this through multiple different AI systems, and the result was frequently the same.

Here's where things start to get interesting, and here's where things are really, really fascinating because these are obviously the benchmarks for this system.

You can see Sense Chat version 5 is over here on the right-hand side, this is Sense Chat version 5.

And then of course what they actually do do, and this pay attention here because I'm going to say some things that I'm going to recall them so you can see right here that they actually do compare this to GPT-4 Turbo, and that actually is to the eagle-eyed viewers that it is the 1106.

This isn't actually the most recent one, but it's not also a most outdated one as well, and we can see that the only benchmarks that it surpasses their model Sense Chat version 5 on is, of course, the math zero shock benchmark right here, and we can see that it's highlighted in the 61% area, and we can see that in this benchmark right here as well.

We can basically see that currently what we're looking at is a situation where across all of the benchmarks compared to GPT-4 Turbo, one of the most recent models.

I don't know why they didn't do the most recent one, I'm guessing just based on where they are in their testing, and of course we do have a comparison with LLaMA 3 70 billion parameters instruct.

We can see that the differences from GPT-4 are honestly, I wouldn't say quite stark, but it doesn't just seem to be a simple percentage gain.

Like, for example, on the MMLU, we can see that this one is rather small, but on the CMMU, I'm guessing that this is a different version of the MMLU, perhaps the Chinese MMLU.

Honestly, I'm not entirely sure.

We can see that it increases, but you can see, like I said, on the different benchmarks here, the gains don't seem to be completely incremental, because if we actually look at the differences, this one is at 80, and this one is at 93.61.

I think what we're looking at here, and of course, some of these benchmarks aren't all on GPT-4.

As in, when I was actually checking some of the other models, I couldn't find things like the race, I couldn't find things like the natural questions on other models, and that's like Gemini Pro and Claude.

And I do actually test them against it, and I'll get to that later on in the video.

But essentially, I think this is rather fascinating that China has, I would say, surpassed GPT-4 Turbo in their model.

And what I would like to see from this model, honestly, is if people could potentially test this model, because it would be interesting to see how it actually compares in terms of users.

And what I mean by that is, I mean that whilst, yes, currently on this chart, what you're currently seeing is GPT-4 only winning in two out of the several different categories here.

And also, even LLaMA 3, we do know that there is a different benchmark, which is rather, rather important.

And the benchmark that I find to be one of the most interesting, because it's not just based on the instance where things could be fine-tuned on the actual benchmarks in terms of just trying to get something that beats things on benchmarks, but isn't actually useful on a day-to-day basis.

You can see here that we do have GPT-4 1106 on the chatbot arena, as I've stated previously.

The chatbot arena is different because it ranks models based on their arena elo, which is determined by votes against other systems.

And basically, all this means is that people test AI systems side by side, and their elo increases when people vote for that system being the most useful.

And it's a blind test, so you don't know which system is which.

And then over time, you get to see which system is actually useful on a day-to-day basis, answering a variety of different questions without the bias of looking at other systems.

It's something that I think is very useful, and we can see that GPT-4 1106 is basically at rank number one.

And of course, like I said, the new one does have a slightly higher elo rating, but I still think that this is rather rather impressive for what we're seeing here.

Something that I also did want to test was I wanted to test this against other models because one thing I realized when looking at this was I was like, okay, whilst this is good, there are other state of the art systems out there.

Because yes, GPT-4 Turbo 1106 is state of the art, but GPT-4 Turbo isn't the only one.

And GPT-3 Turbo and LLaMA 3 70 billion parameters aren't the only ones there.

The other one that most people might be thinking about, and maybe you're not thinking about this, but I certainly am, is I'm thinking about the Claude 3.

Claude 3, if you don't know, it's a model, a state of the art model, and it was released by Anthropic.

I've done this a little bit sloppily, but let me just explain to you what you're currently looking at.

All you're looking at is Claude 3's benchmarks, and we can see that it is compared to GPT-4 in this tab right here.

And essentially, across the board, completely, Claude 3's Opus model does surpass the benchmarks of GPT-4.

What this does mean here, what I've actually done here is I've actually circled these areas where Claude 3 retains its leadership in terms of how far it is, but I've added x's here because these are areas where sense chat v5 actually beats Claude Opus.

In the math benchmarks, which is math problem solving, it does better than Claude Opus.

There we can see it's 61.9, and this one is 60.1.

That's where sense chat v5 reigns supreme.

Then, of course, we can see here on the common sense knowledge, the heliswag, this is 95.4 percent, and then for here on the common sense heliswag, we can see that theirs is actually at 97 percent.

This is actually a huge improvement.

We can see that 95.4 percent to 97.52 percent is a big improvement.

However, Claude actually retains the leadership benchmarks in these areas where we can see the MMLU, the graduate level reasoning, the grade school math.

But something that I do want to add, whilst yes, Claude 3 does have the best on these benchmarks, I would have to be honest and state that it isn't far off.

Like, for example, if we look at the code on the human evaluation at 84.9 percent, or for example, if we look at the big bench hard, we can see it's currently at 82.98 percent, whereas on the big bench hard, it's at 86.8 percent.

I mean, it is pretty interesting to see how these differences are, but the long story short is that basically, Gentschat v5 surpasses GPT-4 Turbo, Claude 3 Opus surpasses GPT-4 Turbo, but in terms of the actual benchmarks, the ones that you do want to care about according to the arena, GPT-4 Turbo's most recent version retains its leadership spot, even if the benchmarks that you know are benchmarks by like math and coding and stuff don't appear to be correct.

People seem to be having a great time with GPT-4 Turbo, but firmly followed by Claude 3 Opus.

And then Google's Gemini 1.5 Pro in the API.

I think this is still rather impressive because it seems that this company has been working very quietly and very diligently in stealth.

Because this is something that did catch people off guard.

There are some other things that are actually really surprising, and one of the most surprising things about this was something in regards to the smaller models.

Because their smaller model is insane, and it's so crazy to the point where I'm wondering to see if some of these things are actually even legit.

One of the things they actually talk about is a writing task, and here's where they compare GPT-4.

You can see right here, GPT-4 to GensChat v5.

And right here, this is where, and funnily enough, this actually does look like the chatbot arena.

I'm wondering if in a couple of days, we are actually going to get GensChat v5 in the arena.

But, they talk about how essentially the summary of this what actually happened here was that they basically had a task to write a college entrance exam essay on innovation based on the classic Chinese novel, Dream of the Red Chamber.

And the author notes that GPT-4's writing style tends to be more rigid and structured, using repetitive phrases.

And in contrast, SenseNova's 5.0 exhibits more free-flowing and divergent writing style, drawing upon a wide range of Chinese cultural and literary references from ancient times to the modern internet age.

Long story short, they're basically trying to say that if we look at the writing tasks in the examples that they've included, GPT-4 loses.

There was also another one right here where they put GPT-4's model against SenseChat v5 in, I guess you could say, a logic/reasoning task.

And they were trying to compare this in terms of what this model would do when it was trying to Figure out the amount of coffee and water consumed.

And honestly, it's pretty hard to decipher what's going on here because, like I said before, it's a translation.

The interpretation might actually not be correct.

And I'm not going to get into the details of that.

But the long story short here was that GPT-4 wasn't able to realize exactly what was going on.

But SenseChat v5, the reasoning steps provided the correct answer.

I will leave the prompt and the explanation in the description.

But it's just very confusing trying to get a logical reasoning task correct.

Because if you've ever seen the examples of these incredibly confusing tasks, one word can truly change the entire outcome of the answer.

And it's important to get the translations 100% because even if you get them 99, you can completely fail the question.

This is probably the most fascinating part of the thing when I was looking through it in terms of the benchmarks.

And in fact, that's actually not true.

It's actually coming up soon.

But right here, and we will get to the most interesting part here.

But right here, this is where we take a look at their visual recognition systems.

And of course, you can see it is compared to Google's Gemini provision, OpenAI's GPT-4 Vision, Quen's VR Max, which is a vision system, and of course, the intern LLM X Composer 2 VL and Step 1 Vision.

There are all of these different ones compared on these benchmarks.

And we can see from what we've seen here, that the top one does surpass them.

One thing that I also saw that was really, really cool was that they also did have image generation.

And by the looks of things, it seems to be very, very photorealistic.

Essentially, the text states that Sense 5.0 sets a new benchmark in terms of AI-powered image generation, demonstrated by its impressive performance in generating nuanced and lifelike portraits.

The following example illustrates the AI's capability to create text-image visuals.

And essentially, using the prompt, a vibrant Asian female portrait with beauty makeup, casual hair, a smile, natural, with movie-grade quality, presenting different expressions and styles.

Sense Nova 5.0 showcases its sophisticated interpretation of textual descriptions and its ability to generate diverse facial expressions and styles that carry the subtlety and richness of film-grade portraits.

That description is pretty good, but I've got to be honest, if this is their text-to-image generation model, I have to say that, while yes, Midjourney might be on this level, this is definitely something that looks really, really good.

I mean, we can't say that this doesn't look completely realistic.

And as for the prompt, I think this is something that does do it really, really well.

Of course, I'm guessing that they did compare this to other systems.

Right here, in fact, it isn't entirely clear, but I'm guessing that these are other systems.

But either way, this one right here does look really, really realistic.

And I have to say that this is pretty impressive.

Here's where we talk about the most interesting bit, because like I said before, we were previously supposed to, but I was thought that the slide was the other slide.

Essentially, this is where they talk about their smaller models, their more compact models that they can use for many different things.

You can see that the one they've highlighted is SenseChat Light, which is 1.8 billion parameters in size.

And it does do a lot better than others at the similar size.

We can see compared to the GEMMA 2 billion parameters by Google, it completely destroys it.

We can see that LLaMA 2, 13 billion parameters, it completely does better than it.

But one thing that I didn't really understand from this and was pretty confusing was, of course, the benchmarks, because they didn't actually do this on traditional benchmarks.

They did this on a different kind of benchmark.

When this text on this left column was actually translated, the benchmarks that we got were, I guess you could say, the words we got were comprehensive score, language comprehension, creativity, reasoning, and the average overall.

One thing that I do think is quite surprising here is that 1.8 billion parameter model does showcase incredible capabilities because time and time again, we've seen that trend to be increased.

One critique that I do have here is that they didn't compare this to Microsoft's Phi 3.

And I'm guessing that Phi 3 was literally just released, so I guess that's okay.

But they didn't actually compare this to LLaMA 3.

You can see here that they compared this to LLaMA 2, 7 billion parameters.

And of course, they state that it beats it and it beats Google's and the other ones like Quen's 1.5.

But even what they actually did was, if we actually look at the original parameters, we can see they actually did compare LLaMA 3, 70 billion instruct.

I'm guessing, okay, and this is just a guess, maybe it isn't true that if the original system beats LLaMA 3, I'm guessing that this smaller system doesn't beat LLaMA 3 because they haven't included it for whatever reasons.

And I'm pretty sure they could have.

I think this is still pretty interesting because they've got a smaller model that is levels above these other smaller models.

But the point here is that based on some other things that I've seen as well they actually do talk about image generation.

And I've got to be honest, the text here is a little bit confusing.

I have tried to understand the context of what it's trying to explain, but it does get a little bit confusing sometimes.

And I will leave a link to the article.

One thing that I actually do talk about that they say that this is going to be a calorie assistant and you can use it to submit images and completely understand your calories.

And I think right now the AI space is heating up quite a lot.

And this is not going to be the only company coming out of China to present their models that do take us by surprise.

But so far, this seems like a very interesting update.

What's also interesting is that the company shares soared more than 30% after announcing its native generative AI model.

This was something that made the company stock price jump 30%. So maybe it could be argued that the benchmarks that they're talking about aren't as good.

But the only way for us to know is, of course, for us to test it in the arena, for other people to do their independent evaluations.

And of course, things are going to be a little bit different because I'm guessing that this is fine-tuned on the Chinese language.

The translations might not be as completely accurate.

I'm guessing maybe if they make an English model, that might be as good or better.

But of course, this is going to be something that we would have to see.

Let me know what you think about SenseTime's new model and if we think China is going to catch up to the US competitors and how this actually impacts the landscape.

I think this is, of course, some very fascinating competition because I think we're going to continue to see models and different companies pour millions and billions of dollars into this industry.

Because, like I said before, I think we are now in that terminal race condition where if one company stops, then they are pretty much giving up the lead.