There's a battle being waged right now in the world of artificial intelligence between large foundational models and smaller open source models, and just this week a new research paper was dropped that promises to append the conversation completely.

Now, if you remember, a few weeks ago I made a video about the letter called We Have No Mode, which was a leaked internal memo from Google that really highlighted how open source models, smaller ones specifically, are iterating so quickly that these large foundational models that Google and OpenAI have are truly at risk.

I found that to be a very compelling paper.

And then, just two weeks ago, another research paper was released that claimed to disprove a lot of the value that these open source smaller models have.

Today, we're going to take a look at all of this and we're going to figure out what's the truth.

We're going to take a look at the new Orca paper that was just dropped this week.

We're going to look at the We Have No Mode document again.

And we're going to take a look at the research paper that came out a couple weeks ago talking about the false promise of imitating proprietary large language models like GPT-4.

Let's go.

So, this is Orca: Progressive learning from complex explanation traces of GPT-4.

This is a new research paper dropped by Microsoft Research, of all companies.

Of course, they made a substantial investment in OpenAI and own a significant portion of that company.

So, for them to release a new research paper illustrating a new technique to make open source smaller models extremely powerful is really fascinating.

Microsoft, as a company, has embraced open source in the years since Satya Nadella took over, and I'm all for it.

This paper is absolutely fascinating and it makes a ton of sense.

But before we get into this paper, let's take a look at those previous documents that I mentioned.

Now, a little over a month ago, this internal memo from Google was released called We Have No Mode.

And the main point of this memo is that open source models are proliferating and iterating so quickly that the gap between models like GPT-4 and PaLM 2 are shrinking very quickly.

The fact that any developer can get their hands on these models and new techniques to train and fine-tune these models are coming out every day.

And we're seeing that from Laura to Q, Laura to now having a ton of different options of how to train and fine-tune these models in really efficient ways and run them on any consumer-grade hardware.

And I agreed with a lot that was in this paper.

Of course, a business mode is not just the technical limitations; there's much more to it than that.

But a lot of the points made in this paper are very valid, and I've seen more innovation in the open source community over these last few weeks than I've seen on these proprietary large models.

But then, a research paper out of UC Berkeley was dropped a couple weeks ago that really challenged the assertions of the We Have No Mode document.

In this research paper, The False Promise of Imitating Proprietary LLMs, they spell out that these open source models are simply just imitating the outputs of these larger models without actually understanding the logic to reach certain output.

The gist of this paper, and what Orca looks to correct, is that these open source models are simply being trained on prompts and responses, which is good for pattern matching.

So, for example, if you're a student in college and you're taking a class, you could probably do pretty well on a lot of tests simply by pattern matching the question to an answer.

But that student is going to have a lot of limitations if one of the questions varies from their pattern matching ability by even just a little bit.

Their ability to reason and figure out what the answer might be becomes highly limited.

Whereas the student who fundamentally and deeply understands a topic won't be thrown off by any variation of the question.

They'll be able to reason and step by step get to the answer because they do truly understand the topic.

And that's really the difference between these large foundational models and the open source imitations of them, as per this paper.

And that brings us to Orca.

Orca challenges the idea that open source models can only really imitate answers and will get thrown off by any variation in the prompts themselves.

And the way they do it seems very obvious in hindsight.

Before we get into the details, Orca outperforms every other open source model and even outperforms ChatGPT, which is GPT-3.5, in a lot of different benchmarks.

Now, of course, it still lags behind GPT4, but the gap continues to close.

So let's take a look at this paper.

Now, they start off the abstract by addressing this imitation concept.

Recent research has focused on enhancing the capability of smaller models through imitation learning, drawing on the outputs generated by large foundational models (LFMs). Again, LFMs are referring to ChatGPT and GPT4.

And they start to outline the limitations of these imitation techniques.

Some that they point out are limited imitation signals from shallow LFM outputs, small-scale homogeneous training data.

And most notably, a lack of rigorous evaluation, resulting in overestimating the small model's capability, as they tend to learn to imitate the style but not the reasoning process of LFMs.

That is really the crux of this paper - how do we start getting these open source models to not just mimic the question-answer pairs, but actually understand how they get from a question to an answer?

And only with that is true intelligence created.

To address these challenges, we develop Orca, a 13 billion parameter model that learns to imitate the reasoning process of LFMs.

Let's pause there for a second.

This model, the Orca model, is only 13 billion parameters, which means it can run on pretty much any modern hardware.

Whereas some of the other models that I've been reviewing recently, like the Guanaco model, require me to rent out a cloud GPU like an A6000 that has 48 gigabytes of VRAM because it's so large (65 billion parameters), and this performs better than that.

Now, here's the key to the paper, here's the key technique: Orca learns from rich signals from GPT4, including explanation traces, step-by-step thought processes, and other complex instructions.

Guided by teacher assistants from ChatGPT.

Now, I'll explain what teacher assistance is in a little bit.

But looking at this sentence, what it's really saying is rather than learning from the prompt and response pairs, we're going to ask these large foundational models to explain their reasoning step by step.

And the smaller open-source models will learn from that.

Truly fascinating.

Now I want to briefly touch on this guided by teacher assistance from ChatGPT.

They have a two-tier teaching process.

One, they take ChatGPT, which is GPT-3.5, and they have a large number of examples to learn from (5 million). Then they take those 5 million and boil it down to the most important 1 million examples and then use GPT4 to continue to train on more complex examples.

So, how does it actually perform?

Orca surpasses conventional state-of-the-art instruction-tuned models such as Vicunia 13B by more than 100 percent in complex zero-shot reasoning benchmarks like Big Bench Hard and 42 on AGI eval.

Big Bench Hard and AGI eval are just sets of tests that they give to these large language models to test their performance.

Orca reaches parity with ChatGPT on the BBH Benchmark and shows competitive performance in professional and academic examinations like SAT, LSAT, GRE, and GMAT, both in zero-shot setting without Chain of Thought.

While trailing behind GPT-4, and again, this last sentence is everything.

Our research indicates that learning from step-by-step explanations, whether these are generated by humans or more advanced AI models, is a promising direction to improve model capabilities and skills.

And just like humans, large language models understanding how something works is much more effective than just being able to pattern match questions and answers.

So, large language models are typically tuned by something called instruction tuning.

You have a set of prompts and you have a set of responses, and those pairs are passed to the open-source model and it learns from that.

This technique is called explanation tuning, where it's not just the prompt and the answer, but an explanation of the reasoning and the logic for how ChatGPT and GPT-4 arrived at an answer.

And so, we can see here when evaluated by GPT-4, and that's called auto-evaluation, Orca 13B actually beats ChatGPT, it beats Bard, and it certainly beats the open-source models based on LLaMA.

And then for zero-shot problems on academic exams, ChatGPT definitely performs better, but Orca 13B is really closing the gap in performance and performs much better than Vicunia 13B.

And for complex zero-shot reasoning tasks in Big Bench Hard, Orca achieves parity with ChatGPT.

And here again, they specifically call out that imitation paper authors assert that model limitation is a false promise.

Since broadly matching ChatGPT using purely imitation would require one, a concerted effort to collect enormous imitation data sets, and far more diverse and higher-quality imitation data than is currently available.

So, one of the biggest problems is these open-source models can't get enough data to use the imitation technique and perform at the same rate as these large foundational models.

Contrary to this assertion, we demonstrate that both conditions one and two are attainable and that it is possible to reduce the gap with proprietary LLMs on multiple zero-shot benchmarks that require sophisticated reasoning.

And here they touch on what the existing open-source models are doing currently to train themselves.

Both Alpaca and WizardLM employ a variant of self-instruct.

So, that's what we've been talking about.

WizardLM introduces the concept of EvoLinst and STRUCK, which gradually rewrites the initial set of instructions into more complex versions, attempting to overcome some of the methods inherent shortcomings.

But with Vicunia and Koala, they demonstrate remarkable performance due to the more human-like conversations and natural instructions in the community-contributed conversations, like those in shared GPT.

So, basically, what they're saying is, as more people are using these open-source models and sharing their data, sharing their instructions, their prompts, and the output, they'll continue to train on those pairs and get better and better.

But there's a limitation with that as well, and it's the same thing that we keep coming back to: models trained on such natural conversations may capture the style but not the reasoning process of the LLM.

So again, they'll be able to pattern match, but they're not going to truly understand the logic and the reasoning behind arriving at the solutions.

Now, the Orca paper puts forth three key contributions.

Number one is explanation tuning.

And again, this is fine-tuning models based on the step-by-step explanation of the reasoning and the logic of how to arrive at a solution.

Let's read this a little bit.

We augment the query-response pairs with detailed responses from GPT4 that explain the reasoning process of the teacher as it generates the response.

And to get the step-by-step reasoning, they're using some of these more modern prompting techniques that we've been learning about, such as explain like I'm five, think step by step, and justify your response.

This forces GPT4 to put forth its reasoning and its logic in the response itself, and that is used to train, and that's what explanation tuning is.

Another issue is scaling the amount of tasks and instructions.

As you'll see in a graph that I'll show in a second, a lot of these open-source models are using a highly limited data set.

But that's where Orca really excels.

We utilize the Flan 2020 collection, and that's a data set of tasks and instructions put forth by Google that has tens of millions of instructions.

So let's quickly take a look at the data sizes for these open-source models.

All of them have in the thousands.

So you can see here that Alpaca has 52,000, Vicunia has 70,000, and WizardLM with the most has 250,000.

Based on the teacher of ChatGPT and some of these other ones like Dolly are human instructed, so they're even more limited because of the limitations of humans.

However, as you can see here, Orca has 5 million, many times more than all of the other open-source models, and it's based on ChatGPT initially.

So that's the initial 5 million pass, and then GPT-4 with a second pass of much more complex tasks and instructions.

So not only are they getting full explanations of query and responses and how they actually reach those responses, but they're getting so many more of them, and they're solving the data scaling issue.

Last is evaluation.

There are a lot of issues with current evaluation techniques for open-source models, but Orca claims to solve these in a few ways.

They use auto-evaluation with GPT-4, so basically asking GPT-4 between two potential responses which one is best.

They also use academic benchmarks like Big Bench, Hard and Truthful QA, and professional and academic exams like the SAT, LSAT, etc.

And lastly, they use safety evaluation from toxic-gen, based really do these responses contain toxic language.

So in Figure 4, they illustrate what the previous techniques do with queries and responses.

So here's a user instruction: Use the data to calculate the median.

Here it says, First, we need to arrange the data in ascending order.

Since there are five numbers, the median is in the middle, which is seven.

And so this is very basic, it's a prompt, a query, and then the response.

Whereas with their new method, they ask GPT-4 to explain.

So the system instruction seems to be the main tool that they use to get ChatGPT and GPT-4 to explain their reasoning.

You are an AI assistant.

User will give you a task.

Your goal is to complete the task as faithfully as you can while performing the task.

Think step by step and justify your steps.

So again, use the given data to calculate the median.

Same prompt to calculate the median, I will follow these steps, and GPT-4 actually outlines step by step how it will figure out what the median is.

That data is then used to train the open source model.

Now, I find it so fascinating that we're using some of these modern prompting techniques like Chain of Thought, like Explain Like I'm Five, that people have been figuring out over the last few months to get better answers from ChatGPT and GPT-4.

And we're using those to get better data to train the open source models with.

And as I mentioned, system messages seem to be the main tool to get ChatGPT and GPT-4 to provide the step-by-step explanations.

And if you play around with the ChatGPT playground or even the API, you'll know that the system messages are a requirement for using either of these tools.

So, here are a few examples.

You will be given a task.

You must generate a detailed and long answer.

Think like you are answering to a five-year-old.

Help as much as you can.

So, it's really just coaxing ChatGPT-4 to explain its reasoning and to be as verbose as possible.

So, let's actually take a look at the difference between these two prompting techniques.

So, for the system message, you are an AI assistant that helps people find information.

User will give you a question.

Your task is to answer as faithfully as you can while answering things step by step and justify your answer.

So, the prompt is: Pick which sentence is not logical.

Sentence A: People in the desert often look forward to flood.

Sentence B: People in the desert often look forward to rain.

Now, it might seem obvious to us why these sentences are different, but there's really just a subtle difference between them.

So, here a typical response might be: There is no reason to look forward to a flood because floods cause damage.

The answer is Sentence A. ChatGPT says, Okay, let's break it down.

Read the sentences carefully.

So, let's talk about what steps am I going to take to answer this.

Identify the differences between them and determine which sentence is not logical.

Then they say, Sentence A: People in the desert often look forward to the flood.

This sentence suggests that people in the desert are eagerly anticipating a flood.

However, floods are typically destructive and dangerous.

So, it seems unlikely that people would look forward to them.

Sentence B: They're looking forward to the rain.

And it says, This sentence suggests that people in the desert are eagerly anticipating rain.

While the desert is a dry environment, rain can be beneficial, providing water for plants and animals.

Then it finishes with, Based on the above analysis, it can be concluded that Sentence A is not logical.

Then we look at GPT-4's answer, and as you can see, I won't read all of it, it's a much more detailed and verbose answer.

Now, in this section, they talk about why ChatGPT as a teaching assistant, assistant to GPT-4, is such a powerful method.

And there's really two reasons for it.

One is a capacity gap, because there's such a large gap between the ORCA model and GPT-4.

Being able to take data from GPT-4 and passing it directly into ORCA, it's going to struggle with imitation.

Whereas if they progressively teach it to get to the GPT-4 level by the intermediate step of ChatGPT, it really performs much better.

This can be viewed as a form of progressive learning or curriculum learning, where the student first learns from easier examples, followed by harder ones again, more and more human-like behavior.

A human doesn't go from learning the basics of addition all the way to calculus in one step; they learn many incrementally more difficult steps of mathematics between addition and calculus.

Next is a simple pragmatic reason: cost and time.

ChatGPT, specifically GPT-3.5 Turbo, is much faster and much less expensive than GPT4.

So that's why they use 5 million examples with ChatGPT and 1 million examples for GPT4.

This graphic shows the performance of these large foundational models: Vicunya and Orca.

And as we can clearly see from questions from the LSAT and the SAT, Orca performs significantly better than Vicunya.

And if we look at the Orca column compared to the ChatGPT column overall, it performs quite similarly but it still does lag behind GPT4.

And they've actually shown that this progressive learning technique really works.

As we can see here, using only GPT4, they were able to achieve a score of 37.18.

Whereas if they use that intermediate step of ChatGPT, they were able to achieve 41.7.

That might seem small, but that is a significant improvement.

And for the big-bench hard results, Orca performs marginally better than ChatGPT on aggregate across all tasks, significantly lags GPT4, and outperforms Vicunya by 113 percent.

Here they give a graphic of the zero-shot performance against all of these different tasks, and you can see that Orca performs substantially better than Vicunya and even across all of them, like it said, it performs better than ChatGPT.

So, what does all this mean?

I find it fascinating for two reasons.

One, open-source models continue to get better at a rapid clip.

New techniques for fine-tuning training are coming out every single day.

So, I think back to that We Have No Mode paper, and it makes a lot of sense still.

I also find it fascinating that GPT4 still seems to have some secret sauce and performs much better than any other model.

So, OpenAI seems to have plenty of mode.

This moat seems to be incrementally getting decreased, but they still do have it.

The last thing that I find fascinating is that this paper is by Microsoft Research.

Microsoft Research owns a significant portion of OpenAI.

So, the fact that they're making research gains in open source is awesome, and OpenAI really must feel that they have a significant moat to work with.

And OpenAI also mentioned a week ago that they're releasing their own open-source model.

So, I think what all of this means is that these large language models will continue to get better and cheaper over time.

Now, Orca's code and dataset are not yet released, but as soon as it is, we're going to review it.

I'm going to show you how to use it, and we'll see how it performs.

If you like this video, please consider giving me a like and subscribe, and I'll see you in the next one.