The latest hype in the AI sphere has included accusations surrounding two heavyweights: OpenAI and DeepSeek.

But unlike the Tyson v. Paul fight, this battle doesn't have a clear winner.

Released on January 20th, DeepSeek's R1 made history within a week, triggering one of the largest single-day losses in US stock market history (a bad day for NVIDIA).

R1's power play was the fact that it performed nearly as well as OpenAI's most advanced models (and allegedly 1000x cheaper to build than GPT-4).

Now, we have an idea of just how they did it.

OpenAI recently accused DeepSeek of violating OpenAI's Terms of Use to train R1 — specifically, through a process called model distillation.

Model distillation is a deep learning technique that leverages powerful AI systems to create smaller, more efficient ones. Distillation isn't an evil in its own right, but it does cross into some ethical gray areas.

The thing is: we know for a fact that DeepSeek uses distillation... but so do others (including OpenAI).

Let's take an objective look at the distillation controversy today.

We'll cover:

Let's dive in.

Model distillation involves knowledge transfer between:

In distillation, the teacher models' outputs become the student model's training data:

As a result, the student model learns not only the answer but the reasoning process, as well.

As the diagram below depicts, the teacher model also helps the student learn nuanced inter-class relationships by providing:

This enables the student model's ability to yield an efficient approximation of the teacher's performance.

One tradeoff of distillation is that you might sacrifice a sliver of accuracy, but that’s an acceptable compromise for many use cases.

Distillation has a distinct appeal over purely fine-tuning a pre-trained LLM. Fine-tuning involves training an LLM for specific tasks on specialized datasets. However, a large, fine-tuned LLMs boast billions of parameters, soaking up loads of computing power (and energy bills).

Distillation offers a clever route route that is more cost-effective:

• Extract the model's core intelligence, and trim the excess
• Create a leaner, distilled model that's faster and cheaper to deploy
• Fine-tune the smaller, distilled model for specialized use cases if needed

Ethical concerns and OpenAI's accusations

If a student model’s training data consists mostly of a teacher model’s outputs, would distillers need explicit permission to harvest that material?

As useful as this technique is, distillation wades into murky ethical waters, where it becomes difficult to draw the line between knowledge transfer and blatant copyright infringement.

While neither OpenAI nor Microsoft have disclosed evidence, OpenAI is accusing DeepSeek of using unfair methods to achieve their low-cost models.

• DeepSeek allegedly exploited OpenAI to perform distillation at scale, using multiple unrelated accounts.
• Microsoft security researchers reportedly detected individuals performing data exfiltration using OpenAI API in late 2024, which they believe to be linked to DeepSeek.

There is some irony in this controversy, as OpenAI itself has also been under fire for scraping people's data without permission to train ChatGPT. OpenAI considered this to be fair use — but not everyone agreed with that (and that's precisely why they were sued for 15 million euros by an Italian watchdog last December).

With that, we could entertain that OpenAI might be trying to position themselves as advocates of data privacy with this accusation.

While the question of whether they inappropriately used OpenAI data remains speculative, there's no doubt that DeepSeek is using distillation to affect powerful, efficient models.