GPT-OSS vs. Qwen vs. Deepseek: Comparing Open Source LLM Architectures

OpenAI recently released its first open-weights model since GPT-2, entering a field led by DeepSeek and Alibaba's Qwen. YC's Ankit Gupta breaks down everything you need to know about these top OSS models, including what sets them apart under the hood. He’ll compare their approaches to mixture-of-experts, long-context training, and post-training techniques that shape reasoning and alignment—and explore how different design choices lead to surprisingly similar performance.

• 0:00Speaker 00

  OpenAI recently dropped GPT-OSs, its first open weights model since GPT-2 in 2019. It's one of the highest-profile open-source model launches since DeepSeek R1 made waves back in January. But how does GPT-OSs compare to the other top open-source models out there architecturally? Let's find out.

• 0:23Speaker 00

  GPT-OSs is one of OpenAI's most anticipated recent launches, a large, fully-open weights model from one of the leading American AI labs. Let's take a closer look at the paper to find out how it was actually engineered and trained. GPT-OSs is a mixture of experts model, available in two sizes: 120 billion parameters and 20 billion parameters. Each token activates the top four experts, meaning only a portion of the total parameters are used at any given time. This allows for efficient inference without sacrificing the benefits of a larger model. Trained as a decoder-only transformer, GPT-OSs incorporates plenty of features typical to modern LLMs. This includes group query attention, a modified attention mechanism that lets multiple query heads share the same key-value pairs to reduce memory use and speed up inference.

• 1:06Speaker 00

  It also includes Sui Glue activations in the feed-forward network layers, which allow for more nuanced transformations than simpler activations like ReLU, as well as rotary positional embeddings, or ROP, which encode token position directly into the attention mechanism to support longer context. Finally, the model also makes use of RMS norm with pre-normalization, a normalization method that scales inputs by their root mean square for more stable training. One standout capability of the model is its 131,000-token context window, which it achieves by applying yarn scaling during pre-training rather than as an inference time adjustment. We'll touch on what this means a little bit later in the video. For GPT-OSs, OpenAI makes use of their open-source O200K Harmony Tokenizer. This byte-parent encoding tokenizer has over 200,000 tokens

• 1:49Speaker 00

  and builds on the O200K tokenizer used in models like GPT-4O. As for the dataset GPT-OSs was trained on, OpenAI has only disclosed the broad strokes. The model was trained on a text-only corpus in the trillions of tokens with a focus on STEM, coding, and general knowledge. Harmful content was filtered out for safety, but beyond that, this little else known publicly. The model was released in a quantized format by default, making it lightweight enough for deployment on modest hardware. This allows it to be run on consumer-grade GPUs, laptops, or other resource-limited hardware. However, there's no un-quantized version available. GPT-OS also underwent substantial post-training for safety and alignment, shaping its default behavior for more controlled outputs. It's worth noting that some in the open-source community

• 2:32Speaker 00

  are experimenting with reducing or removing these layers in order to explore the raw model's capabilities. In the broader landscape of open-source AI, GPT-OS arrives as a fully-equipped, long-context model ready for immediate use. As impressive as it is, however, it's just one of several models in a rapidly-expanding field of open-source LLMs. Qwen-3, the newest family of models developed by Alibaba Cloud, dropped this past April to considerable hype, with benchmark scores that rivaled those of leading open-source-based models like DeepSeek V3 or Llama 4. The Qwen-3 family includes both dense models, which activate all of their parameters for each query, and mixture of expert models, which only activate a small subset of their parameters for each query. The dense models come in seven different size classes, including a 0.6 billion parameter model,

• 3:16Speaker 00

  one of the smallest current-generation open-weight models around, while the MOE models come in two different-sized classes. Architecturally, Qwen-3 dense models are very similar to the Qwen 2.5 models, Alibaba's previous releases. Like Qwen 2.5 in GPT-OS, Qwen-3 incorporates features like group query attention, Swiglu, rope, and RMS norm. Qwen-3's sparse models share the same fundamental architecture as its dense models, but add a mixture of experts layer with 128 total experts, of which eight are activated per token. All Qwen-3 models also use the same tokenizer used in previous Qwen models, which implements byte-level byte-parent codings that allow it to handle any text or symbol without special preprocessing, unlike word- or character-based tokenizers. One of the main things that sets Qwen-3 apart from previous Qwen models is the way it controls the scale of the key query and value projections

• 4:01Speaker 00

  to keep attention score stable at scale. It replaces QKV bias, a static offset that shifts KQ view projections in previous models, with QK norm, a normalization step that dynamically rescales that query and key vectors to maintain constant magnitudes. Data set-wise, Qwen-3 was trained on 36 trillion Training tokens, twice as many as the Quen 2.5 models. In addition to pulling data from multilingual texts, STEM encoding sources and reasoning tasks, Quen 3 also uses Quen 2.5 models to generate trillions of tokens of synthetic data in different formats like textbooks, instructions and code snippets. Quen 3's pre-training occurred in three stages. In stage one, the general stage, models were trained on over 30 trillion tokens covering 119 languages at a sequence length of 4096 tokens.

• 4:46Speaker 00

  In stage two, the reasoning stage, models were trained on an additional 5 trillion higher-quality tokens featuring more STEM, reasoning and coding problems. And in stage three, which the Quen team calls the long context stage, context length was extended to over 32,000 tokens using a bunch of clever algorithmic optimizations, including A B F, a technique to adjust rope so positional signals remain accurate over much longer sequences, yarn to further scale for longer inputs, and dual chunk attention to process sequences efficiently. Together, all of these optimizations allow the models to reason over much longer inputs at inference. Finally, Quen uses a four-step post-training pipeline with two goals: giving users more control over how much reasoning to use for a given query and letting them efficiently distill larger model capabilities into smaller models.

• 5:30Speaker 00

  The first step in the post-training pipeline is a long chain-of-thought cold-start stage, which involves feeding a model a curated dataset of challenging reasoning problems from math, logic and STEM with verifiable reference answers, and then filtering outputs to ensure quality. This is followed by a reasoning RL stage using GRPO, an RL algorithm originally developed by DeepSeek researchers on roughly 4,000 query verifier pairs to strengthen complex problem solving. Personally, I think it's fascinating that it only takes 4,000 pairs to get great results. The third step in the post-training pipeline, thinking mode fusion, is a key Qwen 3 innovation that integrates reasoning and non-reasoning into a single model, letting users switch modes without changing models. Essentially, what developers did in this step was fine-tune the model on a mix of thinking data,

• 6:12Speaker 00

  which includes intermediate reasoning steps, and non-thinking data, which omits them, and then build a chat interface to let users toggle modes. Though this was unique to Qwen when the model first launched, GPT-5 now features a similar toggle. The final step, general RL, broadens capabilities in instruction-following, formatting, preference alignment, tool use in specialized scenarios. Qwen's developers then use strong-to-weak distillation, which allows for the training of smaller models from larger ones. All in all, Qwen 3's performance is very impressive, especially given its relatively small size. But just months earlier, a different model had already raised the stakes in open source. Released in December of last year, DeepSeek's V3 model was one of the most ambitious open-source LLMs to come out of a major lab in recent years.

• 6:53Speaker 01

  A chatbot developed in China. It's called DeepSeek? DeepSeek is such a fundamental change to the economics of what's going on.

• 7:00Speaker 00

  The most downloaded free app in the US. This is an update in what people think is possible. At 671 billion parameters, it's a massive general-purpose-based model, designed for efficiency as much as capability, laying the groundwork for the reasoning-focused R1 model that would follow. We're not going to get into a ton of detail about V3's architecture or training pipeline here because we put out a comprehensive deep dive into it back in February. But high level, the thing to know about V3 is that it's a mixture of experts' model with several hardware and algorithmic optimizations, including training V3 natively in 8-bit rather than 16 or 32-bit, a huge unlock for cutting training costs. And just recently, DeepSeek pushed V3 even further with an updated version. The newly released V3.1 builds directly on the original V3-based checkpoint,

• 7:45Speaker 00

  extending it with a two-phase long-context training approach and adding a hybrid thinking mode that lets the same model switch between reasoning-heavy and lightweight inference. It also improves tool use and agent performance thanks to a more advanced post-training. In practice, this means V3.1 keeps the same core architecture as V3, but delivers stronger reasoning, smarter tool use, and greater performance. One thing that sets V3 apart is that it uses a different attention mechanism than GPT-oss and Qwen 3. In modern LLMs, a lot of the compute and memory is tied up in the KV cache. And so V3 makes use of MLA, which compresses keys and values into a smaller latent space before caching them, then decompresses them during inference. Although MLA is a bit more complex to implement, the previous DeepSeek V2 paper found it delivers greater memory savings

• 8:30Speaker 00

  and better modeling performance than GQA, especially in huge long-context models like this one. And that's just one of several areas where DeepSeek V3 takes a different path. With all that in mind, Let's take a step back. From V3 to Qwen to GPT-OS, how should we think about, at a high level, the differences between these models? One big difference is size. The Qwen 3 model family is the only one of the three to offer both dense and mixture of expert variants, with dense models from 0.6 billion to 32 billion parameters and a mixture of experts lineup that includes a 30 billion parameter model and a 235 billion parameter model. Notably, Qwen's mixture of experts base models matched the dense model's performance with only a fifth as many active parameters. On the other hand, DeepSeek V3 only comes in a mixture of experts architecture

• 9:12Speaker 00

  with 671 billion parameters, of which 37 billion are activated for a given token prediction. So considerably larger than even the biggest Qwen 3 model. GPT-OS sits in the middle. It offers two MOE models, one with 117 billion parameters, of which 5.1 billion are activated for a given token, and a smaller one with 21 billion parameters, of which 3.6 billion are activated for a given token. One of the most interesting technical differences lies in how each model extends its context length. Yarn, short for yet another rope extension, is a technique for stretching the model's rotary positional embeddings so that it can handle far longer sequences than it was originally trained on. Normally, rope starts to break down when you feed it more tokens than its base frequency was set for. But yarn tweaks that frequency so the same embedding space covers much more ground. What's interesting is how the three models here use it differently.

• 9:58Speaker 00

  GPT-OS applies yarn right from pre-training, so its weights have learned to work natively with 131,000 token context. DeepSeek takes a staged approach, fine-tuning after pre-training to first reach 32,000 tokens, then further training to achieve 128,000. Qwen also fine-tunes to 32,000, but skips that additional retraining step. Instead, at inference time, they apply yarn scaling again, increasing the rope base frequency by a factor of four to reach 128,000 tokens without extra retraining. In other words, GPT-OS is born with long context ability. DeepSeek is trained into it step by step, and Qwen pushes the limits of what a 32,000-trained model can do without more long context training. Personally, I think one of the most interesting things about these papers, and the state of the art in deep learning more generally,

• 10:43Speaker 00

  is that a lot of these read as empirical findings. Each lab describes the combination of tools that works well for them, but almost no one gives a First principles justification of why one tool is better than the other. For instance, why MLA is better than GQA, full stop. This is much different from domains like math or theoretical physics, which are all about providing first principles explanations that derive results from axioms or laws. Also, it's interesting that even though most of these models have similar top-line benchmark statistics and use broadly the same tools, like attention mechanisms, activation functions, positional embeddings, and so on, they achieve these similar results using often very different techniques. This is quite surprising. You'd expect that very different training methods would lead to very different results. Also, all of the major models heavily use reinforcement learning as part of the post-training and reasoning portions of their model training efforts.

• 11:28Speaker 00

  And it's fascinating and pretty surprising how some of these RL efforts require very little amounts of data, just 4,000 data pairs in the case of Qwen. Another point here is that it's very opaque what the differences in data sets are between the labs. It's clear from the papers that there's an enormous amount of work happening behind the scenes in data set engineering. This work is probably a significant aspect of the moat that makes these companies comfortable releasing their models. It's very difficult to replicate what they're releasing. So the big takeaway when reading these papers is you shouldn't focus too much on just the benchmark performance or top-line stats like context size. Instead, look at the specific methods that these labs are using to achieve those results. There are tons of high-performing open source models that we didn't discuss in this video, like KIMI K2 or Google Gemma 3. But when you peek under the hood of many of these, you'll find nuanced differences that I find really interesting.

• 12:13Speaker 00

  I hope this gives you a framework for how to understand the latest open source releases and gives you a toolkit to start tinkering with them yourself. Thanks for watching. See you on the next episode.