SemanticScuttle - klotz.me

klotz: gqa*

0 bookmark(s) - Sort by: Date ↓ / Title / - Bookmarks from other users for this tag

1. Inside GPT-OSS: OpenAI’s Latest LLM Architecture

   An in-depth look at the architecture of OpenAI's GPT-OSS models, detailing tokenization, embeddings, transformer blocks, Mixture of Experts, attention mechanisms (GQA and RoPE), and quantization techniques.

   2025-09-27 Tags: llm, gpt-oss, openai, transformer, mixture of experts, moe, attention, gqa, rope, quantization, machine learning, .qwen3–30b-a3b. by klotz
2. Deep-Diving & Decoding The Secrets That Make DeepSeek So Good

   The article explores the architectural changes that enable DeepSeek's models to perform well with fewer resources, focusing on Multi-Head Latent Attention (MLA). It discusses the evolution of attention mechanisms, from Bahdanau to Transformer's Multi-Head Attention (MHA), and introduces Grouped-Query Attention (GQA) as a solution to MHA's memory inefficiencies. The article highlights DeepSeek's competitive performance despite lower reported training costs.

   2025-02-16 Tags: deepseek, multi-head latent attention, mla, attention, transformer, grouped-query attention, gqa, deep learning, llm by klotz
3. Mistral AI vs. Meta: Comparing Top Open-source LLMs: Mistral 7B vs Llama 2 7B and Mixtral 8x7B vs Llama 2 70B

   novel concepts that Mistral AI added to traditional Transformer architectures and we perform a comparison of inference time between Mistral 7B and Llama 2 7B and a comparison of memory, inference time and response quality between Mixtral 8x7B and LLama 2 70B. RAG systems and a public Amazon dataset with customer reviews.

   2024-01-23 Tags: mistral, mixtral, llm, gqa, swa, smoe by klotz

Top of the page

First / Previous / Next / Last / Page 1 of 0