This article explores the critical architectural decision of where to store conversation history when building AI agents. It examines how different storage strategies impact user experience, privacy, cost, and portability. The author compares service-managed versus client-managed storage models and details how modern APIs support both linear threads and forking/branching capabilities.
Key topics include:
* Service-Managed vs. Client-Managed storage tradeoffs
* Linear (single-threaded) vs. Forking-capable conversation models
* Strategies for context window management and compaction such as truncation, summarization, and sliding windows
* How Microsoft Agent Framework abstracts these patterns using AgentSession and ChatHistoryProvider to ensure provider-agnostic code
* Practical implementation examples for the Responses API in different modes
Key topics include:
* Service-Managed vs. Client-Managed storage tradeoffs
* Linear (single-threaded) vs. Forking-capable conversation models
* Strategies for context window management and compaction such as truncation, summarization, and sliding windows
* How Microsoft Agent Framework abstracts these patterns using AgentSession and ChatHistoryProvider to ensure provider-agnostic code
* Practical implementation examples for the Responses API in different modes
A comprehensive curated collection of Large Language Model (LLM) architecture figures and technical fact sheets. This gallery provides a visual and data-driven overview of modern model designs, ranging from classic dense architectures like GPT-2 to advanced sparse Mixture-of-Experts (MoE) systems and hybrid attention models. Users can explore detailed specifications including parameter scales, context windows, attention mechanisms, and intelligence indices for various prominent models.
Key features include:
* Detailed architecture fact sheets for a wide array of models such as Llama, DeepSeek, Qwen, Gemma, and Mistral.
* An architecture diff tool to compare two different model designs side-by-side.
* Comparative analysis across dense, MoE, MLA, and hybrid decoder families.
* Links to original source articles and technical reports for deeper research.
Key features include:
* Detailed architecture fact sheets for a wide array of models such as Llama, DeepSeek, Qwen, Gemma, and Mistral.
* An architecture diff tool to compare two different model designs side-by-side.
* Comparative analysis across dense, MoE, MLA, and hybrid decoder families.
* Links to original source articles and technical reports for deeper research.
The author proposes a 5-layer framework to standardize "harness engineering":
1. **Constraint (Architecture):** Deterministic rules (linters, API contracts).
2. **Context (Dev):** Memory and knowledge injection.
3. **Execution (Platform):** Tool orchestration and sandboxing.
4. **Verification (Dev/QA):** Output validation and error loops.
5. **Lifecycle (SRE):** Monitoring, cost tracking, and recovery.
**Strategic Insight:** While platforms like Anthropic are increasingly absorbing the Context, Execution, and Lifecycle layers, developers must still own **Constraint** and **Verification**. To maximize efficiency on managed platforms, teams should prioritize deterministic constraints (Layer 1) to reduce token waste and improve reliability.
1. **Constraint (Architecture):** Deterministic rules (linters, API contracts).
2. **Context (Dev):** Memory and knowledge injection.
3. **Execution (Platform):** Tool orchestration and sandboxing.
4. **Verification (Dev/QA):** Output validation and error loops.
5. **Lifecycle (SRE):** Monitoring, cost tracking, and recovery.
**Strategic Insight:** While platforms like Anthropic are increasingly absorbing the Context, Execution, and Lifecycle layers, developers must still own **Constraint** and **Verification**. To maximize efficiency on managed platforms, teams should prioritize deterministic constraints (Layer 1) to reduce token waste and improve reliability.
Tansu, an open-source, Apache Kafka-compatible messaging broker, challenges traditional approaches by prioritizing statelessness. Instead of replicating data like Kafka, Tansu delegates durability to external storage, allowing for brokers that are lightweight ("cattle," not "pets") and scale rapidly. It supports various storage backends like S3, SQLite, and Postgres, with a particular emphasis on Postgres integration for streamlined data pipelines. Tansu also offers broker-side schema validation and the ability to directly write validated data to open table formats like Iceberg, Delta Lake, or Parquet. The project is written in Rust and seeks contributors.
Júlio Falbo argues that integrating AI into engineering organizations is hampered by complex connection methods, proposing a solution centered around “SKILL.md” – Markdown files defining tool usage – and “AI Gateways” for centralized orchestration. This combination fosters an “AI-native architecture” prioritizing ease of use, governance, and scalability over bespoke integrations. Ultimately, this approach shifts the focus from complex coding to clear documentation, democratizing AI tool access and boosting productivity.
* Simplifies AI integration via Markdown-based "skills."
* Utilizes AI Gateways for centralized control and security.
* Promotes a convention-over-configuration approach for AI systems.
* Simplifies AI integration via Markdown-based "skills."
* Utilizes AI Gateways for centralized control and security.
* Promotes a convention-over-configuration approach for AI systems.
Developers are replacing bloated MCP servers with Markdown skill files — cutting token costs by 100x. This article explores a two-layer architecture emerging in production AI systems, separating knowledge from execution. It details how skills (Markdown files) encode stable knowledge, while MCP servers handle runtime API interactions. The piece advocates for a layered approach to optimize context window usage, reduce costs, and improve agent reasoning by prioritizing knowledge representation in a version-controlled, accessible format.
This article explains the differences between Model Context Protocol (MCP), Retrieval-Augmented Generation (RAG), and AI Agents, highlighting that they solve different problems at different layers of the AI stack. It also covers how ChatGPT routes prompts and handles modes, agent skills, architectural concepts for developers, and service deployment strategies.
This article details research into finding the optimal architecture for small language models (70M parameters), exploring depth-width tradeoffs, comparing different architectures, and introducing Dhara-70M, a diffusion model offering 3.8x faster throughput with improved factuality.
LLMs are powerful for understanding user input and generating human‑like text, but they are not reliable arbiters of logic. A production‑grade system should:
- Isolate the LLM to language tasks only.
- Put all business rules and tool orchestration in deterministic code.
- Validate every step with automated tests and logging.
- Prefer local models for sensitive domains like healthcare.
| **Issue** | **What users observed** | **Common solutions** |
|-----------|------------------------|----------------------|
| **Hallucinations & false assumptions** | LLMs often answer without calling the required tool, e.g., claiming a doctor is unavailable when the calendar shows otherwise. | Move decision‑making out of the model. Let the code decide and use the LLM only for phrasing or clarification. |
| **Inconsistent tool usage** | Models agree to user requests, then later report the opposite (e.g., confirming an appointment but actually scheduling none). | Enforce deterministic tool calls first, then let the LLM format the result. Use “always‑call‑tool‑first” guards in the prompt. |
| **Privacy concerns** | Sending patient data to cloud APIs is risky. | Prefer self‑hosted/local models (e.g., LLaMA, Qwen) or keep all data on‑premises. |
| **Prompt brittleness** | Adding more rules can make prompts unstable; models still improvise. | Keep prompts short, give concrete examples, and test with a structured evaluation pipeline. |
| **Evaluation & monitoring** | Without systematic “evals,” failures go unnoticed. | Build automated test suites (e.g., with LangChain, LangGraph, or custom eval scripts) that verify correct tool calls and output formats. |
| **Workflow design** | Treat the LLM as a *translator* rather than a *decision engine*. | • Extract intent → produce a JSON/action spec → execute deterministic code → have the LLM produce a user‑friendly response. <br>• Cache common replies to avoid unnecessary model calls. |
| **Alternative UI** | Many suggest a simple button‑driven interface for scheduling. | Use the LLM only for natural‑language front‑end; the back‑end remains a conventional, rule‑based system. |
- Isolate the LLM to language tasks only.
- Put all business rules and tool orchestration in deterministic code.
- Validate every step with automated tests and logging.
- Prefer local models for sensitive domains like healthcare.
| **Issue** | **What users observed** | **Common solutions** |
|-----------|------------------------|----------------------|
| **Hallucinations & false assumptions** | LLMs often answer without calling the required tool, e.g., claiming a doctor is unavailable when the calendar shows otherwise. | Move decision‑making out of the model. Let the code decide and use the LLM only for phrasing or clarification. |
| **Inconsistent tool usage** | Models agree to user requests, then later report the opposite (e.g., confirming an appointment but actually scheduling none). | Enforce deterministic tool calls first, then let the LLM format the result. Use “always‑call‑tool‑first” guards in the prompt. |
| **Privacy concerns** | Sending patient data to cloud APIs is risky. | Prefer self‑hosted/local models (e.g., LLaMA, Qwen) or keep all data on‑premises. |
| **Prompt brittleness** | Adding more rules can make prompts unstable; models still improvise. | Keep prompts short, give concrete examples, and test with a structured evaluation pipeline. |
| **Evaluation & monitoring** | Without systematic “evals,” failures go unnoticed. | Build automated test suites (e.g., with LangChain, LangGraph, or custom eval scripts) that verify correct tool calls and output formats. |
| **Workflow design** | Treat the LLM as a *translator* rather than a *decision engine*. | • Extract intent → produce a JSON/action spec → execute deterministic code → have the LLM produce a user‑friendly response. <br>• Cache common replies to avoid unnecessary model calls. |
| **Alternative UI** | Many suggest a simple button‑driven interface for scheduling. | Use the LLM only for natural‑language front‑end; the back‑end remains a conventional, rule‑based system. |
The article provides practical advice for software architects on how to effectively communicate and deploy ideas through documentation. Key takeaways include:
1. **Focus on ideas, not code**: Architects must organize and deploy ideas to people, not just machines.
2. **Use bullet points**: They help structure information clearly and make documents easy to skim.
3. **Structure with headers**: Break content into sections for easy navigation and quick information retrieval.
4. **Write for the reader**: Prioritize clarity and relevance over perfect formatting or templates.
5. **Organize chronologically**: Group documents by time (year/sprint) rather than topic to improve searchability.
6. **Document types matter**: Specific document formats like architecture overviews, dev designs, and project proposals help manage complex projects.
7. **Keep documents concise and useful**: Aim for point-in-time documentation that remains useful even if outdated.
8. **Share and iterate**: Distribute documents widely and seek feedback to improve them.
1. **Focus on ideas, not code**: Architects must organize and deploy ideas to people, not just machines.
2. **Use bullet points**: They help structure information clearly and make documents easy to skim.
3. **Structure with headers**: Break content into sections for easy navigation and quick information retrieval.
4. **Write for the reader**: Prioritize clarity and relevance over perfect formatting or templates.
5. **Organize chronologically**: Group documents by time (year/sprint) rather than topic to improve searchability.
6. **Document types matter**: Specific document formats like architecture overviews, dev designs, and project proposals help manage complex projects.
7. **Keep documents concise and useful**: Aim for point-in-time documentation that remains useful even if outdated.
8. **Share and iterate**: Distribute documents widely and seek feedback to improve them.
