Architecture
Context & knowledge management
How OperationKit decides what goes in the context window — and why that decision is the primary quality lever in a long-running agent system.
The problem with naive context
Every agent system eventually faces the same question: the model has a finite context window, the relevant knowledge is not finite, and blindly concatenating everything produces worse results than carefully selecting a subset.
This is not a conjecture. Liu et al. (2023) measured it directly: when relevant information is placed in the middle of a long context, model performance degrades substantially compared to placing it at the beginning or end — a finding they call the "lost in the middle" phenomenon. The implication is that more context is not always better context, and position within the window matters as much as presence.
OperationKit's context system is built around three principles that follow from this:
- Signal before noise. High-confidence context (the agent's own identity, the workspace's business rules) loads before lower-confidence context (retrieved KB snippets). The model sees the most important framing first.
- Curation over recall. Every context tier is authored and validated by a human or by an enforced schema — not scraped by a similarity search that can silently return irrelevant results.
- Separation of memory types. Working memory (the current objective), long-term memory (the vault), and procedural memory (skills) are stored and retrieved differently, matching what cognitive science tells us about human memory architecture.
Four-tier instruction stack
When OperationKit spawns a session, it assembles a context from four tiers loaded in order, each narrowing the agent's frame of reference. Tiers load from most-general to most-specific so that a later tier can override a general rule without conflict.
~/.claude/CLAUDE.md
agents/<name>.md
workspaces/<ws>/context.md
users/<id>/profile.md
Beneath the four tiers, dynamic context is injected at session spawn:
- Objective memory — the persistent
NOTES.mdfrom prior sessions on this same task. - Vault KB hits — semantically relevant decisions, research, and patterns retrieved from the knowledge base.
- Recurring failure modes — platform-wide gotchas extracted from reviewer verdicts over the prior 90 days.
- Recently modified files — files changed by other sessions in the last 72 hours, preventing collisions on shared work.
The ordering is deliberate. Stable, high-confidence context (T1–T4) comes before retrieved, potentially stale context (KB hits, failure modes). This front-loads the signal and reduces the chance that a retrieved snippet contradicts a stable instruction at a position the model will attend to less carefully.
Memory model
Park et al. (2023) identified three memory stores in their Generative Agents architecture: in-context (what the agent can see right now), external (a queryable store), and parametric (baked into the model weights). OperationKit maps all three explicitly:
NOTES.md is the agent's scratch pad; ARTIFACT.md is the deliverable record. Both survive backend restarts and are injected into the next session's context automatically.objective: field that ties them back to the board, creating a bidirectional audit trail between what was decided and what produced it.SKILL.md files — repeatable procedures with declared tool dependencies and checkable done-conditions. Skills are loaded by agents, not by retrieval; they are explicit, not inferred.TOOL.md files are vendor-scoped and rewritten when the vendor changes, not when the business process changes.Packer et al. (2023) showed in MemGPT that treating the context window like an OS's working memory — and paging knowledge in and out from external storage — can give an LLM effectively unbounded recall without degrading performance by flooding the window. OperationKit implements a simpler version of this: the vault acts as the backing store, and the session-manager decides what pages in (KB hits, objective memory) and what stays on disk (the full decision history).
Deterministic assembly vs retrieval-augmented generation
Retrieval-Augmented Generation (Lewis et al., 2020) is the dominant approach to grounding language models in external knowledge: run a similarity search over a document store, take the top-k results, and prepend them to the prompt. This is powerful for open-domain question answering where the relevant documents can be identified by semantic proximity to the query.
OperationKit uses a deterministic assembly model for its primary context tiers, and targeted retrieval only for the dynamic KB injection. The distinction matters:
| Property | Pure RAG | OperationKit deterministic assembly |
|---|---|---|
| Selection mechanism | Cosine similarity to the query | Declared edges in YAML frontmatter, validated by okit validate |
| Failure mode | Silent relevance drift; a topically close but contextually wrong document outranks the right one | Dangling edge is a hard validation failure; nothing silent |
| Freshness | Depends on re-indexing cadence | File is the source; no index to desync |
| Scope | Works well for large, heterogeneous corpora | Works well for curated, workspace-scoped knowledge |
| Explainability | Hard to audit why a document was retrieved | Every context item has a declared reason for being there |
The tradeoff is recall vs precision. Pure RAG recovers more knowledge from a large corpus; deterministic assembly delivers more reliable signal from a curated one. OperationKit bets on curation: the skills graph and workspace contexts are authored artifacts that agents and humans maintain, which is more expensive to write but produces fewer hallucinations about what tools and procedures are actually available.
The KB hits injected from the vault (the dynamic layer below T4) are the one place where retrieval runs — but they land after the stable tiers, so a misretrieved snippet cannot override a validated instruction.
Graph consistency and knowledge validity
One underappreciated failure mode in agent systems is the agent confidently referencing a tool, skill, or procedure that doesn't exist or has changed. The context window makes the model's behavior consistent with what's in context — so if the context contains a stale reference, the agent will act on it.
OperationKit addresses this with okit validate, a CLI tool that traverses the full Tools → Skills → Agents dependency graph and fails hard on any dangling edge: a skill that declares a tool not in tools/registry.json, an agent that loads a skill that doesn't exist on disk, or a sub-skill whose parent doesn't declare it. Orphans (tools no skill uses, skills no agent loads) are recorded but not failed — they may be available for future use.
The analogy is a type system for knowledge. Just as a TypeScript compiler catches a reference to an undefined function before the code runs, okit validate catches a reference to an undefined capability before the agent session starts. The graph check runs in CI on every commit that touches agents/, skills/, or tools/.
This connects to a finding from Sumers et al. (2024) in Cognitive Architectures for Language Agents: agents that operate on a consistent, structured knowledge representation outperform those that reason over unstructured or contradictory context. The graph enforces that consistency automatically.
The reviewer loop as verbal reinforcement
Context management is not only about what goes into the window at the start of a session. It's also about what feedback comes back after a session completes, and how that feedback shapes the next session's context.
Shinn et al. (2023) introduced Reflexion: rather than using gradient descent to update model weights from feedback, agents evaluate their own outputs in natural language and store that evaluation in an "episodic memory buffer" — which is then prepended to the next attempt. This produces dramatic improvements on coding and decision-making benchmarks without any fine-tuning.
OperationKit implements a structural version of this. When a working session completes, an autonomous reviewer session runs independently and grades the output. A fail verdict reverts the objective to working and prepends the reviewer's findings to the next session's context. A pass advances to the human review gate.
The critical property is that the reviewer's findings become part of the next session's context automatically — the session-manager injects them. The agent doesn't need to remember its own failure; the harness guarantees the failure is visible. This avoids the common failure mode where a retry session has no knowledge of what went wrong in the prior attempt and repeats the same mistake.
The reviewer is an independent Claude session, not a self-evaluation by the working agent. This matters: a model evaluating its own output is susceptible to the same blind spots that produced the error. An independent session with a fresh context is more likely to catch errors the working session normalized.
References
The engineering decisions above are grounded in published research. The citations below point to the specific papers that informed each design choice.
-
Lost in the Middle: How Language Models Use Long Contexts
Why it matters here: Directly motivates front-loading high-confidence tiers (T1–T4) before retrieved context. Documents the "lost in the middle" degradation that makes position-aware context ordering — not just content selection — a first-class concern.
arxiv.org/abs/2307.03172 ↗ -
Generative Agents: Interactive Simulacra of Human Behavior
Why it matters here: Source of the three-memory taxonomy (in-context, external, parametric) that OperationKit maps to objective memory, vault, and skills. Also introduces recency × importance × relevance as retrieval signals — a framework the vault's KB search partially implements.
arxiv.org/abs/2304.03442 ↗ -
MemGPT: Towards LLMs as Operating Systems
Why it matters here: Establishes the analogy between OS virtual memory and context management: the context window is working memory; external storage pages in and out. OperationKit's split between the objective-memory scratchpad (hot) and the vault (cold) follows this paging model.
arxiv.org/abs/2310.08560 ↗ -
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Why it matters here: The foundational RAG paper — the baseline that OperationKit's deterministic assembly explicitly contrasts with. Understanding where RAG excels (large heterogeneous corpora, open-domain recall) clarifies why a curated graph approach is preferable for workspace-scoped procedural knowledge.
arxiv.org/abs/2005.11401 ↗ -
Reflexion: Language Agents with Verbal Reinforcement Learning
Why it matters here: Demonstrates that injecting a natural-language critique of a prior attempt into the next session's context outperforms naive retry without any weight updates. Directly informs OperationKit's reviewer loop: reviewer findings are prepended to the working session's next spawn context.
arxiv.org/abs/2303.11366 ↗ -
Cognitive Architectures for Language Agents
Why it matters here: Maps agent architectures onto cognitive science frameworks — distinguishing in-context vs external vs parametric storage, procedural vs declarative knowledge, and reactive vs deliberative control. The Tools/Skills/Agents three-layer graph maps onto this taxonomy: tools are declarative facts, skills are procedural memory, agents are the deliberative control layer.
arxiv.org/abs/2309.02427 ↗ -
A Survey on Large Language Model based Autonomous Agents
Why it matters here: Comprehensive survey of memory, planning, and action in LLM agents. Validates OperationKit's multi-tier memory design and documents evidence that agents with persistent external memory consistently outperform stateless agents on multi-step tasks — the primary motivation for the vault and objective-memory system.
arxiv.org/abs/2308.11432 ↗