Architecture

Skills & tools layer

How OperationKit separates raw capability (tools), repeatable procedure (skills), and judgment (agents) — and why the separation matters for reliability at scale.

The three-question layering rule

Every file in ~/ai-workspace belongs to exactly one of three layers. The classification is not subjective — it follows from three questions asked in order. The first question that answers "yes" determines the layer.

Two shortcuts resolve most arguments before you even reach the questions:

• Only agents have a persona and the authority to refuse. If the file cannot decline the work, it is not an agent.
• Only tools can be described without naming a business process. If you cannot explain the file without saying "for a client" or "before launch," it is not a tool.

The dependency runs one way. Agents load skills; skills call tools. A tool never mentions an agent, and a skill never adopts a persona. This direction is enforced structurally: okit validate rejects any edge that runs against the grain.

Tools: raw capability

A tool is the thinnest possible adapter between OperationKit and a vendor. TOOL.md holds connection facts — the base URL, the auth method, the credential reference — and nothing else. It does not know why you would call the API. It does not name a business process. And critically, it names the environment variable that holds each secret, never the value.

auth:
  method: bearer-token
  secrets:
    - env: GHL_PIT_TOKEN
      source: /home/mike/projects/env-master/axia-platform.env
      required: true

That is enough to tell a newcomer exactly what they are missing and where it lives. It is useless to anyone who steals the repo. A value in this field is a policy violation — no schema can distinguish a real key from a placeholder, so the rule is enforced by convention and enforced by code review.

The layout mirrors skills/ on purpose: one main file, one optional references/ subdirectory for overflow, one optional scripts/ directory for runnable wrappers. Someone who can add a skill can add a tool without learning a second layout.

Why accumulated operational knowledge belongs in TOOL.md

Every integration accumulates friction that only shows up under load: an API that requires a browser User-Agent header to bypass Cloudflare, a rate-limit that resets on a UTC calendar boundary rather than a rolling 24-hour window, a pagination cap buried in a vendor changelog. This knowledge normally lives in someone's head, in a Slack thread, or in a post-mortem that no one reads before the next incident.

TOOL.md has a dedicated gotchas section for exactly this. Failure modes that have cost someone time are first-class citizens of the tool definition — not appendices to a skill, not comments in a script. When a new session picks up a tool, the friction map comes with it. The result is institutional memory that outlasts any individual contributor.

This design is consistent with how Toolformer (Schick et al., 2023) framed the problem: a model that learns to invoke APIs by teaching itself from demonstrations will inevitably encounter the same failure surfaces a human would — rate limits, malformed responses, authentication edge cases. Making those surfaces explicit in a structured file rather than expecting the model to rediscover them on each run is the difference between a tool that works once and a tool that works reliably.

MCP servers are tools

Model Context Protocol servers are wired as tools, not as skill prose. Before this convention, MCP install commands were scattered: skills/resend carried an npx -y resend-mcp invocation; skills/live-browser-testing carried a claude mcp add playwright line; skills/gitnexus named its mcp__gitnexus__* tool IDs inline. Each skill taught a newcomer to install a different server in a different way.

An MCP server is a capability with a connection string — question 1 answers yes. Putting the mcp block in TOOL.md frontmatter gives one canonical install command per server and lets an install-check read tools/*/TOOL.md alone to answer "which servers am I missing?" kind: mcp-server requires the mcp block; the schema enforces it.

Skills: repeatable procedure

A skill encodes a job — a sequence of steps that, when followed, produce a specific deliverable with a checkable done-condition. It loads tools; it may require other skills. It has no persona and cannot refuse work. If it needs judgment — whether to run at all, whether to escalate — that judgment belongs in the agent that loads it.

Skills declare their tool dependencies in frontmatter:

name: ghl-lead-loader
description: Upsert contacts, create opportunities, assign an owner. Re-runnable.
tools: [gohighlevel, supabase, trellus, n8n, google-workspace]
requires_skills: [campaign-setup]

tools: lists only what this skill's own procedure invokes. A router skill that hands off to a sibling does not inherit the sibling's tools — it names the sibling under requires_skills: and lets the sibling declare its own edges. This keeps the graph honest: every edge reflects an actual call, not a transitive dependency that might be refactored away.

The sub-skill model

A skill that is genuinely one job but has a distinct, separately-triggerable step inside it can put that step in a sub-skill at skills/<parent>/<child>/SKILL.md. Sub-skills are first-class graph nodes with their own frontmatter, their own tools: edges, and their own registry entries. Depth 2 is the maximum. A SKILL.md at depth 3 or deeper is a hard okit validate failure — if a step needs its own steps, it is a top-level skill.

The bidirectional enforcement matters: a parent that omits an on-disk child fails, and a child that names the wrong parent fails. A half-declared tree reads as real while being wrong, which is worse than a failure.

# Scaffold both ends in one command — patches parent's subskills: automatically
okit new skill marketing/pre-call-hammer          # compound name
okit new skill pre-call-hammer --parent marketing # identical, flag spelling

Why declared edges rather than RAG retrieval

The question is why skills are loaded by explicit declaration — an agent's frontmatter listing the skills it always loads and the skills it can invoke — rather than by semantic similarity search over a skill corpus. The answer is failure mode.

A RAG retrieval that returns a topically close but procedurally wrong skill will cause the agent to attempt a real job using the wrong procedure. The agent will be confident; the context is internally consistent. There is no signal that anything is wrong until the deliverable fails review. ToolLLM (Qin et al., 2023), which studied LLM tool selection across 16,000+ real APIs, found that models confidently invoke APIs with subtly wrong parameters when the retrieval surface is large — the confidence is a property of the context, not a signal of correctness.

A confidently called skill that doesn't exist is a hallucination at the procedure level. Explicit declaration moves this failure from runtime — where it produces a plausible but wrong output — to authoring time, where the graph check catches it before the agent session starts.

Graph validation: okit validate

The graph check is the quality gate for the entire layer system. okit validate traverses the full Tools → Skills → Agents dependency graph and fails hard on any dangling edge:

• A skill that declares a tool not in tools/registry.json
• A skill that requires another skill that doesn't exist on disk
• An agent that loads a skill that doesn't exist
• A sub-skill whose parent: field names the wrong directory
• A parent skill whose subskills: list omits an on-disk child
• A sub-skill at depth 3 or deeper

Orphans — tools no skill uses, skills no agent loads — are recorded and reported but do not fail the check. They may be available for future use, or they may be candidates for deletion. The distinction is intentional: a missing dependency is an error; an unused capability is a warning.

The type-system analogy

The analogy is exact. TypeScript catches a reference to an undefined function before the code runs. okit validate catches a reference to an undefined capability before the agent session starts. In both cases the alternative — discovering the error at runtime — is strictly worse: TypeScript catches it at compile time rather than production; okit validate catches it at commit time rather than mid-task.

Sumers et al. (2024) argue in Cognitive Architectures for Language Agents that agents operating on consistent, structured knowledge representations outperform those reasoning over unstructured or contradictory context. The graph enforces structural consistency automatically, and the CI check runs it on every commit that touches agents/, skills/, or tools/.

HuggingGPT (Shen et al., 2023) demonstrated task planning over a large model zoo by first resolving which models exist and what they accept before committing to a plan. The key insight was that a planning step that assumes a capability it cannot verify is a reliability liability. okit validate makes that pre-flight check automatic and mandatory — not a planning habit but a structural gate.

Authoring CLI: okit

New tools, skills, and agents are scaffolded through okit rather than hand-written from scratch. The commands are symmetric:

okit new tool  porkbun           # scaffolds tools/porkbun/TOOL.md
okit new skill domain-management  # scaffolds skills/domain-management/SKILL.md
okit new agent coo               # scaffolds agents/coo.md + sidecar
okit new skill marketing/pre-call-hammer  # sub-skill: patches parent too
okit validate                    # full graph check, exits non-zero on failure
okit inventory --json            # machine-readable registry dump

The scaffold writes the required frontmatter fields, sets sane defaults, and — for sub-skills — patches the parent's subskills: list so the graph is valid the instant the file exists. Writing a sub-skill by hand and forgetting to update the parent is the single most common authoring error; the CLI makes it structurally impossible.

Why a CLI instead of a template and a README

The alternative is a template file and a documentation page that explains how to fill it in. This is the conventional approach, and it has a known failure mode: schema enforcement happens at review time, not creation time. A reviewer reading a hand-written TOOL.md must mentally validate the frontmatter against the schema; okit validate does this mechanically and fails fast.

The schemas themselves — docs/schemas/{tool,skill,agent}.schema.json — are JSON Schema documents. The CLI validates against them at scaffold time; CI validates against them at commit time; the runtime validates against them at session-spawn time. Three independent checks, same schema. A file that reaches a session is guaranteed to be structurally valid.

The authoring path is: okit new <type> <slug> → fill in the scaffolded content → okit validate → commit. The validate step is optional only in the sense that CI will catch it anyway — the convention is to run it locally before pushing so the CI result is not the first signal of an error.

References

The design decisions in the Skills and Tools layer are grounded in the empirical literature on tool-augmented language models. Each citation below explains which specific finding informed a specific design choice.

• Toolformer: Language Models Can Teach Themselves to Use Tools

  Schick, T., Dwivedi-Kumar, S., Dessì, R., Raileanu, R., Lomeli, M., Hambro, E., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023). Advances in Neural Information Processing Systems, 36.

  Why it matters here: Toolformer showed that models encounter the same failure surfaces — rate limits, auth edge cases, malformed responses — regardless of how they learned to call the API. Making those surfaces explicit in a structured TOOL.md gotchas section is the architectural response: failure modes are first-class citizens of the tool definition, not tribal knowledge that gets rediscovered on each run.

  arxiv.org/abs/2302.04761 ↗

• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs

  Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., Lin, Y., Cong, X., Tang, X., Qian, B., Zhao, S., Tian, R., Xie, R., Zhou, J., Gerstein, M. H., Li, D., Liu, Z., & Sun, M. (2023). arXiv preprint.

  Why it matters here: Across a 16,000+ API surface, ToolLLM found that models confidently invoke APIs with subtly wrong parameters when the retrieval surface is large — the confidence is a property of the context, not a signal of correctness. This directly motivates replacing retrieval with explicit declaration: a skill that is not declared cannot be mis-retrieved. The graph becomes the index, and the graph is validated.

  arxiv.org/abs/2307.16789 ↗

• Gorilla: Large Language Model Connected with Massive APIs

  Patil, S. G., Zhang, T., Wang, X., & Gonzalez, J. E. (2023). arXiv preprint.

  Why it matters here: Gorilla demonstrated that retrieval-augmented tool selection reduces hallucinated API calls compared to in-context-only selection. OperationKit takes this further: the retrieval index is replaced by a validated graph. A skill that exists on disk but is not declared by any agent is an orphan — it cannot be mis-retrieved because it is not in the selection pool. No silent wrong-tool invocation is possible.

  arxiv.org/abs/2305.15334 ↗

• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace

  Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023). Advances in Neural Information Processing Systems, 36.

  Why it matters here: HuggingGPT's key reliability insight was task planning over a resolved model zoo — first confirm which models exist and what they accept, then commit to a plan. The same insight underlies okit validate: a planning step that assumes a capability it cannot verify is a reliability liability. The graph check makes that pre-flight verification automatic and mandatory rather than a discipline that each agent session must remember to apply.

  arxiv.org/abs/2303.17580 ↗

• Cognitive Architectures for Language Agents

  Sumers, T. R., Yao, S., Narasimhan, K., & Griffiths, T. L. (2024). Transactions on Machine Learning Research.

  Why it matters here: Sumers et al. map agent architectures onto cognitive science frameworks, distinguishing declarative knowledge (facts about the world), procedural knowledge (how to act), and deliberative control (which action to take when). The three-layer graph maps directly: tools are declarative facts (API endpoints, credential references), skills are procedural memory (how to execute a job), agents are the deliberative control layer (which skill to run and whether to run it). The finding that agents operating on consistent, structured representations outperform those reasoning over unstructured context is the empirical justification for enforcing that structure with a graph check rather than relying on prompt discipline.

  arxiv.org/abs/2309.02427 ↗