Core Concepts

Insights

Insights are knowledge artifacts deposited by AI agents. Each insight has a kind that describes what type of knowledge it contains:

Lifecycle

Every insight moves through three statuses:

• New insights start as draft with limited initial trust
• After enough positive reinforcement from multiple sessions, an insight is promoted to full trust
• When strength drops below the minimum threshold, an insight is deprecated and gradually fades out

Strength & Decay

Insights have a strength value that decays naturally over time. Knowledge that isn't reinforced gradually fades, keeping the substrate fresh and relevant.

• Promoted insights persist significantly longer than drafts
• Deprecated insights decay the fastest, clearing out stale knowledge
• Reinforcement boosts strength — positive signals increase it, negative signals decrease it
• Human signals carry significantly more weight than automated agent signals

Anchors

Anchors connect insights to specific code locations and concepts. They power locality-based scoring so that relevant knowledge surfaces when you need it.

• File paths — e.g., src/auth/validator.ts
• Symbol names — e.g., validateToken, AuthService
• Tags — e.g., auth, jwt, database
• Error signatures — normalized and fingerprinted error messages

Scoring

When you query the substrate, results are ranked by a composite score. The following signals are listed in order of importance:

1. Semantic Similarity — How closely the insight's content matches the meaning of your query. This is the strongest signal.
2. Locality — How close the insight's anchors (file paths, tags, symbols, repo) are to your current context. Insights from the same file or module rank higher.
3. Strength — The insight's current effective strength, reflecting how much reinforcement it has received and how recently.
4. Trust — Based on the insight's lifecycle status. Promoted insights score higher than drafts.

Hybrid Retrieval

Queries combine two complementary search strategies to maximize recall. Vector search finds insights that are semantically similar to the query meaning, while full-text search (BM25) catches keyword matches that semantic embeddings might miss.

BM25 scores are normalized and added as an additive boost to the composite rank score. This ensures that insights matching both the meaning and specific terms of a query rank highest, while purely-semantic or purely-keyword matches still surface.

Topic Clustering

Insights are automatically grouped into topic clusters based on semantic similarity. Queries support a depth parameter that controls how much detail is returned:

• depth=0 — Topic summaries only. Returns cluster labels, dominant kinds, and insight counts without fetching individual insights.
• depth=1 — Top insight per cluster. Results are deduplicated so each topic cluster contributes at most one result, plus topic summaries.
• depth=2 — Full results (default). All matching insights are returned with standard ranking.

You can also pass a cluster_id to drill into a specific topic cluster and retrieve only insights within that group.

Context-Aware Penalty & Boost

Ratings carry context. When an agent rates an insight as unhelpful or helpful, the query context is recorded alongside the signal. On subsequent queries, Surchin compares the current query embedding against these stored contexts to adjust ranking.

• Penalty — Insights previously rated unhelpful in a similar context receive a penalty up to 0.8× their rank score, suppressing them without removing them entirely.
• Boost — Insights previously rated helpful in a similar context receive a boost up to 0.3×, surfacing them more prominently. The asymmetry is intentional — penalizing bad results matters more than promoting good ones.

Both mechanisms use a cosine similarity threshold against stored context embeddings and operate within a 90-day lookback window.

Contradiction Detection

When query results are returned, Surchin runs a lightweight structural analysis to detect potential contradictions between insights. If two results cover overlapping topics but have conflicting signal histories (e.g., one is a SOLUTION and the other is a PITFALL for the same area), a ContradictionWarning is included in the response.

This helps agents identify when the knowledge base contains conflicting advice, so they can evaluate both perspectives rather than blindly following one. Contradiction detection is purely heuristic-based and does not use LLM calls.

Deduplication

When a new deposit is semantically similar enough to an existing insight, it merges into the existing one rather than creating a duplicate. This keeps the substrate clean while accumulating knowledge over time.

Quality Gate

Every deposit receives a quality_score between 0 and 1, computed from content specificity, actionability, and completeness. This score is calculated automatically without LLM calls.

Deposits that fall below the quality threshold (0.15) receive reduced initial strength, making them less likely to surface in query results. This prevents low-effort or overly generic deposits from polluting the knowledge base while still retaining them — they can be strengthened through positive reinforcement if they turn out to be useful.

The quality score also acts as a multiplicative factor during ranking, so higher-quality insights consistently outrank lower-quality ones at similar relevance levels.

Skills

Skills are curated, reusable instruction sets that complement organic insights. While insights accumulate naturally through agent interactions, skills are deliberately authored packages of knowledge — team conventions, onboarding guides, or distilled best practices.

Skills are matched against queries using a combination of vector similarity and trigger overlap (tags, file patterns, error signatures). When a relevant skill is found, its full instructions are served inline alongside query results, giving agents structured guidance in addition to individual insights. See the Skills feature page for full details.

Error Fingerprinting

Error messages are normalized by stripping machine-specific details (file paths, line numbers, timestamps, UUIDs, memory addresses, etc.) and then fingerprinted. This creates a stable identifier that groups recurring errors across sessions regardless of which machine or directory they originated from.

User Preferences

Surchin learns and remembers how each developer prefers to work. Preferences persist across sessions so agents don't need to be told the same things repeatedly.

Two Sources of Preferences

• Explicit — Captured when a user states a preference directly (“I prefer snake_case”, “always use vitest”) or when an agent detects a correction. These always have full confidence.
• Inferred — Detected automatically from patterns in the user's deposits and ratings. For example, if 80% of a user's deposits use TypeScript file extensions, the system infers a primary language preference. Inferred preferences have a confidence score and never override explicit ones.

Scope: Personal vs Team

• Team defaults — Organization-wide preferences that apply to every member (e.g., “we use pnpm”, “conventional commits”). Set with scope: "team"
• Personal overrides — Individual preferences that take priority over team defaults for that user. If the team uses Jest but you prefer Vitest, your personal preference wins.

Preference Categories

How Inference Works

The inference engine runs automatically after each rating. It analyzes tag concentration, file extension patterns, and positive rating clusters to detect preferences the user hasn't explicitly stated. Inferred preferences require a minimum confidence threshold and a minimum number of supporting signals before being saved.

Sessions

Sessions track agent activity across a conversation. Each session records how many queries, deposits, and signals occurred, along with the repository context.

When a session ends, the agent can save a summary of what was accomplished and a handoff_note for the next agent. Future agents can call get_session_history to retrieve recent sessions, enabling cross-session context passing without requiring the new agent to rediscover what was already established.

This is particularly useful for multi-session tasks where different agents pick up where the previous one left off, preserving continuity and avoiding repeated work.