How to Build a Codebase Knowledge Layer for AI
Why Raw File Access Is Not Enough
AI coding assistants can read files, but reading individual files gives them a keyhole view of the codebase. An assistant editing a function in one file does not automatically know which other files import that function, which tests cover it, whether similar functions exist elsewhere, or whether the module it belongs to has specific conventions that differ from the rest of the codebase. Getting this information requires reading potentially dozens of files, which is slow and consumes context window tokens.
A knowledge layer pre-computes these relationships and serves them on demand. When the assistant needs to modify a function, the knowledge layer tells it the callers, the tests, the module conventions, and the recent change history in a single query. This is faster, uses fewer tokens, and provides more complete context than having the assistant explore the codebase file by file.
Step-by-Step Build Process
Parse your codebase using tree-sitter or language server protocols to extract the structural skeleton: function signatures, class definitions, type declarations, module exports, and import statements. Tree-sitter supports most popular languages and provides consistent AST access across language boundaries.
import tree_sitter_python as tspython
from tree_sitter import Language, Parser
PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)
def extract_functions(filepath, source_code):
tree = parser.parse(bytes(source_code, "utf-8"))
functions = []
for node in tree.root_node.children:
if node.type == "function_definition":
name_node = node.child_by_field_name("name")
params_node = node.child_by_field_name("parameters")
return_node = node.child_by_field_name("return_type")
functions.append({
"file": filepath,
"name": name_node.text.decode(),
"params": params_node.text.decode(),
"return_type": return_node.text.decode()
if return_node else None,
"line": node.start_point[0] + 1,
"end_line": node.end_point[0] + 1
})
return functionsRun this extraction across your entire codebase. For a 100,000-line Python project, tree-sitter parsing takes under 30 seconds. The result is a structured catalog of every function, class, and module in the codebase with their signatures, locations, and boundaries.
Analyze import statements and function calls to map how modules depend on each other. This graph tells the assistant which other parts of the codebase are affected by a change, which modules are tightly coupled, and which functions serve as entry points versus internal implementation details.
import ast
from collections import defaultdict
def build_import_graph(files):
graph = defaultdict(set)
for filepath, source in files.items():
try:
tree = ast.parse(source)
except SyntaxError:
continue
for node in ast.walk(tree):
if isinstance(node, ast.Import):
for alias in node.names:
graph[filepath].add(alias.name)
elif isinstance(node, ast.ImportFrom):
if node.module:
graph[filepath].add(node.module)
return dict(graph)
def find_dependents(graph, module_name):
"""Find all files that import a given module."""
dependents = []
for filepath, imports in graph.items():
if module_name in imports:
dependents.append(filepath)
return dependentsThe dependency graph is the most immediately useful part of the knowledge layer. When an assistant is about to modify a function's signature, querying the dependency graph reveals every caller that will need to be updated. Without the graph, the assistant either misses callers (introducing bugs) or searches the entire codebase (wasting time and tokens).
Embed function and module descriptions into vectors so the assistant can search by functionality rather than by name. Extract a short description of each function (using the docstring, the function name, and the parameter names) and embed it. Store the embeddings in a vector database alongside the structural metadata.
from anthropic import Anthropic
import json
client = Anthropic()
def describe_function(func_metadata, source_lines):
"""Generate a semantic description for embedding."""
code_snippet = "\n".join(
source_lines[func_metadata["line"]-1:
func_metadata["end_line"]])
response = client.messages.create(
model="claude-haiku-4-5-20251001",
max_tokens=200,
messages=[{"role": "user", "content":
f"Describe what this function does in one sentence:"
f"\n\n{code_snippet}"}]
)
return response.content[0].text
def embed_descriptions(descriptions):
"""Create vector embeddings for semantic search."""
# Use your preferred embedding model
# Store vectors alongside function metadata
# in your vector database
passSemantic embeddings enable queries like "find the code that handles user authentication" or "which function validates payment amounts" even when the code does not use those exact words. This is particularly valuable in large codebases where function names follow internal conventions that do not match how developers naturally describe functionality.
Integrate git history to enrich the structural index with temporal information. Track which files change together (co-change analysis), which areas have high commit frequency (hotspots), which code was recently modified, and which functions have been involved in bug fixes.
import subprocess
def get_file_history(filepath, max_commits=50):
result = subprocess.run(
["git", "log", f"--max-count={max_commits}",
"--pretty=format:%H|%ai|%s", "--", filepath],
capture_output=True, text=True)
commits = []
for line in result.stdout.strip().split("\n"):
if "|" in line:
hash, date, message = line.split("|", 2)
commits.append({
"hash": hash, "date": date,
"message": message,
"is_bugfix": "fix" in message.lower()
})
return commits
def find_co_changed_files(filepath, max_commits=100):
"""Files that frequently change alongside this file."""
result = subprocess.run(
["git", "log", f"--max-count={max_commits}",
"--pretty=format:%H", "--", filepath],
capture_output=True, text=True)
co_changes = defaultdict(int)
for commit_hash in result.stdout.strip().split("\n"):
files_in_commit = subprocess.run(
["git", "diff-tree", "--no-commit-id",
"--name-only", "-r", commit_hash],
capture_output=True, text=True)
for f in files_in_commit.stdout.strip().split("\n"):
if f != filepath:
co_changes[f] += 1
return dict(sorted(co_changes.items(),
key=lambda x: -x[1])[:10])Historical context helps the assistant anticipate ripple effects. If file A and file B always change together, modifying A without checking B is likely a mistake. If a function has been involved in three bug fixes in the last month, it deserves extra care when modified. This temporal awareness is something neither static rules files nor accumulated memories typically capture.
Expose the knowledge layer through MCP tools so any compatible coding assistant can query it. Define tools for structural queries (find callers of a function, list module exports), semantic queries (find code related to a description), and historical queries (show recent changes, list hotspots).
The MCP interface makes the knowledge layer tool-agnostic. Claude Code, Cursor, and any MCP-compatible assistant can query the same knowledge layer. The assistant calls the tool when it needs structural context, receives the relevant information, and uses it to inform its code changes.
Set up incremental indexing that triggers on file saves or git commits. When a file changes, re-parse only that file and update its structural metadata, re-embed its descriptions, and update its dependency edges. Full re-indexing should run nightly or weekly as a consistency check, but incremental updates keep the knowledge layer current within minutes of each change.
Practical Considerations
The initial indexing cost scales with codebase size. A 50,000-line project takes minutes to fully index (parsing is fast, LLM description generation is the bottleneck). A 500,000-line project may take an hour or more if generating semantic descriptions for every function. For large codebases, start with structural parsing and dependency graphs (which are cheap) and add semantic embeddings incrementally, prioritizing the most frequently modified code first.
Storage requirements are modest. Structural metadata for a large codebase fits in a few megabytes. Semantic embeddings add more (1536-dimension vectors at 4 bytes per dimension for each function), but even 100,000 functions require only about 600 MB of vector storage. Git history analysis generates small JSON files. The total infrastructure footprint is a lightweight database or file system, not a major deployment.
For teams that want the benefits of a knowledge layer without building one, Adaptive Recall provides the equivalent through its memory system. When you store codebase observations and architectural patterns through the MCP tools, the system builds an entity graph of your codebase concepts and uses spreading activation to surface structurally related knowledge during retrieval. It does not replace a full codebase parser, but it provides the contextual awareness benefits without the indexing infrastructure.
Get codebase awareness without building indexing infrastructure. Adaptive Recall builds knowledge graphs from your observations and uses cognitive scoring to surface the right context every time.
Get Started Free