Do Knowledge Graphs Work with Streaming Data

How Streaming Graph Construction Works

Streaming graph construction processes data as it arrives rather than in bulk. Each new piece of data (a chat message, a support ticket, a log entry, an API event) is analyzed for entities and relationships, and the graph is updated with whatever new information is found. This keeps the graph current with the latest information rather than waiting for a batch processing cycle.

The pipeline for streaming graph updates has five stages: receive the new data, extract entities and relationships, resolve extracted entities against existing graph nodes, detect and handle contradictions with existing triples, and persist the updates. Each stage adds latency, so the total time from data arrival to graph update is typically 1 to 5 seconds depending on the extraction method used.

The Context Problem

Entity extraction works best when it has enough context to understand what entities are being discussed and how they relate. A full document provides rich context: pronouns can be resolved ("it" refers to the service mentioned in the previous paragraph), abbreviations are defined (introduced earlier in the text), and relationships span multiple sentences. A single streaming message ("Redis is down again, can someone check the session service?") provides minimal context.

The result is noisier extraction from streaming data. Entity types may be guessed incorrectly. Relationships may be inferred rather than stated. Pronouns cannot be resolved without prior context. Three strategies help:

Micro-batching: Buffer incoming messages in windows of 10 to 50 and extract from the batch. This provides enough context for the LLM to understand the conversation flow, resolve pronouns, and identify relationships that span multiple messages. The trade-off is a delay (seconds to a few minutes) between data arrival and graph update.

Conversation threading: When streaming data comes from conversations (chat, support tickets), attach previous messages in the thread as context for extraction. The LLM can then understand references to earlier messages and extract entities more accurately.

Lower confidence thresholds: Assign lower initial confidence to entities and relationships extracted from streaming data compared to those extracted from documents. This reflects the higher noise rate and prevents low-quality extractions from influencing retrieval as strongly as well-established facts. As the same entity or relationship appears in multiple streaming messages, confidence increases through corroboration.

Entity Resolution in Real Time

Entity resolution (matching extracted entities to existing graph nodes) is more challenging in streaming contexts because new entities arrive continuously and must be matched against the entire existing graph. Batch processing can afford expensive pairwise comparisons because it runs infrequently. Streaming requires fast matching on every new entity.

The practical approach is a tiered resolution strategy: first check for exact name match (microseconds), then check the alias index (milliseconds), then fall back to fuzzy matching only if the first two miss (tens of milliseconds). LLM-based disambiguation is reserved for entities flagged as ambiguous and processed asynchronously, not blocking the streaming pipeline.

Handling Contradictions

Streaming data frequently introduces contradictions with existing graph state. A chat message says "we switched to Braintree for payments" while the graph says the payments service uses Stripe. In batch processing, you can resolve contradictions after the fact. In streaming, you need a policy that applies immediately: keep both triples with reduced confidence on the older one, flag for human review, or defer resolution to the next batch cycle.

The safest streaming policy is to never delete or overwrite existing triples in real time. Instead, add the new triple at moderate confidence and reduce the confidence of the conflicting triple. The consolidation process (which runs periodically, not in real time) handles the actual resolution after enough evidence has accumulated.

Adaptive Recall's Streaming Model

Adaptive Recall handles streaming data naturally through its memory storage API. Each call to the store tool adds a new memory, extracts entities, and updates the knowledge graph incrementally. Entity resolution runs against the existing graph using fast alias matching. Contradictions are handled through the confidence system, with new information starting at moderate confidence and accumulating support over time. The consolidation process runs periodically to resolve contradictions, merge related memories, and clean up the graph. This gives you the benefits of streaming graph updates without building the extraction pipeline, resolution logic, or contradiction handling yourself.