How to Validate AI Tool Output Before Execution
Before You Start
You need a working function calling implementation and a clear understanding of what each tool does, what side effects it has, and what constitutes a valid vs. invalid call for each tool. Review the schema design guide to make sure your schemas are well-constrained, because good schemas reduce the validation burden by preventing many invalid calls at the model level.
Step-by-Step Implementation
Before executing any tool, validate the model's generated arguments against the tool's JSON schema. Check that all required parameters are present, that parameter types match (string is actually a string, integer is actually an integer), that enum values are from the defined set, that numeric values are within min/max bounds, and that no unexpected parameters are included. Most JSON schema libraries provide validation functions that handle this automatically.
import jsonschema
def validate_tool_call(tool_name, arguments, tool_definitions):
tool_def = next(t for t in tool_definitions if t["name"] == tool_name)
schema = tool_def["input_schema"]
try:
jsonschema.validate(instance=arguments, schema=schema)
return {"valid": True}
except jsonschema.ValidationError as e:
return {
"valid": False,
"error": f"Invalid arguments for {tool_name}: {e.message}",
"path": list(e.absolute_path)
}When validation fails, return the error to the model as a tool result rather than executing the tool. The model can then correct the arguments and retry. Include enough detail in the error for the model to understand what was wrong: "Parameter 'quantity' must be an integer between 1 and 100, but received -5" is actionable. "Validation failed" is not.
Schema validation catches structural errors, but business rule validation catches logical errors that the schema cannot express. A refund amount that exceeds the original order total. A delete operation on a protected record. An API call that would exceed the user's rate limit. A write operation during a maintenance window. These checks require application-specific logic that understands the business context.
BUSINESS_RULES = {
"process_refund": [
lambda args, ctx: (
args["amount"] <= ctx["order_total"],
f"Refund amount ${args['amount']} exceeds order total ${ctx['order_total']}"
),
lambda args, ctx: (
ctx["order_status"] in ["delivered", "returned"],
f"Cannot refund order in '{ctx['order_status']}' status. Must be delivered or returned."
)
],
"delete_record": [
lambda args, ctx: (
args["record_id"] not in ctx["protected_ids"],
"This record is protected and cannot be deleted."
)
]
}
def validate_business_rules(tool_name, arguments, context):
rules = BUSINESS_RULES.get(tool_name, [])
for rule in rules:
passed, message = rule(arguments, context)
if not passed:
return {"valid": False, "error": message}
return {"valid": True}Some operations should never execute without explicit user approval, regardless of how confident the model is. These include operations that create, modify, or delete data; operations with financial implications (charges, refunds, transfers); operations that send communications (emails, messages, notifications); and operations that change system configuration or permissions. For these tools, the validation layer pauses execution and asks the user to confirm.
The confirmation flow works like this: the model generates the tool call, the validation layer intercepts it, formats a human-readable description of what the tool will do, presents it to the user for approval, and either proceeds with execution or returns a "user declined" result to the model. The model then either tries a different approach or acknowledges the user's decision.
REQUIRES_CONFIRMATION = {
"process_refund": "Process a refund of ${amount} for order {order_id}",
"delete_record": "Permanently delete record {record_id}",
"send_email": "Send an email to {recipient} with subject '{subject}'",
"update_subscription": "Change subscription to {new_plan} (billing changes immediately)"
}
def check_confirmation(tool_name, arguments):
template = REQUIRES_CONFIRMATION.get(tool_name)
if template is None:
return {"needs_confirmation": False}
description = template.format(**arguments)
return {
"needs_confirmation": True,
"description": description
}Tool arguments generated by the model can contain content from the user's message, which means they can contain injection payloads if the user (or an attacker manipulating the user's input) is trying to exploit the system. Check tool arguments for SQL injection patterns, command injection characters, path traversal sequences, and other injection vectors before passing them to functions that interact with databases, file systems, or shell commands.
The safest approach is to use parameterized queries and prepared statements in your tool implementations rather than string interpolation, so injection is not possible regardless of input content. But defense in depth is important: validate inputs at the tool call level as well, because defense at a single layer is fragile.
import re
INJECTION_PATTERNS = [
(r"['\";].*(--)|(\/\*)", "Possible SQL injection"),
(r"[;&|`$]", "Possible command injection characters"),
(r"\.\.[/\\]", "Possible path traversal"),
(r"Validation is not just for inputs. Tool results should also be checked before being sent back to the model. A tool might return more data than expected (a database query that returns 10,000 rows instead of 10), sensitive data that should not appear in the model's context (API keys, passwords, SSNs in a customer record), or malformed data that could confuse the model's reasoning. Post-execution validation catches these issues.
Truncate oversized results to a reasonable token budget (typically 2,000 to 5,000 tokens depending on the tool). Redact sensitive fields from results before they reach the model. Validate that the result structure matches what the tool is expected to return, catching cases where an underlying API changes its response format without warning.
SENSITIVE_FIELDS = {"password", "ssn", "api_key", "secret", "token", "credit_card"}
def sanitize_result(result, max_chars=5000):
if isinstance(result, dict):
cleaned = {}
for key, value in result.items():
if key.lower() in SENSITIVE_FIELDS:
cleaned[key] = "[REDACTED]"
else:
cleaned[key] = sanitize_result(value, max_chars)
return cleaned
result_str = json.dumps(result)
if len(result_str) > max_chars:
return {"truncated": True, "preview": result_str[:max_chars], "total_length": len(result_str)}
return resultAdd validation that learns from past failures. Adaptive Recall tracks tool validation patterns so your agent remembers which parameter combinations fail and avoids them in future calls.
Try It Free