If you are wondering how AI agents handle structured vs unstructured data, the short answer is that a common, well-supported design pattern pairs two approaches — one tuned to each data type — and deliberately combines them. For structured data, agents typically issue typed tool calls with JSON-Schema arguments that your application executes against databases and APIs, returning results the agent can filter, aggregate and reconcile. For unstructured data, the common approach is a retrieval pipeline: parse documents, split them into chunks, search by meaning, and generate grounded, cited answers — though vision models can also read a scanned document and emit schema-constrained fields with no retrieval at all. The real engineering happens in the handoff: routing a question to the right mechanism, extracting stable fields out of messy documents, and verifying the result with the right checks for each side.
What structured and unstructured data mean for an AI agent
Structured data: fixed schemas, rows, and deterministic queries
Structured data lives in fixed schemas — tables with named columns, data types, primary keys and relationships. Your CRM, your order database, your product catalog: an agent querying these can ask “total revenue from enterprise customers in Q3” and get an answer computed by arithmetic, not by guesswork. In this application-mediated pattern, the agent never sees the whole table. Instead, it fills in the arguments the input schema defines, and the application runs the query. This is where structured answers feel most trustworthy — but treat that as an impression, not a measured fact: the supplied evidence contains no comparative reliability benchmark across the two mechanisms. A failure here is dangerous precisely because it looks confident. A wrong join, a stale field, an inconsistent customer ID or two teams’ conflicting definitions of “active user” will produce a perfectly formatted, completely wrong number.
Unstructured data: documents, transcripts, and images without a schema
Unstructured data is everything free-form: PDFs, emails, chat logs, meeting transcripts, support tickets, images. It carries the why that tables leave out — the exception clauses in a contract, the reasoning in a design document, the policy detail buried on page 47 of a handbook. An agent usually cannot query it directly; most systems first retrieve the relevant passages and then read them. So unstructured handling is often framed as a two-stage problem: retrieval quality first, generation quality second. If retrieval pulls an outdated policy version or a poorly chunked fragment, the answer is only as good as the evidence it was built on — the model cannot reconstruct facts that the retrieved passages never contained. (The exception is when the agent receives the whole document as input, as with vision models processing scans, where no retrieval step is needed.)
The semi-structured middle: JSON logs and event streams
Semi-structured data — JSON payloads, XML, event streams, patterned log lines — sits between the two. A common approach is to treat it as “unstructured until normalized.” When debugging, the agent retrieves raw JSON as text and reads it like a human would. For analytics, routing and alerting, the stable fields get extracted into structured stores with a schema. The practical rule: keep the raw payload for context, extract the stable fields for computation, and never assume the payload’s shape will stay the same across vendors, versions or time.
Why the data type decides your agent’s design
The data type determines the mechanism, and the mechanism determines the failure modes, the evaluation strategy and the cost profile. Structured verification tends to be straightforward to automate: a deterministic query run against a fixed data snapshot can support repeatable checks, so totals can be reconciled and key metrics regression-tested. Running a query twice tests repeatability under controlled conditions, though — repeatability alone does not show the query answered the original question correctly. Unstructured verification is harder to automate: retrieval precision, citation quality and faithfulness to sources need judgment against human-labeled references. A single evaluation harness can still cover both pipelines, as long as it reports the two metric tracks separately. The numbers from industry engineering accounts point the same way. DataChain, reporting on OpenAI’s and Anthropic’s internal data agents, describes how OpenAI cut a data agent’s tool set from 40 tools to 13 because overlapping tools confused the model, and how Anthropic found that raw retrieval over thousands of past queries moved accuracy less than a single percentage point — while adding curated context and skills lifted Anthropic’s data-agent accuracy from 21% to over 95% (DataChain). These are the companies’ own reported engineering results, relayed by DataChain, not independently verified benchmarks — but the direction is consistent: curated and structured beats raw and large.
How agents work with structured data
Tool calling: a typed interface to databases and APIs
The structured access mechanism is tool calling. Developers declare tools with JSON-Schema input schemas, and the model emits a structured call the application executes. In Anthropic’s Messages API, tools are declared with input_schema, Claude returns tool_use content blocks carrying an id, a name and structured input, and the application threads a tool_result back into the conversation (Anthropic; AWS Bedrock docs). In application-mediated access, the model never touches the database directly — it only fills in the arguments, and your code enforces everything else: authentication, authorization, query limits, timeouts. Note what the schema does and does not cover: it constrains the call’s arguments, not the values your application returns — a typed tool result can still carry the wrong data if the underlying query or data was wrong. Anthropic’s engineering guidance stresses that tool definitions deserve the same prompt-engineering attention as prompts, and that formats should stay close to what models have seen in natural text (Anthropic).
Structured outputs: JSON that is guaranteed to match your schema
Tool calling covers inputs to your systems; structured outputs cover the model’s own response. OpenAI’s Structured Outputs feature takes response_format with a json_schema and enforces schema adherence at decode time in strict mode — for a successful structured response using a supported schema, the output matches the schema. That is scoped to the success path: callers should handle refusals, incomplete responses and API errors before consuming the structured result, since the feature constrains the result’s shape, not whether a result was returned at all. That guarantee covers the output’s format, not its factual correctness — and it says nothing about whether the application is authorized to write it anywhere. Piping output into a database or a downstream API still needs separate checks: that the values are actually right, not just well-formed, and that the write is permitted. LangChain abstracts both providers as response_format on its create_agent, with the validated output returned in the structured_response key of agent state (LangChain). Practical use cases documented by OpenAI (OpenAI cookbook) include populating a database with content extracted from documents — a hint that structured outputs are also the bridge from unstructured input to structured records.
Where structured queries still fail
Schema guarantees do not guarantee correct intent. The documented failure list is short and brutal: wrong joins, stale or mislabeled fields, inconsistent IDs across systems, and “shadow definitions” — the same metric name meaning different things to different teams. StackAI’s treatment of agent data workflows names these failure patterns and the evaluation discipline they demand: query correctness checked against the original intent, deterministic reconciliation of totals, and regression tests on key metrics (StackAI). Log every tool call with its arguments, and treat any decision-driving number as guilty until reconciled.
How agents work with unstructured data
The RAG pipeline: parse, chunk, embed, retrieve, ground
Many unstructured workflows run through retrieval-augmented generation — one well-documented pattern, alongside direct processing options such as vision models reading whole documents. The foundational Lewis et al. paper (NeurIPS 2020, arXiv:2005.11401) combines a pre-trained sequence-to-sequence generator — parametric memory — with a dense vector index — non-parametric memory. The paper found that RAG models generate more specific, diverse and factual language than a parametric-only seq2seq baseline, and that the architecture gives decision provenance plus knowledge updates without retraining (arXiv). In practice the pipeline is: parse documents into text, split into chunks, embed each chunk into a vector, retrieve the chunks most similar to the query, re-rank them, and generate an answer grounded in the retrieved passages with citations — one well-documented example pipeline, not the only way to handle unstructured data. Every step is a potential failure point. Chunking that splits a table across two chunks can scramble the facts inside it. Embeddings that blur near-synonyms retrieve plausible-but-wrong passages. And a model that answers from its own weights instead of the passages produces the classic RAG failure: fluent text with no grounding.
Turning messy documents into clean structured data
One of the most valuable agent patterns is the extraction bridge: vision-language models read scanned or heterogeneous documents and emit schema-constrained fields, turning unstructured documents into structured rows. The canonical example is invoices: one extraction schema can replace dozens of per-supplier templates, and a common design routes failed validations to a human review queue rather than letting them enter the ledger silently. LlamaIndex’s LlamaParse is built for exactly this conversion of PDFs and scans into structured Markdown, JSON or HTML — though its stated capabilities are vendor claims on LlamaIndex’s own site, not independently verified (LlamaIndex). Combined with structured outputs, extraction can become a controlled loop: parse or extract the document content, apply a supported structured-output schema, validate the resulting records, and route failures to review before admitting records downstream. The guarantee stays scoped to the structured-output stage — schema adherence at decode time, not factual correctness — and accuracy plus business-rule checks remain separate steps.
Where unstructured retrieval still fails
The failure modes here are quieter than SQL errors and harder to catch. Version drift: the agent retrieves last year’s policy because the index was never rebuilt. Weak grounding: the answer cites three documents but the key claim comes from none of them. Chunking artifacts: the retrieved passage is true but incomplete in a way that reverses the meaning. StackAI’s guidance puts retrieval precision and recall, citation quality and faithfulness-to-sources at the center of unstructured evaluation (StackAI). And Anthropic’s reported experience — raw retrieval over thousands of past queries moving accuracy less than a point, per DataChain’s account — is a warning that, in that data-agent setting, adding more documents did not move the needle while curated context and skills did. Treat curation as a diagnostic option rather than a rule: check evidence coverage, freshness, chunking and retrieval quality before deciding whether to add documents or improve the existing material.
How AI agents handle structured vs unstructured data pipelines together
Dual retrieval: SQL for the numbers, search for the reasons
The highest-value agent workflows are neither purely structured nor purely unstructured — they are the deliberate handoff between the two. The dual-retrieval pattern is straightforward: compute the numbers with SQL, fetch the reasons with semantic search, then reconcile. A customer-churn question runs an aggregation over the CRM for the churn rate and the segments, then pulls support tickets and call transcripts to find the stated reasons. Neither side alone answers the real question — the numbers without the reasons are unexplained; the reasons without the numbers are anecdotal (StackAI).
Tool routing: the cheapest reliable path to a verified answer
Before retrieving anything, the agent must decide which mechanism to use. The tool-routing heuristic StackAI describes is practical: if the question requires aggregation, start with structured tools (leading with document search is rarely right); if it asks what a policy says, start with unstructured retrieval; if it is mixed, run parallel retrieval and reconcile (StackAI). This is also where the curation lesson lands hardest. Per DataChain’s account of the two companies’ engineering, OpenAI reduced a data agent’s tool set from 40 to 13 because overlapping tools confused the model about which to call — the bottleneck was discovery (mapping the question to the right tables and entities), not SQL generation. And Anthropic’s jump from 21% to over 95% came from curated context and skills, not from a larger index.
Knowledge graphs and metadata as the glue
Between the two pipelines sits a layer that many teams skip and later regret: entity resolution. The customer “Acme Corp” in the CRM, “acme-corp” in the support tickets and “ACME Corporation” in the contracts are the same entity, and matching them reliably is an entity-resolution problem in its own right — SQL matches exact keys and vector search matches meaning, but reconciling the three spellings into one identity still takes deliberate design. Knowledge graphs and well-maintained metadata — canonical entity IDs, table and column descriptions, document provenance and freshness timestamps — give the agent the map it needs to connect structured rows with unstructured passages about the same thing.
Governance, evaluation, and trust
Permissions and PII across both data types
Governance has to cover both sides from ingestion; retrofitting access controls later is where permission gaps tend to appear. Structured data needs row-level security — the agent’s SQL must only see the rows the user is allowed to see. Unstructured data needs document-level access controls — the agent’s index must respect the same ACLs as the document store, or you get “permission laundering”: an agent summarizing a document the user was never allowed to open (StackAI). Add PII detection and redaction that covers OCR paths, data minimization so the agent only retrieves what the task needs, and you have a solid baseline. If your agent can read everything, a misconfigured index or a changed ACL is all it takes for a summary to surface a document the user was never meant to see.
Measuring structured and unstructured quality differently
Keep the two evaluation tracks distinct inside one harness. Structured evaluation checks query correctness against the original intent, deterministic reconciliation and regression tests on key metrics; unstructured checks retrieval precision and recall, citation quality and faithfulness to sources (StackAI). Build a golden set of questions with known answers for each side, run it on every pipeline change, and track the two scores separately — blending them into one number lets the structured side’s easy wins drown out the unstructured side’s quiet failures.
Observability: log tool calls, passages, and outputs
One of the most valuable reliability controls is the “verify” step after “act.” Once agent outputs influence decisions — a number in a report, a summary sent to a customer — the agent loop should include a check against the evidence before the output goes out. Observability makes that check possible: log every tool call with its arguments, every retrieved passage with its source and version, and every output the agent produced. That trail is what lets you debug a wrong answer (was it the join, the chunk, or the grounding?) and build the regression sets the previous section needs.
Practical takeaways
- Match the approach to the data type. Structured questions typically go through typed tool calls; unstructured questions usually go through retrieval (parse, chunk, embed, retrieve, ground) — or, for single documents, direct processing like vision-based extraction. Do not force one approach onto the other.
- Design the handoff, not just the pipelines. Dual retrieval (SQL for numbers, search for reasons), tool routing by question type, and schema-on-read extraction are where the highest-value agent workflows live.
- Curate, don’t accumulate. DataChain’s account of OpenAI and Anthropic engineering points one way: fewer, better-described tools (40 cut to 13) and curated context (21% to over 95% accuracy) beat raw scale. These are reported figures from the companies’ own accounts, not independent benchmarks — treat them as direction, not guarantees.
- Verify with the right checks per side. Reconcile structured numbers deterministically; check unstructured answers for retrieval precision, citation quality and faithfulness. Track the two scores separately, even inside one harness.
- Govern from ingestion. Row-level security for tables, document-level ACLs for indexes, PII redaction including OCR paths, and full observability — tool calls, passages, versions, outputs — before the agent touches production data.
