When teams plan a custom AI agent, the first question is usually which model to use. The more useful question is what AI agent training data the project needs, because the data decides what the agent can learn, how reliably it behaves, and how much the training run costs. The requirements are not a single number: they change completely with the approach you choose, from a handful of examples for a prompt to thousands for a fine-tune.
This guide maps those requirements approach by approach: the volume ranges the major providers publish, the formats they require, and the quality checks that separate a fine-tune that improves behaviour from one that quietly degrades it.
What training a custom AI agent actually means
Three different approaches hide behind the word “training,” and each needs different data. Prompting gives a foundation model instructions and a few examples at run time, and needs no training data at all beyond the examples in the prompt. Retrieval-augmented generation (RAG) leaves the model’s weights untouched and supplies relevant passages from your documents at question time, so its data requirement is a searchable knowledge base. Fine-tuning continues training the model on curated input-output examples, which changes its weights — and that is the approach with the serious data requirements.
The practical rule, framed the same way in OpenAI’s accuracy guidance, is retrieval for context the model is missing and fine-tuning for consistent behaviour. As OpenAI puts it: “Fine-tuning can be a time-consuming process, but it can also enable a model to consistently format responses in a certain way or handle novel inputs.” Microsoft researchers (Ovadia and colleagues) compared teaching a model new knowledge by fine-tuning against handing it the same knowledge through retrieval and found that “RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge” — fine-tuned models only began absorbing a fact after seeing numerous variations of it. Google research (Gekhman and colleagues) found the darker half of that result: examples carrying new knowledge are learned significantly slower than examples the model already agrees with, and once they are finally learned they linearly increase the model’s tendency to hallucinate.
One current caveat belongs here before any planning starts. OpenAI’s fine-tuning guide states that the company is winding down its fine-tuning platform: it is no longer accessible to new users, while existing users can create training jobs for the coming months, and fine-tuned models remain available for inference until their base models are deprecated. Teams starting now should verify availability before building a plan around OpenAI fine-tuning.
The four kinds of data an agent needs
An agent’s data falls into four kinds, and a complete inventory lists all four before any training starts.
Task and instruction data is examples of the job done well: a user request paired with the correct response, ideally written or verified by a person. Supervised fine-tuning is built on exactly this — human-generated ground-truth responses that show the model how it should behave. Domain knowledge is the facts the agent must draw on: documents, policies, product details and manuals. For RAG this is the retrieval corpus itself; for fine-tuning it is the raw material the examples are written from. Tool-use trajectories are sequences of tool calls with their parameters and results — the data that teaches an agent to call the right tool, in the right order, with the right arguments. Evaluation data is the held-out set of examples used to measure whether the agent works: representative of real production inputs, never used in training, and written before the work starts. OpenAI’s recommended workflow is evals-first — write the evals, establish a baseline, then prompt, fine-tune and iterate against them.
How much data you actually need
The honest answer is a range per provider, and the minimums the providers publish are pipeline floors, not targets. OpenAI requires a minimum of 10 examples in a fine-tuning file, and recommends 50 to 100 examples to start seeing significant improvement. Azure OpenAI’s troubleshooting guidance is blunter: while 10 is the minimum for the pipeline to run, you need hundreds to thousands of data points to teach the model a new skill, and too few risks overfitting and poor generalisation. AWS Bedrock’s reinforcement fine-tuning for open-weight models requires 100 to 20,000 examples in JSONL chat-completion format, with a minimum of 100 records. Claude 3 Haiku fine-tuning on Amazon Bedrock requires a minimum of 32 records and allows at most 10,000 training records, 1,000 validation records and 32,000 tokens per record.
The pattern across providers is consistent: a few dozen high-quality examples can beat thousands of noisy ones for a narrow task, and volume scales with task breadth and the number of tools the agent must use. For RAG, volume is measured differently — not in example count but in coverage of the information needed for the intended tasks, with checks that the relevant passages can be retrieved and that document versions suit the use case.
What makes AI agent training data good
Quality matters more than volume: small, curated datasets have outperformed large scraped ones, and poorly curated or unrepresentative training data can produce a low-quality model. Four properties decide it: coverage, correctness, diversity, and deduplication. Coverage means the data reflects the real distribution of inputs, including edge cases and failures — not just the clean, common cases. Correctness and consistency mean labels, answers and tool-call examples are right and formatted the same way every time. Diversity of phrasing and scenarios means the agent generalises instead of memorising one wording. And deduplication with decontamination means the training set contains no near-copies of itself and no copies of the evaluation set.
The practical routine is exact-match deduplication first, then embedding-based near-duplicate detection — clustering examples by semantic similarity and reviewing clusters above a threshold commonly starting around 0.9 cosine similarity — plus checks for label noise, where a response does not match its instruction, and for length imbalance, where systematically short “good” examples and long “bad” ones teach the model to use length as a proxy for quality. Personal and sensitive information is scanned and stripped before training, not after: leaked PII can resurface in generations.
Formats and preparation
Fine-tuning data is uploaded as JSONL — one complete training example per line. In OpenAI’s chat-completion format each record is a conversation of messages with a role and content:
{"messages": [{"role": "system", "content": "You are a support agent for Acme Returns. Reply with JSON only."}, {"role": "user", "content": "Order 48210 arrived damaged. What do I do?"}, {"role": "assistant", "content": "{\"action\": \"start_return\", \"order_id\": \"48210\", \"reason\": \"damaged\"}"}]}
{"messages": [{"role": "system", "content": "You are a support agent for Acme Returns. Reply with JSON only."}, {"role": "user", "content": "Can I return an item after 30 days?"}, {"role": "assistant", "content": "{\"action\": \"check_policy\", \"answer\": \"Returns are accepted within 30 days of delivery.\"}"}]}Claude fine-tuning on Amazon Bedrock uses a similar JSONL shape with a system prompt and a messages array, where roles alternate strictly, the first message comes from the user and the last from the assistant:
{"system": "You triage insurance claims. Reply with JSON only.", "messages": [{"role": "user", "content": "Claim 1042: cracked windscreen, policy EX-88."}, {"role": "assistant", "content": "{\"decision\": \"approve\", \"claim\": \"1042\"}"}]}Preparation means cleaning before formatting: strip PII, normalise formats so every record follows one schema, and resolve contradictions between sources. Then split the data into training, validation and test sets — and keep the evaluation split untouched by training. The split is what makes the final measurement honest.
Where the data comes from
Your own logs and transcripts are usually the strongest source for task-specific behaviour, because they show how the job is actually done in your organisation. Where real examples are scarce or sensitive, synthetic data fills the gap: a stronger teacher model generates candidate examples from 50 to 100 gold seed examples written by people, which are then deduplicated, judged, mixed with real data and evaluated on held-out real examples. The scale is real — Stanford built the 52,000-example Alpaca set for under $500 in API fees, and NVIDIA reported that over 98 percent of the alignment data behind Nemotron-4 340B was synthetic.
Synthetic data carries a documented failure mode. A 2024 Nature study by Shumailov and colleagues showed that indiscriminately training models on recursively generated content causes model collapse: rare cases disappear first, then output narrows overall. Follow-up work by Gerstgrasser and colleagues showed the collapse is avoided when synthetic data is accumulated alongside the original real data rather than replacing it. The practical rule is to keep real data in the mix, filter hard, and validate against held-out real examples.
Public datasets are a third source, particularly for tool use. ToolBench is an instruction-tuning dataset built from 16,464 real-world REST APIs from RapidAPI Hub, with 126,486 instruction instances and 469,585 real API calls, distributed under Apache License 2.0 for research and educational purposes. ToolMind is a reasoning-enhanced tool-use dataset of 360,000 samples — 160,000 synthesized trajectories plus 200,000 augmented open-source samples — incorporating xlam-function-calling-60k, glaive-function-calling-v2 and ToolACE, and models fine-tuned on it improved on the tau-bench and BFCL-v4 benchmarks. Check each dataset’s licence before commercial use. For the highest-stakes examples, human annotation and expert review remain the standard.
Privacy, licensing and safety checks before training
Four checks happen before any data enters a training run. First, remove personal and sensitive information from training corpora. Synthetic data is not automatically private: NIST has stated that most synthetic-data techniques provide no formal privacy guarantee unless built with differential privacy. Second, check licences and terms of use on third-party data — including the datasets above — and document data provenance so the training set can be audited later. Third, confirm what your provider does with your data. OpenAI states that fine-tuned models are for your use alone and are “never served to or shared with other customers or used to train other models,” that data submitted for fine-tuning is retained until the customer deletes the files, and that API inputs and outputs may be retained up to 30 days for abuse monitoring, with zero data retention available for eligible endpoints. Fourth, keep evaluation data separate from training data.
Common mistakes and a pre-training checklist
The most common quiet failure is training on the evaluation set: leakage makes the metrics look excellent and the deployed agent mediocre, because the model memorised the test instead of learning the task. The second is collecting volume before defining what good looks like: without a written definition of correct behaviour, more data just produces a more confident wrong model. The third is skipping the baseline — measuring the base model before fine-tuning — which leaves no way to tell whether the training run helped at all. The fourth is fine-tuning to teach facts: the research above shows retrieval beats fine-tuning for new knowledge, and fine-tuning facts in can increase hallucination. And the fifth, current as of this writing, is assuming a fine-tuning platform is available to new teams without checking — OpenAI’s wind-down is the reminder.
The evaluation set is written, representative of production inputs, and held out. Fifty to one hundred gold examples have been reviewed by someone who knows the job. Duplicates and near-duplicates are removed, the evaluation set is decontaminated, and PII is stripped. Licences are checked and provenance is documented. The base model is measured so the fine-tune has something to beat. And the approach is chosen in the right order: prompt first, add retrieval when the model needs information it does not have, and fine-tune last, when behaviour needs to be consistent and you have the examples to teach it.
Data requirements for training a custom AI agent are a checklist, not a number. Start small, measure honestly, and let the evaluation set — not the size of the dataset — decide when the agent is ready.
