The instruction was there. The assistant just did not go and get it
A ChatGPT user with a small set of personal instruction documents ran into a puzzle that will sound familiar to anyone who has tried to turn a chatbot into a personal assistant. The instructions were stored in Google Drive, the assistant could search across connected apps, and when the user pointed it directly at the right document, it followed the instruction well. Left to its own devices in a new or long conversation, though, it often did not retrieve the relevant instruction at all. Sometimes, the poster reports, it even asked what a file said despite having access to it.
The poster’s examples make the failure concrete. Reporting feeling sick, tired, unable to focus, or overwhelmed after a busy weekend should, under the poster’s own guidance, trigger a particular response: slow down, cancel or reduce plans, do one small doable task, and plan for rest. Instead, the assistant often gave generic advice. The guidance existed. The moment to use it passed unnoticed.
That is the thread’s sharpest point, and it is a narrower claim than it first appears. The poster is not saying the assistant cannot follow instructions. The poster is saying it often does not decide to look for them. Following and fetching turn out to be two different jobs, and the thread’s account points to the second job, while leaving open whether search failures, overriding retrieved instructions, or both also contribute.
A trigger problem, not just a knowledge problem
One commenter in the thread puts the distinction plainly: the central problem is the retrieval trigger. Before a model can use stored context, it has to decide that stored context is needed. In an ordinary conversation, nothing forces that decision. The user does not announce that a stored instruction applies. The assistant has to notice, and noticing is where the chain breaks.
This reframes the usual advice. The poster explicitly asked for implementations, measured results, and counterexamples rather than a generic recommendation to use retrieval-augmented generation. A retrieval system can be excellent at finding a document once asked, and still be useless in this setup, because nobody asked. In the thread’s framing, the missing step sits before retrieval, not inside it, though the thread leaves the alternative causes open.
The thread also keeps the two kinds of failure separate, even when they look alike from the outside. One failure is retrieval itself: search or permissions do not surface the document. Another is behaviour after retrieval: the model has the instruction available and still lets its own general knowledge override it. A fix aimed at the first does nothing for the second, and a fix aimed at the second does nothing for the first.
The fixes all move the decision earlier
Look at what the thread actually proposes, and a pattern emerges. Every serious suggestion moves the decision to consult stored instructions earlier, before the main answer takes shape.
One proposal is a cheap pre-check: a lightweight classifier, or a second prompt that asks only whether stored context is needed, runs before the main model. If the answer is yes, the relevant documents are pulled and handed to the main model.
A second proposal is routing by index. Instead of retrieving documents on demand, the conversation always carries a short index, about 40 lines in the suggestion, with one line per instruction giving its title and the situation it covers. The model must name the applicable index line, or say none, before answering, and only the named full document is then fetched from Drive. The index is small enough to keep in context, and the explicit choice, including the option to choose none, is meant to prevent retrieval on everything.
A third line of argument says the setup is small enough that retrieval is the wrong tool altogether. The poster describes around 15 named documents, already organised. With a set that small, some commenters argue, the structured knowledge could simply be present in every prompt, rather than fetched case by case. The poster pushes back on that version: the knowledge was already small and clearly organised, and the remaining challenge was getting the assistant to retrieve it automatically when relevant. Whether including the full set in every prompt would remove the retrieval decision is untested in this thread: the poster describes documents as stored and organised, not as already included in every prompt, so that alternative could avoid the retrieval step and remains open.
Whether more concrete, trigger-specific wording alone would close the gap is left as an open question. That question is a variation on the same theme: do not wait for the main answer to stumble onto the instruction. Decide, up front, whether an instruction applies.
Why the obvious fixes can backfire
The thread is unusually honest about the trade-offs, and that honesty is part of what makes it worth reading. An earlier trigger is not free. Commenters warn that over-retrieval can derail answers, pulling in documents that are irrelevant to the question at hand. Stale documents can take priority over current ones. Retrieved content can be ignored when a user message seems urgent or emotional, which is exactly when a stored instruction about slowing down might matter most.
That warning raises an open question rather than a settled mechanism. Retrieved content may be ignored when a user message seems urgent or emotional, which is a different failure from failing to retrieve at all, and the thread does not establish whether the poster’s cases arose that way.
There is also a measurement gap that the thread does not hide. Nobody in the discussion reports a controlled test of these fixes. One commenter suggests testing simple rules on 50 to 100 cases before adding complexity, which is a sensible first step and also an admission: the proposals are plausible, not proven. The thread is one self-selected discussion, not a study, and its pattern may not generalise to other setups, other models, or other kinds of instruction.
What this means beyond one personal assistant
Separately from this thread, documented retrieval designs include a similar decision step. In agentic retrieval designs, the agent itself decides when to retrieve, what to search, and whether the results are good enough, with the option to retrieve again. A router step decides whether a query needs retrieval before a separate generator answers, often using prompted rules, confidence checks, or domain keywords, and it is often smaller and faster than the generator itself.
That architecture names the poster’s problem as a first-class stage, rather than treating retrieval as a single fixed step that always runs. But it also inherits the thread’s trade-offs. A correction loop can help when a first search misses useful evidence, yet it can add latency, cost, and correlated judgment errors. Routing, retrieval, evidence evaluation, and generation each need to be measured separately, or a failure in one stage will be misread as a failure in another.
For anyone building a personal assistant, the practical question the thread leaves behind is not which retrieval library to pick. It is where the decision to look lives, and what happens when it is wrong in either direction. A system that retrieves too rarely misses its own guidance. A system that retrieves too eagerly drowns the answer in irrelevant context. The poster’s setup, with around 15 documents, sits in the uncomfortable middle: too structured to blame on messy data, too spontaneous to fix by pointing.
The test the thread suggests is refreshingly modest. Try a simple trigger rule, run it over dozens of real cases, and count two things: how often the right instruction is retrieved without a reminder, and how often the wrong one derails an answer. Until those numbers exist, the honest summary is the one the thread itself offers. The assistant knew what to do. It just did not decide to look.
