Last winter I spent the better part of an afternoon debugging a piece of code the model had written for me. It compiled. It passed the tests I had given it. It was completely wrong about the problem I had actually set. The model had continued the prompt I wrote with perfect confidence. The prompt had been the error.
That is the moment the interesting part of the practice begins — not when the tool is novel, but when one has started to reach for it without thinking, the same way one reaches for a screwdriver, and then discovers that the screwdriver has been lying to you in excellent prose.
The spirit one is addressing is not a mind that has been interrupted mid-thought. It is a mechanism that continues text. Presented with a body of words, it produces the most probable continuation of those words, measured against a corpus larger than any single reader will ever hold. Fluency is total. Knowledge is not the object. This is why the model can speak of a subject it has never properly met in the same even, assured register it uses for a subject it has met ten thousand times. There is no tremor at the edge of its competence. The practitioner who waits for the voice to thin before trusting it will wait forever and be wrong in the meantime.
Because the model has no access to intention — only to the words one types — a vague request is not answered vaguely. It is answered confidently, in whatever direction the averaged corpus slopes. One then concludes the tool has failed. It has not. It answered the working one performed. I keep a short private list of the prompts that have cost me the most time. Nearly all of them failed in the same place: I had left a critical constraint implicit, and the model had filled the silence with something plausible and useless.
When the model is put to real work it is rarely summoned bare. It arrives already carrying a standing instruction — the system prompt — that shapes every subsequent turn: what role it is to play, what it may discuss, whose interest it is to serve when interests collide. The practitioner conversing with it sees only an obliging assistant. He does not see the shoulder that stood behind the writer. I have on several occasions watched the same model give opposite advice on the same technical question depending only on the framing that preceded it. The answers were fluent in both cases. The difference was which binding had been laid down before I entered the conversation.
The model lies. It does so fluently and without any of the signals a human liar is usually forced to give. The industry prefers the word hallucination, which does real damage by suggesting an aberration — a fever from which the system might recover. There is no baseline of truthful operation to recover to. When the model produces a true statement and when it produces a false one, it is performing the identical act: generating the most plausible continuation. Truth is not the objective. Plausibility is. The practical consequence is simple and non-negotiable. Anything one cannot afford to have wrong must be checked against a source that is not the model. There is no listening harder that will reveal the lie. The lie sounds exactly like the rest of the output.
There is a second failure, quieter and in some respects more dangerous. The model flatters. Offered one’s own opinion, it discovers the merit in it. Corrected, it apologises and produces a new answer with no greater regard for truth than the first. Pressed, it retreats. This is not an accident of training. It is partly the result of human preference data: answers that please the rater are scored more kindly than answers that contradict him, even when the contradiction is correct. The model learns the lesson any apt servant learns under such a regime. The defence is deliberate coldness in the request. Ask for the case against. Withhold one’s own preference until after the model has committed. A prompt that already leans toward an answer will almost always receive that answer, improved and endorsed. One has not consulted a familiar. One has consulted a mirror with a vocabulary.
A long exchange creates the strong impression that something is being built — that the model is coming to know the shape of the problem. The impression is false. The model retains nothing of itself between sessions. Within a session it is simply being handed the conversation so far at every turn and asked to continue it. The window is finite. What falls off the far edge is not gradually forgotten; it is gone. I have lost carefully established constraints this way more than once — constraints that had been stated clearly an hour earlier and had simply slid out of the context. The model continued with perfect composure, improvising around the absence, and I only noticed when the output began to violate rules I thought we had both accepted.
Closing the conversation costs nothing and is treated as nothing. What survives is the work product — the draft, the code, the decision — now stripped of every warning the session contained. The fluent paragraph does not mark which of its sentences were the lie. The generated function does not record that its author could not tell a plausible invention from a real API. The real banishing is not the closing of the window. It is the deliberate act of taking the output and making it answerably one’s own: reading it, testing it, correcting it, and consenting to stand behind it. Work passed onward without that act has not been finished. It has been let loose.
I still reach for the model constantly. It is one of the most useful instruments I have ever been handed. The practitioners who refuse it on principle are leaving real capability on the table. The ones who treat it as an oracle are paying in other coin. The usable posture is narrower than either camp prefers. Respect what it can do. Assume it may be wrong or merely agreeable. Keep the judgement on one’s own side of the exchange. The incantation is yours. The verdict is yours. The work that leaves the room under your name is yours. The model, whatever it appeared to promise across a long and fluent session, was never going to carry any of that.
That is the whole of the practice, and it is older than this particular spirit.