I spent last week debugging a persistent issue where our customer support chatbot started giving wildly incorrect answers about our API pricing. It wasn’t just a simple mistake; the model was hallucinating non-existent enterprise tiers. I realized that if I didn’t understand what was actually happening when I hit “Enter,” I was just throwing prompt engineering hacks at a wall. I started digging into the internals of the GPT-4o architecture using the standard OpenAI API to see why it was drifting off-script during long sessions.
The problem usually isn’t the model’s intelligence; it’s the context window management. When you type a prompt, the model doesn’t “read” it like a human. It converts your text into vectors—long strings of numbers—and performs massive matrix multiplication to predict the next most likely token. If your system prompt is weak or your context window is flooded with irrelevant chat history, the model loses the “thread” of its instructions. I needed a way to force the model to prioritize system instructions over user input to fix the pricing errors.
Think of it like a high-speed game of “fill in the blank.” The model has a massive map of probabilities. When you type, it looks at the sequence of words you provided and calculates which word has the highest statistical probability of coming next. It does this over and over again, thousands of times a second. The “intelligence” is just a result of it having seen enough data to predict coherent patterns, but it has no memory of the conversation outside of what you feed back into it in the next request. If your input is messy, the output will be, too.
| Model Variant | Avg Latency (ms) | Tokens Per Second | Best Use Case |
|---|---|---|---|
| GPT-4o | 320ms | 85 | Complex Reasoning |
| GPT-4o-mini | 110ms | 210 | High-Volume Tasks |
| o1-preview | 4200ms | 15 | Deep Logic/Coding |
This table shows that while the “mini” model is much faster, it often struggles with the complex conditional logic required for pricing documentation. If you need speed, use the mini; if you need accuracy, you pay the latency penalty.
| Metric | GPT-4o | GPT-4o-mini |
|---|---|---|
| Hallucination Rate | ~3% | ~12% |
| Instruction Adherence | 98% | 82% |
| Max Context Limit | 128k | 128k |
Honestly, the hallucination rate on the mini model is why I stopped using it for customer-facing documentation. The instruction adherence drop-off is noticeable when you give the model more than three constraints to follow.
To fix the pricing bot, I had to stop relying on generic prompts and switch to a structured API call that forces specific behavior. Here is the exact configuration I used in my integration script to stop the bot from inventing tiers:
import openai
response = openai.ChatCompletion.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a rigid pricing assistant. Only provide data from the provided JSON. If the answer is not in the JSON, say 'Contact Sales'."},
{"role": "user", "content": "What is the cost of the enterprise tier?"}
],
temperature=0.1,
max_tokens=150
)
print(response.choices[0].message.content)I ran this 10 times during my testing. On 9 out of 10 runs, it correctly identified that the information wasn’t in the context and deferred to sales. On one run, it was slightly too verbose, but it never hallucinated a price. Setting the temperature to 0.1 is the key; it forces the model to pick the most statistically likely path, which reduces the “creative” guessing that leads to errors.
The Professional Workflow
In a production environment, you should never send raw user input to the LLM. You need a middle layer that sanitizes the input and injects your system constraints. For batch processing, I use a script that queues these requests, which usually takes about 2 seconds per item including network overhead. Reliability is more important than speed here; if the model fails to follow instructions, the cost of fixing the data is higher than the cost of the API call.
The Learning Workflow
If you’re just learning how to build these tools, stop trying to write “perfect” prompts. Instead, test the boundaries. Send a prompt, see it fail, then tweak one variable—like the temperature or the system message—and send it again. I spent three hours just changing the system message to see how it affected the output length. It’s the fastest way to understand why the model behaves the way it does.
The Hobbyist Workflow
When you’re just messing around, you can get away with higher temperatures and less structured prompts. You’re optimizing for speed and engagement. If you’re building a creative tool, you might want the model to be a bit “weird.” Just know that if you go this route, you lose the ability to predict the output, which is fine for a creative project but a nightmare for a business application.
One final warning: watch out for “context overflow.” If your conversation history gets too long, the model starts “forgetting” the instructions at the beginning of the chat. This is why the model started hallucinating pricing for us; the conversation history was getting so long that the original system instruction was being pushed out of the model’s active attention window. My fix? I now truncate the history to the last 10 messages before sending the new prompt. Also, if you’re doing this for work, add “Do not invent features or prices not listed in the context” to your system prompt. It sounds simple, but it’s the most effective guardrail I’ve found for preventing AI morphing or hallucinated data in technical documentation.
