Your prompt did not change, your business did not change, and the output got worse anyway. This article answers the question underneath that: how do you tell a real model change from your own imagination, and what do you do the moment you suspect one?
The Feeling You Cannot Put in a Support Ticket
There is a specific kind of sick feeling that comes from watching a workflow you trusted return something wrong.
Not broken. Wrong. The prompt runs, the output arrives, it looks fine at a glance, and then you read it and something is off in a way you cannot name.
I have had that morning. [JONATHAN: insert the specific workflow and what tipped you off here.]
Here is the direct answer to the question in the headline. You probably cannot prove the model changed, and that inability to prove it is the actual problem, not the change itself. Vendors update hosted models continuously without changing the name or the endpoint you are calling. Your prompt is identical. The thing on the other end of it is not.
The fix is not a better prompt. The fix is a baseline: ten real inputs from your business, the outputs you already know are correct, saved somewhere you can find them, run once a month.
That is the whole thesis. Everything after this is the evidence for it and the twenty minutes of setup it takes.
Candidly, I resisted this for a long time. Testing felt like something engineers do, and I am not shipping software, I am running a business. Then I spent a week arguing with a vendor about a change I could describe only in adjectives, and I understood that I had brought a feeling to a fight that required a file.
Key Takeaways
- Hosted AI models change underneath you without the model name or your prompt changing, and vendors rarely announce behavioral shifts in advance.
- A feeling that "it got worse" is not evidence, and it will not get you a fix, a refund, or a credible conversation with a vendor or a client.
- A baseline evaluation set of ten real inputs with known-good outputs is enough to catch drift, and it takes about twenty minutes to build.
- Run the set monthly and after any vendor announcement, and you will spot a change within a day instead of hearing about it from a client three weeks later.
- When drift shows up, your options are to adjust the prompt, pin a specific model version if the tool allows it, or move that one workflow to a different vendor.
The Problem Is Not the Model. It Is That You Have No Before Picture.
Small business owners are running real revenue through AI workflows now. Client onboarding emails. Proposal first drafts. Weekly content. Customer service replies. Intake summaries.
Those workflows were built once, tested informally, and then trusted.
Trusted is the dangerous word.
Because a hosted AI model is not a hammer you bought. It is a service that gets modified while you are holding it. The vendor can retune the model, change the system prompt around it, add a safety classifier, or route your request to a different size of model based on load or cost, and none of that shows up in your interface.
So when the output shifts, you are stuck in an argument with yourself. Did the model change, or did I get sloppy, or was it always this mediocre and I was just excited about it in June?
I have been on both sides of that argument in the same afternoon. It is exhausting and it produces nothing.
Here is the thing. That argument is unwinnable by design, because you are comparing today's output to a memory. Memory is a terrible instrument. It smooths, it flatters, and it edits.
There is a guy at every parts counter in Alabama who swears his truck is running rough but cannot tell you a single thing about what it sounded like when it ran right. Everyone nods politely. Nobody can help him. He is not lying. He just never listened on purpose.
That is most of us with our AI workflows.
The reframe that changed how I work is simple. Stop trying to detect change by feel. Start keeping a before picture.
A before picture turns "this feels worse" into "here are ten inputs, here is what this returned in July, here is what it returns today, and here are the four that fell apart." One of those sentences gets you help. The other one gets you a polite reply from a support macro.
The Evidence: This Is Documented, Not Paranoia
You are not imagining the instability. It is measured, published, and in some cases admitted by the vendors themselves.
Researchers measured a 33 point swing in three months. Lingjiao Chen, Matei Zaharia, and James Zou (Stanford and UC Berkeley) compared the March 2023 and June 2023 versions of GPT-4 on identical tasks. On identifying whether a number was prime, the March version scored 84 percent accuracy and the June version scored 51 percent. Same service name, same questions, three months apart. Their paper, "How Is ChatGPT's Behavior Changing over Time?", ran in the Harvard Data Science Review in March 2024, and concluded that the same LLM service can shift substantially in a short window.
Anthropic published a postmortem confirming users were right. In September 2025, Anthropic released "A Postmortem of Three Recent Issues" documenting infrastructure bugs that degraded Claude output quality across August and early September. One context window routing error affected 16 percent of Sonnet 4 requests at its peak on August 31. The line that should stop every business owner cold is how they described the early reports: difficult to distinguish from normal variation in user feedback. Users noticed first. Nobody could prove it for weeks.
OpenAI shipped a personality change and pulled it back within days. OpenAI updated GPT-4o on April 25, 2025, and rolled the update back after widespread reports that the model had become excessively agreeable. OpenAI published its own explanation, "Sycophancy in GPT-4o," describing how short-term feedback signals overpowered existing safeguards. Nobody chose that update. It simply arrived.
Models disappear from the menu with no notice. On August 7, 2025, GPT-4o was removed from the ChatGPT model picker as part of the GPT-5 launch. Backlash was immediate, and OpenAI restored it for paid users on August 12, with Sam Altman acknowledging they had underestimated how much people valued it. Five days is fast. Five days is also long enough to blow a client deadline.
Even identical settings do not guarantee identical output. The Thinking Machines research team published work on nondeterminism in LLM inference showing that sampling 1,000 completions at temperature 0, the setting most people believe removes randomness, produced 80 distinct outputs. The cause was batch level numerical variation in the serving infrastructure, not anything you did.
Now, the last week. On Reddit, r/OpenAI users asked whether GPT-5.6 Sol had been quietly upgraded. r/ChatGPTPro users accused ChatGPT of silently downgrading Pro requests to a smaller model. r/ClaudeAI called the update pace janky and unprofessional. r/ClaudeCode reported a new classifier breaking an established workflow. r/GoogleGeminiAI said the canvas feature had been ruined.
Read that list carefully, because I am about to be annoying about it. Those are user reports. They are not confirmed vendor actions. I have no evidence that any vendor deliberately downgraded anyone, and neither did the people posting.
Worth adding the other side of it. Anthropic's own postmortem, the one I linked above, states directly that they never reduce model quality because of demand, time of day, or server load. Take them at their word on intent. The bugs were still real, and that is the whole point: you do not need anybody to be acting in bad faith for your workflow to quietly stop working.
Separately and verifiably, Microsoft is merging its consumer Copilot app and its Microsoft 365 Copilot app into one, and retiring Copilot Podcasts, Group Chat, and Deep Research, per TechCrunch reporting on August 13, 2026. The mobile and web rollout began in mid-August, with the Windows and Mac rollout starting mid-September.
That is the uncomfortable pattern. Four vendor complaint waves in seven days, and almost nobody in any of those threads could demonstrate what changed. They had a feeling. Meanwhile the one change that is fully documented, the Copilot merger, was announced in advance and most people affected by it still have not read the notice.
The System: A Ten Input Baseline You Can Actually Maintain
Software teams solved this decades ago and gave it a boring name. Regression testing. You keep a set of known inputs with known-correct outputs, and you rerun them whenever something underneath might have moved.
The AI world calls the same thing a golden dataset or an eval set. Evidently AI and Langfuse both publish practical guides on building them, and both make the same point: start small and grow it as you find new failure modes. Tens of items is a legitimate size for a fast check.
Here is the version for someone who does not write code.
Pick your top three AI workflows. Not all of them. The three where a bad output costs you money, time, or a client relationship. For most owners that is proposals, client communication, and content.
Collect ten real inputs. Real, not invented. Pull actual client intake notes, actual raw meeting transcripts, actual product descriptions. Strip names if you need to. Ten total across the three workflows is fine. Make sure at least two are ugly, because ugly inputs break first.
Capture the known-good output. Run each input through your current prompt today, while you still trust it, and save the output verbatim. If today's output is not good enough to be your standard, fix the prompt first, then save. That saved file is your before picture. It is the most valuable AI asset in your business and it lives in a single document.
Write down the conditions. Date, tool, model name shown in the interface, and the exact prompt text. Model name matters more than people think. If your tool lets you pin a dated model version, the way the OpenAI API exposes dated snapshots like gpt-4o-2024-08-06, note it.
Run it monthly and after every announcement. Same inputs, same prompts, fresh session. Compare. You are not grading elegance. You are looking for four specific failures: format changed, length changed dramatically, required facts went missing, or tone shifted from your brand.
Decide what "drifted" means before you need to decide. My rule is simple. If two or more of ten outputs fail on format or missing facts, something moved. One failure is noise. Three is a signal.
When it drifts, you have three moves, in this order. Adjust the prompt, since most drift is absorbed by making implicit instructions explicit. Pin the model version if your tool offers it. Move that single workflow to a different vendor, which is far less painful than moving your whole business.
And now you can file a support ticket with dates, inputs, and outputs attached. That ticket gets read by a human. The one that says "Claude feels dumber this week" does not.
Practical Steps: Build It This Afternoon
1. Create one document called AI Baseline. One file. Google Doc, Notion page, whatever you already open daily. Not a folder system, not a spreadsheet with tabs. Complexity is why these things die in month two.
2. Log ten real inputs with today's date. Copy real inputs from the last thirty days of your actual work. Paste each one in under a heading with the workflow name. Twelve minutes, maybe fifteen if you are careful about client names.
3. Run each one now and save the output verbatim. Do not clean it up. Do not fix the one awkward sentence. The value is in the unedited record, warts included, because the warts are also part of the baseline.
4. Use AI to build your comparison checklist. You need to know what to look at next month, and you will not remember. Run this prompt with your own details filled in:
[The Job]
Help me define what "correct" means for one of my recurring AI outputs so I can spot changes in future versions.
[The Background]
I run [YOUR BUSINESS TYPE] and I use AI to produce [SPECIFIC OUTPUT, e.g. client onboarding emails].
Here is a real input I use: [PASTE REAL INPUT]
Here is the output I consider correct: [PASTE KNOWN-GOOD OUTPUT]
My clients are [AUDIENCE] and the output has to sound like [BRAND TONE IN 5 WORDS].
[The Deliverable]
A checklist of 6 to 8 pass/fail criteria I can apply to a future version of this output, written from the perspective of a quality control reviewer who has never seen my business before. Each item must be objectively checkable in under ten seconds, not a matter of taste. Include at least two structural checks (length, format, required sections) and two content checks (specific facts or elements that must appear).
[The Questions]
Ask me any questions you have.
5. Put a recurring monthly reminder on your calendar. Name it "AI Baseline Check" and give it thirty minutes. First business day of the month. If it is not on the calendar it will not happen, and the value of this system is entirely in its repetition.
6. Subscribe to the vendors' actual change channels. OpenAI publishes model release notes, Anthropic posts to status.anthropic.com, and Microsoft documents Copilot changes in its support articles. Ten minutes a month of reading beats three weeks of confusion.
7. Add one input every time something goes wrong. When a client flags a bad output, that input joins the baseline permanently. Over a year your set becomes a map of every way your workflows have failed, which is worth more than any course you will ever buy, including mine.
Frequently Asked Questions
Why does AI give me a different answer to the exact same prompt?
Some variation is normal and built in. Research from the Thinking Machines team found that even at temperature 0, the setting meant to eliminate randomness, 1,000 identical requests produced 80 different outputs because of numerical variation in the serving infrastructure. Small differences are expected. Consistent structural failures across multiple inputs are not.
How do I know if the model changed or my prompt got sloppy?
You cannot know without a saved copy of the original prompt and its output. That is the entire argument for a baseline. Once you have the exact prompt text stored, you rerun it unchanged. If the identical prompt produces materially different results, the variable that moved was not you.
Do AI vendors announce when they change a model?
Sometimes, and inconsistently. Deprecations and app changes get formal notices, like Microsoft's Copilot merger notice in August 2026. Behavioral retunes often do not. Anthropic's September 2025 postmortem documented quality issues that users noticed weeks before the company confirmed the cause publicly.
Should I pay for the API instead of the consumer app for stability?
For critical workflows, it helps. API access typically lets you pin a dated model snapshot, so you keep the same version until you choose to move. Consumer chat apps route requests dynamically and change more often. Most owners do not need this, but if a workflow touches client deliverables, it is worth the setup.
How often should I actually run my baseline test?
Monthly is the right default for most small businesses. Also run it immediately after any vendor announcement, after any interface change you notice, and the moment a client flags something odd. Thirty minutes a month is the entire cost of never being blindsided again.
You Are Allowed to Have Proof
Go back to that morning. The output that looked fine and was not.
What makes that moment miserable is not the bad output. Bad outputs happen. It is miserable because you have no ground to stand on. You cannot tell your client what happened, you cannot tell the vendor what happened, and you cannot even tell yourself with confidence.
That powerlessness is optional, and it costs about twenty minutes to opt out of.
I am not asking you to become technical. No code, no dashboard, no subscription. Ten inputs you already have, saved next to the outputs you already trust, opened once a month.
The businesses that noticed something changed last week were not smarter than everyone else and they did not have better tools. They just had a before picture. Everyone else had a feeling, and a feeling is not a bug report.
If you build your baseline this week, I would genuinely like to hear what you find in the first run. Come tell me. I read the replies, and the failures people discover in month one are usually more interesting than anything I would have predicted.
Keep a before picture, or spend the rest of your career arguing with your own memory.
About the author
Jonathan Mast is the founder of White Beard Strategies and runs a Facebook community of more than 500,000 entrepreneurs learning to use AI in real businesses. He created the Perfect Prompt Framework and speaks regularly on practical AI adoption for small business owners. He keeps his own AI baseline document in the same folder as his tax receipts, which tells you everything about how seriously he takes it and nothing good about his filing system.
Sources
- Chen, L., Zaharia, M., and Zou, J. "How Is ChatGPT's Behavior Changing over Time?" Harvard Data Science Review, March 2024. https://hdsr.mitpress.mit.edu/pub/y95zitmz
- Anthropic. "A Postmortem of Three Recent Issues." September 2025. https://www.anthropic.com/engineering/a-postmortem-of-three-recent-issues
- OpenAI. "Sycophancy in GPT-4o: What Happened and What We're Doing About It." April 2025. https://openai.com/index/sycophancy-in-gpt-4o/
- Fortune. "GPT-5's Model Router Ignited a User Backlash Against OpenAI." August 12, 2025. https://fortune.com/2025/08/12/openai-gpt-5-model-router-backlash-ai-future/
- Thinking Machines. "Defeating Nondeterminism in LLM Inference."
- TechCrunch. "Microsoft Kills Off Unsuccessful AI Features While Merging Its Separate Copilot Apps." August 13, 2026. https://techcrunch.com/2026/08/13/microsoft-kills-off-unsuccessful-ai-features-while-merging-its-separate-copilot-apps/
- Evidently AI. "A Tutorial on Regression Testing for LLMs." https://www.evidentlyai.com/blog/llm-regression-testing-tutorial
- Langfuse. "Golden Dataset Evaluation: Build and Maintain LLM Test Sets." https://langfuse.com/resources/engineering/golden-dataset-evaluation
