Last week I told you AI had raised the ceiling on how much one person can build but had not touched judgment. Research published yesterday says one slice of judgment is now the fastest-improving thing in the field, so this article answers the question I got wrong: which parts of judgment are actually moving, and which parts still belong to you.
Last week I told you something I now think was half wrong.
I said AI had raised the ceiling on scope. One person can now build what used to take a team of six. I believed that then and I believe it harder now.
Then I said judgment was untouched. That part was wrong, and I would rather fix it in front of you than let it quietly age.
Here is the straight answer to the question in the headline. AI is measurably improving at one specific slice of judgment, the part where you run an experiment and correctly read what came back, and on at least one benchmark it is improving faster than almost anything else being tracked. It is not improving at the part upstream of that, where you decide what is worth building at all and what to deliberately leave out. Those are two different skills, and the field just handed us evidence that they are moving at very different speeds.
Candidly, I do not find this hard to say. I have owned bigger things in public than a wrong read on a benchmark. I have been to prison. I built a seven figure agency and lost it and built it again. A wrong call about where a ceiling sits costs me nothing I actually value, and leaving it uncorrected would cost me the only thing I do value, which is you trusting what I tell you.
So here is the revision. The ceiling that moved is scope, plus the experimental half of judgment. The ceiling that has not moved is taste: deciding what is worth doing, and deciding what to leave out on purpose.
Key Takeaways
- AI is improving fast at the experimental part of judgment: designing a test and correctly reading the result.
- AI is not improving at the part upstream of the experiment: choosing what is worth building and what to deliberately exclude.
- The headline number driving this story is one benchmark, published by the organization that built it, with three months of data and wide confidence intervals.
- The same paper contains a second measurement showing no acceleration at all, and almost nobody is quoting it.
- Your position now rests on what you know that nobody else in your market knows, and on your willingness to leave things out.
The Problem: I Drew The Line In The Wrong Place
The line I drew last week was clean and comfortable. Machines handle execution. Humans handle judgment. Everybody nods.
It was too clean. Judgment is not one thing. It is at least three things stacked on each other, and I treated them as a single block.
Layer one is choosing the problem. What is actually worth my next ninety days.
Layer two is designing the test. Given the problem, how do I find out cheaply whether I am right.
Layer three is reading the result. The number came back. What does it mean, and what do I do next.
When I said judgment was untouched, I was picturing layer one and claiming all three. That is sloppy thinking, which is exactly what I tell people to avoid. The evidence that landed yesterday hits layers two and three hard and barely touches layer one.
Here is the thing about why that matters to you. If you are a coach, a consultant, an agency owner, a small business owner, you have been told for two years that your safety is "judgment." That word has been doing an enormous amount of load-bearing work in a lot of keynotes, including some of mine. If it covers three different skills and only one is holding, then "lean into judgment" is not advice. It is a slogan.
I owe you better than a slogan. Jack Clark, who writes the Import AI newsletter, did the same thing I am doing this week. He quoted his own earlier position against himself, then moved his forecast. That is the standard.
The Evidence: Four Numbers And One Nobody Is Quoting
Here is what moved, with every source named. An article about honesty with numbers that carried an unsourced statistic would be worthless.
One. The taste benchmark. P-Zero Research published TasteVal, measuring experimental research taste across eight tasks representative of frontier AI development: picking the problem, designing the experiment, reading the result. They scored models against 24 human experts, at least two per task, recently from organizations including OpenAI, Google DeepMind, NVIDIA and Microsoft Research. Their finding, quoted exactly: "The experimental research taste of frontier models has doubled every 3.0 months since December 2025 (95% CI 1.7-5.0 months). The best model, Opus 5.5, significantly exceeds our human baseline with a compute multiplier of 2.30x (95% CI 1.15-4.37x), at roughly 1/30 of our expert baseliners' average cost per attempt."
That caveat travels with the claim every time it appears, including here. One benchmark. Three months of trend data. Confidence intervals wide enough to drive a truck through. Published by the organization that built it, with the tasks deliberately unreleased, so nobody outside can replicate it. And it measures one slice of judgment, not judgment in general. I reached the primary paper, by Oliver Jaffe and Dane Sherburn, on arXiv, not just the newsletter summary. That turned out to matter.
Two. The number almost nobody is quoting. Because I read the paper instead of the roundup, I found a second sentence in the same abstract: "Measured by final normalized performance, frontier models show no trend break, doubling every 14.6 months."
Read that twice. Same paper, same eight tasks, same models. Measured one way, taste doubles every three months and the world is on fire. Measured another way, nothing accelerated at all. Only one of those numbers went viral.
That is not a scandal. It is how measurement works. But it is the best illustration I have seen of why you do not reorganize a business around a headline number.
Three. Benchmarks in general are shakier than they look. The Oxford Internet Institute led a review of 445 AI benchmarks with 42 researchers, published at NeurIPS 2025 as "Measuring What Matters: Construct Validity in Large Language Model Benchmarks." Only 16 percent of the reviewed studies used statistical methods when comparing model performance. Roughly half measured abstract ideas like reasoning without defining those words. Lead author Andrew Bean put it plainly: without shared definitions and sound measurement, it is hard to know whether models are genuinely improving or just appearing to.
Four. The gap between a benchmark and a Tuesday. In July 2025, METR ran a randomized controlled trial with 16 experienced developers on 246 real tasks in their own repositories. The ones using AI tools took 19 percent longer. They had predicted they would be 24 percent faster. After living through the slowdown, they still believed AI had sped them up by 20 percent.
Then METR did the thing I respect most here. In February 2026 they published a post calling their own follow-up unreliable, because too many developers refused to work without AI and skewed the sample. They said plainly that developers are probably faster now than their 2025 numbers suggested, and that their own data was weak evidence of how much. A research nonprofit marking down its own headline result in public.
And the disagreement. TLDR AI's newsletter the next morning, October 13, argued close to the opposite: that expensive-loop tasks require building theories, managing proxies, budgeting time and compute, and diagnosing what went wrong when feedback is scarce, and that models do not learn those things in cheap-loop training. I could not verify that issue through a public archive, so take it as a newsletter's argument, not a measurement.
Two credible sources, same week, opposite directions. I will not resolve that by picking a favorite. It is a live disagreement among people who know more than I do.
What Actually Changed For Me
Here is my revised read, and it is narrower and more useful than what I said last week.
The experimental half of judgment is commoditizing. Run the test, read the number, decide what to try next. That work is getting cheaper and faster, and if your value to your clients lives there, your position is weaker this month than it was last month.
The part holding is upstream of the experiment. It is deciding what is worth testing, and it is deciding what to leave out.
The clearest evidence I have for that is not a benchmark. It showed up five separate times in one day in forums where people post what they actually built. An urban planner with no coding ability built a browser-based simulator running over roughly 481,000 Chicago buildings. His own words about the experience: "The domain knowledge was the part Claude couldn't do for me: which policies to include, how to model them, and what to leave out on purpose."
Read that last clause again. What to leave out on purpose.
A blind developer shipped an iPhone app to version five, having started with a small project to get weather data into an accessible format. He knew what accessible meant because he lives it. Nobody handed him that spec.
A software engineer whose game engine got one-shotted put it this way: "While the generated code itself is almost flawless every time, actually getting it into a place where I'd consider it complete and polished is where my energy went." The one-shot was free. Two days of back and forth followed.
And on October 13, Isar Meitis published episode 335 of Leveraging AI with Dima Negodiuk, who runs five businesses with no team and no coding background. His lesson was not "automate everything." It was start with your most boring repetitive processes, and keep humans in control as the workflows get complicated.
Every one of them owned the same thing: the decision about what the thing should be and what it should not include. They handed over everything else, cheerfully, and shipped.
Jack Clark moved his own forecast on the back of the taste result, writing: "I now think it's more likely we'll see full RSI become possible in 2027, whereas before I thought it was more likely in 2028." Forecasts move. In the 2022 AI Impacts survey, researchers' aggregate fifty percent estimate for high-level machine intelligence sat around 2060. In the 2023 survey of 2,778 researchers it had moved to around 2047. Thirteen years, gone in twelve months.
There is a much older version of this point. Michael Polanyi wrote in 1966 that we can know more than we can tell. The economist David Autor built an analysis of automation on that in his 2014 paper on Polanyi's paradox, arguing that work requiring adaptability, common sense and situated judgment is hardest to hand to a machine. Sixty years later, the urban planner said it in his own words.
Seven Things To Do With This
1. Write down the one thing you know that your market does not. Not your service list. The specific pattern you have seen two hundred times that a newcomer in your field has never seen once. If you cannot fill one page, that is your real finding today.
2. Audit your week against that page. Mark every hour last week that used that knowledge. Most people find it is under five hours out of forty. The gap between those numbers is your actual problem, and AI did not create it.
3. Practice deciding what to leave out. Take your next offer or project and cut one feature you were proud of. Write down why. Leaving things out is a taste decision, and taste is the last thing to commoditize. Almost nobody trains that muscle.
4. Learn to read a headline AI claim without getting played. Four questions, every time. Who published it, and did they also build the thing being measured. How long is the trend, because three months of data is a rumor. How wide are the confidence intervals. And is there a second number in the same paper pointing the other way, because with TasteVal there was.
5. Always find the second source before you act. Reading the primary paper instead of the summary is the only reason I found the 14.6 month number. Then a newsletter argued the opposite the next morning. One source is a story. Two sources pointing in opposite directions is the actual state of the field, and you hold your position loosely when that is what you are looking at.
6. Put the model on the layer it is good at. Hand it the experiment. Keep the question. Try this prompt, built on the Perfect Prompt Framework:
[The Job] Help me design the cheapest possible test of a business assumption I am about to bet money on. This is for me, the owner of [YOUR BUSINESS], and it matters because I cannot afford to learn this the slow way.
[The Background] My assumption is [THE SPECIFIC ASSUMPTION]. My market is [WHO YOU SERVE]. What I already know from experience is [THE THING YOU KNOW THAT OTHERS DO NOT]. Constraints: [BUDGET], [TIMEFRAME]. Leave out anything requiring hiring, new subscriptions, or more than [NUMBER] hours of my personal time.
[The Deliverable] Three tests, ranked cheapest first. For each: what I do, what number I watch, what result proves me wrong, what result proves me right. One page, plain language, written from the perspective of a skeptical operator who has watched businesses waste money on research.
[The Questions] Ask me any questions you have.
7. Revise in public when you are wrong. Not as a performance. As a habit. Jack Clark moved a forecast and said so. METR marked down its own study and said so. I got the line wrong last week and I am saying so here. Those are the people worth listening to in five years.
Frequently Asked Questions
Is AI going to replace business judgment entirely?
Not on current evidence. The experimental part of judgment, designing a test and reading the result, is improving quickly on at least one benchmark. The part where you decide what is worth building and what to deliberately leave out shows no comparable movement. Treat those as separate skills with separate timelines.
What is TasteVal and should I care about it?
TasteVal is a benchmark from P-Zero Research measuring how well AI models design experiments and interpret results across eight AI research tasks, scored against 24 human experts. You should care that it exists and care much less about its headline number, which rests on three months of data published by the organization that built the benchmark.
How do I tell whether an AI benchmark claim actually matters to my business?
Ask who published it and whether they built the thing being measured, how long the trend runs, how wide the confidence intervals are, and whether a second metric in the same paper disagrees. If a claim cannot survive those four questions, it should not change how you spend Monday.
Why do AI researchers keep changing their predictions?
Because the evidence keeps changing and the honest ones update. Between the 2022 and 2023 AI Impacts surveys, researchers' aggregate estimate for high-level machine intelligence moved roughly thirteen years earlier. Revision is a sign of seriousness, not confusion. Be more suspicious of the forecaster who never moves.
If AI gets better at judgment, what should I actually be learning?
Learn your own domain deeper, and learn to exclude. The people shipping real things right now contributed the specific knowledge of what their work should contain and what it should leave out, then handed the building to a machine. That combination of deep domain knowledge and ruthless exclusion is the scarcest thing in your market.
What I Actually Want You To Take From This
I came back to tell you I drew the line in the wrong place, and I want to be clear about what that does not mean.
It does not mean you should be afraid. I am not afraid of AI taking anyone's work. What I fear is someone reading a headline that says something doubles every three months, deciding there is no point, and walking away from the exact moment they were built for.
Because look at who actually shipped this month. A city planner who could not code. A blind developer on version five. A man running five businesses with no team. None of them won because they were technical. They won because they knew something specific and true about a real problem, and they were willing to decide what to leave out.
That is the whole game now. Not the volume you can produce, because everyone can produce volume. The judgment about what deserves to exist at all.
So spend your hours on the thing you know that nobody else in your market knows. Get ruthless about what you leave out. And when the next headline says something is doubling every three months, hold it loosely and go find the second source.
I was wrong about where the line sits. I am not wrong about this: the part of you that decides what is worth building is still the most valuable asset you own, and nobody has shipped a benchmark for it yet.
If you are working out where your own line sits, come find me. I would rather have that conversation in public than be right in private.
About the Author
Jonathan Mast is the founder of White Beard Strategies, where he provides AI coaching and mentorship for entrepreneurs who want results rather than hype. He leads a Facebook community of more than 500,000 members, runs the AI Insiders membership, created the Perfect Prompt Framework, and speaks on practical AI adoption for small business. He has been to prison, built a seven figure agency, lost it, and built it again, which is roughly why publicly correcting a wrong call about a benchmark does not keep him up at night.