Building products requires a strange mix of hubris and humility. The confidence that some kind of answer is out there balanced with a very real sense of “I don’t know.”

Or, at least “I don’t know, yet.”

Because not knowing drives curiosity, learning, exploration and everything that makes building for a living such a never-ending adventure. Related to my prior post “It’s in the way that you use it”, poor use of AI damages this learning path. Rather than providing more paths to really build mastery, its bias towards helpfulness and credibility makes it easy to take AI answers at face value.

Too easy, as it turns out. New research confirms this:

Merely having access to AI advice nearly eliminated participants’ willingness to suspend judgment,even though the advice was usually wrong.

Oof. Access to AI advice reduced willingness to say “I don’t know” from 44% to 3%. Response accuracy dropped from 27% to 9%. Confidence, meanwhile, rose from 30% to 76%. This is consistent with earlier research from BCG.

Why agents aren’t enough

Amidst all of the well-deserved excitement about the unreasonable effectiveness of AI — and in particular, agents operating in loops — what is easily lost is the question:

What is providing the error correction signal in your agentic loop?

Product development is one of the easiest domains for this. Compilers, linters, e2e testing, o11y — all of these help catch bad guesses on the part of an LLM, and since it’s in a loop, the agent can continue on its merry way, converging on a demonstrably correct answer.

Abstract mathematics seems to be the next one — sucks to be you, Jacobian Conjecture — where rigorous tools like Lean can be used to conclusively verify the results.

Absent error correction signals, even the most capable agents are still guessing. A lot.

I suspect much of the AI schism comes down to whether your agentic loop can be built with high-quality error detection and feedback. Or not.

I see a lot of products claiming to generate answers via agents. It’s obvious that many of them have no error signal and the whole product is just LLMs making credible guesses.

Perhaps if more developers started from “I don’t know” they’d be more likely to build loops that can actually win.