Show HN: Reliably Incorrect – explore LLM reliability with data visualizations
Developer Adam Sohn built "Reliably Incorrect," an interactive data visualization project that maps out where and how large language models fail, even when they sound completely certain. It matters because AI teams deploying LLMs in production still lack good diagnostic tools for...
Adam Sohn, a developer publishing at adamsohn.com, has released a project called "Reliably Incorrect," a hands-on data visualization tool designed to expose a core problem with large language models: they get things wrong with total confidence. The project went live at adamsohn.com/reliably-incorrect and was submitted to Hacker News on May 2025 under the "Show HN" category, where it drew focused if modest engagement from the technical community.
Why This Matters
The AI industry has spent years selling confidence in LLM outputs while quietly admitting the hallucination problem exists but is hard to quantify. Sohn's project attacks that gap head-on by making failure modes explorable rather than theoretical. MIT researchers published findings in January 2025 documenting specific shortcomings that make LLMs structurally less reliable than users assume, and yet most teams deploying these models in production still have no practical diagnostic layer. Tools like this one, free and open to explore in a browser, put that diagnostic capability directly in front of the developers who need it most.
Daily briefing from 50+ sources. Free, 5-minute read.
The Full Story
The centerpiece of Sohn's project is an illustration that most engineers working with LLMs will recognize immediately. The project uses a Rubik's Cube state analysis task to show exactly how an LLM reasons through a complex spatial problem, tracking the model's internal logic step by step and exposing the points where it drifts confidently into error.
The cube scenario laid out in the project is specific and deliberate. Sohn presents a partially scrambled cube with a defined configuration: the up face has red stickers in the top row with white filling the remainder, the down face is mostly yellow with orange in the bottom row, the right face shows red on the left columns and yellow on the right column, and the left face has white on the left column with orange filling the rest. The front face is all green and the back face is all blue. A solved cube it is not, and the task of figuring out which moves return it to a solved state is genuinely difficult.
What the project documents is how the LLM walks through this problem. The model correctly identifies the standard cube notation, correctly names which faces border the back face, and then begins tracing the index-level mappings for a B move, the clockwise rotation of the back face. It maps out which stickers on the U row cycle to R, which R stickers cycle to D, and so on. The logic appears rigorous. The notation is precise. And yet the model loses the thread when it has to hold multiple simultaneous state changes in memory and reason about how a cycle of four face segments resolves to a solution.
This is the core of what "Reliably Incorrect" is trying to show. The failure is not random noise. It is not a simple factual error where the model retrieved the wrong data. It is a systematic breakdown in a specific type of reasoning task, one requiring precise positional tracking across multiple steps, and the model's confidence never wavers even as the logic starts to drift. The visualization layers over this analysis to help users see which categories of tasks produce this pattern.
That distinction matters enormously for anyone building AI agents or deploying LLMs in real workflows. Errors that cluster around predictable failure modes are manageable if you know where those modes are. Errors that look random are not. Sohn's project argues that LLM failures are the former, and that visualization is the right tool for making those clusters visible.
Key Details
- Adam Sohn published the project at adamsohn.com/reliably-incorrect in 2025.
- The Hacker News "Show HN" submission received 2 points and generated 4 comments.
- The demonstration uses a 6-face Rubik's Cube with a specific partially scrambled configuration involving 6 distinct color states.
- MIT published research in January 2025 identifying structural shortcomings in LLM reliability across specialized reasoning domains.
- Factagora released an API-based LLM validation service, described explicitly as "an API that catches what your LLM confidently got wrong," representing one commercial response to the same problem.
- The project targets the "Show HN" audience, primarily developers and engineers evaluating tools for production deployment.
What's Next
The immediate value of "Reliably Incorrect" is as a reference project for teams building evaluation frameworks around their own LLM deployments. Expect similar visualization-based diagnostic tools to emerge throughout 2025 as the evaluation and interpretability category matures from academic research into developer tooling. The real test will be whether projects like this one get picked up and extended by the broader open-source community into reusable frameworks rather than staying as standalone demos.
How This Compares
The LLM reliability problem is not new, but the approaches to solving it are still fragmenting across the industry with no clear standard. Factagora's API service represents one commercial approach: build a validation layer that sits between your application and the LLM and flags outputs that are likely wrong. That is a useful product, but it is a black box solving a transparency problem with more black boxes. Sohn's visualization approach goes in the opposite direction, prioritizing understanding over correction. These are genuinely different philosophies, and for teams that need to explain AI behavior to stakeholders or regulators, the understanding-first approach has a practical advantage.
Compare this also to the broader academic work coming out of institutions like the University of Illinois Urbana-Champaign, which has examined how LLMs process and interpret data visualizations. That research mostly treats the model as the analyzer of visual information. Sohn flips the relationship and uses visualization to analyze the model, which is a smarter framing for practitioners who need accountability, not capability demonstrations.
What makes the timing of this project notable is the skeptical reassessment happening across the industry right now. Commentators including the "Entertainment Strategy Guy" Substack have argued in 2025 that LLMs may actually be degrading data analysis workflows by giving users plausible-sounding wrong answers with authoritative tone. Sohn's work is a direct, practical response to that critique. It does not argue that LLMs are useless. It argues that they fail in specific, findable ways, and that finding those ways is the job of developers, not something to leave to chance in production. That is a mature and defensible position, and it is the kind of AI tool the industry needs more of right now.
FAQ
Q: What does "reliably incorrect" mean in the context of AI? A: It means an LLM consistently produces wrong answers in predictable categories of tasks, not randomly across all tasks. The danger is that the model states these wrong answers with the same confident tone it uses for correct ones, making errors hard to spot without a systematic way to examine where failures cluster.
Q: Why is a Rubik's Cube used to test LLM reliability? A: Solving a scrambled Rubik's Cube requires tracking precise positional changes across multiple steps simultaneously, which is a known weak point for language models. It is a clean, verifiable problem where right and wrong are unambiguous, making it a useful benchmark for exposing the kind of multi-step spatial reasoning failures Sohn documents.
Q: How can developers use this kind of visualization in real projects? A: Teams can use the same approach to map failure patterns specific to their use cases, such as legal document analysis or financial data interpretation, and then build targeted review checkpoints where the model's reliability drops. Check the AI Agents Daily guides section for practical walkthroughs on building LLM evaluation layers into production pipelines.
Sohn's "Reliably Incorrect" is a small project with a clear-eyed argument: LLM failures are systematic enough to be mapped, and visualization is one of the best ways to do the mapping. As the AI news cycle moves past capability announcements toward accountability and deployment reality, that argument is going to get louder. Subscribe to the AI Agents Daily weekly newsletter for daily updates on AI agents, tools, and automation.
Get stories like this daily
Free briefing. Curated from 50+ sources. 5-minute read every morning.
