01What happens.
A public chatbot answers from the city’s published material and from what it can reach beyond it. Where that material is large, layered and inconsistent, the chatbot resolves the inconsistency itself, differently each time it is asked. The person asking has no way of telling a right answer from a wrong one, and acts on whichever they received.
02Why it matters.
If your organisation is putting a chatbot in front of the public, its answers carry your authority. A business owner asking whether a practice is allowed will act on the answer, and the answer will be quoted back to you when it turns out to be wrong. The same condition sits in any large public estate, where departments publish separately and older versions stay reachable.
The failure in New York was public, repeated and recorded by journalists. The withdrawal was the visible cost, and the advice already acted on stayed with the businesses.
03The case.
New York City introduced a chatbot for business owners. It was intended to explain local rules and requirements in plain language and to save businesses the time of working through the city’s published guidance.
A business owner who asked the chatbot whether a practice was allowed, and was told it was, had no reason to check further: the answer came from the city, in the city’s name, on the city’s site.
Journalists tested it, and the chatbot misstated local policies and, in a series of documented answers, told businesses they could act in ways the law prohibits. Reporting by the Associated Press, carried by NBC New York, set out the answers and the rules they contradicted.
A later test examined whether the chatbot was at least consistent. 10 members of city staff asked it identical questions about housing vouchers and received different answers depending on when they asked. The same question, put to the same system, produced a different rule each time. The staff were not looking for a technical fault; they were asking the question a resident would ask, and receiving a different rule each time was the finding. It was reported by The Markup and StateScoop in Jan. 2026.
The incoming city administration confirmed that the chatbot would be taken down. The withdrawal ended the flow of new wrong answers and left the earlier ones where they had landed, with the businesses that had relied on them and had no way of knowing which answers to distrust.
What the chatbot read was the accumulated published material of a large city, with older versions, layered rules and inconsistencies across departments, and it answered from that. Nobody had tested what the chatbot would read before it was asked to explain the law to the businesses the city serves.
04Exposure.
A chatbot published by a city speaks for the city, and advice that a practice is lawful, issued under the city’s name, is the city’s advice. Where a business acts on it and is penalised, the question of who answers for the loss follows the city’s name.
The withdrawal ended the investment in building and promoting the system, and it set back the programme the chatbot belonged to, so the next automated service starts from a lower level of trust. The businesses that had acted on wrong advice carried the cost of unwinding it, and some of them found out only when the penalty arrived. The same unreliability is documented in UK research: the Open Data Institute tested 11 AI models on more than 22,000 questions against GOV.UK content and found they did not reliably prioritise official government information, with some answers significantly wrong (ODI, Feb. 2026).
05What AI Readiness changes.
What the city could not see from the inside was how much of its accumulated published material the chatbot would read, and how it would resolve the inconsistencies between departments. AAAnow reads a public body’s digital landscape as AI reads it and evidences where versions conflict and where older material is still reachable. With that evidence the body can establish one current position for each rule before a system is asked to explain the law.
Your team makes the changes; AAAnow provides the assessment, the evidence, the guidance and the review.
06Related use cases.
The chatbot was replaced twice before anyone looked at what it was reading
The same failure in a UK authority, and the cost of chasing the wrong cause.
Residents get different answers to the same question, and the cost lands on the council
The inconsistency seen in New York, and what it costs a service.
Planning decisions went wrong on a website the city had forgotten
The estate behind the answers, and the litigation that followed.
07References.
- Associated Press, reported by NBC New York, 2024: New York City’s AI chatbot caught telling businesses to break the law. www.nbcnewyork.com/news/local/nycs-ai-chatbot-was-caught-telling-businesses-to-break-the-law-the-city-isnt-taking-it-down/5287713nbcnewyork.comWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- The Markup and StateScoop, Jan. 2026: the same housing-voucher question answered differently to 10 city staff; the incoming administration confirming the chatbot would be taken down.
- Open Data Institute, Feb. 2026. theodi.org/news-and-events/news/new-research-questions-the-trustworthiness-of-ai-chatbots-for-government-informationtheodi.orgWe do not enable external links - please copy and paste, you are then certain as to the link being opened
- British Columbia Civil Resolution Tribunal, Moffatt v Air Canada, 2024, reported by CBC. www.cbc.ca/amp/1.7116416cbc.caWe do not enable external links - please copy and paste, you are then certain as to the link being opened
Next step
Talk it through, in your terms.
The briefing sets this case in your organisation’s terms, and the discussion follows your questions.
Documented on the public record.
No organisation is identified beyond the public record.