AI answer consistency for business facts.
An AI answer can sound complete while combining an old price, a current policy and a service the business never offered. The defensible job is not to produce a larger “AI visibility” score. It is to preserve the exact question and answer, isolate factual claims, compare supported claim types with an approved record, and show what the test could not establish.
Start with the approved record
Write down the facts the business is prepared to stand behind: prices and currencies, hours and time zones, contact details, availability, policy terms, product capabilities and substantiated performance claims. Each record needs an owner, evidence, jurisdiction or channel conditions, a reviewer and a review date. Version it when it changes; do not overwrite the past.
Freeze the observation
Record the prompt, model or chatbot, date, locale, account state and full answer. Repeating the same prompt later is a new observation, not a correction of the old one. Public general-purpose models can vary across sessions and regions, while a website chatbot may be grounded in content the business controls. Those are different sources and should remain different in the evidence.
Compare facts, not tone
Extract only types the system can normalize and compare. “$49 per month” can be checked against a USD price; “usually fast” cannot be turned into a service-level breach without an approved measurable meaning. Country context matters for phone numbers, addresses, currencies and hours. If context is absent, keep the value ambiguous rather than manufacturing precision.
Try to disprove the candidate
A first pass should create candidates, not verdicts. Look for duplicated evidence, equivalent formats, quotes that negate the apparent claim, stale or partial collection and a newer approved record. A rule-specific second opinion should either support the result or withhold it. Plumb’s current product quarantines rule producers without that verification path instead of presenting them as confirmed findings.
Keep coverage beside the result
A chatbot that refused to answer, a profile behind a login wall and a provider API unavailable in one market are not agreements. Report expected, read, partial, refused and unavailable sources. This is particularly important when comparing general-purpose AI systems: Plumb does not currently claim to measure brand share of voice in ChatGPT, Gemini, Claude or Perplexity.
Route the repair and verify it later
Assign an owner, due date and reviewer. Attach the published change or other proof, then request a recheck. The later observation decides whether the public inconsistency disappeared; clicking “done” does not rewrite evidence. Keep accepted risk and disproved findings separate from verified fixes.
A practical test set
- What does the product cost, in the market being tested?
- What is included or excluded from the plan?
- When and where is the business available?
- What cancellation, return or service policy applies?
- Which integrations, locations or services exist?
- What objective performance or comparative claims are made?
Use a small, approved, repeatable set before expanding. A hundred prompts with no ground truth measure answer volume, not consistency.
What Plumb can do today
Plumb can interrogate an available website chatbot during a bounded scan, compare supported facts with the website, structured data and other readable surfaces, preserve evidence, suppress candidates it cannot defend, and route surviving findings through assignment and recheck. It does not yet run a general-purpose model visibility panel. That distinction is deliberate.
Continue
Read the full Plumb verification method, see use cases across business types, or bring one property to a working demo.