When does an AI visibility change deserve attention?

A daily answer can change without the underlying long-run tendency changing. A dashboard can also move because it asked a different mix of questions. Those are different explanations and need different checks.

This experiment compares a pooled two-proportion test with a question-matched Mantel–Haenszel test. The matched formula is checked against Parse’s implementation on 12 fixtures. We test a conventional two-sided threshold of 1.96 and a stricter matched threshold of 3.29, with agreement between the pooled and matched directions for the stricter rule.

These tests are components, not the full production detector. The simulation does not reproduce all evidence floors, missing-run rules, rival comparisons, collection changes or notification behavior. No production false-alert rate follows from these results.

Each question has a fixed probability of counting the brand, ranging from 10% to 70%. In the question-mix scenario those probabilities do not change, but high-probability questions receive more observations in the later period and low-probability questions receive fewer.

The pooled test flagged 1,999 of 2,000 comparisons, or 99.95%. The matched test at 1.96 flagged 98, or 4.9%; the stricter matched rule flagged none. Here the per-question behavior is unchanged: the pooled percentage moves because its weights changed.

This is a deliberately strong stress test, not an estimate of how often Parse experiences composition changes. It demonstrates why a change analysis must separate question mix from behavior within the same questions.

In a separate scenario, every question’s probability increases by ten percentage points. The pooled test detects 84.1% of repetitions, the matched test at 1.96 detects 87.95%, and the stricter matched rule detects 44.15%. A lower false-alert rate has a cost: more missed changes.

With unchanged independent answers, the respective flag rates are 3.1%, 4.3% and 0.05%. These are false flags relative to the known unchanged simulation target. They describe these specific probabilities, sample sizes and assumptions.

All rates are estimates from 2,000 repetitions. The result table includes Wilson intervals for Monte Carlo uncertainty. Zero observed flags does not establish a zero underlying probability.

In the final scenario, each of two seven-day blocks per period gets a common shift of plus or minus ten percentage points, shared by all questions. The long-run baseline remains fixed, but actual period probabilities can differ. This approximates temporary changes affecting many answers together.

The pooled test flags 57.95% of repetitions, the matched test 60.1%, and the stricter matched rule 37.25%. These are flags during temporary shared shifts. It would be misleading to call all of them statistical false positives: some simulated periods truly have different conditional probabilities.

The practical limit is persistence. Question matching alone cannot tell whether a change will last or whether independent evidence supports it. Repeated observations during one shared shift provide less information about long-run behavior than the same number of independent observations.

A per-comparison threshold is not a promise about the number of alerts across a whole product. Even an individually calibrated test can produce many false flags when applied to many brands, markets, engines and overlapping periods.

This study does not estimate the product-wide false discovery rate. That requires replaying the actual comparison schedule and selection rules, including how an alert is chosen and deduplicated. A stronger evaluation would use held-out stable periods, known interventions where available, missing-run stress tests and explicitly simulated day or week dependence.

Practical effect size also matters. A statistically detectable change may be too small to act on, while a large uncertain change may justify investigating answer evidence. Decide that distinction before looking at the most dramatic results.

Check that the questions, engines, language and location are comparable. Count distinct questions as well as answers. Inspect missing runs, extraction failures and any collection or model change. Read examples of both gained and lost brand occurrences.

Compare the same questions where possible, show the size of the change, and state the dates covered. Treat a claimed cause separately: a visibility change after a content edit does not establish that the edit caused it.

For automated alerts, validate both false flags and missed changes under the intended deployment conditions. Recheck calibration when collection or classification changes. The downloadable script makes this limited experiment reproducible; it does not certify current alerts.