How many prompts do you need to measure AI visibility?

Updated October 6, 2026. The previous version described 1.96 times a pooled error estimate as a 95% band and extrapolated a universal prompt requirement. Its arithmetic did not establish that the interval covered the target 95% of the time. We have removed those coverage and prompt-count claims.

This version supplies a frozen row-level panel and executable sampling study. It measures how much small samples differ from that panel. It does not estimate all buyer conversations, validate a confidence interval or certify the precision of every Parse page.

For each market and engine, we retained questions with at least ten usable observations and selected the five most frequently counted brands. This produced 40 series. For each series, the reference is the average of the individual question-level observed rates, giving every eligible question equal weight.

We then sampled questions without replacement and sampled answers within each question without replacement. Each design was repeated 200 times per series, for 8,000 resamples per design. These resamples reuse observed data; they are not new independent AI answers.

The ten-answer designs are directly comparable on the same series. Ten different questions had a root-mean-square error of 13.1 percentage points; five questions answered twice had 15.4 points; one question answered ten times had 27.3 points. Root-mean-square error summarizes the distance from the frozen reference, giving larger misses more weight.

At 50 answers, five questions answered ten times had an error of 11.1 points, compared with 7.4 points for ten questions answered five times. Broader question coverage helped in this panel at both tested answer totals.

This does not mean repeating a question is useless. Repeats reveal variation in its answers. New questions cover different needs. The right balance depends on the quantity being measured and how different the questions are. Closely paraphrased questions may add little new coverage.

The downloads include median absolute error and the 95th percentile of absolute error. A percentile of these resampling errors is descriptive; it is not a validated 95% confidence interval for a new market or future period.

We selected four markets deliberately to span software, employment services, physical goods and insurance. Within each, a deterministic ordering selected up to 20 public, visible, non-customer-written questions. One selected question had no observations, leaving 79. The engines were ChatGPT Search and Google AI Mode.

This is a reproducible selection, not a probability sample of people or all possible buyer questions. There are no buyer-volume weights. Frequent brands were selected using the whole panel, so these findings do not establish performance for rare brands. Market and engine results are equally represented through their selected series.

Answer weighting and question weighting answer different questions. This experiment weights questions equally. A product percentage based on all collected answers gives more weight to questions with more successful runs. Compare the method and denominator before comparing two percentages.

We excluded 340 observations whose brand and selection arrays were missing or did not align. Exclusions were uneven: one in CRM, 147 in mattresses, 18 in payroll and 174 in pet insurance. We did not convert those observations into evidence that a brand was absent.

Six repeated text hashes occurred within the same question and engine. Their observations were retained because they were separate collected answers, but identical text and repeated dates do not create independent evidence. Collection gaps and incomplete extraction can bias the retained sample.

The stored brand selection flags define this dataset’s counted occurrences. This is not an independent audit of extraction correctness. An incorrect stored brand would affect the analysis; that uncertainty is separate from sampling error.

The extraction requested April 9 through October 5, a 180-day window ending before October 6. Actual observations begin May 29. Requesting 180 days does not create 180 days of data for each question.

On 40 series with a common eligible question set, the median absolute difference between the last 30 days and the available observations in the requested 180-day window was 5.7 percentage points. This compares different time averages, not which one is more accurate.

A longer window can supply more observations and smooth short-term changes. It can also retain older model behavior. Publish the requested window, actual date coverage, answer count and distinct question count together. A 180-day public index and a shorter monitoring view serve different purposes.

Start by stating the target: a fixed list of tracked questions, a market taxonomy or a separately justified population of topics. Cover the relevant topics and document how questions are generated, selected, added or removed. Keep engines and collection conditions visible.

Estimate uncertainty using the actual dependence structure. Answers share questions, dates and sometimes sources; treating every answer as an independent coin toss can overstate precision. Check interval coverage on held-out data or controlled simulations before advertising a confidence level.

For decisions, predefine the smallest change worth acting on. Then test whether the chosen design can detect it at an acceptable false-alert rate. This study supports broader coverage within its panel. It does not justify a universal minimum such as 50 runs, 145 prompts or a guaranteed five-point margin.