AI visibility varies by question type. Here is the measured spread.

Pilot · 3 repetitions per query · published 2026-11-02

The short version

We asked 15 B2B SaaS buying questions three times each and recorded which brands each answer named. Across the 45 answers, only 6 of 15 questions returned the same brand set in all three runs. The first-mentioned brand was stable in 9 of 15. The “how should I choose” questions were the least stable: mean Jaccard 0.697, and no stable first brand in five tries.

This is a pilot with 3 repetitions per query, not the 30-repetition run. Run-to-run variance is measured as Jaccard overlap between the three runs, not as a single score. Every result number below is recomputed from the published CSV when this page is built; the CSV is the record, this page is the reading.

What we asked, and how many times

Five B2B SaaS categories, three question shapes per category, three runs per question: 15 queries and 45 answers. The three shapes are a best-of shortlist question, an alternatives to the category leader question and a how should I choose question. The questions are in the CSV, one per row, unchanged.

The answers were sampled through an API, not a consumer chat window, with one grounded model: Perplexity Sonar. The CSV records the resulting brand sets, not the model name or the answer text. The paid diagnostic is a different surface and a different reading; this page does not claim to measure ChatGPT, Gemini or Google AI Overviews.

How the brand sets were extracted

1. Each category has a fixed candidate list of six brands, written before sampling and stored with the run script. Only names on that list are counted.
2. For each answer, every candidate brand is matched case-insensitively as a substring. The position of the first character match is kept; the brand is added to the set once, however many times it appears.
3. The set for a run is the de-duplicated list of candidate brands found in that answer. The top-1 is the candidate with the smallest first-character position, not a judgement about which brand was recommended most strongly.
4. An answer that mentions none of the six candidates produces an empty set. Two of the 45 answers in this pilot are empty; both are background-jobs how-to-choose runs.

How Jaccard is calculated

For one query, take the three run sets A, B and C. There are three pairs: A–B, A–C and B–C. Each pair is scored as |A ∩ B| / |A ∪ B| — the brands both runs share, divided by the brands either run named. An empty-versus-empty pair scores 1.0; an empty-versus-non-empty pair scores 0.0. The per-query value is the arithmetic mean of the three pair scores, rounded to three decimals in the CSV. A per-query value of 1.000 means all three runs returned the same set.

Results by query

What the 45 answers show

By question type

The spread is not uniform: Email API 1.000; Product analytics 0.956; Documentation platform 0.911; Background jobs platform 0.696; User authentication 0.650. The category labels are the ones in the CSV; the mean Jaccard here is the mean of that category's three per-query values.

What moved, and what did not

Instability is concentrated in the how-to-choose questions. None of the five had a stable top-1, and the mean Jaccard there is 0.697, against 0.902 for best-of and 0.929 for alternatives. The background-jobs how-to-choose query is the extreme case: two of its three runs returned no tracked brand at all.

Membership can be stable while order is not. The email-API how-to-choose question returned the same five brands in every run, but the first-mentioned brand moved between SendGrid and Postmark, so its top-1 is not stable even though its Jaccard is 1.000. The largest set swing is in user authentication: best-of returned 4, 6 and 4 brands, with Firebase and Supabase appearing only in the middle run; the how-to-choose answer grew from 1 to 2 to 4 brands.

Limitations

1. Pilot scale and one surface. 15 queries × 3 runs = 45 answers, from one grounded model sampled once. There is no confidence interval, no second model, no consumer-UI comparison and no repeat over time. These numbers describe this pilot; they are not a category-wide estimate.
2. Extraction is a fixed list plus substring matching. A brand the answer names outside the six-name candidate list is invisible here. A substring mention is counted even when it is not a recommendation, and top-1 is the earliest character position rather than a judged ranking. The set is a presence record, not a quality score.
3. Empty answers and unpublished wording. Two of the 45 answers named none of the tracked brands. The Jaccard rule scores an empty-versus-empty pair as 1.0, which raises the background-jobs how-to-choose mean to 0.333. The CSV publishes brand sets only, not the raw answer text, so a reader can check the arithmetic but cannot re-read the wording.

The raw data

Download ai-visibility-variance-pilot-2026-11-01.csv (15 rows, 13 columns). The file is served from this site with a stable filename; it is the same file this page is generated from. Columns: category, question_type, question, run1_brands, run2_brands, run3_brands, jaccard_mean, top1_run1, top1_run2, top1_run3, top1_stable, identical_set, brands_count. Brand sets are semicolon-separated; empty cells are empty sets.

What comes next

The next measurement is a 30-repetition run per query on the same fixed question set, so the same questions get a distribution instead of three points. That run is a development-pipeline task, not part of this publication: it needs its own sampling log and its own dated CSV. This page will keep reporting 3 repetitions per query until that file exists, and will link the new file rather than overwrite this one.