Why we ran this

A large amount of commercial advice now rests on a single observation: someone asks a chatbot which vendors it recommends, screenshots the answer, and treats that list as a ranking. Whether that observation means anything depends on a measurable property nobody publishes: how much the answer changes when the same question is asked again.

This note measures that property directly. It does not attempt to explain why models name particular vendors, and it is not a quality ranking of the products named. It is a stability measurement, and its only claim is about variance.

What we did

We wrote eight buying questions covering software infrastructure, business software and professional services, all framed for a United States buyer. Each question asked for up to five names and explicitly requested names only, so that the output could be parsed without interpretation.

Each question was sent ten times to each of two models, as independent requests with no shared conversation history and no change to sampling parameters. Every response was parsed into an ordered list of at most five names, normalised to lower case. For each model and category we then compared all 45 possible pairs of runs and averaged their set overlap.

A pilot pass exposed a measurement artifact worth reporting: with a low output cap, some responses were cut mid-word, producing truncated fragments that a naive parser counts as distinct vendors. Anyone reproducing this work should confirm that responses complete before treating name variance as model behaviour. The published dataset comes from the second pass, with output caps high enough that no response was truncated.

What the numbers show

Stability is not a single property of a model. Both models were near-deterministic on some categories and visibly unstable on others, and they agreed on which categories were which. Observability, e-signature, payroll and vector databases all sit at the stable end. SEO agencies and cyber insurance sit at the unstable end in both models.

The pattern is consistent with how much settled public reference material exists per category. Categories with a small, well-documented set of widely compared products produce repeatable answers. Categories made up of many similar service firms, where public comparison material is thin and largely promotional, produce answers that shift between runs.

The two models also differ in breadth rather than only in stability. gpt-5.4-mini drew on a wider candidate pool and rotated through it; gemini-3.8-flash returned a narrower set and repeated it, reaching perfect overlap on two categories. Neither behaviour is better as such, but they imply different things for anyone reading a single answer as a ranking.

What follows from this, and what does not

Follows: a single observed answer is a sample, not a measurement. In the least stable categories tested, no vendor appeared in all ten runs of one model, so an appearance in one session says little about standing. Any claim about being recommended by an AI system should state how many runs were observed and on which model and date.

Follows: measurement needs to be per category. A method calibrated on observability tooling will overstate stability for service categories by a wide margin.

Does not follow: that stable names are better products, that unstable categories are being manipulated, or that these figures will hold for other models, other phrasings, retrieval-enabled search products, or later model versions. Each of those is a separate question and would need its own collection run.

How this becomes a recurring series

This is release v1 of a recurring measurement. We intend to repeat the identical prompt set at intervals, adding models as they become available, so that changes over time can be read against a fixed baseline rather than against memory. Prompts will not be reworded between releases; new prompts will be added as new series rather than replacing existing ones.

The raw per-run data is published below in JSON and CSV. If you reproduce this and get materially different figures, we want to hear about it and will publish corrections against this version number.