Agreement is not comparable across brands of different breadth. A brand judged against one query cannot be wrong, so it reaches 100% mechanically. Without a query floor those brands occupy the top of the agreement ordering and drive the correlation negative.

Mean agreement falls as breadth rises. Perfect scores collapse with it.

• Brands on 1 queryn=2079.4% mean agreement, 50% at 100%
• Brands on 2 queriesn=975.9% mean agreement, 22.2% at 100%
• Brands on 3 to 4 queriesn=2074.8% mean agreement, 15% at 100%
• Brands on 5 to 8 queriesn=3772.3% mean agreement, 2.7% at 100%
• Brands on 9 or more queriesn=6670.2% mean agreement, 3% at 100%

• Recall.Share of the query's Exact-labelled brands the model named.
• Precision (judged). Of the named brands ESCI judged, the share labelled Exact. This is the fair precision, since it ignores brands ESCI never covered.
• Leader. Whether the model named the Exact brand with the most judgements in the dataset.
• Off-corpus. Named brands ESCI never judged for the query. Includes products released after ESCI and brands outside its results, so it is not a hallucination rate.

Brands are matched to ESCI by normalised name. The run cost about $0.12 in API usage. Generated 2026-08-02.

• ExactProduct matches the query intent71%
• SubstituteNot exact, but a reasonable alternative20.2%
• ComplementUsed with the product, not instead of it1.7%
• IrrelevantDoes not match the intent7.1%

A human rated every product against the query intent. Exact is the ground truth a recommendation would be scored against.

Four defects, each handled in the numbers above.

• 23 brands carried multiple spellings and were collapsed into one record. 1MORE and 1More are one company. Left split they divide the ground-truth set, which understates recall before a model is even involved.
• 36 entries are product titles, not brands, because some sellers fill the field with keywords. Flagged rather than deleted, and excluded by default. 1 of them also carried split spellings, so these two counts overlap.
• 12 entries are aliases that could not be resolved. A name written in another script with a romanisation in brackets resolves cleanly, so コールマン(Coleman) folds into Coleman. A bare one does not: キヤノン is Canon, but there is nothing to key on without an alias table. They are excluded from the substitute graph, where they would otherwise read as a brand competing with itself, and left in the brand list.
• Only 602 of 3,928 brands appear against three or more queries. The usable brand universe is 15% of the raw count. The remainder are one-off marketplace sellers.

The substitute graph is where these surface. Uncleaned, the strongest rivalries are companies paired with themselves: 1MORE / 1More, Coleman / コールマン(Coleman), Canon / キヤノン.

• Temporal mismatch. The judgements predate the paper's publication in June 2022. Model responses are current, so a model naming a 2025 product this dataset never saw is a dataset gap, not a model error. The paper states no collection window, so publication date is the only firm upper bound.
• Relevance is not market share. Exact means the product matched the query intent, not that it sells well. A brand can be relevant and obscure.
• Near-duplicate queries. The sample holds several variants of the same search, so a print-on-demand seller can clear a three-query threshold without being recognisable in any ordinary sense. Deduplicating by query stem would fix this and has not been done.
• Amazon-shaped intent. These are product searches, not open-ended advice questions, so findings apply to purchase-adjacent prompts.

US locale only. The corpus is ordered by locale, so evenly spaced offsets draw from the US block at the front and the final offsets return no US rows. The sample is therefore even across the US portion, not across all 2.68M rows. ESCI has no category column, so the unit of analysis is the query, not the product category.

Reddy et al., Shopping Queries Dataset (ESCI), arXiv:2206.06588. Generated 2026-08-02 from a 0.4009% sample of 2,680,364 rows. Aggregates are precomputed and committed, both because the source API rate-limits anonymous access and because frozen numbers stay citable.