How well does Parse identify and classify brands?

First, an extractor must find a name and resolve which entity it means. Then a classifier decides how the answer treats that entity. A brand can appear in a warning, a comparison, a source citation or a recommendation. Counting all of these as recommendations changes the meaning of the number.

The historical evaluation starts with supplied brand-answer pairs. It tests whether the classifier counts those pairs as recommendations; it does not measure how many brand names the extractor missed. The fresh source review examines the identification boundary, but is a small diagnostic casebook rather than an accuracy benchmark.

On the 165 held-out cases, 89 recommendations were correctly counted and 56 non-recommendations were correctly excluded. There were 16 extra recommendations and four missed recommendations. Overall agreement was 87.9%.

Precision was 84.8%: 89 of the 105 counted recommendations matched the reference. Recall was 95.7%: 89 of the 93 reference recommendations were found. These are different questions. The net overcount was 12 cases, or 7.3 percentage points of this selected test set; that is not a correction factor for Parse pages.

Recommended and alternative labels count as positive in this historical evaluation. This definition belongs to this experiment and should not be substituted for the definition printed beside a product metric.

The test included 75 recommended, 18 alternative, 14 compared, 29 incidental, 22 warned-against, one listed and six not-present cases. Eight of 14 comparisons and eight of 29 incidental references were counted as recommendations. None of the 22 warnings or six not-present cases was counted.

Six not-present cases cannot establish a reliable false-match rate. The single listed case cannot establish performance for factual product lists. The downloadable table also reports exact treatment-label agreement, which is stricter than the recommendation-versus-other decision.

The production question is therefore not just “How accurate is the model?” It is “Which kinds of language does it mistake for a recommendation, and how often do those kinds occur in the answers being measured?”

These are retained historical model evaluations, not a new independent human audit of the current production system. The cases combine an earlier audit, recent answers and historical answers. Answers span November 2025 through September 2026. Calibration and test sets share no answer, but the selected cases are not a random sample of all Parse observations.

Nine blind model labellers contributed labels. Disagreements with a comparison classifier were adjudicated, while many agreements were retained without a fresh adjudication. That process can preserve shared mistakes and favor the comparison classifier. A small earlier quality check also found disagreements in the labels.

Two held-out labels were corrected in the retained evaluation. With those corrections, agreement is 145 of 165; with the original labels it is 143 of 165, or 86.7%. Both versions are downloadable. We report counts rather than a population confidence interval because selection and correlated cases limit that interpretation.

A separate model reviewer read 24 public answers without seeing Parse’s stored extracted names. It marked 222 explicit brand, product or publisher entries and attached source quotes and uncertainty notes. These are annotations, not 222 independently verified entities.

The answers come from CRM, mattresses, payroll and pet insurance, with three questions per market and one answer per question on each of two engines. Selection required a short answer and an existing nonempty extraction. That excludes complete extraction failures and makes this unsuitable for estimating overall recall.

The casebook shows concrete decisions a benchmark must settle. In source-02, the answer names Freshsales while the stored entity is Freshworks. In source-11, branded materials appear beside mattresses and publishers. In source-24, the answer recommends research sources, while a regulator also appears in the stored extraction. A literal name comparison would mix aliases, parent-company resolution, source mentions and actual errors.

We publish the answer packet, blind annotations and stored names separately so readers can inspect these distinctions. We have not converted disagreements into a headline identity accuracy score. Resolving them requires a declared entity policy and adjudicated labels, including ambiguous names, misspellings and complete misses.

The historical case table contains pseudonymous answer groups, labels and predictions. The script reproduces the error counts and checks that calibration and test answers do not overlap. It supports an arithmetic audit, but without the historical full answer texts it does not support independent relabelling of those cases.

The fresh source casebook does include public answer text, enabling independent annotation of that small sample. It does not verify the factual truth of the AI answers themselves.

A stronger next evaluation would freeze the current extractor, sample answers before observing success or failure, have independent annotators label every eligible entity, adjudicate disagreements, and publish precision and recall separately by company, product, ambiguous name and treatment class. Those results are not yet established here. Current pages should be read with their answer evidence, collection dates and metric definitions.