# Do you need UI scraping to track AI visibility? We tested it. > We ran one buying-intent prompt 120 times across ChatGPT, Gemini, and Google AI Overviews, four ways: a DIY tracker from memory, with search, UI scraping… Source: https://contengi.com/insights/llm-tracking-accuracy-report Insights · Research report We ran one buying-intent prompt 120 times across ChatGPT, Gemini, and Google AI Overviews, four different ways: a DIY tracker answering from memory, the same tracker with live search, a paid UI-scraping tool, and a person typing it into the apps by hand. Then we scored every answer with the same code. By [Stefan Maritz](https://www.linkedin.com/in/stefan-maritz/) · Published 15 October 2026 · 14 min read 120answers across three AI engines 4ways of getting them, from memory to hand-typed 68% vs 68%DIY from memory and UI scraping, brand overlap with the hand-typed answers $1.59total API cost of the DIY side of the test The short version Nothing agrees with itself. Ask the same question twice and you get a different answer, whichever method you use, the hand-typed one included. The brand list overlaps 69 to 79% from one run to the next. The methods sit about as far from each other as they sit from themselves. A DIY tracker answering from memory overlapped 67.8% with the manual runs on brands, UI scraping overlapped 68.0%. The winners come out the same every way. Gusto and OnPay appear in all 110 unique answers, QuickBooks in 109, and Gusto is named first in 102 of them. Cited sources are noise for everyone. Two runs of the same method share 31 to 41% of their sources, two different methods 5 to 15%. We saw the same at Backbase across thousands of answers: around 80% overlap between a DIY tracker and AirOps, and around 70% from one run to the next inside both. ## How this started I published a guide on The Workflow on [building your own AI visibility tracker](https://www.theworkflow.digital/p/build-your-own-ai-visibility-tracker). A small app on Supabase and Vercel sends brand, category, and comparison prompts to the models through one AI gateway, stores every answer, and scores how often your brand gets mentioned and cited. A 50-prompt run costs a few dollars, and the hosting sits on free tiers. Ugi Djuric, the founder of ContentMonk, pushed back in the comments. His argument had two parts. First, answers pulled through an API are not the answers your customers see in the apps, which is why tools like Peec, Profound, Otterly, and ContentMonk scrape the chat interfaces instead. Second, once you add servers and API costs, building it yourself costs about the same as a subscription. > “So you end up paying the same amount for essentially the fake and lower quality data.” Both points are fair, and the first one is testable. I'd already compared AirOps against API answers on a much larger set of thousands of answers and found them close, so we agreed on a public test. Ugi picked 10 buying-intent prompts across industries and ran each one 10 times through ContentMonk's UI scraping on ChatGPT, Gemini, and Google AI Overviews, from the United States. Then he [shared the full results](https://app.contentmonk.io/shared/prompt-check/AEwM4wX9uqpdE8E__56anoPl?prompt=1), every answer and every source. Credit where it's due, that is how this kind of argument should be settled. ## What we tested We took his second prompt, word for word: “Best payroll software for a US small business with under 20 employees” Then we answered it four ways, 10 times on each engine. If a model with no search at all, answering purely from what it learned in training, lands as close to the apps as a scraper does, then live search is not what decides which brands get recommended. Method ChatGPT Gemini Google AI Overview DIY, from memory GPT-6 Luna, search off10 answers Gemini 3.5 Flash, search off10 answers Full overview via DataForSEO10 answers DIY, with search GPT-6 Luna, web search on10 answers Gemini 3.5 Flash, Google Search on10 answers Same 10 overviews as above10 answers UI scraping ChatGPT app (gpt-6)10 answers Gemini app (3.5 Flash)10 answers Google results page10 answers Manual ChatGPT app, by hand10 answers Gemini app, by hand10 answers Google results page, by hand10 answers 120 answers in total, 110 of them unique: an AI Overview is always a Google search, so both DIY setups share the same 10. DIY, from memory GPT-6 Luna and Gemini 3.5 Flash through the Vercel AI Gateway with web search switched off, so they answer from training data alone. Google AI Overviews through DataForSEO. DIY, with search The same models with live web search on (OpenAI web search, Google Search grounding). Google AI Overviews through DataForSEO. UI scraping ContentMonk's own run, scraped from the ChatGPT, Gemini, and Google interfaces in the United States and shared publicly by its founder. Manual The prompt typed into the real ChatGPT, Gemini, and Google apps by hand, 10 times on each, every answer and source copied out. ### How we scored every answer We measured three things. Brands named: one extraction model lists every payroll product an answer recommends, and a brand only counts if its name appears in the text, so review sites and publishers don't count as brands. Sources cited: the domains each answer links to. What the answer says: text embeddings turn each answer into a point in space, and we measure how close two answers sit in meaning, ignoring formatting. Every method went through the same scorer, ContentMonk's answers included, so any difference comes from the answers themselves. What overlap means Overlap is the share of items two answers have in common. If one answer names Gusto, OnPay, QuickBooks, and Patriot, and another names Gusto, OnPay, QuickBooks, and ADP, they share three of the five brands between them: 60%. We worked that out for every pair of answers and took the average. ## Finding one: nothing agrees with itself Before you compare methods, you need to know how much one method moves on its own. It moves a lot. Type the prompt into the apps twice and the brand lists overlap 69.2%. UI scraping repeats itself at 76.9%, the DIY tracker with search at 70.2%, and answering from memory is the steadiest of the lot at 78.9%. This is the floor every AI visibility number stands on. A tool that shows you one answer per prompt is showing you one roll of the dice, however it got that answer. Brands named DIY memory78.9% DIY search70.2% UI scraping76.9% Manual69.2% Sources cited DIY memoryAI Overviews only17.7% DIY search40.5% UI scraping39.6% Manual31.4% What the answer says DIY memory94.1% DIY search93.4% UI scraping93.4% Manual92.9% Average overlap between two answers from the same method, ChatGPT, Gemini, and AI Overviews averaged. Answering from memory produces no citations, so its source figure comes from its AI Overviews alone. ## Finding two: the methods sit as far apart as each one sits from itself On brands, the DIY tracker answering from memory overlapped 67.8% with the hand-typed answers. UI scraping overlapped 68.0%. That is a tie, and both sit inside the range a single method moves by itself. Against each other, memory-only DIY and UI scraping overlapped 74.7%, closer than two hand-typed runs get to each other. The answers themselves say nearly the same thing. Every pair of methods lands between 87 and 90% similarity in meaning, a few points under the 93 to 94% you get when one method repeats itself. Brands named, method against method DIY memoryvsManual67.8% UI scrapingvsManual68.0% DIY searchvsManual62.0% DIY memoryvsUI scraping74.7% DIY searchvsUI scraping70.5% Dashed box: how much each method agrees with itself, run to run (69 to 79%). All three measures, both kinds of overlap Brands named 69\-79% 62\-75% Sources cited 31\-41% 5\-15% What the answer says 93\-94% 87\-90% ## Engine by engine Averages hide detail, so here is brand overlap with the hand-typed answers for each engine. The faded bar is the manual runs against themselves, the noise you would get from simply asking again. ChatGPT DIY memory78.5% DIY search70.1% UI scraping76.1% Manual vs itself77.3% Gemini DIY memory61.7% DIY search52.8% UI scraping63.0% Manual vs itself59.5% Google AI Overview DIY memory63.1% DIY search63.1% UI scraping64.9% Manual vs itself70.7% ChatGPT Memory-only DIY matched the hand-typed answers best of all, 78.5% against 76.1% for UI scraping and 77.3% for the manual runs against themselves. Search pulled it down to 70.1%: with search on, the API version writes shorter answers and cites less pages than the app does. Gemini A tie: 61.7% for memory-only DIY, 63.0% for UI scraping. Gemini moves more than any other engine, even against itself, where two hand-typed runs only overlap 59.5%. Google AI Overviews 63.1% for DIY, 64.9% for UI scraping, and 82.4% between the two, close to the 85.1% scraping gets against itself. Our first pass used a provider that returned the collapsed overview, about 600 characters. Switching to one that loads the expanded text closed the gap. ## The brands that count came out the same For a marketer, the question that pays the bills is who gets recommended. Gusto and OnPay appear in all 110 unique answers, QuickBooks in 109, Patriot in 90, and Gusto is the first brand named in 102 of them. Every method, every engine, same story at the top. Further down it gets noisy. Square, ADP, and Rippling drift in and out depending on the engine and the run, which is exactly why one answer per prompt tells you so little about where you stand. Gusto110 / 110 OnPay110 / 110 QuickBooks109 / 110 Patriot90 / 110 Square57 / 110 ADP53 / 110 Rippling23 / 110 Xero18 / 110 Every method and every engine combined. Gusto is the first brand named in 102 of the 110 answers. ## Sources are noise, whichever tool you use Cited sources move far more than brands. Two runs of the same method share 31 to 41% of their cited domains, and two different methods share 5 to 15%. Where both had sources to compare, the DIY tracker came out closer to the hand-typed answers than scraping did, 13.9% against 11.5% overall and 20.8% against 13.2% on ChatGPT. At these levels the sensible read is that individual cited pages move too much to track one by one, whatever you pay for the data. The domains that keep coming back across many runs are the signal worth watching. 08 · We have seen this before ## The same result at Backbase, over thousands of answers This wasn't our first run at the question. At Backbase I compared the same DIY setup against AirOps over thousands of answers. Two tests, two different paid tools, one conclusion: a well-built DIY tracker and a paid platform disagree with each other about as much as each one disagrees with itself. ~80%overlap between the DIY tracker and AirOps on cited brands, sources, and answers ~70%overlap from one run to the next inside each tool, DIY and AirOps alike ## What it cost The whole DIY side of this test, every API answer with and without search, the AI Overviews, and the scoring, cost $1.59. Here is what each answer costs at October 2026 prices, measured on the runs above. One answer API cost ChatGPT answer, with web search $0.033 Gemini answer, with Google Search grounding $0.096 ChatGPT answer, from memory $0.0003 Gemini answer, from memory $0.023 Google AI Overview, full text via DataForSEO $0.0024 100 prompts, three engines Daily, a month Weekly, a month DIY from memory, plus AI Overviews $77 $11 DIY with search, plus AI Overviews $394 $57 API costs only. The tracker runs on free tiers of Supabase and Vercel. Live search is where the money goes, Gemini's Google Search grounding most of all, because it bills per search and runs about five searches per answer. Answering from memory costs next to nothing, and in this test it matched the apps at least as well. Whether that justifies a monthly AEO subscription is your call, but the data quality argument for one is a lot weaker than it sounds. ## Where scraping still earns its keep One prompt is one prompt, and a B2B software question with clear market leaders is where methods agree most. So we ran a second check on three more of Ugi's prompts, comparing the DIY tracker against his scraped answers. Software and product prompts held up. Local search did not. On “Best emergency plumber in Austin, TX”, UI scraping only agreed with itself 15% of the time on ChatGPT, and the DIY tracker overlapped 7% with it. On Gemini it was 40% against 19%. Local answers depend on where the person searching is standing, and the apps handle that better than an API call. If you track plumbers, dentists, gyms, or restaurants, scraping is worth paying for. ## What we would do with this Run every prompt more than once and report the share of runs that name you. One answer per prompt is one roll of the dice. Start from memory-only runs for brand perception. Add live search when you need to see citations, and budget for it, because search is where the bill comes from. Run weekly. Answers move between runs far more than they move between days, so daily tracking buys you more noise. Track the domains that keep coming back across many runs, and stop worrying about individual cited pages. Pull AI Overviews through a provider that returns the expanded text. A collapsed overview hides half the brands. Type your five most important prompts into the apps by hand once a month. It takes 20 minutes and tells you whether your tracker still lines up with what people see. This is also how [LLM tracking inside Contengi](https://contengi.com/features/llm-tracking) works, and you can try a single check on your own brand with the [LLM visibility check](https://contengi.com/llm-tracker). ## Methodology ContentMonk ran its check on 13 October 2026 from the United States and recorded the models as gpt-6, gemini-3.5-flash, and Google AI Overview. We ran the DIY tracker and the manual runs on 14 and 15 October 2026. The DIY tracker used GPT-6 Luna and Gemini 3.5 Flash through the Vercel AI Gateway, told only the date and that the user is in the United States, with no instructions about naming brands or adding links. Brands were extracted by one model for every method, merged into vendors (QuickBooks Payroll and QuickBooks Online count as QuickBooks), and only counted when the name appears in the answer text. Names that only appear in a follow-up question, like an accounting software picker, don't count. Answer similarity uses OpenAI's text-embedding-3-large on the answer text with links and formatting stripped. Manual answers were copied from the apps, which carries source names rather than links, so manual source counts are a floor. Both DIY setups use the same 10 Google AI Overviews, since an overview is always a Google search. Overlap, all engines Brands Sources Answers DIY memory vs Manual 67.8% 15.2%\* 86.6% UI scraping vs Manual 68.0% 11.5% 89.8% DIY search vs Manual 62.0% 13.9% 86.8% DIY memory vs UI scraping 74.7% 5.4%\* 88.0% DIY search vs UI scraping 70.5% 15.1% 87.9% DIY memory, run to run 78.9% 17.7%\* 94.1% DIY search, run to run 70.2% 40.5% 93.4% UI scraping, run to run 76.9% 39.6% 93.4% Manual, run to run 69.2% 31.4% 92.9% \* Answering from memory produces no citations, so these source figures come from its AI Overviews alone. Read the original guide on [The Workflow](https://www.theworkflow.digital/p/build-your-own-ai-visibility-tracker) or ContentMonk's [shared results](https://app.contentmonk.io/shared/prompt-check/AEwM4wX9uqpdE8E__56anoPl?prompt=1) . [See LLM tracking in Contengi](https://contengi.com/features/llm-tracking) FAQ ## Common questions ### Is API data less accurate than scraping the ChatGPT or Gemini interface? Not in a way that changed the result in our test. A DIY tracker answering from memory overlapped 67.8% with hand-typed answers on the brands it named, UI scraping overlapped 68.0%, and two hand-typed runs only overlapped 69.2% with each other. Every method lands inside the noise that a single method produces on its own. ### Why did answering from memory match the apps so well? For a category with established leaders, the model already knows who they are. Live search adds citations and fresher prices, but it barely changes who gets recommended. On ChatGPT the memory-only answers matched the manual runs at 78.5%, ahead of UI scraping at 76.1%. ### How many times should I run each prompt? More than once. Two runs of the same method only shared 69 to 79% of their brands, so a single answer can easily show a brand that the next run drops. We ran 10 per engine, and reporting the share of runs that name a brand is far steadier than any single answer. ### Can I track which pages AI engines cite? You can, but expect it to move a lot. Two runs of the same method shared 31 to 41% of their cited domains, and different methods shared 5 to 15%. Track the domains that recur across many runs, and treat single citations as anecdotes. ### What did the DIY side of the test cost? $1.59 in API calls for every DIY answer with and without search, the AI Overviews, and the scoring. The tracker itself runs on free tiers of Supabase and Vercel. ### Does this hold for local searches? Less so. On "Best emergency plumber in Austin, TX", UI scraping only agreed with itself 15% of the time on ChatGPT, and the DIY tracker overlapped 7% with it. Local results depend on where the searcher is, and the app handles that better than an API call does. Keep exploring ## Related features and reads In the platform [### LLM tracking Track how ChatGPT, Gemini, Claude, Perplexity, and Grok talk about your brand.](https://contengi.com/features/llm-tracking) [### LLM visibility check Run a one-off check on your own brand.](https://contengi.com/llm-tracker) [### How it works The models behind Contengi and what each one costs.](https://contengi.com/how-it-works) From the blog [Every guide on AI visibility and answer engine optimisation](https://contengi.com/ai-visibility) [Content engineering: the role, the skills, and the tools](https://contengi.com/content-engineering) [See all posts](https://contengi.com/blog) ## See how AI engines talk about your brand Contengi tracks your brand across five AI engines and connects it to the content you publish, set up with you from day one. [Request beta access](https://contengi.com/beta-access) [Try it out](https://contengi.com/demo)