Mostly yes, and far more than the world's biggest websites. Of the 1,642 working UAE business and other sites that Claude cited for 500 buyer questions, 94% had a robots.txt file, only 4.4% blocked any AI bot, and just 12 (0.7%) blocked a bot that AI search or answers use. 77% had Organization schema, 40% an llms.txt file, and only 2.7% a nearly empty homepage without JavaScript. Among the world's top 1,000 sites with a working homepage, 17% blocked an AI bot and 21% were nearly empty.
But being ready is not what set the most-cited sites apart. Sites cited three or more times passed all seven of our basic checks about as often as sites cited once (71% against 68%). The basics look like the entry ticket, not the prize. And AI cited these sites; that does not prove these signals are why.
Why we ran this
On 21 October 2026 we published our UAE AI visibility study: 500 real buyer questions in English and Arabic across 10 industries. Claude, with live web search, cited 4,634 sources, and 79% were businesses' own websites and other sites outside the big directories (see which websites AI cites).
Most advice about "getting ready for AI" is a checklist: allow the AI bots, add llms.txt, add schema, make the page work without JavaScript. Do the sites AI actually picks follow it, and how do they compare with the rest of the web? So we crawled every one of them, plus the world's 1,000 most popular sites, on 22 October 2026.
How we did it
- The AI-cited sites. We took all 4,634 sources from the first study and grouped them by website: 1,954 sites. With the first study's own rules we split them into 1,861 business and other sites and 93 platforms (directories, review sites, social media, news, Wikipedia, government). The business group is led by businesses' own websites but also holds blogs and smaller directories. For a stricter view we marked 451 sites whose address matches a business the same answer named, such as a clinic cited in an answer that recommended it.
- The baseline. The top 1,000 sites of the Tranco list, a research ranking of popular websites (list 262W9, 20 October 2026). Many are infrastructure domains with no website, so every comparison uses only sites with a working homepage: 1,642 of the 1,861 AI-cited sites and 558 of the 1,000 global sites.
- A polite crawl. Our crawler named itself honestly as AdjoltzResearchBot, made at most 4 requests at a time, gave each 8 seconds and obeyed robots.txt itself. It fetched robots.txt, /llms.txt, the homepage and up to two sitemap addresses per site, on 22 October 2026 from 07:01 to 08:27 Dubai time. It never pretended to be an AI bot.
- The checks. For 14 crawlers we read whether robots.txt blocks them from the homepage, with the matching rules of the robots.txt standard (RFC 9309). On the homepage we measured the text in the HTML itself, without running JavaScript, and read the schema markup (JSON-LD), language tags, title, description, canonical and noindex tags.
Finding 1: almost none of them block AI bots
robots.txt is a small text file that tells crawlers what they may visit. 94.0% of the AI-cited sites had one. Only 72 (4.4%) blocked any of the 12 AI crawlers we checked, and none blocked Googlebot or Bingbot. Among the global top sites, 17.0% blocked an AI crawler.
What matters is the split between bots that collect training data and bots that fetch pages for AI search and answers. OpenAI, Anthropic and Perplexity document these as separate bots. Almost every block we found was on the training side:
| Bot (company) | What its company says it does | Blocked by AI-cited UAE sites (1,642) | Blocked by global top sites (558) |
|---|---|---|---|
| OAI-SearchBot (OpenAI) | Shows sites in ChatGPT search | 0% | 4.5% |
| ChatGPT-User (OpenAI) | Visits a page when a user asks | 0.3% (5) | 6.1% |
| Claude-SearchBot (Anthropic) | Improves Claude's search results | 0.2% (4) | 5.0% |
| Claude-User (Anthropic) | Visits a page when a user asks | 0.2% (4) | 5.7% |
| PerplexityBot (Perplexity) | Shows sites in Perplexity search, not used for training | 0.5% (8) | 7.7% |
| GPTBot (OpenAI) | Training data | 2.5% (41) | 10.9% |
| ClaudeBot (Anthropic) | Training data | 2.4% (40) | 10.8% |
| Google-Extended (Google) | Use in Gemini training and grounding; no effect on Google Search | 1.9% (32) | 9.1% |
| Applebot-Extended (Apple) | Use in Apple's AI training; it does not crawl | 1.7% (28) | 6.8% |
| CCBot (Common Crawl) | Builds an open archive of the web that anyone can download | 3.3% (55) | 12.7% |
| Bytespider (ByteDance) | Reportedly gathers training data (Cloudflare) | 2.9% (48) | 12.9% |
| Meta-ExternalAgent (Meta) | Training and product indexing | 1.9% (32) | 8.6% |
| Googlebot and Bingbot | Search engines | 0% | 0% |
One detail stood out. 44 of the sites Claude cited block ClaudeBot, Anthropic's training crawler, yet its search cited them the day before (5 also block Claude-SearchBot, and 5 Claude-User). We do not know how Anthropic's search tool builds its index, so we read this narrowly: blocking a training bot did not keep these sites out of Claude's answers.
robots.txt does not show firewalls. 116 AI-cited sites (6.2%) refused or challenged our honest bot with a "403 Forbidden" or a "Just a moment..." page, even though their robots.txt allowed it. A firewall can turn away an AI crawler the same way. Cloudflare's managed robots.txt, which adds Disallow rules for a list of AI crawlers, was on only 1.3% of them.
Our reading. For the sites AI already cites, robots.txt is not the problem: they let the search and answer bots in. If your site blocks one of those five bots, that is the first thing to fix. You can check in seconds with our free AI crawler checker. The full list of bots, with each company's documentation, is in our AI crawlers guide.
Finding 2: llms.txt is common among them, but it does not separate them
llms.txt is a plain text file at the root of a website that summarises the site for AI tools. Jeremy Howard proposed it in September 2024; it is not an official standard. We counted it only when /llms.txt answered with real text, not a web page or an error.
652 AI-cited sites (40.0%) had one, against 18.6% of the global top sites, far more than we expected. Part of the reason is WordPress: 56% of the AI-cited sites run on it, and a third of their llms.txt files (33.4%) open with a note saying an SEO plugin (Yoast, Rank Math or All in One SEO) or Wix made them. Among the global sites, 3.9% did.
But it did not separate the sites AI cited most: 38.6% of sites cited once had one, and 45.9% of sites cited three or more times. By industry it ranged from 32% (restaurants) to 48% (aesthetics, business setup).
Do not expect a file to do the work. If you have an llms.txt, make sure it says who you are, where you work and links to your real service pages. But nothing in our data shows that AI picks a site because of it.
Finding 3: their homepages work without JavaScript
Many websites build the page in the browser with JavaScript. In Vercel's December 2024 analysis, OpenAI's and Anthropic's crawlers downloaded JavaScript files but did not run them, so they missed text that only appears after the script runs. Google's Gemini and Apple's crawler did render pages.
We measured the text in the homepage HTML itself. The AI-cited sites had a median of 8,214 characters, and only 45 (2.7%) had under 500 characters, about 80 words. The global top sites had a median of 4,601, and 21.0% had under 500: the list is full of web apps such as google.com, youtube.com and spotify.com.
Restaurants were the exception among UAE sites: a median of 4,875 characters of text and 4.8% nearly empty, the most of any industry.
Finding 4: they say who they are with schema
Schema markup (structured data) is code in the page that states facts machines read without guessing: this is a dental clinic, this is its address, these are its hours and profiles. Google says Organization markup helps it understand an organisation and tell it apart from others.
| Schema on the homepage | AI-cited UAE sites (1,642) | Named-business sites (404) | Global top sites (558) |
|---|---|---|---|
| Any JSON-LD at all | 83.7% | 89.6% | 46.4% |
| Organization (any type of organisation) | 76.8% | 82.2% | 37.3% |
| LocalBusiness (or a type of it, such as Dentist) | 36.6% | 52.7% | 0.4% |
| FAQPage | 19.5% | 30.7% | 3.2% |
| Review or rating | 12.5% | 21.8% | 1.8% |
| Product | 2.3% | 4.0% | 1.4% |
| sameAs links to other profiles | 53.7% | 62.9% | 30.5% |
The named-business sites, the clinics, garages and firms AI recommended by name and also cited, had the most complete markup: over half had LocalBusiness schema and almost a third FAQ markup. The global sites are mostly platforms and brands, so for them the fair comparison is Organization, at 37%.
The markup was usually sound. Only 0.7% of the AI-cited sites had a JSON-LD block that would not parse. The most common types were WebSite, Organization, SearchAction (the site search box), ImageObject and WebPage.
Finding 5: Arabic signals are thin
A page tells machines its language with its lang tag, and hreflang tags link the English and Arabic versions of a page to each other. In the first study, Arabic answers were thinner and sometimes drifted to Saudi businesses, so we looked at the sites cited in Arabic answers separately.
| Cited in English answers (1,050) | Cited in Arabic answers (679) | Cited only in Arabic answers (592) | |
|---|---|---|---|
| Homepage language tag is Arabic | 0.9% | 56.1% | 63.9% |
| Homepage links an Arabic version (hreflang) | 15.1% | 38.4% | 35.3% |
| Either Arabic signal | 15.4% | 79.2% | 82.1% |
| LocalBusiness schema | 39.6% | 34.9% | 31.2% |
| FAQPage schema | 23.3% | 15.2% | 12.7% |
| llms.txt | 43.5% | 36.8% | 33.9% |
| Median text in the HTML (characters) | 9,578 | 6,769 | 6,322 |
| All seven basics | 73.0% | 66.3% | 64.2% |
Two readings. One in five sites cited in Arabic answers (20.8%) showed no Arabic signal on the homepage at all: if they have Arabic pages, the homepage does not tell machines where they are. And the sites English answers cited were almost never bilingual in a way machines can see: only 15% linked an Arabic version from the homepage. Across the whole AI-cited set, 22.4% had an Arabic hreflang tag, against 10.6% of the global top sites.
The sites cited in Arabic were also less complete: less schema, less text in the HTML, fewer passing all the basics. That fits the first study: there is less good Arabic material for AI to use, and it is less well labelled.
The full comparison
| Check (homepage) | AI-cited UAE sites (1,642) | Named-business sites (404) | Global top sites (558) |
|---|---|---|---|
| robots.txt present | 94.0% | 95.5% | 75.6% |
| Blocks at least one AI bot | 4.4% | 3.0% | 17.0% |
| Blocks an AI search or answer bot | 0.7% | 0.2% | 9.1% |
| llms.txt | 40.0% | 46.8% | 18.6% |
| HTTPS | 98.6% | 99.0% | 95.0% |
| Under 500 characters without JavaScript | 2.7% | 2.2% | 21.0% |
| Title tag | 98.5% | 99.3% | 93.2% |
| Meta description | 91.3% | 95.3% | 82.4% |
| Canonical tag | 91.5% | 94.6% | 69.7% |
| Homepage set to noindex | 0.4% | 0.7% | 1.8% |
| Working sitemap | 91.4% | 94.1% | 59.5% |
| Organization schema | 76.8% | 82.2% | 37.3% |
| Arabic hreflang | 22.4% | 19.1% | 10.6% |
| Passes all seven basics | 69.8% | 76.5% | 26.3% |
The seven basics are our own checklist: robots.txt, no AI search or answer bot blocked, HTTPS, a title and meta description, at least 500 characters of text in the HTML, Organization schema, and a working sitemap. Together they mean an AI crawler can reach the site, read it and tell whose it is.
By industry
| Industry | Working sites | Blocks an AI bot | llms.txt | LocalBusiness schema | FAQPage schema | Arabic signal | All seven basics |
|---|---|---|---|---|---|---|---|
| Dental clinics | 164 | 5.5% | 42.3% | 47.6% | 23.8% | 35.4% | 76.8% |
| Aesthetics and dermatology | 186 | 2.7% | 47.8% | 43.0% | 15.1% | 46.2% | 72.6% |
| Law firms | 181 | 3.9% | 38.7% | 36.5% | 18.2% | 44.2% | 71.8% |
| Home services | 245 | 2.0% | 40.8% | 49.0% | 24.5% | 42.0% | 71.4% |
| Business setup | 199 | 2.5% | 47.7% | 34.7% | 28.6% | 40.7% | 71.4% |
| Real estate | 156 | 6.4% | 39.4% | 31.4% | 16.0% | 39.1% | 67.9% |
| Automotive | 230 | 6.5% | 38.6% | 36.1% | 23.0% | 43.0% | 67.0% |
| Education and childcare | 179 | 6.1% | 33.0% | 20.1% | 18.4% | 44.1% | 64.8% |
| Fitness and wellness | 162 | 8.6% | 36.4% | 27.8% | 9.3% | 35.2% | 63.0% |
| Restaurants and cafes | 124 | 10.5% | 32.3% | 22.6% | 4.8% | 45.2% | 62.9% |
A site cited in two industries counts in both rows. "Arabic signal" means an Arabic language tag or hreflang link on the homepage. Clinics, home services and law firms had the most complete sites. Restaurants and fitness had the most AI bot blocks and the least FAQ markup, and in the first study their answers leaned most on "best of" lists and platforms such as TripAdvisor, TimeOut and ClassPass.
Cited once or cited often: readiness hardly changes
| AI-cited UAE sites | Cited once (1,082) | Cited twice (289) | Cited 3 or more times (271) |
|---|---|---|---|
| Passes all seven basics | 68.1% | 75.1% | 70.8% |
| Blocks an AI bot | 3.6% | 6.9% | 4.8% |
| llms.txt | 38.6% | 39.6% | 45.9% |
| LocalBusiness schema | 35.1% | 38.8% | 40.2% |
| FAQPage schema | 17.5% | 22.1% | 24.7% |
| Arabic signal | 35.1% | 43.9% | 52.0% |
The sites AI came back to most had a little more markup and more Arabic signals, but on the basics they looked like the sites cited once. Something other than technical readiness decided how often a site was used. The first study points to what: whether a page answers the exact question asked, with the area, the service and the price.
What this means for your business
Correlation, not cause. AI cited these sites; that does not prove the signals we checked are why. Sites that rank well are usually built by people who also install SEO plugins, add schema and keep sitemaps, so the gap against the global sites can come from that shared cause. Read the checks as the standard your site is compared with, not as a switch that gets you named.
Still, the AI-cited sites set a clear floor. Each fix below is cheap and none is guesswork:
- Let the search and answer bots in. Open yoursite.com/robots.txt and make sure OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot and Bingbot are not blocked. Decide about training bots (GPTBot, ClaudeBot, Google-Extended and the rest) separately. Our AI crawler checker reads the file for you.
- Check your firewall too. 6.2% of the AI-cited sites turned away an honest new bot. If you use Cloudflare or a security plugin, make sure AI search bots are not challenged.
- Put your words in the HTML. Open "View page source" on your homepage and search for a sentence from the page. If it is not there, crawlers that do not run JavaScript cannot read it.
- Say who you are in schema. Organization, or LocalBusiness with the right type (Dentist, LegalService, RealEstateAgent), with your exact name, address, phone, hours and sameAs links to your Google Business Profile, Instagram and LinkedIn.
- Label your Arabic pages. Set lang="ar" on Arabic pages and link each English page to its Arabic twin with hreflang, both ways.
- Keep a working sitemap and list it in robots.txt. 91% of the AI-cited sites had one; 60% of the global sites did.
- Add llms.txt last, if you like. Our free llms.txt generator writes a draft, but our data does not show that the file matters.
- Then do the work that does separate sites: a page for each real buyer question, answered plainly, in English and Arabic. Our first study shows how answers are built from those pages, and the free AI visibility checker shows where you stand.
Our free AI check asks 10 real buyer questions about your business on ChatGPT and Gemini (Google AI) and tells you who gets named. We do not promise that these fixes get you named; they remove the reasons not to be.
Method details and limits
- One assistant, one day. The AI-cited sites come from Claude's answers with Anthropic's web search tool on 21 October 2026; other assistants cite partly different sites. We crawled the next day, so a few sites may have changed.
- The baseline is not a control group. The global top 1,000 are mostly platforms, apps and big brands, not UAE businesses that AI did not cite. A fair set of uncited UAE businesses is hard to build, so we used a public, reproducible list as a reference point.
- Homepage only. Schema, FAQ markup and Arabic pages often live on inner pages, and a site can block AI bots from some folders and still count as allowed here.
- robots.txt is a stated policy. We never tested with an AI bot's name, so firewall blocks of AI bots are not counted. Sites that refused our own bot are left out of the rates and may be the strictest. Counting every site whose robots.txt answered, 5.4% of the AI-cited sites block an AI bot and 1.6% a search or answer bot.
- Rough grouping. The business group, sorted by address, holds some blogs and smaller directories. The 451 named-business sites, the strict view, point the same way.
- What we could not reach. Of the 1,861 AI-cited business and other sites, 116 refused or challenged our bot, 60 had a robots.txt we could not reach (the standard then says stay away), 17 asked unknown bots to stay away, 16 answered with an error and 10 were down, leaving 1,642.
- Plugins. We only know a plugin made an llms.txt file when the file says so, so the real share is likely higher.
Download the data
One row per website (2,940 rows): its group, how often and in which industries and languages AI cited it, its Tranco rank if any, and every check, including all 14 bots. Website addresses and technical results only, nothing about people. Download the CSV, and please credit Adjoltz and link to this page.