Ahmed Fayed Logo AF ligature mark with Ahmed Fayed wordmark. Software Engineer & Systems Builder

Online store SEO

Why your store isn't on Google: robots.txt and noindex mistakes

18 Oct 2026 ·2 min read · اقرأ بالعربية

Two things decide whether Google can list your store: robots.txt, the file that tells crawlers which URLs they may fetch, and the noindex rule, which tells search engines not to show a page. One wrong line in either can take products, categories or the whole store out of Google while the store keeps selling normally, so the loss often goes unnoticed until search traffic drops. The two also interfere with each other. Google says noindex only works when the page is not blocked by robots.txt, because a blocked page is never fetched and its noindex is never read. A blocked URL can still appear in results, just without its content.

How to check

  1. Open https://yourdomain.com/robots.txt. Look for Disallow: / under User-agent: *, for rules that cover product or category paths, and for rules that block CSS or JavaScript files.
  2. Note the status code. Under the robots.txt standard (RFC 9309), a server error (5xx) means crawlers must treat the whole site as blocked, while a 404 means everything is allowed.
  3. Open a product page's source and search for noindex. Also check the response headers for X-Robots-Tag, which carries the same rule from the server.
  4. In Search Console, URL Inspection tells you whether a URL is blocked by robots.txt or excluded by noindex.
  5. Run the free audit. Store Audit by Ahmed Fayed reads your robots.txt rules per crawler and checks the homepage, product and category templates for noindex.

Search engines and AI crawlers

AI companies use separate robots.txt names (user-agent tokens) for training and for search. Blocking the wrong one removes you from AI search results you may want.

Token Owner What the sources say
GPTBot OpenAI Training. Obeys robots.txt.
OAI-SearchBot OpenAI ChatGPT search. Obeys robots.txt.
ChatGPT-User OpenAI Fetches pages on a user's request. Since December 2025, OpenAI says robots.txt rules "may not apply".
ClaudeBot, Claude-SearchBot, Claude-User Anthropic All three obey robots.txt, including Crawl-delay (February 2026).
PerplexityBot, Perplexity-User Perplexity The first obeys robots.txt; the second "generally ignores" it.
Google-Extended Google A robots.txt token for Gemini training and grounding. It does not affect Google Search.

To keep your content out of OpenAI training while staying in ChatGPT search, block only the training token:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

How to fix

Salla

Salla's default robots.txt allows everything except one /*<iframe pattern, so problems usually come from edits. A live Salla store checked for this article served WooCommerce rules such as /?add-to-cart= and /my-account, which match no Salla URL. Remove copied rules or restore the default file. I haven't confirmed where the robots.txt editor sits in the current Salla dashboard; ask Salla support if you can't find it. Salla controls the robots meta tag, so if a product or category template carries noindex, send the affected URLs to Salla support.

Zid

The editor is at Dashboard › Online Store › SEO › Robots.txt file, on the Growth and Professional plans only. It validates syntax, offers a "restore defaults" button, and warns that Disallow: / de-indexes the store. The default file on a live Zid store blocked /cart, /api/, /js/, review and search URLs, and about 600 named bots. Check that list for any search or AI crawler you want. Whether Disallow: /js/ affects how Google renders a Zid theme is unconfirmed, so test a product page with URL Inspection. As on Salla, noindex output is controlled by the platform.

Shopify

The only way to change robots.txt is the robots.txt.liquid theme template. Shopify Support doesn't help with these edits, and Shopify warns that mistakes can cause "loss of all traffic". Keep the robots.default_groups loop that prints Shopify's default rules and add your rules to it. A plain-text replacement drops those defaults and any updates Shopify makes to them later.

WordPress/WooCommerce

Check Settings › Reading first. When search engine visibility is set to discourage indexing, WordPress adds noindex to every page. Then check your SEO plugin's settings for the affected content type. Some noindex is intended: WordPress noindexes internal search results, and WooCommerce noindexes the cart, checkout and my-account pages.

Custom-coded

Check the server and CDN configuration for an X-Robots-Tag header, and the templates for a noindex left over from a staging environment. Send the final robots meta in the HTML the server returns. Google says that when the original HTML contains noindex, it may skip rendering, so removing the tag with JavaScript may not work.

Common mistakes

  • Blocking a URL in robots.txt to remove it from Google. Use noindex and keep the URL crawlable until it drops out.
  • Writing noindex inside robots.txt. Google doesn't support it.
  • Blocking CSS or JavaScript files that pages need to render.
  • Blocking every OpenAI or Anthropic token when you only meant to opt out of training.
  • Letting robots.txt return a server error during maintenance.

Run the free Store Audit by Ahmed Fayed to check your store for these problems. If you'd like help fixing this, contact me on WhatsApp +20 102 702 0701 or at ahmed-fayed.com.

Sources

Indexing

More in this series

Back to guides