User-Agent: * Disallow: /index.html Disallow: /help-search # Percorsi di checkout — aggiunti il 16/09/2026, su approvazione del # proprietario. NON sono pagine da indicizzare e ognuna di esse CREA UNA # SESSIONE STRIPE: /checkout-direct carica page/js/checkout-direct.js, che # chiama l'endpoint che apre la sessione, e Googlebot esegue JavaScript. # Il 14/09/2026 sono nate 2.498 sessioni Stripe in un giorno solo (2.487 a meno # di 60 secondi l'una dall'altra), quasi tutte da 66.249.73.x (Googlebot) e # 116.179.33.20 (Baidu), con UNA sola pagata. # Questo file e' condiviso da tutti gli host shoppi.*: nessuno di loro ha # motivo di far indicizzare un checkout. # robots.txt resta pero' solo un'indicazione. Il pixel Meta e' protetto a parte # da una guardia sullo user-agent in checkout-stripe.php e # checkout-stripe-web.php, che salta l'evento InitiateCheckout per i crawler. Disallow: /checkout-direct Disallow: /checkout Disallow: /cart Disallow: /plugins/checkout/ Disallow: /plugins/cart/ # AJAX endpoints are linked from the UI (e.g. the language switcher) but are not # pages: they answered 200 with an identical 553-byte body, so crawlers indexed # dozens of duplicates of the same non-page. Disallow: /plugins/ # shoppi.cloud partner kit (24/09/2026): public for whoever has the link, kept # out of search because the decks carry commission figures. The same rules sit # in sites/shoppi.cloud/robots.txt; they are repeated here because the nodes # whose nginx lacks the per-host try_files serve THIS file for every host. The # paths exist only on shoppi.cloud, so they change nothing for the other sites. Disallow: /partners/kit Disallow: /downloads/partner-kit/ # NOTE: /search is deliberately NOT disallowed. Those pages carry # , and a crawler has to be able to # FETCH a page to read that tag. Blocking them here would leave the ones already # indexed stuck: disallowed URLs can still be indexed from inbound links, and the # noindex would never be seen. Crawl-and-drop is what actually removes them. # --- Applebot: crawl-delay (23/09/2026) ------------------------------------ # Measured, not guessed: in a 12.000-request slice of the access log Applebot # was 9.945 of them — 83% of everything the fleet served. It was asking faster # than four cores shared with the primary Mongo can answer, and the result was # a crawl that mostly failed: 4.611 of its requests ended in 499 (Applebot gave # up waiting) and 1.007 in 504. Every abandoned request still held a PHP worker # for up to 60 seconds, which made the next one slower still. # # So this is not a block and it is not an SEO sacrifice: Apple already was not # getting the pages. A one-second delay gives it a crawl that returns 200 # instead of 499, and gives everyone else the machine back. Raise or remove it # once the box has headroom. Applebot documents support for Crawl-delay. User-agent: Applebot Crawl-delay: 1 # The rules above are for `*`; a crawler with its own group reads ONLY that # group, so without these lines it had no Disallow at all (27/09/2026: # Applebot was crawling /plugins/ and creating phantom favourites). Disallow: /index.html Disallow: /help-search Disallow: /checkout-direct Disallow: /checkout Disallow: /cart Disallow: /plugins/checkout/ Disallow: /plugins/cart/ Disallow: /plugins/ Disallow: /partners/kit Disallow: /downloads/partner-kit/ # Same treatment for the crawlers that bring no traffic and no buyers. They are # welcome, just not at full speed. User-agent: Baiduspider Crawl-delay: 5 # The rules above are for `*`; a crawler with its own group reads ONLY that # group, so without these lines it had no Disallow at all (27/09/2026: # Applebot was crawling /plugins/ and creating phantom favourites). Disallow: /index.html Disallow: /help-search Disallow: /checkout-direct Disallow: /checkout Disallow: /cart Disallow: /plugins/checkout/ Disallow: /plugins/cart/ Disallow: /plugins/ Disallow: /partners/kit Disallow: /downloads/partner-kit/ User-agent: GPTBot Crawl-delay: 2 # The rules above are for `*`; a crawler with its own group reads ONLY that # group, so without these lines it had no Disallow at all (27/09/2026: # Applebot was crawling /plugins/ and creating phantom favourites). Disallow: /index.html Disallow: /help-search Disallow: /checkout-direct Disallow: /checkout Disallow: /cart Disallow: /plugins/checkout/ Disallow: /plugins/cart/ Disallow: /plugins/ Disallow: /partners/kit Disallow: /downloads/partner-kit/ User-agent: Amazonbot Crawl-delay: 5 # The rules above are for `*`; a crawler with its own group reads ONLY that # group, so without these lines it had no Disallow at all (27/09/2026: # Applebot was crawling /plugins/ and creating phantom favourites). Disallow: /index.html Disallow: /help-search Disallow: /checkout-direct Disallow: /checkout Disallow: /cart Disallow: /plugins/checkout/ Disallow: /plugins/cart/ Disallow: /plugins/ Disallow: /partners/kit Disallow: /downloads/partner-kit/ User-agent: DotBot Crawl-delay: 10 # The rules above are for `*`; a crawler with its own group reads ONLY that # group, so without these lines it had no Disallow at all (27/09/2026: # Applebot was crawling /plugins/ and creating phantom favourites). Disallow: /index.html Disallow: /help-search Disallow: /checkout-direct Disallow: /checkout Disallow: /cart Disallow: /plugins/checkout/ Disallow: /plugins/cart/ Disallow: /plugins/ Disallow: /partners/kit Disallow: /downloads/partner-kit/ Sitemap: https://shoppi.app/sitemaps/zh_cn/sitemap.xml Sitemap: https://shoppi.app/sitemaps/sg_sg/sitemap.xml Sitemap: https://shoppi.app/sitemaps/ro_ro/sitemap.xml Sitemap: https://shoppi.app/sitemaps/pl_pl/sitemap.xml Sitemap: https://shoppi.app/sitemaps/nl_nl/sitemap.xml Sitemap: https://shoppi.app/sitemaps/it_it/sitemap.xml Sitemap: https://shoppi.app/sitemaps/hk_hk/sitemap.xml Sitemap: https://shoppi.app/sitemaps/fr_fr/sitemap.xml Sitemap: https://shoppi.app/sitemaps/es_es/sitemap.xml Sitemap: https://shoppi.app/sitemaps/en_us/sitemap.xml Sitemap: https://shoppi.app/sitemaps/en_gb/sitemap.xml Sitemap: https://shoppi.app/sitemaps/de_de/sitemap.xml Sitemap: https://shoppi.app/sitemaps/ca_ca/sitemap.xml Sitemap: https://shoppi.app/sitemaps/au_au/sitemap.xml # svacant.com (25/09/2026). Two of the three nodes still lack the per-host # try_files for /robots.txt, so they serve THIS file for svacant.com too, and # svacant's own sites/svacant.com/robots.txt is seen only 1 time in 3. Listing # its sitemap here makes it discoverable either way; for the shoppi.* hosts a # sitemap on another host changes nothing. Remove once all nodes have the patch. Sitemap: https://svacant.com/sitemap.xml