# ============================================================ # Ultimate Tools Baazar — Robots.txt # https://ultimatetoolsbaazar.com/ # ============================================================ # ==================== ALL CRAWLERS (default) ==================== User-agent: * Allow: / # Block admin/draft/private directories (if any exist later) Disallow: /admin/ Disallow: /private/ Disallow: /draft/ Disallow: /*.json$ Disallow: /*?*utm_ Disallow: /*?q= # Allow all static assets Allow: /*.css$ Allow: /*.js$ Allow: /*.png$ Allow: /*.jpg$ Allow: /*.jpeg$ Allow: /*.webp$ Allow: /*.svg$ Allow: /*.ico$ Allow: /*.woff2$ # ==================== GOOGLE ==================== User-agent: Googlebot Allow: / Crawl-delay: 1 User-agent: Googlebot-Image Allow: / Allow: /*.png$ Allow: /*.jpg$ Allow: /*.jpeg$ Allow: /*.webp$ # ==================== BING ==================== User-agent: Bingbot Allow: / Crawl-delay: 1 # ==================== YAHOO ==================== User-agent: Slurp Allow: / Crawl-delay: 1 # ==================== DUCKDUCKGO ==================== User-agent: DuckDuckBot Allow: / Crawl-delay: 1 # ==================== YANDEX ==================== User-agent: Yandex Allow: / Crawl-delay: 2 # ==================== BAIDU ==================== User-agent: Baiduspider Allow: / Crawl-delay: 2 # ==================== SOCIAL MEDIA PREVIEWS ==================== User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: LinkedInBot Allow: / User-agent: WhatsApp Allow: / User-agent: TelegramBot Allow: / # ==================== BLOCK BAD BOTS ==================== User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: PetalBot Disallow: / User-agent: Bytespider Disallow: / # ==================== SITEMAPS ==================== Sitemap: https://ultimatetoolsbaazar.com/sitemap.xml