User-agent: * Disallow: /cgi-bin/ User-agent: AhrefsBot Crawl-Delay: 10 User-agent: SemrushBot Crawl-Delay: 3 User-agent: MJ12bot Crawl-Delay: 10 User-agent: SemrushBot-SA Crawl-Delay: 3 User-agent: SemrushBot-BA Crawl-Delay: 3 User-agent: SemrushBot-SI Crawl-Delay: 3 User-agent: SemrushBot-SWA Crawl-Delay: 3 User-agent: SemrushBot-CT Crawl-Delay: 3 User-agent: SemrushBot-BM Crawl-Delay: 3