User-Agent: MJ12bot Disallow: / User-Agent: MJ12bot/v1.4.5 Disallow: / user-agent: AhrefsBot disallow: / User-agent: Baiduspider Disallow: / user-agent: AhrefsBot/5.0 disallow: / user-agent: AhrefsBot/5.1 disallow: / User-agent: Baiduspider/2.0 Disallow: / User-agent: 360Spider Disallow: / User-agent: Steeler Disallow: / User-agent: Steeler/3.5 Disallow: / User-agent: DotBot/1.1 Disallow: / User-agent: BLEXBot/1.0 Disallow: / User-agent: spbot/5.0.1 Disallow: / User-agent: Uptimebot/1.0 Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: Aboundex/0.3 Disallow:/ User-agent: UCY/Nutch-1.2 Disallow:/ User-agent: SeznamBot/3.2 Disallow:/ User-agent: BLock Genieo/1.0 Disallow:/ User-agent: Mail.RU_Bot/2.0 Disallow:/ User-agent: LTX71 Disallow:/ User-agent: Mozilla/5.0 (compatible; MSIE or Firefox mutant;) Daum 4.1 Disallow:/ User-agent: InfluenceBot Disallow:/ User-agent: semanticbot Disallow:/ User-agent: SurveyBot/2.3 Disallow:/ User-agent: Dataprovider/6.92 Disallow:/ User-agent: wotbox Disallow: / User-agent: NetSeer crawler/2.0 Disallow: / User-agent: GrapeshotCrawler/2.0 Disallow: / User-agent:woobot/1.1 Disallow: / User-agent: woobot/2.0 Disallow: / User-agent: HubSpot Connect 2.0 Disallow: / User-agent: YandexBot/3.0 Disallow: / User-agent: Exabot/3.0 Disallow: / User-agent: WbSrch/1.1 Disallow: / User-agent: seoscanners.net/1 Disallow: / User-agent: Scrapy/1.1.0 Disallow: / User-agent: SemrushBot/1.1~bl Disallow: / User-agent: Sogou web spider/4.0 Disallow: / User-agent: spbot/5.0.2 Disallow: / User-agent: seoscanners.net/1 Disallow: / User-agent: Daum 4.1 Disallow: / User-agent: Scrapy/1.1.0rc4 Disallow: / User-agent: grapeshot Disallow: / User-agent: viz/Nutch-1.7 Disallow: / User-agent: oBot/2.3.1 Disallow: / User-agent: AdnormCrawler Disallow: / User-agent: CrazyWebCrawler-Spider Disallow: / User-agent: istellabot/t.1 Disallow: / User-agent: Mechanize/2.7.2 Ruby/2.0.0p481 (http://github.com/sparklemotion/mechanize/) Disallow: / User-agent:GrapeshotCrawler/2.0 Disallow: / User-agent: DomainAppender /1.0 Disallow: / User-agent: SiteExplorer/1.0b Disallow: / User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1 Disallow: / User-agent: CRAZYWEBCRAWLER 0.9.10 Disallow: / User-agent: Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0 Disallow: / User-agent: bd/Nutch-1.7 Disallow: / User-agent: DomainSONOCrawler/0.1 Disallow: / User-agent: Dataprovider/6.101 Disallow: / User-agent: SemrushBot/1.1~bl Disallow: / User-agent: ZoomInformation Bot Disallow: / User-agent: Uptimebot/0.2.43 Disallow: / User-agent: MojeekBot/0.6 Disallow: / User-agent: DowntimeDetector/1.0 Disallow: / User-agent: nutch-1.4/Nutch-1.4 Disallow: / User-agent: Cliqzbot/1.0 Disallow: / User-agent: TinEye-bot/0.51 Disallow: / User-agent: Gluten Free Crawler/1.0 Disallow: / User-agent: Qwantify/2.2w Disallow: / User-agent: psbot/0.1 Disallow: / User-agent: psbot-page Disallow: / User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: DomainAppender /1.0 Disallow: / User-agent: vebidoobot Disallow: / User-agent: SemrushBot/1.1~bl Disallow:/ User-agent: MixrankBot Disallow:/ User-agent: SemrushBot/1.2~bl Disallow:/ User-agent: Plukkie/1.6 Disallow:/ User-agent: SiteExplorer/1.1b Disallow:/ User-agent: AhrefsBot/5.2 Disallow:/ User-agent: okhttp/2.5.0 Disallow:/ User-agent: YisouSpider Disallow:/ User-agent: DataMiner/3.14 Disallow:/ User-agent: Mozilla/5.0 (X11; Ubuntu; Linux armv7l; rv:17.0) Gecko/20100101 Firefox/17.0 Disallow:/ User-agent: NetcraftSurveyAgent/1.0 Disallow:/ User-agent: Daum/4.1 Disallow:/ User-agent: Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) Disallow:/ User-agent: Gobyus Disallow:/ User-agent: Googlebot Allow:/ User-agent: Googlebot/2.1 Allow: / User-agent: Googlebot-Image Allow:/ User-agent: Adsbot-Google Allow:/ User-agent: Googlebot-Mobile Allow:/ User-agent: Go-http-client/1.1 Disallow:/ User-agent: Scrapy/1.4.0 Disallow:/ User-agent: GarlikCrawler/1.2 Disallow:/ User-agent: Virusdie crawler/3.0 Disallow:/ User-agent: Qwantify/2.4w Disallow:/ User-agent: spbot/5.0.3 Disallow:/ User-agent: Barkrowler 0.1.6 Disallow:/ User-agent: SeaMonkey/17.8 Disallow:/ User-agent: Contacts-Crawler/0.2 Disallow:/ User-agent: CipaCrawler/3.0 Disallow:/ User-agent: AwarioRssBot/1.0 Disallow:/ User-agent: IAS crawler Disallow:/ User-agent: 007ac9 Crawler Disallow:/ User-agent: SeaMonkey/3.6 Disallow:/ User-agent: DnyzBot/1.0 Disallow:/ User-agent: Cliqzbot/2.0 Disallow:/ User-agent: Barkrowler/0.7 Disallow:/ User-agent: Neangan Spider/Nutch-2.3.1 Disallow:/ User-agent: BUbiNG Disallow: / User-agent: Gigabot Disallow: / User-agent: Barkrowler/0.7 Disallow: / User-agent: SurdotlyBot/1.0 Disallow: / User-agent: Apache OpenOffice/4.1.5 Disallow: / User-agent: TinEye-bot/1.31 Disallow: / User-agent: Iceweasel/44.0 Disallow: / User-agent: curl/7.45.0-DEV Disallow: / User-agent: Tarantula/Tarantula-1.14 Disallow: / User-agent: tracemyfile/1.0 Disallow: / User-agent: Quick-Crawler Disallow: / User-agent: curl/7.52.1 Disallow: / User-agent: CCBot/2.0 Disallow: / User-agent: MauiBot Disallow: / User-agent: Naaraa/Nutch-1.14 Disallow: / User-agent: PulsePoint-Ads.txt-Crawler/1.0 Disallow: / User-agent: Dispatch/0.11.3 Disallow: / User-agent: Python-urllib/2.7 Disallow: / User-agent: pyspider/0.3.9 Disallow: / User-agent: Wayback Machine Live Record Disallow: / User-agent: Apache-HttpClient/4.5.2 Disallow:/ User-agent: PhantomJS/2.1.1 Disallow:/ User-agent: bidswitchbot/1.0 Disallow:/ User-agent: Applebot/0.1 Disallow:/ USer-agent: Photon/1.0 Disallow:/ User-agent: SemrushBot/2~bl Disallow:/ User-agent: Scrapy/1.5.1 Disallow:/ User-agent: elenPublicWebCrawler Disallow:/ User-agent: Presto/2.8.131 Disallow:/ User-agent: Barkrowler/0.9 Disallow: / User-agent: ZoominfoBot Disallow: / User-agent: evc-batch/2.0 Disallow: / User-agent: WebDataStats/1.0 Disallow: / User-agent: VelenPublicWebCrawler/1.0 Disallow:/ User-agent: libwww-perl/6.08 Disallow:/ User-agent: Scrapy/1.6.0 Disallow:/ User-agent: MJ12bot/v1.4.8 Disallow:/ User-agent: Xenu Link Sleuth/1.3.8 Disallow:/ User-agent: zgrab/0.x Disallow:/ User-agent: MojeekBot/0.7 Disallow:/ User-agent:SEOkicks Disallow:/ User-agent: Darwin/19.3.0 Disallow: / User-agent: DomainStatsBot/1.0 Disallow: / User-agent: cortex/1.0 Disallow: / User-agent: PetalBot Disallow: /