Alright, all you misbehaving spiders that don't flush your DNS cache (I'm looking at you Yahoo! Slurp and Adobe Feeds), I'm killing my old box today. So while currently you're transparently proxying through to the new one because you won't respect my TTLs in DNS, that's gonna stop. Here's the list of user agents that are still hitting my old box even though DNS (with 1 hour TTLs) has been updated for 36+ hours:
Adobe Feeds AideRSS/2.0 (aiderss.com) Baiduspider+(+http://www.baidu.com/search/spider.htm) Baiduspider+(+http://www.baidu.jp/spider/) ColdFusion Gigabot/3.0 (http://www.gigablast.com/spider.html) magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net) Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; GTB6; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; MS-RTC EA 2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0; MSOffice 12) Mozilla/4.0 (compatible; NaverBot/1.0; http://help.naver.com/customer_webtxt_02.jsp) Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html) R6_CommentReader(www.radian6.com/crawler) R6_FeedFetcher(www.radian6.com/crawler) SkyGrid/1.0 (+http://skygrid.com/partners) Sosoblogspider+(+http://help.soso.com/soso-blog-spider.htm) Sosospider+(+http://help.soso.com/webspider.htm) SurveyBot/2.3 (Whois Source) Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)
Comments are closed.