Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The problem is not Googlebot.

robots.txt on ticker.picolisp.com says "Disallow: /", but ticker.picolisp.com redirects to picolisp.com/21000, and the robots.txt on picolisp.com says "Disallow:". If he wants Googlebot to stop crawling those URLs, he needs to add "Disallow: /21000" to picolisp.com.



Hmmmm, maybe he just did: http://picolisp.com/robots.txt




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: