A couple problems with weblogs.com and editthispage.com
I noticed one weblogger from editthispage.com had submitted her weblog several times. Since her weblog was one of the "originals" that I had in my index (I started with a list of about 100 weblogs in the testing stages) I checked out why it wasn't making it into the index.
Do this search: please slow down
You'll notice all the weblogs.com and editthispage.com pages have a message to effect of:
Please Slow Down. Your crawler is hitting our servers too hard.
I checked my scan logs and requests are generally at least a minute apart and more likely even further apart because this only affects about 40 pages which get re-indexed every 24 hours.
Anyway the message is misleading because it's not really the case.
I remember way early on, I encountered this problem with Doc Searls weblog and I emailed Userland to ask what was up. They said they didn't allow crawling of Userland sites at all. I promptly forgot about it and the implications. At the time, I figured OK, so I can't spider Doc's page. I didn't even think about all the other people who had their weblogs hosted by Userland.
I've emailed Userland again and asked them to help me out on this. Hopefully, I can get weblogs.com and editthispage.com webloggers indexed in the near future.
 |
Comments disabled.
