Confusing Web Crawlers
It's been almost a month since I published my infinite directory of web pages. I was curious to see what web crawlers would think of this infinitely deep tree of links. Interestingly enough, most crawlers (like Google and Altavista) only downloaded a small number of pages before giving up. Others were a bit more persistent.
Screen shot of www.siroker.com server stats
The web crawler for AskJeeves seemed to have had a field day when they came across my directory. Whereas most web crawlers downloaded far less than 1 MB of total data from my web server, good old Jeeves decided he wanted almost 100 MB before giving up. That may not seem like a lot of data, but keep in mind that each page in my infinite directory is only 2.36 KB! Take a look at the server stats and you'll see that Jeeves made 46,634 total hits on my web site. A hundred times as many as Google!
Since I posted my original proof of the infinite size of the web, an observant reader named Christopher Everett commented that the directory was not infinite as I had claimed. Indeed, he was correct. I forgot to upload the rest of infinity. I just did.
Try starting at page #9999999999999999999999999999999999 and let me know if you ever finish.
You're very techincal.
I think a screenshot with a circle over what portion of the server stats we should be interested by would be nice. But that's just a suggestion.
Thanks for updating about your LIFE. Fascinating stuff. Heh, just wondering if there's more that's going on than just work.
Hi Dan,
Very interesting post. I thought I'd share a little research I did to follow up. Doing a Google search on “site:www.siroker.com dan siroker” yielded some interesting results.
Searching something similar, and including infinite in the search of your name turned up a big fat nothing on the infinite pages. However, I found something infinitely more interesting than that. There appears to be another Dan Siroker in the world. Just thought that it was funny!
How'd you get all these crawlers to do their deeds to you? I haven't gotten listed on google yet :/
Admittedly, there is not much going on in my life other than work right now. However, I did recently write about a whole year of my life.
Matt, believe it or not but that supposed other Dan Siroker is actually me! Freshman year I took an online course in bioinformatics and one of the lectures was about protein physics. In order to pass the class we were required to contribute to the online discussion forums. I actually ended up doing really well in the class even though I barely watched any of the online lectures!
To answer your question regarding getting crawled: it's only a matter of time. Make sure that you have inbound links to your web site from others. You could also try adding yourself to google.
Wow…my name on a page other than a post…heck..i feel honored. ha!
Thanks for the “upgrade” and the futher proof of concept here. Heck, now the web is infinite!
Worst blog, ever.
Usagi7 (12:15:41 PM): I'm a big fan of the
“Best/worst ____, ever” sentence paradigm.
e.g.:
Worst day, ever.
Saibok (12:15:41 PM): You seriously are. I was going to respond pejoratively to your blog comment along those lines, but I didn't want to shatter your fragile emotional state.
amazing article. stumble across it when i was wandering ard the web looking for some statistics on number of websites in the world.
so what exactly did you do to prove your infinite-web claim? I am how do you generate this long series of web pages on your site. (I don't know anything about web publishing, you can say)