Tuesday, August 31, 2010

Search Engines to explore the "Invisible" Web.

The Invisible Web refers to the part of the WWW that’s not indexed by the search engines. Most of us think that that search powerhouses like Google and Bing are like the Great Oracle…they see everything. Unfortunately, they can’t because they aren’t divine at all; they are just web spiders who index pages by following one hyperlink after the other.

But there are some places where a spider cannot enter. Take library databases which need a password for access. Or even pages that belong to private networks of organizations. Dynamically generated web pages in response to a query are often left un-indexed by search engine spiders.

To get a more precise idea of the nature of this ‘Dark Continent’ involving the invisible and web search engines, read what Wikipedia has to say about the Deep Web. The figures are attention grabbers – the size of the open web is 167 terabytes. The Invisible Web is estimated at 91,000 terabytes. Check this out – the Library of Congress, in 1997, was figured to have close to 3,000 terabytes!

So how do we get this information?

Some of these are invisible web search engines with specifically indexed information:

The WWW Virtual Library
Complete Planet
Info Please
Deep Peep
Incy Wincy

Enjoy your new vessels to explore the "Invisible Web!"



  1. So, it's like there is the known universe, which we know through the search engines, and there is 1000x more of the unknown universe, which isn't on the search engines!? Holy crap.

  2. Exploring the invisible web, probably not the best idea. They're going to find a bunch of information that they wished they didn't find. That, and it's a fluffing large waste of time, and invation of privacy. If it's not being shown publicly, it's for a reason.
    That's my two cents :)

  3. infomine ftw, definitely the best

  4. interesting. i learn't something

  5. hey man, check out my blog i am sure its very relevant to your interest.

    smoochies n' poopies.

  6. everyone thinks google is the best, google barely scratches the surface of the world wide web