In spite of their seemingly immense index of trillions of Web pagers, there is much that a search of common search engines, the free "surface" Web (e.g., Google, Bing, Yahoo) cannot find, which requires specialized search tools and techniques. Early research concluded that the "deep web" is by some estimates 500 to 5,000 times larger than the free surface web (BrightPlanet, 2009; Lesk, 1998; Madhavan et al, 2009; Wright, 2009); more recently researchers concluded that the Deep Web was growing exponentially and was unquantifiable.
The Deep Web goes by many names: Dark Web, Hidden Web, Invisible Web, Invisible Continent, and DeepNet. There are many reasons for the Dark Web. The deep web contains information that may not (for copyright or programming reasons) or cannot be indexed in a text-based web search engine. This includes databases, computer software, multimedia, PDF files and more. For example, BrightPlanet claimed that over 350,000 specialty databases exist whose data is not indexed by the standard public web search engines. That is, the search engines cannot or will not look into these databases and index the information that is found there. Further, data such as pictures, audio and video files do not contain text content that is searchable, unless someone has specifically created accompanying text data. Facebook, for example, does not allow Google to index its information resources. Further, much of the Web is behind a password. There are also many who prefer to remain anonymous as both searchers and information providers, for example, for illegal activity or political protection. Finally, there is the Covert Web, activity which is even further hidden, whether corporate, government or criminal espionage which seeks to unlock or keep locked a variety of digital doors and data.
The web sites below provide some access to much of this deep
and seemingly invisible set of resources. The databases will have their own
private indexes of their information. Some
deep web systems require a fee payment for access. Nearby libraries will have paid special fees to numerous deep web resources. If denied access, check to make sure that the
fee has not already been paid by some organization. Your
nearest library will have specialists interested in providing you with the
access or passwords and links that will be needed to reach deeper into this territory, access for
which the library has often already paid. To find more current information
on this topic, use the search engines to find information on "searchable
databases" or the "deep web".
Basu, S. (2010). Ten Search Engines to Explore the Invisible Web. http://www.makeuseof.com/tag/10-search-engines-explore-deep-invisible-web/
BrightPlanet (2009). Fact Sheet. Retrieved December 5, 2009 from http://www.brightplanet.com/resource-library/fact-sheet/ see also http://quod.lib.umich.edu/cgi/t/text/text-idx?c=jep;view=text;rgn=main;idno=3336451.0007.104
Deep Web Primer. (2014). BrightPlanet. http://www.brightplanet.com/deep-web-university-2/deep-web-a-primer/
"Deep Web" (2014-current). Wikipedia. http://en.wikipedia.org/wiki/Deep_Web
Michael, Lesk. (1997). How much information is there in the world? Retrieved December 5, 2009 from http://www.lesk.com/mlesk/ksg97/ksg.html.
Madhavan, J., Afanasiev, L., Antova, L., and Halevy, A. (2009). Harnessing the deep web: Present and future. In 4th Biennial Conference on Innovative Data Systems Research (CIDR). Retrieved December 5, 2009 from http://www-db.cs.wisc.edu/cidr/cidr2009/Paper_115.pdf
O'Neill, P. H. (2013, October 14). How to Search the Deep Web. http://www.dailydot.com/technology/how-to-search-the-deep-web/
Wright, A. (February 22, 2009). Exploring a ‘Deep Web’ That Google Can’t Grasp. New York Times. Retrieved November 5, 2009 from http://www.nytimes.com/2009/02/23/technology/internet/23search.html
Updated March 13, 2014 Parent frame - Pageauthor's Houghton