Many Internet users are enamored by the fact that a vast number of interconnected host computers exist that are off the beaten path, meaning they do not always show up in a Google or Bing search.
Like the New World of past centuries, the Deep Web and Dark Web are out there somewhere, just waiting to be discovered by brave souls that are willing to challenge conventional wisdom and online traditions.
Introducing the Dark Web
Although the terms are often used interchangeably, strictly defined, Dark Web and Deep Web are quite different.
The Dark Web is the part of the Internet that is composed of host machines that cannot be reached through conventional means. This usually means that router settings have deliberately or accidentally made accessing these hosts virtually impossible.
“Unreachable,” of course, is in the eye of the beholder. After all hosts that are on the “Dark Web” are still connected, they just cannot be reached via traditional methods. In this regard, some computer hosts actually are reachable, only that they are reachable by means that are not conventional.
Alternative DNS systems
The news has been replete with examples of the United States Government seizing domain names, making their owners unable to do business online. To combat this apparent threat to online freedom, domain naming systems (DNS) other than the ubiquitous Internet Corporation for Assigned Numbers and Names (ICANN) have been developed.
Host machines that use peer-to-peer DNS systems such as the Dot-P2P project make it possible to create Internet hosts over which governments have no control, meaning that the survivability of web sites is dramatically increased.
Crime and Terror links
Because the Dark Web is not connected to the Internet via traditional means, hosts are harder to find and connect to site operators. This makes it an appealing haven to purveyors of child pornography and other illegal content. Similarly, terror groups have been discovered to haunt the Dark Web as part of their effort to recruit and radicalize new members.
Although the Dark Web has some nebulous uses, many of them are innocuous. For example people who want to communicate and associate online without fear of being shut down by a government agency.
Because the Dark Net consists of inaccessible hosts, the concept of creating a Dark Net search engine is problematic at best. Although the ability to search alternative DNS formats exists, there is no clear way to make sure to find all hosts that use a particular system.
Further complicating any effort to create a Dark Net search engine is the need to cobble together a variety of alternative DNS configurations as well as hosts that do not use any DNS at all.
Still, some search functions are available on different Dark Net services. For example, the alternative DNS, Freenet, has some limited search capabilities.
Understanding Deep Web
Despite Google and other search engines indexing hundreds of billions of Web pages, most of their crawlers do not index the wealth of information known as the Deep Web.
The Deep Web refers to online content contained on traditional hosts that are accessible by ICANN DNS servers. The Deep Web is similar to the Dark Web, however, because most of its content is not reachable via the major search engines. Still, because hosts themselves are reachable, from most computers, the content can usually be accessed via specialized search engines and other tools.
An almost innumerable number of databases litter the Internet, containing information that some experts say could amount to more than 500 times the size of the Web indexed by popular search engines.
Because they offer special services, many Deep Web search and data harvesting services (e.g. Bright Planet) are premium sites that require payment prior before search results are rendered. However, many alternatives to give Internet users free access to the Deep Web such as the Deep Peep search engine are also available.
Deep Web content often includes public records, data on individuals and businesses, as well as property information
Although the Dark Web and Deep Web have a variety of features that can benefit Internet users, each of them requires special information before they can be adequately accessed, indexed and searched.
Storm, Darlene, “P2P DNS to take on ICANN after US domain seizures”, https://blogs.computerworld.com/17444/p2p_dns_to_take_on_icann_after_us_domain_seizures
Thomas, Keir, “Pirate DNS Could Hatch a Lawless Darknet”, https://www.pcworld.com/businesscenter/article/212263/pirate_dns_could_hatch_a_lawless_darknet.html
Stevens, Tim. (2009). Regulating the ‘Dark Web’: How a Two-Fold Approach can Tackle Peer-to-Peer Radicalisation. RUSI Journal: Royal United Services Institute for Defence Studies, 154(2), 28-33.
Mufti, Sheryar. (2010). Web search software hacks into secretive online forums. New Scientist, 205(2755), 20.
Wright, Alex. “Exploring a ‘Deep Web’ That Google Can’t Grasp”, https://www.nytimes.com/2009/02/23/technology/internet/23search.html?_r=1&ref=business
Image Credit: Wikimedia Commons/Hank van Helvete