Jump to content

Deep web: Difference between revisions

From Wikipedia, the free encyclopedia
Browse history interactively
← Previous edit
Content deleted Content added
VisualWikitext
No edit summary
Restored revision 1257606050 by Demt1298 (talk)
 
Line 1: Line 1:
{{short description|Content of the World Wide Web that is not indexed by search engines}}
The '''deep Web''' (or '''Deepnet''', '''invisible Web''' or '''hidden Web''') refers to [[World Wide Web]] content that is not part of the [[surface Web]] [[index (search engine)|indexed]] by [[search engine]]s. It is estimated that the deep Web is several orders of magnitude larger than the surface Web.<ref name=bergman2001>{{cite journal |first=Michael K. |last=Bergman | title = The Deep Web: Surfacing Hidden Value | journal = The Journal of Electronic Publishing | year = 2001 | month = Aug | volume = 7 | issue = 1 | url = http://www.press.umich.edu/jep/07-01/bergman.html}}. According to that paper, the study was originally published on July 26, 2000, with data then updated to 2001.</ref>
{{About | the part of the World Wide Web not indexed by traditional search engines}}
{{Distinguish |Dark web}}
{{Use mdy dates|date=May 2022}}


The '''deep web''',<ref>{{cite book |citeseerx= 10.1.1.90.5847 |chapter-url= http://www.iadisportal.org/digital-library/the-mechanics-of-a-deep-net-metasearch-engine |chapter=The Mechanics of a Deep Net Metasearch Engine |pages=1034–6 |last= Hamilton |first= Nigel |year= 2019–2020 |editor1-first= Pedro |editor1-last= Isaías |editor2-first= António |editor2-last= Palma dos Reis |title= Proceedings of the IADIS International Conference on e-Society |publisher= IADIS Press |isbn= 978-972-98947-0-1 }}</ref> '''invisible web''',<ref>{{cite journal |last1= Devine |first1= Jane |last2= Egger-Sider |first2= Francine |title= Beyond google: the invisible web in the academic library |journal= The Journal of Academic Librarianship |date= August 2021 |volume= 30 |issue= 4 |pages= 265–269 |doi= 10.1016/j.acalib.2004.04.010 }}</ref> or '''hidden web'''<ref name="cthw">{{cite journal|title= Crawling the Hidden Web|journal= 27th International Conference on Very Large Data Bases|date= September 11–14, 2001|first1= Sriram|last1= Raghavan|first2= Hector|last2= Garcia-Molina|url=http://ilpubs.stanford.edu:8090/725/}}</ref> are parts of the [[World Wide Web]] whose contents are not [[Search engine indexing|indexed]] by standard [[web search engine|web search-engine program]]s.<ref>{{Cite web |last=Maor |first=Etay |title=Council Post: Lessons Learned From Tracing Cybercrime’s Evolution On The Dark Web |url=https://www.forbes.com/councils/forbestechcouncil/2024/02/02/lessons-learned-from-tracing-cybercrimes-evolution-on-the-dark-web/ |access-date=2024-09-22 |website=Forbes |language=en}}</ref> This is in contrast to the "[[surface web]]", which is accessible to anyone using the Internet.<ref>{{cite web |title= Surface Web |url= https://www.computerhope.com/jargon/s/surface-web.htm |publisher= Computer Hope |access-date= June 20, 2018}}</ref> [[Computer science|Computer scientist]] Michael K. Bergman is credited with inventing the term in 2001 as a search-indexing term.<ref name=":0">
==Naming==
{{cite news
| last = Wright
| first = Alex
| title = Exploring a 'Deep Web' That Google Can't Grasp
| work =[[The New York Times]]
| date = February 22, 2009
| url = https://www.nytimes.com/2009/02/23/technology/internet/23search.html?th&emc=th
| access-date = September 2, 2019
| quote = [...] Mike Bergman, a computer scientist and consultant who is credited with coining the term Deep Web.
}}
</ref>


Deep web sites can be accessed by a direct [[URL]] or [[IP address]], but may require entering a password or other security information to access actual content.<ref>Madhavan, J., Ko, D., Kot, Ł., Ganapathy, V., Rasmussen, A., & Halevy, A. (2008). Google's deep web crawl. Proceedings of the VLDB Endowment, 1(2), 1241–52.</ref><ref>{{cite web |last=Shedden |first=Sam |date=June 8, 2014 |title=How Do You Want Me to Do It? Does It Have to Look like an Accident? – an Assassin Selling a Hit on the Net; Revealed Inside the Deep Web |url=https://www.questia.com/article/1G1-370513892/how-do-you-want-me-to-do-it-does-it-have-to-look |url-access= |url-status=dead |archive-url=https://web.archive.org/web/20200301174912/https://www.questia.com/article/1G1-370513892/how-do-you-want-me-to-do-it-does-it-have-to-look |archive-date=March 1, 2020 |access-date= |newspaper=[[Sunday Mail (Scotland)|Sunday Mail]]}}</ref> Uses of deep web sites include [[web mail]], [[online banking]], [[cloud storage]], restricted-access [[social-media]] pages and profiles, and [[web forums]] that require registration for viewing content. It also includes [[paywall]]ed services such as [[video on demand]] and some online magazines and newspapers.
[[Michael Bergman]] has said that Jill Ellsworth coined the term "invisible Web" in 1994 to refer to [[website]]s that are not registered with any search engine.<ref name=bergman2001/> Bergman cited a January 1996 article by Frank Garcia in which Ellsworth was quoted using the term (but did not say she coined it in 1994):<ref>Garcia, Frank (January 1996). "Business and Marketing on the Internet". ''Masthead'' 9 (1). (Citation from Flynn-Burhoe, Maureen (19 December 2006). "[http://papergirls.wordpress.com/2006/12/19/the-ultimate-guide-to-the-invisible-web/ The Ultimate Guide to the Invisible Web]". ''oceanflynn @ Digg''.) ([http://web.archive.org/web/19961205083117/http://tcp.ca/Jan96/BusandMark.html Electronic copy] archived by the [[Internet Archive]].)</ref>
<blockquote>"It would be a site that's possibly reasonably designed, but they didn't bother to register it with any of the search engines. So, no one can find them! You're hidden. I call that the invisible Web."</blockquote>


==Terminology==
Another early use of the term "invisible Web" was by Bruce Mount (Director of Product Development) and Matthew B. Koll (CEO/Founder) of Personal Library Software, Inc. (PLS) when describing the @1 deep Web tool.{{fact|date=February 2008}} The term was used in a December 1996 press release from PLS.<ref>Personal Library Software (Dec 1996). "PLS introduces AT1, the first 'second generation' Internet search service". ([http://web.archive.org/web/19971021232057/www.pls.com/news/pr961212_at1.html Archived by the Internet Archive].)</ref>


The first conflation of the terms "deep web" and "[[dark web]]" happened during 2009 when deep web search terminology was discussed together with illegal activities occurring on the [[Freenet]] and [[darknet]].<ref>{{cite news|last1=Beckett|first1=Andy|title=The dark side of the internet|url=https://www.theguardian.com/technology/2009/nov/26/dark-side-internet-freenet |date=November 26, 2009 |access-date=August 9, 2015}}</ref> Those criminal activities include the commerce of personal [[Password|passwords]], [[Identity document forgery|false identity documents]], [[Drug|drugs]], [[firearms]], and [[child pornography]].<ref>{{cite video|url=https://www.youtube.com/watch?v=GEwmYk15ZcU| archive-url=https://ghostarchive.org/varchive/youtube/20211113/GEwmYk15ZcU| archive-date=November 13, 2021 | url-status=live|author=D. Day|title=Easiest Catch: Don't Be Another Fish in the Dark Net|publisher=[[TEDx Talks]]|location=Wake Forest University}}{{cbignore}}</ref>
The first use of the specific term 'deep Web' occurred in that same 2001 Bergman study.<ref name=bergman2001/>


Since then, after their use in the media's reporting on the black-market website [[Silk Road (marketplace)|Silk Road]], media outlets have generally used 'deep web' [[Skunked term|synonymously]] with the [[dark web]] or [[darknet]], a comparison some reject as inaccurate<ref name="confusion">{{cite web|url=http://www.brightplanet.com/2014/03/clearing-confusion-deep-web-vs-dark-web/|title=Clearing Up Confusion – Deep Web vs. Dark Web|publisher=BrightPlanet|date=March 27, 2014}}</ref> and consequently has become an ongoing source of confusion.<ref>{{cite news|last1=Solomon|first1=Jane|title=The Deep Web vs. The Dark Web|url=http://blog.dictionary.com/dark-web/ |date=May 6, 2015 |access-date=May 26, 2015}}</ref> ''[[Wired (website)|Wired]]'' reporters [[Kim Zetter]]<ref>{{cite news|last1=NPR Staff|title=Going Dark: The Internet Behind The Internet|url=https://www.npr.org/sections/alltechconsidered/2014/05/25/315821415/going-dark-the-internet-behind-the-internet|access-date=May 29, 2015|date=May 25, 2014}}</ref> and [[Andy Greenberg]]<ref>{{cite news|last1=Greenberg|first1=Andy|title=Hacker Lexicon: What Is the Dark Web?|url=https://www.wired.com/2014/11/hacker-lexicon-whats-dark-web/ |date=November 19, 2014 |access-date=June 6, 2015}}</ref> recommend the terms be used in distinct fashions. While the deep web is a reference to any site that cannot be accessed by a traditional search engine, the dark web is a portion of the deep web that has been hidden intentionally and is inaccessible by standard browsers and methods.<ref>{{cite web|title=The Impact of the Dark Web on Internet Governance and Cyber Security|date=January 20, 2014|url=https://www.ourinternet.org/sites/default/files/publications/GCIG_Paper_No6.pdf|access-date=January 15, 2017|archive-date=January 16, 2017|archive-url=https://web.archive.org/web/20170116173141/https://www.ourinternet.org/sites/default/files/publications/GCIG_Paper_No6.pdf|url-status=dead}}</ref><ref>{{cite book|last1=Lam|first1=Kwok-Yan|last2=Chi|first2=Chi-Hung|last3=Qing|first3=Sihan|title=Information and Communications Security: 18th International Conference, ICICS 2016, Singapore, Singapore, November 29 – December 2, 2016, Proceedings|publisher=Springer|isbn=9783319500119|url=https://books.google.com/books?id=uraVDQAAQBAJ&pg=PA174 |language=en |date=November 23, 2016 |access-date=January 15, 2017}}</ref><ref>{{cite web|title=The Deep Web vs. The Dark Web {{!}} Dictionary.com Blog|url=http://blog.dictionary.com/dark-web/|publisher=Dictionary Blog |date=May 6, 2015 |access-date=January 15, 2017}}</ref><ref>{{cite book|last1=Akhgar|first1=Babak|last2=Bayerl|first2=P. Saskia|last3=Sampson|first3=Fraser|title=Open Source Intelligence Investigation: From Strategy to Implementation|publisher=Springer|isbn=9783319476711|url=https://books.google.com/books?id=39zTDQAAQBAJ&pg=PA112 |language=en |date=January 1, 2017 |access-date=January 15, 2017}}</ref><ref>{{cite news|title=What is the dark web and who uses it?|url=https://www.theglobeandmail.com/technology/tech-news/what-is-the-dark-web-and-who-uses-it/article26026082/|newspaper=The Globe and Mail|access-date=January 15, 2017}}</ref>{{excessive citations inline|date=June 2024}}
==Size==
In 2000, it was estimated that the deep Web contained approximately 7,500 terabytes of data and 550 billion individual documents.<ref name=bergman2001/> Estimates, based on extrapolations from the study entitled ''How much information is there?'', from [[University of California, Berkeley]], show that the deep Web consists of about 91,000 [[terabyte]]s. By contrast, the surface Web, which is easily reached by search engines, is only about 167 terabytes. The [[Library of Congress]] contains about 11 terabytes, for comparison.<ref>[http://www.sciencefriday.com/pages/2007/Jul/hour2_072707.html ''Hour Two: Depression Medication / Baby Talk / Search Engines''], [[Science Friday]], [[National Public Radio]], July 27, 2007</ref><ref>The unpublished paper [http://www.lesk.com/mlesk/ksg97/ksg.html ''How much information is there in the world?''], by [[Michael Lesk]] in 1997, estimated that in 1997, the [[Library of Congress]] had between 200 [[terabyte]]s and 3 [[petabyte]]s.</ref>


==Non-indexed content==
==Deep resources==
Bergman, in a paper on the deep web published in ''The Journal of Electronic Publishing'', mentioned that Jill Ellsworth used the term ''[[Invisible Web]]'' in 1994 to refer to websites that were not registered with any search engine.<ref name="bergman2001">{{cite journal |first= Michael K | last= Bergman | title=The Deep Web: Surfacing Hidden Value | journal=The Journal of Electronic Publishing |date=August 2001 | volume=7 | issue=1 | url=http://quod.lib.umich.edu/cgi/t/text/text-idx?c=jep;view=text;rgn=main;idno=3336451.0007.104 |doi=10.3998/3336451.0007.104| doi-access=free | hdl=2027/spo.3336451.0007.104 | hdl-access=free }}</ref> Bergman cited a January 1996 article by Frank Garcia:<ref>{{cite journal
Deep Web resources may be classified into one or more of the following categories{{Fact|date=October 2007}}:
| last = Garcia
* ''Dynamic content'' - [[Dynamic Web page|dynamic pages]] which are returned in response to a submitted query or accessed only through a form (especially if open-domain input elements e.g. text fields are used; such fields are hard to navigate without domain knowledge).
| first = Frank
| title = Business and Marketing on the Internet
| journal = Masthead
| volume = 15
| issue = 1
| date = January 1996
| url = http://tcp.ca/Jan96/BusandMark.html
| access-date=February 24, 2009 |archive-url=https://web.archive.org/web/19961205083117/http://tcp.ca/Jan96/BusandMark.html |archive-date=December 5, 1996}}</ref>


<blockquote>
* ''Unlinked content'' - pages which are not linked to by other pages, which may prevent [[Web crawling]] programs from accessing the content. This content is referred to as pages without backlinks (or inlinks).
It would be a site that's possibly reasonably designed, but they didn't bother to register it with any of the search engines. So, no one can find them! You're hidden. I call that the invisible Web.
</blockquote>


Another early use of the term ''Invisible Web'' was by Bruce Mount and Matthew B. Koll of [[Personal Library Software]], in a description of the No. 1 Deep Web program found in a December 1996 press release.<ref name="PLS">@1 started with 5.7 terabytes of content, estimated to be 30 times the size of the nascent World Wide Web; PLS was acquired by AOL in 1998 and @1 was abandoned. {{cite press release |title=PLS introduces AT1, the first 'second generation' Internet search service |publisher=Personal Library Software |date=December 1996 |url=http://www.pls.com/news/pr961212_at1.html |access-date=February 24, 2009 |url-status=dead |archive-url=https://web.archive.org/web/19971021232057/http://www.pls.com/news/pr961212_at1.html |archive-date=October 21, 1997 }}</ref>
* ''Private Web'' - sites that require registration and login (password-protected resources).


The first use of the specific term ''deep web'', now generally accepted, occurred in the aforementioned 2001 Bergman study.<ref name=bergman2001/>
* ''Contextual Web'' - pages with content varying for different access contexts (e.g. ranges of client IP addresses or previous navigation sequence).


==Indexing methods==
* ''Limited access content'' - sites that limit access to their pages in a technical way (e.g., using the [[Robots Exclusion Standard]], [[CAPTCHA]]s or pragma:no-cache/cache-control:no-cache [[HTTP header]]s), prohibiting search engines from browsing them and creating [[cache]]d copies.
Methods that prevent web pages from being indexed by traditional search engines may be categorized as one or more of the following:


# '''Contextual web''': pages with content varying for different access contexts (e.g., ranges of client IP addresses or previous navigation sequence).
* ''Scripted content'' - pages that are only accessible through links produced by [[JavaScript]] as well as content dynamically downloaded from Web servers via [[Macromedia Flash|Flash]] or [[AJAX]] solutions.
# '''Dynamic content''': [[Dynamic Web page|dynamic pages]], which are returned in response to a submitted query or accessed only through a form, especially if open-domain input elements (such as text fields) are used; such fields are hard to navigate without [[domain knowledge]].
# '''Limited access content''': sites that limit access to their pages in a technical manner (e.g., using the [[Robots Exclusion Standard]] or [[CAPTCHA]]s, or no-store directive, which prohibit search engines from browsing them and creating [[web cache|cached]] copies).<ref>{{cite journal|title=Hypertext Transfer Protocol (HTTP/1.1): Caching|website=[[Internet Engineering Task Force]]|year=2014|doi=10.17487/RFC7234 |url=http://tools.ietf.org/html/rfc7234#section-5.2.2.3|access-date=July 30, 2014|editor-last1=Fielding |editor-last2=Nottingham |editor-last3=Reschke |editor-first1=R. |editor-first2=M. |editor-first3=J. |last1=Fielding |first1=R. |last2=Nottingham |first2=M. |last3=Reschke |first3=J. }}</ref> Sites may feature an internal search engine for exploring such pages.<ref>[[Special:Search]]</ref><ref>{{Cite web|url=https://archive.org/search.php|title=Internet Archive Search}}</ref>
# '''Non-HTML/text content''': textual content encoded in multimedia (image or video) files or specific [[file formats]] not recognised by search engines.
# '''Private web''': sites that require registration and login (password-protected resources).
# '''Scripted content''': pages that are accessible only by links produced by [[JavaScript]] as well as content dynamically downloaded from Web servers via [[Adobe Flash|Flash]] or [[Ajax (programming)|Ajax]] solutions.
# '''Software''': certain content is hidden intentionally from the regular Internet, accessible only with special software, such as [[Tor (anonymity network)|Tor]], [[I2P]], or other darknet software. For example, Tor allows users to access websites using the [[.onion]] server address anonymously, hiding their IP address.
# '''Unlinked content''': pages which are not linked to by other pages, which may prevent [[web crawling]] programs from accessing the content. This content is referred to as pages without [[backlink]]s (also known as inlinks). Also, search engines do not always detect all backlinks from searched web pages.
# '''Web archives''': Web archival services such as the [[Wayback Machine]] enable users to see archived versions of web pages across time, including websites that have become inaccessible and are not indexed by search engines such as Google. <ref name=":0" />The Wayback Machine may be termed a program for viewing the deep web, as web archives that are not from the present cannot be indexed, as past versions of websites are impossible to view by a search. All websites are updated at some time, which is why web archives are considered Deep Web content.<ref>{{cite web|last1=Wiener-Bronner|first1=Danielle|title=NASA is indexing the 'Deep Web' to show mankind what Google won't|url=http://fusion.net/story/145885/nasa-is-indexing-the-deep-web-to-show-mankind-what-google-wont/|publisher=Fusion|date=June 10, 2015|access-date=June 27, 2015|quote=There are other simpler versions of Memex already available. "If you've ever used the Internet Archive's Wayback Machine", which gives you past versions of a website not accessible through Google, then you've technically searched the Deep Web, said [[Chris Mattmann]].|archive-date=June 30, 2015|archive-url=https://web.archive.org/web/20150630010143/http://fusion.net/story/145885/nasa-is-indexing-the-deep-web-to-show-mankind-what-google-wont/|url-status=dead}}</ref>


==Content types==
* ''Non-HTML/text content'' - textual content encoded in multimedia (image or video) files or specific [[file formats]] not handled by search engines.
While it is not always possible to discover directly a specific web server's content so that it may be indexed, a site potentially can be accessed indirectly (due to [[Vulnerability (computing)|computer vulnerabilities]]).


To discover content on the web, search engines use [[web crawler]]s that follow hyperlinks through known protocol virtual [[port (computer networking)|port numbers]]. This technique is ideal for discovering content on the surface web but is often ineffective at finding deep web content. For example, these crawlers do not attempt to find dynamic pages that are the result of database queries due to the indeterminate number of queries that are possible.<ref name=":0" /> It has been noted that this can be overcome (partially) by providing links to query results, but this could unintentionally inflate the popularity of a site of the deep web.
==Accessing==
To discover content on the Web, search engines use [[web crawler]]s that follow [[hyperlink]]s. This technique is ideal for discovering resources on the [[surface Web]] but is often ineffective at finding deep Web resources. For example, these crawlers do not attempt to find dynamic pages that are the result of database queries due to the infinite number of queries that are possible. It has been noted that this can be (partially) overcome by providing links to query results, but this could unintentionally inflate the popularity (e.g., [[PageRank]]) for a member of the deep Web.


[[DeepPeep]], [[Intute]], [[Deep Web Technologies]], [[Scirus]], and [[Ahmia.fi]] are a few search engines that have accessed the deep web. Intute ran out of funding and is now a temporary static archive as of July 2011.<ref>{{cite web | url=http://www.intute.ac.uk/faq.html | title=Intute FAQ, dead link | access-date=October 13, 2012}}</ref> Scirus retired near the end of January 2013.<ref>{{cite web|title=Elsevier to Retire Popular Science Search Engine|url=http://library.bldrdoc.gov/newsarc/201312.html|website=library.bldrdoc.gov|date=December 2013|access-date=June 22, 2015|quote=by end of January 2014, Elsevier will be discontinuing Scirus, its free science search engine. Scirus has been a wide-ranging research tool, with over 575 million items indexed for searching, including webpages, pre-print articles, patents, and repositories.|archive-url=https://web.archive.org/web/20150623002452/http://library.bldrdoc.gov/newsarc/201312.html|archive-date=June 23, 2015|url-status=dead}}</ref>
One way to access the deep Web is via [[federated search]] based search engines. Search tools such as [http://www.science.gov/ Science.gov] is being designed to retrieve information from the deep Web. These tools identify and interact with searchable databases, aiming to provide access to deep Web content.


Researchers have been exploring how the deep web can be crawled in an automatic fashion, including content that can be accessed only by special software such as [[Tor (anonymity network)|Tor]]. In 2001, Sriram Raghavan and Hector Garcia-Molina (Stanford Computer Science Department, Stanford University)<ref name=raghavan2000>{{cite web
Another way to explore the deep Web is by using human crawlers instead of algorithmic crawlers. In this paradigm referred to as [[Web harvesting]], humans find interesting links of the deep Web that algorithmic crawlers can't find. This [[human-based computation]] technique to discover the deep Web has been used by the [[StumbleUpon]] service since February 2002.
| author = Sriram Raghavan
| first2 = Hector
| last2 = Garcia-Molina
| title = Crawling the Hidden Web
| publisher = Stanford Digital Libraries Technical Report
| year = 2000
| url = http://ilpubs.stanford.edu:8090/456/1/2000-36.pdf
| access-date = December 27, 2008
| archive-date = May 8, 2018
| archive-url = https://web.archive.org/web/20180508094122/http://ilpubs.stanford.edu:8090/456/1/2000-36.pdf
| url-status = dead
}}</ref><ref>{{cite conference |first=Sriram |last=Raghavan |author2=Garcia-Molina, Hector | year=2001 | title=Crawling the Hidden Web | book-title=Proceedings of the 27th International Conference on Very Large Data Bases (VLDB) | pages=129–38 | url=http://www.dia.uniroma3.it/~vldbproc/017_129.pdf }}</ref> presented an architectural model for a hidden-Web crawler that used important terms provided by users or collected from the query interfaces to query a Web form and crawl the Deep Web content. Alexandros Ntoulas, Petros Zerfos, and Junghoo Cho of [[University of California, Los Angeles|UCLA]] created a hidden-Web crawler that automatically generated meaningful queries to issue against search forms.<ref>{{cite web
| first1 = Ntoulas
| last1 = Alexandros
| first2 = Petros | last2 = Zerfos | first3 = Junghoo | last3 = Cho
| title = Downloading Hidden Web Content
| publisher = [[UCLA]] Computer Science
| year = 2005
| url = http://oak.cs.ucla.edu/~cho/papers/ntoulas-hidden.pdf
| access-date = February 24, 2009}}</ref> Several form query languages (e.g., DEQUEL<ref>{{cite journal
| first1 = Denis
| last1 = Shestakov
| first2 = Sourav S. | last2 = Bhowmick | first3 = Ee-Peng | last3 = Lim
| title = DEQUE: Querying the Deep Web
| journal = Data & Knowledge Engineering |volume=52 |issue=3
| pages = 273–311
| year = 2005
| doi = 10.1016/S0169-023X(04)00107-7
| url = http://www.inf.ufsc.br/~r.mello/deepWeb/querying/DKE2005-Sourav.pdf
}}</ref>) have been proposed that, besides issuing a query, also allow extraction of structured data from result pages. Another effort is DeepPeep, a project of the [[University of Utah]] sponsored by the [[National Science Foundation]], which gathered hidden-web sources (web forms) in different domains based on novel focused crawler techniques.<ref>{{cite conference
| first1 = Luciano
| last1 = Barbosa
| first2 = Juliana
| last2 = Freire
| author2-link = Juliana Freire
| title = An Adaptive Crawler for Locating Hidden-Web Entry Points
| conference = WWW Conference 2007
| year = 2007
| url = http://www.cs.utah.edu/~lbarbosa/publications/ache-www2007.pdf
| access-date = March 20, 2009
| archive-date = June 5, 2011
| archive-url = https://web.archive.org/web/20110605082603/http://www.cs.utah.edu/~lbarbosa/publications/ache-www2007.pdf
| url-status = dead
}}</ref><ref>{{cite conference
| first1 = Luciano
| last1 = Barbosa
| first2 = Juliana
| last2 = Freire
| author2-link = Juliana Freire
| title = Searching for Hidden-Web Databases
| conference = WebDB 2005
| year = 2005
| url = http://www.cs.utah.edu/~lbarbosa/publications/webdb2005.pdf
| access-date = March 20, 2009
| archive-date = June 5, 2011
| archive-url = https://web.archive.org/web/20110605082629/http://www.cs.utah.edu/~lbarbosa/publications/webdb2005.pdf
| url-status = dead
}}</ref>


Commercial search engines have begun exploring alternative methods to crawl the deep web. The [[Sitemap Protocol]] (first developed, and introduced by Google in 2005) and [[Open Archives Initiative Protocol for Metadata Harvesting|OAI-PMH]] are mechanisms that allow search engines and other interested parties to discover deep web resources on particular web servers. Both mechanisms allow web servers to advertise the URLs that are accessible on them, thereby allowing automatic discovery of resources that are not linked directly to the surface web. Google's deep web surfacing system computes submissions for each HTML form and adds the resulting HTML pages into the Google search engine index. The surfaced results account for a thousand queries per second to deep web content.<ref>{{cite conference | first1 = Jayant | last1 = Madhavan | first2 = David | last2 = Ko | first3 = Łucja | last3 = Kot | first4 = Vignesh | last4 = Ganapathy | first5 = Alex | last5 = Rasmussen | first6 = Alon | last6 = Halevy | title = Google's Deep-Web Crawl | publisher = VLDB Endowment, ACM | conference = PVLDB '08, August 23-28, 2008, Auckland, New Zealand | year = 2008 | url = https://homes.cs.washington.edu/~alon/files/vldb08deepweb.pdf | access-date = April 17, 2009 | archive-date = September 16, 2012 | archive-url = https://web.archive.org/web/20120916104001/http://homes.cs.washington.edu/~alon/files/vldb08deepweb.pdf | url-status = dead }}</ref> In this system, the pre-computation of submissions is done using three algorithms:
In [[2005]], [[Yahoo!]] made a small part of the deep Web searchable by releasing Yahoo! Subscriptions. This search engine searches through a few subscription-only Web sites. Some subscription websites display their full content to search engine robots so they will show up in user searches, but then show users a login or subscription page when they click a link from the search engine results page.
# selecting input values for text search inputs that accept keywords,
# identifying inputs that accept only values of a specific type (e.g., date) and
# selecting a small number of input combinations that generate URLs suitable for inclusion into the Web search index.


In 2008, to facilitate users of [[Tor (anonymity network)#Hidden services|Tor hidden services]] in their access and search of a hidden [[.onion]] suffix, [[Aaron Swartz]] designed [[Tor2web]]—a proxy application able to provide access by means of common web browsers.<ref name=RELEASE>{{cite web|last=Aaron|first=Swartz|title=In Defense of Anonymity|url=http://www.aaronsw.com/weblog/tor2web|access-date=February 4, 2014}}</ref> Using this application, deep web links appear as a random sequence of letters followed by the .onion [[top-level domain]].
== Crawling the deep Web ==

Researchers have been exploring how the deep Web can be crawled in an automatic fashion. Raghavan and Garcia-Molina (2001) presented an architectural model for a hidden-Web crawler that used key terms provided by users or collected from the query interfaces to query a Web form and crawl the deep Web resources. Ntoulas et al. (2005) created a hidden-Web crawler that automatically generated meaningful queries to issue against search forms. Their crawler generated promising results, but the problem is far from being solved.

Since a large amount of useful data and information resides in the deep Web, search engines have begun exploring alternative methods to crawl the deep Web. [[Google]]’s [[Google Sitemap|Sitemap Protocol]] and [[mod oai]] are mechanisms that allow search engines and other interested parties to discover deep Web resources on particular Web servers. Both mechanisms allow Web servers to advertise the URLs that are accessible on them, thereby allowing automatic discovery of resources that are not directly linked to the surface Web.

[[Federated search]] by subject category or vertical is an alternative mechanism to crawling the deep Web. Traditional engines have difficulty crawling and indexing deep Web pages and their content, but deep Web search engines like [[CloserLookSearch]], [http://www.science.gov/ Science.gov] and [[Northern Light Group|Northern Light]] create specialty engines by topic to search the deep Web. Because these engines are narrow in their data focus, they are built to access specified deep Web content by topic. These engines can search dynamic or password protected databases that are otherwise closed to search engines.

== Classifying resources ==

It is difficult to automatically determine if a Web resource is a member of the surface Web or the deep Web. If a resource is indexed by a search engine, it is not necessarily a member of the surface Web since the resource could have been found using the [[Sitemap Protocol]], [[mod oai]], [[OAIster]], etc. instead of traditional crawling. If a search engine provides a [[backlink]] for a resource, one may assume that the resource is in the surface Web. Unfortunately, search engines do not always provide all backlinks to resources. Even if a backlink does exist, there is no way to determine if the resource providing the link is itself in the surface Web without crawling all of the Web. Furthermore, a resource may reside in the surface Web, but it has not yet been found by a search engine. Therefore, if we have an arbitrary resource, we cannot know for sure if the resource resides in the surface Web or deep Web without a complete crawl of the Web.

The concept of classifying search results by topic was pioneered by Yahoo! Directory search and is gaining importance as search becomes more relevant in day to day decisions. However, most of the work here has been in categorizing the surface Web by topic. This classification poses a challenge while searching the deep Web whereby two levels of categorization are required. The first level is to categorize sites into vertical topics (health, travel, automobiles, etc.) and sub-topics according to the nature of the content underlying their databases. Several deep Web directories are under development such as [[OAIster]] by the [[University of Michigan]], [http://infomine.ucr.edu INFOMINE] at the [[University of California at Riverside]] and [[DirectSearch]] by [[Gary Price]] to name a few.

The second, more difficult, challenge is to categorize and map the information extracted from multiple deep Web sources according to end-user needs. Deep Web search reports cannot display URLs like traditional search reports. End users expect their search tools to not only find what they are looking for quickly, but to be intuitive and user-friendly. In order to be meaningful, the search reports have to offer some depth to the nature of content that underlie the sources or else the end-user will be lost in the sea of URLs that do not indicate what content lies underneath them. The format in which search results are to be presented varies widely by the particular topic of the search and the type of content being exposed. The challenge is to find and map similar data elements from multiple disparate sources so that search results may be exposed in a unified format on the search report irrespective of their source.

==History==
The first commercial deep Web tool was @1 from Personal Library Software (PLS), announced December 12th, 1996 in partnership with large content providers. According to a December 12th, 1996 press release, @1 started with 5.7 terabytes of content which was estimated to be 30 times the size of the nascent World Wide Web.<ref>AOL (Dec 1996) press release ''[http://web.archive.org/web/19971021232106/www.pls.com/news/pr961212_aol.html announcing AOL's participation in @1]''</ref> PLS was acquired by AOL in 1998 and @1 was abandoned.


==See also==
==See also==
{{Portal|Internet}}
*[[Federated search]]
* [[Clearnet (networking)]]
*[[Robots Exclusion Standard]]
*[[Surface Web]]
* [[DARPA's Memex program]]
*[[Web crawler]]
* [[Deep linking]]
*[[Web Harvesting]]
* [[Deep Web Technologies]]
* [[Intellectual dark web]]
*[[Dark internet]]
*[[Darknet]]
* [[Darknet market]]
* [[Darknet]]
* [[Dark web]]
* [[Tor (network)]]
* [[List of Tor onion services]]


==References==
==References==
{{reflist|30em}}
<references/>


==Further reading==
==Further reading==
{{refbegin|30em}}
* Barker, Joe (Jan 2004). ''[http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html Invisible Web: What it is, Why it exists, How to find it, and Its inherent ambiguity]'' UC Berkeley - Teaching Library Internet Workshops.
* {{cite web | last = Barker | first = Joe | date = January 2004 | url = http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html | title = Invisible Web: What it is, Why it exists, How to find it, and its inherent ambiguity | publisher = University of California, Berkeley, Teaching Library Internet Workshops | access-date = July 26, 2011 | archive-url = https://web.archive.org/web/20050729110408/http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html | archive-date = July 29, 2005 | url-status = dead | df = mdy-all }}.
* {{cite journal |first=Michael K. |last=Bergman | title = The Deep Web: Surfacing Hidden Value | journal = The Journal of Electronic Publishing | year = 2001 | month = Aug | volume = 7 | issue = 1 | url = http://www.press.umich.edu/jep/07-01/bergman.html}}
* {{cite web | first = Saikat | last = Basu | url = http://www.makeuseof.com/tag/10-search-engines-explore-deep-invisible-web/ | title = 10 Search Engines to Explore the Invisible Web | publisher = MakeUseOf.com | date = March 14, 2010 }}.
* Gruchawka, Steve (June 2006). ''[http://techdeepweb.com/ How-To Guide to the Deep Web]'' TechDeepWeb.com, [http://TechDeepWeb.com/ http://TechDeepWeb.com]
* {{cite web | last = Ozkan | first = Akin | date = November 2014 | url = http://akinozkan.com/deep-web-derin-internet/ | title = Deep Web /Derin İnternet | access-date = November 6, 2014 | archive-date = November 8, 2014 | archive-url = https://web.archive.org/web/20141108182138/http://akinozkan.com/deep-web-derin-internet/ | url-status = dead }}.
* Hamilton, Nigel (2003). [http://turbo10.com/papers/deepnet.pdf ''The Mechanics of a Deep Net Metasearch Engine''] - 12th World Wide Web Conference poster.
* {{cite web | last = Gruchawka | first = Steve | date = June 2006 | url = http://techdeepweb.com/ | title = How-To Guide to the Deep Web | access-date = February 28, 2007 | archive-date = January 5, 2014 | archive-url = https://web.archive.org/web/20140105152032/http://techdeepweb.com/ | url-status = dead }}.
* {{cite conference |first=Bin |last=He |coauthors= Chang, Kevin Chen-Chuan | year = 2003 | title = Statistical Schema Matching across Web Query Interfaces | booktitle = Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data | url = http://eagle.cs.uiuc.edu/pubs/2003/unifiedschema-sigmod03-hc-mar03.pdf}}
* {{cite web | last = Hamilton | first = Nigel | year = 2003 | url = http://www2003.org/cdrom/papers/poster/p170/poster/poster.html | title = The Mechanics of a Deep Net Metasearch Engine | publisher = 12th World Wide Web Conference }}.
* {{cite journal |first=Bin |last=He |coauthors= Patel, Mitesh; Zhang, Zhen; Chang, Kevin Chen-Chuan | title = Accessing the Deep Web: A Survey | journal = Communications of the ACM (CACM) | pages = 94-101 | year = 2007 | month = May | volume = 50 | issue = 2 | url = http://doi.acm.org/10.1145/1230819.1241670}}
* {{cite conference |first=Panagiotis G. |last=Ipeirotis |coauthors=Gravano, Luis; Sahami, Mehran | year = 2001 | title = Probe, Count, and Classify: Categorizing Hidden-Web Databases | booktitle = Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data | pages = 67-78 | url = http://qprober.cs.columbia.edu/publications/sigmod2001.pdf}}
* {{cite conference | first1 = Bin | last1 = He | last2 = Chang | first2 = Kevin Chen-Chuan | year = 2003 | title = Statistical Schema Matching across Web Query Interfaces | book-title = Proceedings of the 2003 ACM SIGMOD International Conference on Management of Data | url = http://eagle.cs.uiuc.edu/pubs/2003/unifiedschema-sigmod03-hc-mar03.pdf | archive-url = https://web.archive.org/web/20110720085124/http://eagle.cs.uiuc.edu/pubs/2003/unifiedschema-sigmod03-hc-mar03.pdf | archive-date = July 20, 2011 }}
* {{cite news | last = Howell O'Neill | first = Patrick | date = October 2013 | work = The Daily Dot | url = http://www.dailydot.com/technology/how-to-search-the-deep-web/ | title = How to search the Deep Web }}.
* {{cite journal |last=King |first=John D. |coauthors= Li, Yuefeng; Tao, Daniel; Nayak, Richi | title = Mining World Knowledge for Analysis of Search Engine Content | journal = Web Intelligence and Agent Systems: An International Journal | pages = 233-253 | year = 2007 | month = November | volume = 5 | issue = 3 | url = http://sky.fit.qut.edu.au/~kingj2/downloads/king07mining.pdf}}
* {{cite conference | first1 = Panagiotis G. | last1 = Ipeirotis | last2 = Gravano | first2 = Luis | last3 = Sahami | first3 = Mehran | year = 2001 | title = Probe, Count, and Classify: Categorizing Hidden-Web Databases | book-title = Proceedings of the 2001 ACM SIGMOD International Conference on Management of Data | pages = 67–78 | url = http://qprober.cs.columbia.edu/publications/sigmod2001.pdf | access-date = September 26, 2006 | archive-url = https://web.archive.org/web/20060912141432/http://qprober.cs.columbia.edu/publications/sigmod2001.pdf | archive-date = September 12, 2006 | url-status = dead | df = mdy-all }}
* {{cite journal |last=McCown |first=Frank |coauthors= Liu, Xiaoming; Nelson, Michael L.; Zubair, Mohammad | title = Search Engine Coverage of the OAI-PMH Corpus | journal = IEEE Internet Computing | pages = 66-73 | year = 2006 | month = Mar/Apr | volume = 10 | issue = 2 | url = http://library.lanl.gov/cgi-bin/getfile?LA-UR-05-9158.pdf}}
* {{cite journal | last1 = King | first1 = John D. | last2 = Li | first2 = Yuefeng | last3 = Tao | first3 = Daniel | last4 = Nayak | first4 = Richi | title = Mining World Knowledge for Analysis of Search Engine Content | journal = Web Intelligence and Agent Systems | pages = 233–53 | date = November 2007 | volume = 5 | issue = 3 | url = http://sky.fit.qut.edu.au/~kingj2/downloads/king07mining.pdf | access-date = July 26, 2011 | archive-url = https://web.archive.org/web/20081203104452/http://sky.fit.qut.edu.au/~kingj2/downloads/king07mining.pdf | archive-date = December 3, 2008 | url-status = dead | df = mdy-all }}
* {{cite conference |first=Alexandros |last=Ntoulas |coauthors=Zerfos, Petros; Cho, Junghoo | year = 2005 | title = Downloading Textual Hidden Web Content Through Keyword Queries | booktitle = Proceedings of the Joint Conference on Digital Libraries (JCDL) | pages = 100-109 | url = http://oak.cs.ucla.edu/~ntoulas/pubs/ntoulas_hidden_web.pdf}} [http://oak.cs.ucla.edu/~cho/papers/ntoulas-hidden.pdf Extended version]
* {{cite journal | last1 = McCown | first1 = Frank | last2 = Liu | first2 = Xiaoming | last3 = Nelson | first3 = Michael L. | last4 = Zubair | first4 = Mohammad | title = Search Engine Coverage of the OAI-PMH Corpus | journal = [[IEEE Internet Computing]] | pages = 66–73 | date = March–April 2006 | volume = 10 | issue = 2 | url = http://library.lanl.gov/cgi-bin/getfile?LA-UR-05-9158.pdf | doi = 10.1109/MIC.2006.41 | s2cid = 15511914 }}
* Personal Library Software (Dec 1996) press release ''[http://web.archive.org/web/19971021232057/www.pls.com/news/pr961212_at1.html announcing @1 as an "Invisible Web" search service]''
* {{cite book |last=Price |first=Gary |authorlink=Gary Price |coauthors=Sherman, Chris |title=The Invisible Web : Uncovering Information Sources Search Engines Can't See |year=2001 |month=July |publisher=CyberAge Books |location= |isbn=0-910965-51-X }}
* {{cite book |last1=Price |first1=Gary | last2 = Sherman | first2 = Chris | title= The Invisible Web: Uncovering Information Sources Search Engines Can't See |date=July 2001 |publisher=CyberAge Books |isbn=978-0-910965-51-4 }}
* Shestakov, Denis (June 2008). ''[https://web.archive.org/web/20140706120641/https://oa.doria.fi/handle/10024/38506 Search Interfaces on the Web: Querying and Characterizing]''. TUCS Doctoral Dissertations 104, University of Turku
* {{cite conference |first=Sriram |last=Raghavan |coauthors=Garcia-Molina, Hector | year = 2001 | title = Crawling the Hidden Web | booktitle = Proceedings of the 27th International Conference on Very Large Data Bases (VLDB) | pages = 129-138 | url = http://www.dia.uniroma3.it/~vldbproc/017_129.pdf}}
* {{cite news | url = https://www.washingtonpost.com/archive/business/2008/12/11/firms-push-for-a-more-searchable-federal-web/c4f48ea6-6b4b-421c-8603-f53ace3358e9?itid=sr_1_1ccbe264-886d-4fc1-8603-4cd596c7bf20 | title = Firms Push for a More Searchable Federal Web | newspaper =[[The Washington Post]] | date = December 11, 2008 | page = D01 | first = Peter | last = Whoriskey}}
* Wright, Alex (Mar 2004). ''In Search of the Deep Web,'' Salon.com, [http://archive.salon.com/tech/feature/2004/03/09/deep_web/index_np.html http://www.salon.com/tech/feature/2004/03/09/deep_web/]
* {{cite news | last = Wright | first = Alex | date = March 2004 | url = http://www.salon.com/tech/feature/2004/03/09/deep_web/ | title = In Search of the Deep Web | work = Salon | archive-url = http://archive.salon.com/tech/feature/2004/03/09/deep_web/index_np.html | archive-date = March 9, 2007 }}.
* {{cite podcast | last = Scientists | first = Naked | date = December 2014 | url = http://www.thenakedscientists.com/HTML/podcasts/naked-scientists/show/20141202/ | title = The Internet: the good, the bad and the ugly – In-depth exploration of the Internet and the Dark Web by Cambridge University's Naked Scientists }}
* {{cite thesis |last1=King |first1=John D. | title= Search Engine Content Analysis |date=July 2009 |publisher=Queensland University of Technology| url=https://eprints.qut.edu.au/26241/1/John_King_Thesis.pdf}}
{{refend}}


==External links==
==External links==
*{{commons category-inline}}
* [http://blog.relactions.com/2007/08/travel-industry-and-deep-web-exclusive.html Travel industry and Deep Web] interview with Marcus P. Zillman
*{{Wiktionary-inline}}

{{Authority control}}


[[Category:World Wide Web]]
<!--Category:Search indexing?-->


[[Category:Dark web]]
[[bg:Дълбока мрежа]]
[[Category:Internet search engines]]
[[da:Dybe net]]
[[Category:Internet terminology]]
[[de:Deep Web]]
[[fr:Web profond]]
[[Category:2000s neologisms]]
[[it:Web invisibile]]
[[ru:Глубокая паутина]]

Latest revision as of 18:54, 17 November 2024

This article is about the part of the World Wide Web not indexed by traditional search engines. For other uses, see Deep web (disambiguation).
Not to be confused with Dark web.

The deep web,[1] invisible web,[2] or hidden web[3] are parts of the World Wide Web whose contents are not indexed by standard web search-engine programs.[4] This is in contrast to the "surface web", which is accessible to anyone using the Internet.[5] Computer scientist Michael K. Bergman is credited with inventing the term in 2001 as a search-indexing term.[6]

Deep web sites can be accessed by a direct URL or IP address, but may require entering a password or other security information to access actual content.[7][8] Uses of deep web sites include web mail, online banking, cloud storage, restricted-access social-media pages and profiles, and web forums that require registration for viewing content. It also includes paywalled services such as video on demand and some online magazines and newspapers.

Terminology

[edit]

The first conflation of the terms "deep web" and "dark web" happened during 2009 when deep web search terminology was discussed together with illegal activities occurring on the Freenet and darknet.[9] Those criminal activities include the commerce of personal passwords, false identity documents, drugs, firearms, and child pornography.[10]

Since then, after their use in the media's reporting on the black-market website Silk Road, media outlets have generally used 'deep web' synonymously with the dark web or darknet, a comparison some reject as inaccurate[11] and consequently has become an ongoing source of confusion.[12] Wired reporters Kim Zetter[13] and Andy Greenberg[14] recommend the terms be used in distinct fashions. While the deep web is a reference to any site that cannot be accessed by a traditional search engine, the dark web is a portion of the deep web that has been hidden intentionally and is inaccessible by standard browsers and methods.[15][16][17][18][19][excessive citations]

Non-indexed content

[edit]

Bergman, in a paper on the deep web published in The Journal of Electronic Publishing, mentioned that Jill Ellsworth used the term Invisible Web in 1994 to refer to websites that were not registered with any search engine.[20] Bergman cited a January 1996 article by Frank Garcia:[21]

It would be a site that's possibly reasonably designed, but they didn't bother to register it with any of the search engines. So, no one can find them! You're hidden. I call that the invisible Web.

Another early use of the term Invisible Web was by Bruce Mount and Matthew B. Koll of Personal Library Software, in a description of the No. 1 Deep Web program found in a December 1996 press release.[22]

The first use of the specific term deep web, now generally accepted, occurred in the aforementioned 2001 Bergman study.[20]

Indexing methods

[edit]

Methods that prevent web pages from being indexed by traditional search engines may be categorized as one or more of the following:

  1. Contextual web: pages with content varying for different access contexts (e.g., ranges of client IP addresses or previous navigation sequence).
  2. Dynamic content: dynamic pages, which are returned in response to a submitted query or accessed only through a form, especially if open-domain input elements (such as text fields) are used; such fields are hard to navigate without domain knowledge.
  3. Limited access content: sites that limit access to their pages in a technical manner (e.g., using the Robots Exclusion Standard or CAPTCHAs, or no-store directive, which prohibit search engines from browsing them and creating cached copies).[23] Sites may feature an internal search engine for exploring such pages.[24][25]
  4. Non-HTML/text content: textual content encoded in multimedia (image or video) files or specific file formats not recognised by search engines.
  5. Private web: sites that require registration and login (password-protected resources).
  6. Scripted content: pages that are accessible only by links produced by JavaScript as well as content dynamically downloaded from Web servers via Flash or Ajax solutions.
  7. Software: certain content is hidden intentionally from the regular Internet, accessible only with special software, such as Tor, I2P, or other darknet software. For example, Tor allows users to access websites using the .onion server address anonymously, hiding their IP address.
  8. Unlinked content: pages which are not linked to by other pages, which may prevent web crawling programs from accessing the content. This content is referred to as pages without backlinks (also known as inlinks). Also, search engines do not always detect all backlinks from searched web pages.
  9. Web archives: Web archival services such as the Wayback Machine enable users to see archived versions of web pages across time, including websites that have become inaccessible and are not indexed by search engines such as Google. [6]The Wayback Machine may be termed a program for viewing the deep web, as web archives that are not from the present cannot be indexed, as past versions of websites are impossible to view by a search. All websites are updated at some time, which is why web archives are considered Deep Web content.[26]

Content types

[edit]

While it is not always possible to discover directly a specific web server's content so that it may be indexed, a site potentially can be accessed indirectly (due to computer vulnerabilities).

To discover content on the web, search engines use web crawlers that follow hyperlinks through known protocol virtual port numbers. This technique is ideal for discovering content on the surface web but is often ineffective at finding deep web content. For example, these crawlers do not attempt to find dynamic pages that are the result of database queries due to the indeterminate number of queries that are possible.[6] It has been noted that this can be overcome (partially) by providing links to query results, but this could unintentionally inflate the popularity of a site of the deep web.

DeepPeep, Intute, Deep Web Technologies, Scirus, and Ahmia.fi are a few search engines that have accessed the deep web. Intute ran out of funding and is now a temporary static archive as of July 2011.[27] Scirus retired near the end of January 2013.[28]

Researchers have been exploring how the deep web can be crawled in an automatic fashion, including content that can be accessed only by special software such as Tor. In 2001, Sriram Raghavan and Hector Garcia-Molina (Stanford Computer Science Department, Stanford University)[29][30] presented an architectural model for a hidden-Web crawler that used important terms provided by users or collected from the query interfaces to query a Web form and crawl the Deep Web content. Alexandros Ntoulas, Petros Zerfos, and Junghoo Cho of UCLA created a hidden-Web crawler that automatically generated meaningful queries to issue against search forms.[31] Several form query languages (e.g., DEQUEL[32]) have been proposed that, besides issuing a query, also allow extraction of structured data from result pages. Another effort is DeepPeep, a project of the University of Utah sponsored by the National Science Foundation, which gathered hidden-web sources (web forms) in different domains based on novel focused crawler techniques.[33][34]

Commercial search engines have begun exploring alternative methods to crawl the deep web. The Sitemap Protocol (first developed, and introduced by Google in 2005) and OAI-PMH are mechanisms that allow search engines and other interested parties to discover deep web resources on particular web servers. Both mechanisms allow web servers to advertise the URLs that are accessible on them, thereby allowing automatic discovery of resources that are not linked directly to the surface web. Google's deep web surfacing system computes submissions for each HTML form and adds the resulting HTML pages into the Google search engine index. The surfaced results account for a thousand queries per second to deep web content.[35] In this system, the pre-computation of submissions is done using three algorithms:

  1. selecting input values for text search inputs that accept keywords,
  2. identifying inputs that accept only values of a specific type (e.g., date) and
  3. selecting a small number of input combinations that generate URLs suitable for inclusion into the Web search index.

In 2008, to facilitate users of Tor hidden services in their access and search of a hidden .onion suffix, Aaron Swartz designed Tor2web—a proxy application able to provide access by means of common web browsers.[36] Using this application, deep web links appear as a random sequence of letters followed by the .onion top-level domain.

See also

[edit]

References

[edit]
  1. ^ Hamilton, Nigel (2019–2020). "The Mechanics of a Deep Net Metasearch Engine". In Isaías, Pedro; Palma dos Reis, António (eds.). Proceedings of the IADIS International Conference on e-Society. IADIS Press. pp. 1034–6. CiteSeerX 10.1.1.90.5847. ISBN 978-972-98947-0-1.
  2. ^ Devine, Jane; Egger-Sider, Francine (August 2021). "Beyond google: the invisible web in the academic library". The Journal of Academic Librarianship. 30 (4): 265–269. doi:10.1016/j.acalib.2004.04.010.
  3. ^ Raghavan, Sriram; Garcia-Molina, Hector (September 11–14, 2001). "Crawling the Hidden Web". 27th International Conference on Very Large Data Bases.
  4. ^ Maor, Etay. "Council Post: Lessons Learned From Tracing Cybercrime's Evolution On The Dark Web". Forbes. Retrieved September 22, 2024.
  5. ^ "Surface Web". Computer Hope. Retrieved June 20, 2018.
  6. ^ a b c Wright, Alex (February 22, 2009). "Exploring a 'Deep Web' That Google Can't Grasp". The New York Times. Retrieved September 2, 2019. [...] Mike Bergman, a computer scientist and consultant who is credited with coining the term Deep Web.
  7. ^ Madhavan, J., Ko, D., Kot, Ł., Ganapathy, V., Rasmussen, A., & Halevy, A. (2008). Google's deep web crawl. Proceedings of the VLDB Endowment, 1(2), 1241–52.
  8. ^ Shedden, Sam (June 8, 2014). "How Do You Want Me to Do It? Does It Have to Look like an Accident? – an Assassin Selling a Hit on the Net; Revealed Inside the Deep Web". Sunday Mail. Archived from the original on March 1, 2020.
  9. ^ Beckett, Andy (November 26, 2009). "The dark side of the internet". Retrieved August 9, 2015.
  10. ^ D. Day. Easiest Catch: Don't Be Another Fish in the Dark Net. Wake Forest University: TEDx Talks. Archived from the original on November 13, 2021.
  11. ^ "Clearing Up Confusion – Deep Web vs. Dark Web". BrightPlanet. March 27, 2014.
  12. ^ Solomon, Jane (May 6, 2015). "The Deep Web vs. The Dark Web". Retrieved May 26, 2015.
  13. ^ NPR Staff (May 25, 2014). "Going Dark: The Internet Behind The Internet". Retrieved May 29, 2015.
  14. ^ Greenberg, Andy (November 19, 2014). "Hacker Lexicon: What Is the Dark Web?". Retrieved June 6, 2015.
  15. ^ "The Impact of the Dark Web on Internet Governance and Cyber Security" (PDF). January 20, 2014. Archived from the original (PDF) on January 16, 2017. Retrieved January 15, 2017.
  16. ^ Lam, Kwok-Yan; Chi, Chi-Hung; Qing, Sihan (November 23, 2016). Information and Communications Security: 18th International Conference, ICICS 2016, Singapore, Singapore, November 29 – December 2, 2016, Proceedings. Springer. ISBN 9783319500119. Retrieved January 15, 2017.
  17. ^ "The Deep Web vs. The Dark Web | Dictionary.com Blog". Dictionary Blog. May 6, 2015. Retrieved January 15, 2017.
  18. ^ Akhgar, Babak; Bayerl, P. Saskia; Sampson, Fraser (January 1, 2017). Open Source Intelligence Investigation: From Strategy to Implementation. Springer. ISBN 9783319476711. Retrieved January 15, 2017.
  19. ^ "What is the dark web and who uses it?". The Globe and Mail. Retrieved January 15, 2017.
  20. ^ a b Bergman, Michael K (August 2001). "The Deep Web: Surfacing Hidden Value". The Journal of Electronic Publishing. 7 (1). doi:10.3998/3336451.0007.104. hdl:2027/spo.3336451.0007.104.
  21. ^ Garcia, Frank (January 1996). "Business and Marketing on the Internet". Masthead. 15 (1). Archived from the original on December 5, 1996. Retrieved February 24, 2009.
  22. ^ @1 started with 5.7 terabytes of content, estimated to be 30 times the size of the nascent World Wide Web; PLS was acquired by AOL in 1998 and @1 was abandoned. "PLS introduces AT1, the first 'second generation' Internet search service" (Press release). Personal Library Software. December 1996. Archived from the original on October 21, 1997. Retrieved February 24, 2009.
  23. ^ Fielding, R.; Nottingham, M.; Reschke, J. (2014). Fielding, R.; Nottingham, M.; Reschke, J. (eds.). "Hypertext Transfer Protocol (HTTP/1.1): Caching". Internet Engineering Task Force. doi:10.17487/RFC7234. Retrieved July 30, 2014.
  24. ^ Special:Search
  25. ^ "Internet Archive Search".
  26. ^ Wiener-Bronner, Danielle (June 10, 2015). "NASA is indexing the 'Deep Web' to show mankind what Google won't". Fusion. Archived from the original on June 30, 2015. Retrieved June 27, 2015. There are other simpler versions of Memex already available. "If you've ever used the Internet Archive's Wayback Machine", which gives you past versions of a website not accessible through Google, then you've technically searched the Deep Web, said Chris Mattmann.
  27. ^ "Intute FAQ, dead link". Retrieved October 13, 2012.
  28. ^ "Elsevier to Retire Popular Science Search Engine". library.bldrdoc.gov. December 2013. Archived from the original on June 23, 2015. Retrieved June 22, 2015. by end of January 2014, Elsevier will be discontinuing Scirus, its free science search engine. Scirus has been a wide-ranging research tool, with over 575 million items indexed for searching, including webpages, pre-print articles, patents, and repositories.
  29. ^ Sriram Raghavan; Garcia-Molina, Hector (2000). "Crawling the Hidden Web" (PDF). Stanford Digital Libraries Technical Report. Archived from the original (PDF) on May 8, 2018. Retrieved December 27, 2008.
  30. ^ Raghavan, Sriram; Garcia-Molina, Hector (2001). "Crawling the Hidden Web" (PDF). Proceedings of the 27th International Conference on Very Large Data Bases (VLDB). pp. 129–38.
  31. ^ Alexandros, Ntoulas; Zerfos, Petros; Cho, Junghoo (2005). "Downloading Hidden Web Content" (PDF). UCLA Computer Science. Retrieved February 24, 2009.
  32. ^ Shestakov, Denis; Bhowmick, Sourav S.; Lim, Ee-Peng (2005). "DEQUE: Querying the Deep Web" (PDF). Data & Knowledge Engineering. 52 (3): 273–311. doi:10.1016/S0169-023X(04)00107-7.
  33. ^ Barbosa, Luciano; Freire, Juliana (2007). An Adaptive Crawler for Locating Hidden-Web Entry Points (PDF). WWW Conference 2007. Archived from the original (PDF) on June 5, 2011. Retrieved March 20, 2009.
  34. ^ Barbosa, Luciano; Freire, Juliana (2005). Searching for Hidden-Web Databases (PDF). WebDB 2005. Archived from the original (PDF) on June 5, 2011. Retrieved March 20, 2009.
  35. ^ Madhavan, Jayant; Ko, David; Kot, Łucja; Ganapathy, Vignesh; Rasmussen, Alex; Halevy, Alon (2008). Google's Deep-Web Crawl (PDF). PVLDB '08, August 23-28, 2008, Auckland, New Zealand. VLDB Endowment, ACM. Archived from the original (PDF) on September 16, 2012. Retrieved April 17, 2009.
  36. ^ Aaron, Swartz. "In Defense of Anonymity". Retrieved February 4, 2014.

Further reading

[edit]
[edit]
  • Media related to Deep web at Wikimedia Commons
  • The dictionary definition of deep web at Wiktionary
Retrieved from "https://en.wikipedia.org/w/index.php?title=Deep_web&oldid=1258019590"