~ Early Web Links ~

Robotstxt.org is the definitive reference for understanding web robots, crawlers, and spiders, covering everything from how to write a robots.txt file to blocking unwanted bots from your site. It includes a robots database, a robots.txt syntax checker, an IP lookup tool, and a comprehensive FAQ making it an essential stop for webmasters and developers.

People and Blogs is a weekly newsletter series created by Manu that spotlights interesting individuals and their personal blogs through interview-style profiles. The archive features hundreds of bloggers and their sites, making it a rich discovery resource for anyone looking to explore the indie web beyond social media.

Edited by Paul Wise and Jed Wing, this unofficial specification documents Microsoft's HTML Help system in exhaustive technical detail, covering the CHM file format, LZX compression, XML structure, and all associated configuration options. Reverse-engineered without NDA restrictions and released under the GNU GPL, it serves as a rare open reference for developers working with or building tools around the proprietary .chm format.

Search Engine History.com, published by Aaron Wall, traces the full arc of search engine development from Vannevar Bush's 1945 vision of hypertext through the rise of Google, Yahoo, and Microsoft. Covering early directories, meta search, SEO, pay-per-click advertising, and legal battles, it serves as a comprehensive reference for anyone curious about how the modern web's information retrieval systems came to be.

Published in D-Lib Magazine in April 2005, this academic article by Tony Hammond and colleagues at Nature Publishing Group provides a comprehensive review of social bookmarking tools, covering platforms like del.icio.us, tagging systems, RSS feeds, and the emerging concept of the social web. It offers a detailed look at how user-driven bookmarking and tagging were reshaping information management on the early Web, making it a fascinating historical snapshot of Web 2.0 in its infancy.

Xah Lee's Unicode Search is a comprehensive reference tool for finding Unicode characters, emojis, and symbols by name, ID, or hex code, covering everything from math symbols to ancient scripts like Cuneiform and Linear B. Visitors can browse an extensive categorized index spanning emojis, currencies, IPA characters, box art, and dozens of world writing systems, making it an invaluable bookmark for developers and writers alike.

Secret Onions is a community-maintained Tor hidden service link directory focused on legitimate content, tools, and knowledge rather than scammy dark web marketplaces. Curated with an anti-surveillance, pro-privacy philosophy, it organizes onion links into categories like forums, email services, file hosting, software, and search engines while actively filtering out scam and marketplace sites.

The Search Engine Map is an interactive visual reference that maps all English-language search engines, showing what type each is and where they source their organic results. It distinguishes crawler-based engines from metasearch engines and illustrates the relationships between them in a network graph format.

L-Lists is a collaborative list-making platform developed by Statistical Consultants Ltd, where registered users can create, contribute to, and browse curated lists covering everything from search engines to science fiction films. The site functions as a crowd-sourced reference hub, offering a structured directory of lists on technology, games, media, and more.

A preserved page from the DMOZ Open Directory Project explaining the structure and internationalization of the World category, which supported 90 languages and allowed volunteer editors to build out non-English directory sections. It provides editor guidance, FAQ links, and details on how subcategories were organized across languages for one of the web's most ambitious human-curated link directories.