A Standard for robot exclusion documents the methods used to exclude robots from a server by creating a file which specifiec access policies for robots.

Grabber is the web indexing spider for Direct Hit's World Wide Web search engine.

This paper contains an evaluation of the Standard for Robots Exclusion, identifies some of its problems and feature requsts, and recomends future work.

Information about the Mercator project and frequently asked questions about Mercator.

The Robots META tag is a simple mechanism to indicate to visiting Web Robots if a page should be indexed, or links on the page should be followed.

The following document is intended to highlight some issues involving the current standard for robot exclusion, as well as to propose some suggestions for future expansion of the standard.

This guide is aimed at Web Server Administrators who want to use the Robots Exclusion Protocol.

Search Engine World - Robots.txt Validator

Search engine programs are called "robots" or "spiders", because they follow links on the Web to discover new pages. These are fairly fragile and both search administrators and webmasters need to understand what links they can and can't follow.

Robots generator is a free web tool to create server side or client side robots to control search engine indexing and spidering of your site.

The Anatomy of a Large-Scale Hypertextual Web Search Engine research paper written by Sergey Brin and Lawrence Page founders of Google. In this paper they present Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext.

Frequently asked questions (FAQs) about web robots and spiders. About WWW robots, indexing robots, administrator information, and documentation on how to use the robots exclusion protocol.

Web Robots are programs that traverse the Web automatically. Some people call them Web Wanderers, Crawlers, or Spiders. These pages have further information about these Web Robots.

Database of web robots overview. Find crawler contact information and review robots by type.

HTML Author's Guide to the Robots META tag.

Larbin is a web crawler (also called (web) robot, spider, scooter...). It is intended to fetch a large number of web pages to fill the database of a search engine. With a network fast enough, Larbin should be able to fetch more than 100 millions pages on a standard PC.

Checks the contents of a site's robots.txt against that contained in the latest specification, along with providing warnings on the use of new features that are not yet widely deployed.

The Scouting Spider of Global ScoutNet provides a full searchable database for Scouting and Guiding related pages on the world wide web, a free access counter for homepages and a Scouting and Guiding Webring feature to connect pages together.

Listing of Search Engine IP Addresses

IP Addresses, Hosts, and User Agents of the top Search Engines. Information to assist you in detecting when a spider has downloaded a page from your site.

Every wonder what the Search Engines see? Try this search engine spider simulator to find out.

Ian Peacock describes the Robots Exclusion Protocol and reports on a analysis of the use of this protocol by UK Universities and Colleges.

A list of the 360 known spiders.

Free cloaking scripts, spider lists, forums, and tons of information.

Nice link to a robots list. View robots by their name, by robot type, and the contact.

Web Robots FAQ's, Robots Exclusion, A list of Robots, Robots Mailing List, Articles and papers, and related sites.

This tool allows you to make your own robots.txt file online, easy and fast.

This utility will perform some simple checking of the server-wide robots.txt file for any URL.