We describe the design, prototyping and evaluation of ARC, a system for automatically compiling a list of authoritative Web resources on any (sufficiently broad) topic.

This paper studies in what order a crawler should visit the URLs it has seen, in order to obtain more "important" pages first. Obtaining important pages rapidly can be very useful when a crawler cannot visit the entire Web in a reasonable amount of time.

Technical paper describing Mercator and the architecture of a scalable web crawler.

The Anatomy of a Large-Scale Hypertextual Web Search Engine research paper written by Sergey Brin and Lawrence Page founders of Google. In this paper they present Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext.

Paper discusses efficient techniques for computing PageRank, a ranking metric for hypertext documents. Several methods are discussed analyzing the convergence of PageRank based on induced ordering of the pages.

The Systems Research Center (SRC) is one of Compaq's Corporate Research labs.

A Cornell University computer scientist has developed a new method of searching the World Wide Web that uses the way sites are linked to one another, rather than their text content, to find the most valuable sites on a given topic.

In this paper we propose a new approach to authoritative ranking, which we call Hilltop. Our approach is based on the same assumptions as the other connectivity algorithms, namely that the number and quality of the sources referring to a page are a good measure of the page's quality.

Explains how search engines rank web pages by determining relevancy though analyzing keyword location, frequency and through other methods.

Robots, spiders and other user agents: a resource for WebMasters

Free cloaking scripts, spider lists, forums, and tons of information.

The Term Vector Database allows fast access to indexing terms for Web pages.

Web Robots FAQ's, Robots Exclusion, A list of Robots, Robots Mailing List, Articles and papers, and related sites.

WTMS: A System for Collecting and Analysing Topic-Specific Web Information

This document supplements the mod_rewrite reference documentation. It describes how one can use Apache's mod_rewrite to solve typical URL-based problems webmasters are usually confronted with in practice.

This module uses a rule-based rewriting engine (based on a regular-expression parser) to rewrite requested URLs on the fly.

Jon Kleinberg's researches algorithms which exploit the combinatorial structure of networks and information. This includes techniques for analyzing and modeling link structure in the World Wide Web.

Here you can find fulltexts and bibliographic data of a number of electronic publications of the Stanford InfoLab.

Universal Resource Identifiers in WWW: A Unifying Syntax for the Expression of Names and Addresses of Objects on the Network as used in the World-Wide Web