A spider is a program used by a search engine to index the World Wide Web content. Spiders captures specific information about a page some capture the title and the first 1,000 characters of content, some captures the title and "description" meta tag, some look for only the keywords and some use the combination of all of these methods.