Crawling, Indexing and Serving are three very important concepts in the search engine business. Every search engine implements these three concepts in some way or the other, using different algorithms. Knowing how they work can help you in optimizing your website to get more visitors by way of search engines.
To understand these concepts, we will first need to learn about the Google bot. The Google bot is the search bot or web spider software used by Google to crawl through documents, web pages on the web and index them. It is used to create a huge index of web pages which Google stores in its database. This database index is then accessed and queried whenever any user searches for something. The Google bot is identified by a particular user agent string. Webmasters can exclude the Google bot from indexing their pages by editing the robots.txt file in that particular directory.
Crawling
Crawling is the process in which the Google Bot crawls and discovers new pages and also updates indexed pages. Crawling is accomplished by using a complex algorithm which determines which pages are to be indexed. Search Engine Optimization is usually done in order to facilitate crawling and indexing of your websites to achieve higher SERP (Search Engine Result Page) rankings.
Indexing
After the crawling process is finished, Google compiles an index of all the pages the Google Bot retrieves after crawling. All relevant information about a webpage including location, title, meta tags, keywords, attributes etc is stored in the index. The index is then accessed to retrieve results whenever the user searches for a particular term. Google currently has an index of about several billion web pages, much more than any other search engine.
Serving
Serving is the next piece of the search puzzle. Whenever a user searches for a particular term, the Google searches its index, sorts all the results retrieved and displays on them on the results page. Google uses a complex ranking algorithm called Page Rank in order to determine the importance of a particular page. The results are then displayed according to their relevance to the search term, the page rank, and many other factors.
SEO is mainly done in order to ensure that the crawling and indexing of your website is done in the best possible way. Ranking highly on the result pages can drive a lot of organic traffic to your website resulting in higher revenues.