What happens when you type a search query

When you type a question into Google, Bing, or another search engine, you are not searching the entire internet in real time. Instead, you are searching a massive database the search engine built beforehand by crawling billions of web pages and storing information about them. The search engine matches your words against that stored database, ranks the results by relevance and authority, and shows you the top matches in seconds.

The process has three main stages: crawling (finding pages), indexing (storing information about them), and ranking (deciding which results to show first). Understanding how these work explains why some pages appear at the top of your results and others do not, and why a page you visited yesterday might not show up today.

Key Takeaways

  • Search engines use automated programs called crawlers to visit web pages, follow links, and discover new content across the internet.
  • Indexing stores information about each page's content, structure, and links so the search engine can retrieve it when ready when you search.
  • Ranking algorithms score pages based on relevance to your query, the authority of the site, and hundreds of other factors to order results from most to least useful.
  • Your search history is stored separately from how search engines index the public web, and the two systems work independently.

How crawlers discover and visit web pages

Search engines send out automated programs called crawlers (also called spiders or bots) that visit web pages the same way you do in a browser. They follow links from one page to another, reading the HTML code that makes up each page. When a crawler finds a page it has never seen before, it notes the URL and adds it to a queue to be processed.

Crawlers start from pages they already know about — usually the homepages of major websites — and follow every link they find. If a page links to ten other pages, the crawler visits those ten pages and follows their links in turn. This process repeats across the entire web, though crawlers prioritize pages that are linked to frequently or from authoritative sites, because those tend to be more important.

Website owners can control what crawlers see by using a file called robots.txt in the root directory of their site. This file tells crawlers which parts of the site to visit and which to skip. A page can also include a meta tag that tells crawlers not to index it, even if they visit it. Without these signals, crawlers will attempt to visit and index everything they can reach.

How search engines store and organize page information

Indexing is the process of reading a page's content and storing information about it in a database. When a crawler visits a page, the search engine extracts the text, identifies headings and links, notes the page's title and description, and records when the page was last updated. All of this information gets stored in an index — essentially a giant catalog organized so the search engine can look up information when ready.

The index does not store the entire text of every page. Instead, it stores a compressed version that includes the words on the page, their location (heading, body text, link anchor), and how often they appear. The search engine also records metadata: the page's title, the domain it lives on, the date it was last crawled, and the pages that link to it. This structure allows the search engine to answer your query without having to read billions of full web pages every time you search.

Indexing happens continuously. When a crawler revisits a page and finds changes, the search engine updates its index. Pages that change frequently get crawled more often; pages that rarely change get crawled less often. A brand-new page might take days or weeks to appear in search results because crawlers have not found it yet, or because the search engine has not gotten around to indexing it.

How ranking algorithms decide which results to show first

Once the search engine has indexed a page, it still needs to decide whether that page is relevant to your specific query and how much authority it has. Ranking algorithms score pages on hundreds of factors and sort results from highest to lowest score. The exact factors are proprietary — Google does not publish its complete ranking formula — but the major ones are well understood.

Relevance is the first factor. If you search for "how to fix a leaky faucet," the search engine looks for pages that contain those words or related terms, in prominent places like the title or headings. A page titled "How to Fix a Leaky Faucet" ranks higher than a page that mentions faucets once in the middle of an article about plumbing history.

Authority is the second major factor. The search engine estimates a page's authority by counting how many other pages link to it, and how authoritative those linking pages are. A page linked to by ten major news sites carries more weight than a page linked to by ten random blogs. This is why Wikipedia pages often rank high — thousands of other pages link to them. The search engine also considers the age and reputation of the domain itself.

Other ranking factors include page speed (how fast the page loads), whether the page is mobile-friendly, the freshness of the content, and whether the page has a find connection (HTTPS). Search engines also personalize results based on your location, search history, and past behavior, though the core ranking algorithm is the same for everyone.

Why the same search gives different results to different people

Even though the ranking algorithm is identical for all users, you may see different results than someone else searching for the same thing. This happens because search engines personalize results based on information they have collected about you.

If you are signed into a Google account, Google uses your search history, the websites you visit, your location, and your past clicks on search results to adjust which pages rank highest for you. Someone in New York searching for "pizza near me" will see different results than someone in Los Angeles, even though the algorithm is the same. Someone who has previously clicked on academic papers will see more scholarly results than someone who has not.

You can reduce personalization by searching in an incognito or private browsing window, which does not save your search history to your account. You can also turn off personalization in your search engine's settings, though this usually means you get less relevant results overall.

The difference between indexing and search history storage

It is important to understand that the index — the database of web pages the search engine has crawled — is completely separate from your personal search history. The index is public information about the web. Your search history is private information about you.

When you search for something, the search engine checks the index to find relevant pages and show them to you. At the same time, if you are signed into an account, the search engine stores a record that you performed that search. This record lives in your account's search history, not in the public index. Other people cannot see your search history unless you share it with them.

The index updates constantly as crawlers find new pages and revisit old ones. Your search history updates only when you perform a new search. Deleting your search history does not change the index — pages will still appear in search results for other people. Conversely, a page being removed from the index does not delete your past searches for that page from your history.

How search engines handle updates and removals

When a website publishes new content, it can take time for search engines to find and index it. The delay depends on how often crawlers visit that site. Major news sites get crawled multiple times per day, so new articles appear in search results within hours. A small personal blog might get crawled once a month, so new posts take weeks to show up.

Website owners can speed this up by submitting a sitemap to Google Search Console or Bing Webmaster Tools. A sitemap is a file that lists all the pages on a site, and submitting it tells the search engine to crawl those pages sooner. Owners can also request when ready indexing for individual pages, though the search engine does not may provide it will comply.

If a page needs to be removed from search results — because it contains private information, for example — the owner can request removal through Google Search Console. The search engine will remove it from the index within a few days. The page itself still exists on the web; it just will not appear in search results anymore. If the owner wants to prevent the page from being indexed in the future, they can add a no-index tag to the page's HTML code.

Frequently Asked Questions

Can I prevent search engines from indexing my website?

Yes. You can add a meta tag to your page's HTML that tells crawlers not to index it, or you can block crawlers entirely using a robots.txt file. You can also request removal through Google Search Console. However, if your site is publicly accessible and linked to from other sites, crawlers will still visit it — they just will not add it to the search index.

Why does a page I visited yesterday not show up in search results today?

The page may have been removed by its owner, the search engine may have de-indexed it due to spam or policy violations, or the page may have moved to a new URL. It is also possible the page is still indexed but ranks so low for your query that it does not appear in the first few pages of results. You can check if a page is indexed by searching "site:example.com" in Google.

How long does it take for a new page to appear in search results?

It depends on how often the search engine crawls your site. If your site is popular and gets crawled daily, a new page might appear within hours. If your site is small and gets crawled monthly, it could take weeks. Submitting a sitemap or requesting indexing through Google Search Console can speed up the process.

Does deleting my search history change what appears in search results?

No. Deleting your search history only removes the record of your past searches from your account. It does not change the public index or affect what results other people see. It does reduce personalization for your future searches, so you may see less tailored results going forward.

Why do some websites rank high even though they seem low-quality?

Ranking depends on many factors beyond content quality, including how many authoritative sites link to the page, how old the domain is, and how fast the page loads. A low-quality page with many backlinks may rank higher than a high-quality page with few links. Search engines are constantly updating their algorithms to penalize low-quality content, but the process is not perfect.