Technical SEO Guide: Crawling, Indexing, Core Web Vitals and Audits

Technical SEO is the infrastructure work that lets search engines crawl your site, index the right pages and serve them with a fast, secure experience. If Google cannot reach a page or indexes the wrong version of it, the quality of the content does not matter.

This guide follows the order I use in a technical audit. Deeper topics such as structured data, site migrations and e-commerce filters have their own guides, linked where they come up.

Short answer: Technical SEO makes sure your site is crawlable, correctly indexable, fast, complete on mobile and secure. Its core tools are robots.txt, canonical tags, noindex, XML sitemaps and Search Console; the good thresholds for Core Web Vitals are LCP ≤ 2.5 s, INP ≤ 200 ms and CLS ≤ 0.1.

Crawlability: how do you manage robots.txt and crawl budget?

robots.txt tells crawlers which paths not to fetch, but it does not control indexing. To keep a page out of the index, use noindex and leave the page crawlable; Google cannot see a noindex tag on a page blocked in robots.txt.

# Example robots.txt
User-agent: *
Disallow: /cart/
Disallow: /account/
Disallow: /*?sort=

Sitemap: https://www.example.com/sitemap.xml

According to Google's documentation, crawl budget mainly matters for very large sites (roughly a million pages or more) or sites with over 10,000 pages whose content changes daily. On smaller sites the real problem is important pages that never get discovered. On large sites, waste usually comes from:

• URLs multiplied by filter, sort and session parameters (handled in the e-commerce SEO guide).
• Redirect chains and internal links to 404s.
• Empty search and category pages returning soft 404s.
• Slow server responses; Google crawls a struggling server more slowly.

You can watch this in the Crawl stats report in Search Console. Why the often-confused llms.txt file is no substitute for robots.txt is covered in my llms.txt guide.

Indexing: how do canonical tags, noindex and XML sitemaps work together?

Not every crawled page is indexed. Google picks one canonical URL among duplicate versions; your job is to guide that choice with consistent signals.

Update lastmod only when content has genuinely changed; Google ignores priority and changefreq.

The Page indexing report in Search Console

The report groups non-indexed pages by reason. Pages you set to noindex or redirected are expected there. Focus on "Crawled – currently not indexed" (often thin content or overlap with similar pages), "Discovered – currently not indexed" (internal linking or server capacity) and cases where Google chose a different canonical than yours (conflicting signals). For a single URL, the URL Inspection tool shows the rendered HTML and the canonical Google selected. Add these reports to the monthly template in the SEO reporting guide.

How do site architecture and internal linking affect crawling?

Google finds new pages largely by following links. A page with no internal links looks unimportant even if it is in your sitemap.

• Important pages are a few clicks from the home page.
• Links are real <a> elements with an href.
• Anchor text describes the target page.
• Internal links point to the final URL, not through redirects.

The content plan decides which topic lives on which page; I cover topic cluster architecture in the content strategy guide.

Core Web Vitals: what are the targets for LCP, INP and CLS?

INP replaced FID in March 2024. Google assesses the thresholds at the 75th percentile of real-user data.

Decide based on field data (the Search Console Core Web Vitals report, PageSpeed Insights); Lighthouse is a lab tool and does not measure INP directly. Better vitals will not lift weak content, but they make the page usable. Metric-by-metric steps are in my Core Web Vitals article.

Mobile-first indexing, HTTPS and redirect chains

Google indexes sites primarily with its smartphone crawler; text, links or structured data missing on mobile do not exist for Google. The Mobile Usability report and the Mobile-Friendly Test were retired at the end of 2023, so I now check with Lighthouse, Chrome DevTools device emulation and the URL Inspection tool.

HTTPS is one of the page experience signals. Every HTTP address should 301 to HTTPS in one hop, with no mixed content (images or scripts loaded over HTTP).

Use 301 or 308 for permanent moves and 302 or 307 for temporary ones. Chains such as A → B → C slow crawling; every old URL should reach its destination in one hop. For redirect maps during domain or URL changes, see the site migration SEO guide.

How does Google see JavaScript-generated content?

Google can process JavaScript, but it crawls the HTML first and renders later. Content that only exists after rendering may be discovered late, and you should not assume every search engine or AI crawler handles JavaScript as well as Google.

• Use server-side rendering or static generation where possible so the main content is in the initial HTML.
• Do not change title, canonical or robots tags with JavaScript after load.
• Content behind tabs or "load more" buttons should be in the DOM on load; Googlebot does not click.
• Confirm in the URL Inspection tool that the rendered HTML contains the main text and links.

How do you set up hreflang for international sites?

hreflang tells Google about language or regional versions of the same content. It is not a ranking signal; it matches the right version with the right user.

<link rel="alternate" hreflang="tr" href="https://www.example.com/rehber/ornek/" />
<link rel="alternate" hreflang="en" href="https://www.example.com/en/guides/ornek/" />
<link rel="alternate" hreflang="x-default" href="https://www.example.com/en/guides/ornek/" />

• Each version lists itself and the others; links must be reciprocal.
• Use ISO 639-1 language codes with optional ISO 3166-1 regions (tr, en, en-GB); "uk" means Ukrainian.
• List canonical URLs only.
• Do not force IP-based redirects; Googlebot mostly crawls from the US.

How do you run a technical SEO audit and prioritise the findings?

I start by comparing Search Console data with a crawl from a tool such as Screaming Frog or Sitebulb: URLs in the sitemap but not indexed, or with no internal links, give the quickest findings. Then I check vitals, mobile rendering and rendered HTML on sample pages from each template.

Add affected URL counts, an owner and effort to each finding and the list becomes a plan. I check structured data with the steps in the schema structured data guide. To work through an audit together, see my SEO consulting scope or book a call.