Oliver Zeman
Your Pages Aren't Ranking Because Google Can't Find Them
Fix crawlability and indexing issues in technical SEO before they silently kill your rankings.
You published a page. You optimized the title tag, added internal links, even built a few backlinks. And it still doesn't rank.
Before you rewrite the content or chase more links, check something more fundamental: can Google actually find and store that page? Because if crawlability or indexing is broken, nothing else you do will matter.
This is the part of technical SEO that most site owners skip — and it's exactly why pages stay invisible. For the full picture of everything technical SEO covers beyond this, see our technical SEO overview.
What Crawlability and Indexing Actually Mean
These two terms get lumped together, but they describe two separate steps in how Google works.
Crawling is discovery. Googlebot follows links across the web, starting from known URLs and moving outward. When it lands on your page, it downloads the HTML and queues it for analysis.
Indexing is storage. After crawling, Google processes what it found — the text, links, structured data, signals of quality — and decides whether to add the page to its database. Only indexed pages can appear in search results.
The order matters: crawling must happen before indexing. But crawling doesn't guarantee indexing. Google crawls billions of pages and only indexes a fraction of them.
So you can have a page that Google has visited multiple times and still chosen not to index. That's a different problem than a page Google has never seen at all — and each requires a different fix.
Why Google Skips Pages It Has Already Crawled
This is the most frustrating scenario in technical SEO: Google Search Console shows your URL as "Crawled — currently not indexed." The bot visited. It just didn't file the page.
There are a few reasons this happens.
Thin or Duplicate Content
Google's quality filters are aggressive. If a page has minimal original text, closely mirrors another page on your site, or looks like a near-duplicate of something already in the index, Google will often crawl it and pass.
Pagination pages, filtered product URLs (like /shoes?color=black&size=10), and auto-generated tag archive pages are common offenders. They're technically accessible but add no unique value.
Soft 404s
A soft 404 is a page that returns a 200 HTTP status code but contains almost no content — an empty category page, a search result with zero results, or a deleted product page that was never properly redirected. Google sees it as a waste of crawl time and ignores it.
Slow Rendering or JavaScript Dependency
If your page's content only appears after JavaScript executes, Google may crawl the raw HTML, find it mostly empty, and move on. Google does render JavaScript eventually — but it's a two-wave process, and pages that depend entirely on client-side rendering often get deprioritized.
The fix: make sure your core content is in the initial HTML response, not injected by JavaScript after load.
Low Internal Link Equity
Pages that aren't linked from anywhere prominent on your site get treated as low-priority. If the only path to a page is through a deeply nested URL or a footer link with no anchor text context, Google may crawl it rarely and index it reluctantly.
The Crawl Budget Problem (And When It Actually Matters)
Crawl budget is the number of pages Googlebot will crawl on your site within a given timeframe. For most small and medium sites — under a few thousand pages — this isn't a real constraint. Google will find everything.
But it becomes a real problem when your site generates thousands of low-value URLs that eat up that budget before Google gets to your important pages.
Common crawl budget killers:
- Faceted navigation (e.g., e-commerce filter combinations creating thousands of unique URLs)
- Session IDs appended to URLs
- Infinite scroll or calendar-based archives with no end
- Duplicate pages accessible via
wwwand non-www,httpandhttps - Redirect chains longer than two hops
If your site has 500 real pages but generates 50,000 crawlable URLs through parameter combinations, Googlebot may never reliably reach your most important content.
The solution isn't to block everything — it's to be deliberate. Use robots.txt to disallow parameter-based URLs that add no indexable value. Use canonical tags to point duplicates to the preferred version. Keep your XML sitemap clean and limited to pages you actually want indexed.
robots.txt: The File That Can Accidentally Break Everything
Your robots.txt file lives at yourdomain.com/robots.txt. It tells crawlers what they're allowed to access. One wrong line can block your entire site from being crawled.
The most catastrophic version looks like this:
User-agent: *
Disallow: /
That single directive tells every bot to crawl nothing. It sounds extreme, but it happens — often after a developer accidentally pushes a staging configuration to production.
A well-configured robots.txt looks more like this:
User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/
Disallow: /cart/
Sitemap: https://www.yourdomain.com/sitemap.xml
You're blocking admin areas and transactional pages that have no business appearing in search results, while leaving everything else open. And you're pointing crawlers directly to your sitemap.
One important nuance: robots.txt controls crawling, not indexing. If a page is linked from another site, Google can still discover and index it even if you've blocked it in robots.txt. To prevent indexing, you need a noindex meta tag on the page itself.
XML Sitemaps: Your Direct Line to Googlebot
An XML sitemap is a file that lists every URL on your site you want Google to know about. Think of it as a map you hand directly to the crawler rather than making it find everything through links.
Submitting your sitemap through Google Search Console doesn't guarantee indexing — but it does speed up discovery, especially for new pages or sites with weak internal linking.
A few rules for a clean sitemap:
- Only include URLs you want indexed. Don't list pages with
noindextags. - Keep it under 50,000 URLs per file (use a sitemap index for large sites).
- Update it automatically when you publish new content — most CMS platforms handle this.
- Remove 404 pages and redirected URLs promptly.
If you're running a large site, consider splitting your sitemap by content type: one for blog posts, one for product pages, one for category pages. This makes it easier to diagnose which sections have indexing problems.
Internal Linking: How Google Decides What Matters
Googlebot doesn't just find pages through sitemaps. It follows links. The structure of your internal links is one of the strongest signals you can send about which pages deserve to be crawled frequently and indexed prominently.
A page with zero internal links pointing to it — an "orphan page" — is effectively invisible to crawlers navigating your site. Even if it's in your sitemap, the absence of internal links signals that you don't consider it important.
Strong internal linking means:
- Every page is reachable within three clicks from your homepage
- Your most important pages receive links from multiple places (navigation, body content, related posts)
- Anchor text is descriptive and relevant, not generic ("click here" helps no one)
- You link contextually within body copy, not just in sidebars and footers
This is also where site architecture pays off. A flat hierarchy — where category pages link directly to individual posts or products — passes more link equity and makes crawling more efficient than a deeply nested structure.
How to Check Your Indexing Status Right Now
You don't need a paid tool to diagnose basic indexing problems. Start with these free methods.
Google Search Console (GSC) is your most important tool. Go to the Pages report under Indexing. You'll see a breakdown of indexed pages, excluded pages, and the reason for exclusion. The most useful exclusion categories to investigate:
| GSC Status | What It Means | What to Do |
|---|---|---|
| Crawled — currently not indexed | Google visited but chose not to index | Improve content quality, add internal links |
| Discovered — currently not indexed | Google knows the URL exists but hasn't crawled it | Check crawl budget, improve internal linking |
Excluded by noindex tag |
You (or your CMS) told Google not to index it | Verify this is intentional |
| Duplicate, Google chose different canonical | Google picked a different version of the page | Audit and fix canonical tags |
| Blocked by robots.txt | Crawler was told not to access the page | Review robots.txt |
| Soft 404 | Page exists but has almost no content | Add content or redirect to a relevant page |
The site: search operator gives you a rough count: type site:yourdomain.com into Google. The number of results is an approximation of how many pages are indexed. If it's dramatically lower than your actual page count, you have an indexing problem.
URL Inspection in GSC lets you check any individual URL. It tells you the last crawl date, whether the page is indexed, and what Google actually rendered when it visited.
Canonical Tags: Telling Google Which Version to Index
When the same content is accessible at multiple URLs, Google has to pick one to index. If you don't tell it which to prefer, it will guess — and it might guess wrong.
Canonical tags solve this. Add a <link rel="canonical" href="https://yourdomain.com/preferred-url/"> tag to the <head> of your page, pointing to the version you want indexed.
Common situations where canonicals matter:
httpvshttpsversions of the same pagewwwvs non-www- URLs with and without trailing slashes
- Product pages accessible through multiple category paths
- Syndicated content published on multiple sites
One mistake to avoid: setting a canonical to a URL that itself has a noindex tag. That creates a contradiction Google has to resolve, and it usually doesn't go the way you want.
Crawlability and AI Search Engines
Here's something most technical SEO guides don't mention yet: crawlability matters for AI-powered search engines too.
Tools like Grok, ChatGPT Search, and Perplexity use crawlers to discover and read web content. If your pages are blocked by robots.txt, inaccessible due to JavaScript rendering issues, or simply not indexed, AI engines can't cite you either.
This is increasingly important as more searches happen through AI interfaces rather than traditional SERPs. A page that Google can't crawl is also a page that Grok can't read when forming an answer about your industry, your product, or your brand.
If you want to know whether AI engines are actually citing your brand in their responses, Lookelo's brand monitoring tracks exactly that — across traditional search and AI-generated answers. It's a fast way to see whether your technical SEO work is translating into real visibility.
A Practical Crawlability Audit Checklist
Run through this list every time you launch a new site, migrate a domain, or notice a drop in organic traffic.
Crawl access
- robots.txt is not blocking important pages or resources
- robots.txt includes your sitemap URL
- No accidental
Disallow: /in production
Sitemap health
- XML sitemap is submitted in Google Search Console
- Sitemap only includes indexable URLs (no
noindexpages, no redirects, no 404s) - Sitemap updates automatically when new content is published
Indexing controls
-
noindextags are only on pages you intentionally want excluded - Canonical tags point to the correct preferred URL
- No canonical-to-noindex conflicts
Site architecture
- Every important page is reachable within 3 clicks from the homepage
- No orphan pages (pages with zero internal links)
- Internal anchor text is descriptive
Technical health
- All pages return correct HTTP status codes (200 for live pages, 301 for permanent redirects, 404 for truly missing pages)
- No redirect chains longer than two hops
- Core content is in the HTML, not dependent on JavaScript to render
- Site loads on HTTPS with a valid SSL certificate
Monitoring
- GSC Pages report reviewed monthly
- Crawl errors investigated and resolved within 30 days
- New content verified as indexed within 1–2 weeks of publishing
The Connection Between Technical SEO and Organic Visibility
Getting crawlability and indexing right doesn't directly boost your rankings — it removes the ceiling on them. A page that isn't indexed can't rank, no matter how good the content is. A page that's crawled inefficiently gets updated in the index slowly, so fresh content takes longer to surface.
Think of technical SEO as the foundation. Content and backlinks build on top of it. If the foundation has cracks, you're working against yourself. For the mechanics of how Google actually ranks pages once they're indexed, see how Google ranks websites.
The good news: most crawlability and indexing problems are fixable. They're not algorithm mysteries or competitive battles — they're configuration issues with clear solutions. A single afternoon in Google Search Console, combined with the checklist above, is often enough to find and fix the biggest blockers on a typical site.
And once you've cleaned up the technical layer, you can track whether your fixes are actually translating into visibility — in both traditional search and AI-generated answers. Run a free brand visibility check with Lookelo to see where you stand right now.
FAQ
What's the difference between crawlability and indexability?
Crawlability is about access — can Google's bot reach and download your page? Indexability is about inclusion — after crawling, does Google decide to store the page in its database? A page can be crawlable but not indexable (if it has a noindex tag or thin content), and a page can be indexable in theory but never crawled (if it has no internal links and isn't in your sitemap).
Why does Google say "Crawled — currently not indexed"?
This means Googlebot visited your page but chose not to add it to the index. The most common causes are thin or duplicate content, soft 404 behavior, poor internal linking, or the page being too similar to other pages on your site. Improving content quality and adding strong internal links from relevant pages usually resolves it.
Does robots.txt prevent indexing?
No. robots.txt controls crawl access, not indexing. If your page is linked from an external site, Google can discover and index it even if your robots.txt blocks crawling. To prevent indexing, use a noindex meta robots tag on the page itself.
What is crawl budget and do I need to worry about it?
Crawl budget is the number of pages Googlebot will crawl on your site in a given period. For most sites under a few thousand pages, it's not a practical concern — Google will find everything. It becomes relevant for large e-commerce or news sites that generate thousands of low-value URLs through filters, parameters, or archives. If you're in that category, use robots.txt and canonical tags to guide crawlers toward your most valuable pages.
How long does it take for Google to index a new page?
It varies. A new page on a well-established site with strong internal links can be indexed within hours. A page on a newer site with weak authority might take days or weeks. Submitting the URL through the URL Inspection tool in Google Search Console can speed up the process, though it doesn't guarantee immediate indexing.
Do AI search engines like Grok or ChatGPT use crawlability?
Yes. AI-powered search engines use their own crawlers to discover and read web content. If your pages are blocked, inaccessible due to JavaScript issues, or not indexed, AI engines can't include your content in their answers. This makes technical SEO relevant not just for Google rankings but for AI citation visibility as well.
How do I check if a specific page is indexed?
Use the URL Inspection tool in Google Search Console for the most accurate answer. You can also type site:yourdomain.com/your-page-url into Google — if it appears in results, it's indexed. If it doesn't appear, that's a signal to investigate further in GSC.