Index/no-index refers to the directives that tell search engines whether a page may appear in their index, set through a robots meta tag or an X-Robots-Tag HTTP header. A noindex directive is not the same as a robots.txt block: blocking crawls stops a bot from reading a page, while noindex lets it read the page and then keeps it out of results.
For marketers, deciding which pages carry index versus noindex controls exactly which URLs compete for rankings, clicks, and AI citations. Get it wrong and you can bury a key page from Google, or flood the index with thin pages that dilute the ones you want found.
Index/no-index is a pair of indexing instructions that operate at the individual page level, telling a crawler like Googlebot whether to store a URL in the search index or drop it. The default state is index, so a page is eligible for search results unless you explicitly apply a noindex directive. You set noindex two ways: a robots meta tag in the page's HTML head, or an X-Robots-Tag in the HTTP response header, which also works for non-HTML files like PDFs.
The directive only takes effect if the crawler can reach the tag. If you block a URL in robots.txt, the bot never fetches the page, never sees the noindex, and the URL can still surface in results as a bare link. Indexing directives also stack with canonical tags, sitemaps, and internal links, all of which shape what search engines ultimately store.
Indexing directives sit alongside crawl directives and canonicalization in the technical SEO toolkit, but they answer a narrower question: should this specific page exist in the index at all. Because AI answer engines pull from the same crawled and indexed corpus, a noindex on a page also removes it from the pool that tools like AirOps track for citations.
Resources: how crawlability and indexation decide whether AI answer engines can cite a page
Indexing directives run through a set sequence, from the moment a crawler requests a URL to the moment it decides whether the page can appear in results.
Crawl request: A search bot fetches the URL, reading both the HTTP headers and the page's HTML head before doing anything else.
Directive check: The bot looks for an X-Robots-Tag in the header and a robots meta tag in the HTML, then reads the index or noindex value.
Access gate: If robots.txt blocked the URL, the bot never reaches this step, so a noindex it cannot see has no effect.
Index decision: With a readable noindex, the engine drops the URL from its index; with index or no directive at all, the page stays eligible.
Recrawl and refresh: On later visits the bot rechecks the directive, so removing a noindex lets a page return to the index over subsequent crawls.
The index status tells you whether a page can appear in search and AI answers. It does not tell you whether that page will rank or get cited, which depends on relevance, structure, and authority.
Resources: why indexation and technical SEO stay essential when you extend SEO into AI search
Your index decisions set the boundary of your visibility. Every URL you allow into the index is a page that can win rankings, clicks, and AI citations, and every noindex is a deliberate choice to keep a page out of that competition. Treating those choices as an afterthought is how teams lose traffic they never knew they had.
Wasted crawl budget: When thin, duplicate, or filter-generated pages stay indexable, search engines spend crawl and ranking resources on URLs that will never convert, starving your priority pages of attention.
Accidental deindexing: A stray noindex left on a template after a site migration can silently pull whole sections out of Google, and the drop often goes unnoticed until organic traffic craters weeks later.
AI citation eligibility: Answer engines can only cite pages they have crawled and indexed, so a noindex on a valuable resource removes it from every AI answer before the content quality even matters.
SEO managers use index/no-index to keep thin tag, filter, and pagination pages out of Google so crawl budget flows to revenue pages.
Content strategists use index/no-index to noindex outdated or duplicate articles while they consolidate topic clusters into stronger canonical pages.
Demand gen leads use index/no-index to hide gated thank-you and landing pages from search while keeping paid campaign traffic clean.
Crawling is a bot fetching a page, while indexing is the engine choosing whether to store that page for search results, and a URL must stay crawlable for any noindex directive placed on it to ever register with the engine.
The robots meta tag lives in the HTML head and covers standard web pages, while the X-Robots-Tag sits in the HTTP response header and can apply a noindex to PDFs, images, and other non-HTML files a meta tag cannot reach.
Index bloat is the slow buildup of thin, duplicate, or auto-generated URLs in the search index, and selective noindexing is the main lever for trimming it back so your strongest pages carry more relative authority in your category.
Direct crawl budget toward the pages that actually drive pipeline.
Remove duplicate and thin URLs from Google before they dilute rankings.
Confirm index status fast in Google Search Console with the URL Inspection tool.
Protect private, staging, and thank-you pages from showing up in search.
Keep your indexed footprint clean so AI answer engines pull from your best content.
Recover lost visibility by finding and removing accidental noindex tags.
Default to index and noindex by exception, so you only pull pages when there is a clear reason to remove them.
Never combine a robots.txt block with a noindex on the same URL, because the bot cannot read a directive on a page it is not allowed to fetch.
Audit index status after every migration or redesign, since template-level noindex tags are the most common cause of sudden traffic loss.
Use the X-Robots-Tag for non-HTML files like PDFs and images, where a meta tag has nowhere to live.
Verify changes in Google Search Console, using URL Inspection to confirm a page is indexed or excluded as you intended.
Keep a documented list of noindexed URLs, so future team members understand why each page sits outside the index.
Avoid using noindex as a quick fix for duplicate content when a canonical tag is the right tool. Noindex removes a page from results entirely, while canonicalization consolidates signals to a preferred version and keeps the content working for you.
AirOps: Tracks which of your indexed pages actually earn citations across ChatGPT, Perplexity, and Google AI Overviews, so you can see whether an indexable page is pulling its weight in AI search.
Google Search Console: Reports index coverage and lets you inspect any URL to confirm whether Google has indexed it or honored a noindex directive.
Screaming Frog: Crawls your site to flag every page carrying a noindex tag or blocked in robots.txt, surfacing indexing mistakes in bulk.
Pull your indexed pages. Open Google Search Console and export the Pages report to see every URL Google currently indexes. This takes an afternoon, needs no budget approval, and gives you the baseline for everything that follows.
Flag the pages that should not be there. Review the list for thin, duplicate, staging, or internal-search URLs that add no value to searchers or AI answers.
Apply noindex where it belongs. Add a robots meta tag to HTML pages, or an X-Robots-Tag for PDFs and other files, on each URL you flagged.
Confirm the crawler can see it. Check that none of those URLs are blocked in robots.txt, since a bot has to fetch the page to read the noindex.
Recheck and monitor. Use URL Inspection to confirm each change took effect, then watch your indexed count over the following weeks to catch anything that slips back in or gets deindexed by accident.
Index/no-index directives tell search engines whether a specific page is allowed to appear in their results at all.
You set them with a robots meta tag in the HTML or an X-Robots-Tag in the HTTP header.
A noindex only works if the page stays crawlable, so never block the same URL in robots.txt.
A stray noindex left on a template can quietly deindex whole sections of a site after a migration.
The leverage is focus: trimming low-value URLs concentrates crawl attention and authority on the pages you most want found and cited.
Index/no-index and a robots.txt disallow solve different problems, and confusing them is a common technical SEO mistake. A noindex directive lets a crawler fetch the page, read it, and then keep it out of the search index, so the page is understood but excluded. A robots.txt disallow stops the crawler from fetching the page at all, which means it never sees any noindex tag you placed there. That distinction has a practical trap: if you disallow a URL in robots.txt and also add noindex, the bot cannot read the noindex, and the URL can still appear in results as a bare link with no description. When you want a page gone from search, keep it crawlable and use noindex. Save the robots.txt disallow for controlling crawl access to sections you do not want bots spending time on, like faceted navigation or infinite calendar pages.
Audit your index status on a set cadence and after any major site change, instead of only when traffic drops. A quarterly review works for most sites: export the indexed pages from Google Search Console, scan for URLs that should not be there, and confirm your intended noindex tags are still in place. On top of that schedule, run a check immediately after a migration, redesign, CMS change, or template update, because those are the moments when a noindex gets applied site-wide by accident. High-velocity sites that publish or generate many pages, like large ecommerce catalogs, benefit from a monthly look, since index bloat builds faster there. The goal is to catch two failure modes early: valuable pages that fell out of the index, and low-value pages that crept back in. A recurring check turns index management into a maintained system instead of a fire drill.
A page usually lingers in results after you add noindex because Google has not recrawled it yet. Indexing directives only take effect the next time a bot fetches the page and reads the tag, and recrawl timing varies widely by site authority, crawl frequency, and how deep the page sits in your architecture. A page crawled daily may drop within days, while a rarely visited URL can persist for weeks. Two other causes are common. First, the page may be blocked in robots.txt, so the bot never reaches the noindex it needs to obey. Second, the tag may be placed incorrectly, such as in the page body instead of the head, or served only after JavaScript runs. To speed removal on a priority URL, confirm the page is crawlable, verify the tag renders in the raw HTML, then request indexing through URL Inspection so Google recrawls sooner.
You can strongly influence which pages Google indexes, but you do not get absolute control. A noindex directive is the most reliable signal you have: Google honors a correctly implemented, crawlable noindex and will drop the page from its index on the next crawl. What you cannot force is the reverse. Adding index or removing a noindex makes a page eligible, but it does not guarantee inclusion, because Google still decides whether the page is worth indexing based on quality, duplication, and demand. Low-value or near-duplicate pages can be passed over even when nothing blocks them. So treat noindex as a dependable off switch and treat indexing as an invitation the engine can decline. Focus your energy on making the pages you want indexed genuinely worth storing, with unique content and clear structure, then use directives to remove the pages that dilute that set.
A healthy index/no-index balance means nearly every indexed URL is a page you would actively want a searcher or AI engine to find. There is no universal number, but a useful benchmark is this: the count of indexed pages in Google Search Console should closely track the count of valuable, unique pages you know you have published. Large gaps signal a problem in either direction. Far more indexed pages than real content usually points to index bloat from filters, parameters, or duplicates that need noindex. Far fewer indexed pages than expected suggests accidental noindexing or crawl blocks removing pages you want found. Instead of chasing a ratio, watch the trend: a stable or slowly growing indexed count that matches your publishing pace is healthy, while sudden swings deserve investigation. Check the Pages report in Search Console monthly, and reconcile any excluded URLs against your intent.