How do I send the SEO noindex header to prevent permalinks of past deploys from being indexed by search engines like Google?

Issues
1

For SEO reasons, I want to make sure the permalinks (the URLs that end in moovweb.io that are created for every deploy) do not get indexed by search engines like Google, Bing, etc. but not prevent the production URL of my site from being indexed.

2

You can solve this by using CDN-as-JavaScript to send the X-Robots-Tag: noindex header whenever the domain ends in moovweb.io. You can do this by putting something like this as one of the first statements in your router:

  .match(
    {
      headers: {
        host: /.*.moovweb.io/,
      }      
    },
    ({ setResponseHeader }) => {
      setResponseHeader('X-Robots-Tag', 'noindex')
    })

Here’s a full router example:

const { Router } = require('@xdn/core/router')
const { nextRoutes } = require('@xdn/next')
const { API, SSR, cacheResponse } = require('./cache')
const prerenderRequests = require('./xdn/prerenderRequests')

module.exports = new Router()
  .prerender(prerenderRequests)
  .match(
    {
      headers: {
        host: /.*.moovweb.io/,
      }      
    },
    ({ setResponseHeader }) => {
      setResponseHeader('X-Robots-Tag', 'noindex')
    })
  .match('/service-worker.js', ({ serviceWorker }) => {
    serviceWorker('.next/static/service-worker.js')
  })
  .match('/', cacheResponse(SSR))
  .match('/api', cacheResponse(API))
  .match('/s/:categorySlug*', cacheResponse(SSR))
  .match('/api/s/:categorySlug*', cacheResponse(API))
  .match('/p/:productId', cacheResponse(SSR))
  .match('/api/p/:productId', cacheResponse(API))
  .use(nextRoutes)
  .fallback(({ proxy }) => proxy('legacy'))
1 Like
3

Would the robots.txt file function similarly with no index in coding? https://yemlihatoker.com/robots.txt

4

They are similar but subtly different: robots.txt tells search engines which pages they can crawl and noindex tells them what they should index. I’d recommend Googling for “noindex vs robots.txt” for more details.

8

Could you clarify how you plan to handle routing for dynamic subdomains? For example, your provided regex moovweb. seems to cover the base, but what if there are specific subdomains or edge cases that need to be excluded from this rule? Should any Moovweb subdomains be indexed at all, or do all of them need the noindex tag?

9

To prevent search engines from indexing the moovweb.io permalinks while still allowing your production URL to be indexed, add a robots.txt file to your project. In this file, specify the disallow rule for the moovweb.io URLs like so:
User-agent: *
Disallow: /*.moovweb.io
This will block search engines from indexing these deploy URLs without affecting your production site’s indexing.