Creating `robots.txt` in the App Router

A robots file tells crawlers which paths they may fetch. Creating robots.txt in the App Router means adding one file at the root of the app directory, either as plain text or as a module that returns an object, and Next.js serves it at /robots.txt. The public folder is not involved.

The output below was verified with Next.js 16.3 in an App Router project, requesting the file from a production build. The generated file is prerendered, so it costs nothing per request.

The static version

If the rules never change, a plain text file is the whole job. Put it at the top level of the app directory next to your root layout.

text

User-Agent: *
Allow: /
Disallow: /private/
 
Sitemap: https://acme.com/sitemap.xml

Save that as app/robots.txt and the same content is served at /robots.txt. Nothing is generated and nothing is parsed, which makes this the right choice for a simple site with one environment.

The sitemap line matters more than the rules for most projects. It is how crawlers find the URL list described in generating a sitemap.

One detail catches people moving from an older project. A file in the public folder is also served at that path, so a project holding both has two files claiming the same URL. The official guidance is to use the app convention for static metadata files, so keep exactly one and keep it in the app directory.

The generated version

Swap the file for robots.ts when the output depends on your data or environment. The default export returns an object, and the type keeps the field names honest.

typescript

// app/robots.ts
import type { MetadataRoute } from 'next'
 
export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', allow: '/', disallow: '/dash/' },
    sitemap: 'https://acme.com/sitemap.xml',
  }
}

Requesting /robots.txt returns the text form of that object with a text/plain content type, ending in the sitemap line. The build output lists the route as a prerendered static entry, so the function runs once during the build rather than per request.

This function runs on the server at build time. It can read environment variables or query a database, and none of that reaches the browser.

Use the generated form when the answer is computed and the static form when it is fixed. A file that hardcodes the same three rules gains nothing from being a function.

Rules for specific crawlers

Passing an array to the rules field produces one block per entry. This is how you treat a search crawler differently from an aggressive scraper.

typescript

// app/robots.ts
export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: 'Googlebot', allow: '/', disallow: '/private/' },
      { userAgent: ['Applebot', 'Bingbot'], disallow: '/', crawlDelay: 10 },
    ],
    host: 'https://acme.com',
  }
}

The rendered file contains a Googlebot block, then a block with two user agent lines sharing one rule, with a crawl delay line, and the host line at the end. An array of agent names produces several User-Agent lines in the same block rather than duplicated blocks.

Crawl delay is a hint that not every crawler honors, and Google ignores it. Treat rate limiting at the server or CDN as the real control.

Blocking a bot outright is a blunt instrument. It stops that agent fetching anything, which is fine for a scraper and costly if you later want the same company's search product to index the site.

Non-standard directives

Some engines read directives that are not in the standard, such as a request rate for one bot. Next.js 16.3 added a field that passes them through untouched.

typescript

// app/robots.ts
const rules = [
  { userAgent: '*', allow: '/' },
  { userAgent: 'SeznamBot', allow: '/', other: { 'Request-Rate': '10/1m' } },
]

Those entries render as a normal block for everyone plus a SeznamBot block whose extra line appears verbatim. Keys keep their casing and array values emit one line each, scoped to that agent.

Reach for this only when a specific engine documents a directive you need. Standard rules are understood everywhere, and an unknown line is simply skipped by every crawler that does not recognize it.

Blocking preview environments

The most valuable use of the generated form is keeping non-production deployments out of search results. Branch on an environment variable that only production sets.

typescript

// app/robots.ts
import type { MetadataRoute } from 'next'
 
const isProduction = process.env.SITE_ENV === 'production'
 
export default function robots(): MetadataRoute.Robots {
  if (!isProduction) {
    return { rules: { userAgent: '*', disallow: '/' } }
  }
  return { rules: { userAgent: '*', allow: '/' } }
}

A build without that variable serves a file that disallows everything, and a production build serves the real rules. Both outputs were confirmed by building twice with different environment values.

Read the variable at module scope, as shown, so the value is fixed for the build rather than looked up on each call. Any variable your host sets for production works, and the name is yours to choose.

Pair this with a noindex tag on the same environments. Crawling and indexing are separate decisions, and a link from elsewhere can put a blocked URL in an index without it ever being fetched.

What a robots file cannot do

This is where the file gets misused, because two different jobs look similar from the outside.

A disallowed path cannot have its noindex tag read, since the crawler never fetches the page. If a URL is already indexed and you want it gone, leave it crawlable and serve the tag, which the SEO checklist covers alongside the other indexing controls.

Never treat a robots file as security. It is a public document that lists the paths you would rather people not visit.

Common mistakes

Each of these produces a file that looks right in the repository and behaves differently in production.

• Keeping both a static robots.txt and a robots.ts, where the static file wins and the generated one is silently ignored.
• Putting the file in the public folder in a project that also has the app convention, which makes the winner harder to reason about.
• Disallowing a path and then wondering why its noindex tag has no effect.
• Blocking asset paths that pages depend on, which stops crawlers rendering the page properly.
• Hardcoding a production sitemap URL in a file that also serves preview deployments.