robots.txt Generator

Build robots.txt visually with multiple user-agent groups, Allow / Disallow rules, sitemap lines, and advanced options.

robots.txt Generator

The robots.txt generator helps you build crawler rules with a visual editor, including multiple user-agent groups, Allow / Disallow paths, sitemap lines, Host, and Crawl-delay fields. It works well for launches, SEO updates, and migration checklists.

How do you use it?

  1. Choose a preset or add your own rule groups.
  2. Enter user agents and attach Allow or Disallow paths to each group.
  3. Add sitemap lines and optional Host or Crawl-delay values when needed.
  4. Preview the final file, then copy or download the generated robots.txt.

Features

  • Supports multiple crawler groups and advanced directives
  • Includes quick presets for common robots.txt patterns
  • Builds the file locally in real time without a backend

Why use this tool?

  • Avoid formatting mistakes in hand-written robots.txt files
  • Manage different crawler policies faster
  • Review launch, SEO, and migration rules in one place

Related Dev Tools

User-Agent Parser

Parse user agent strings locally in your browser to identify browser, OS, device, engine, and CPU details.

URL Parser and Builder

Parse URLs into components and build URLs from individual components. Bidirectional URL analysis and construction tool.

JSON Formatter

Validate, format and highlight JSON data.

JSON Schema 生成器

本地 json-schema-generator 思路，在浏览器本地通过现有的 JSON 数据快速生成对应的 JSON Schema，支持多种 Draft 规范及高级配置。

JSON Schema 验证器

本地 json-schema-validator 思路，在浏览器本地通过 JSON Schema 严格验证 JSON 数据，支持多种 Draft 规范及全面格式校验。

Prettier Code Formatter

Format JavaScript, TypeScript, JSON, HTML, CSS, Markdown, YAML, and more locally in your browser with Prettier.

OpenAPI to TypeScript

Generate TypeScript types from OpenAPI 3.0 / 3.1 YAML or JSON documents locally in your browser with common codegen options.

Code Screenshot Generator

Generate syntax-highlighted code screenshots locally in your browser and export them as SVG, PNG, JPG, or HTML.

SQL Formatter / Linter

Format SQL locally in your browser and check for common issues like SELECT *, unsafe updates or deletes, and missing semicolons.

cURL Converter

Convert cURL commands locally in your browser into JavaScript, Node.js, Python, Go, PHP, HTTP, JSON, and more.

Docker Run to Compose

Convert docker run commands locally in your browser into docker-compose config and flag unsupported options.

JSONPath Tester

Test JSONPath queries locally in your browser and inspect matched values and paths.

JMESPath Tester

Test JMESPath queries locally in your browser and inspect the returned results.

JSON to YAML

Convert JSON to YAML locally in your browser with copy and download support.

JSON to TOML

Convert JSON to TOML locally in your browser with nested tables and array-of-table output.

TOML to JSON

Convert TOML to JSON locally in your browser with support for key-value pairs, tables, and arrays of tables.

YAML to TOML

Convert YAML to TOML locally in your browser with nested tables and array-of-table output.

TOML to YAML

Convert TOML to YAML locally in your browser with support for key-value pairs, tables, and arrays of tables.

JSON to XML

Convert JSON to XML locally in your browser with support for common attribute and text-node mappings.

JSON to CSV

Convert JSON object arrays to CSV locally in your browser with delimiter, header, and formula-escaping options.

CSV to JSON

Convert CSV to JSON locally in your browser with header parsing, type detection, and empty-line handling.

XML to JSON

Convert XML to JSON locally in your browser with control over attributes, text nodes, and array output.

YAML to JSON

Convert YAML to JSON locally in your browser with copy and download support.

.gitignore Generator

Quickly generate .gitignore templates for different languages and frameworks.

Sitemap XML Generator

Generate sitemap.xml or sitemapindex files with optional image, video, and news extensions.

chmod Calculator

Convert Unix file permissions between numeric and symbolic notation locally in your browser, with a linked checkbox matrix and ready-to-copy chmod command.

UUID Generator

Generate single or batch UUID v4 values locally in your browser with multiple output formats.

UUID Decoder

Decode UUID details locally in your browser, including version, variant, Base64, integer form, and v1 timestamp data.

NanoID Generator

Generate NanoID values locally in your browser by length, charset, and batch size.

ULID Generator

Generate lexicographically sortable ULIDs locally in your browser and inspect the embedded timestamp.