robots.txt Directive Counter
Counts User-agent lines, Allow/Disallow/Crawl-delay lines and Sitemap URLs in pasted robots.txt text. Does not generate rules or test path access.
About this robots.txt directive counter
Counts User-agent lines, Allow/Disallow/Crawl-delay lines and Sitemap URLs in pasted robots.txt text. Does not generate rules or test path access Validates and converts interface schemas client-side without transmitting private endpoint specifications.
API contract & interface schema logic
Scans pasted lines for selected directive labels and returns counts, Sitemap values and a warning if User-agent is missing.
Worked schema conversion scenario
Scenario: Summarize directive counts and a Sitemap value in pasted robots.txt text.
Sample input:
Processing: Count matching directive labels and collect Sitemap values; it does not test a URL path or apply RFC 9309 matching.
Illustrative output:
Schema version & syntax constraints
userAgentGroups counts User-agent lines rather than logical groups. The page does not generate robots.txt, evaluate Allow/Disallow precedence, validate Sitemap URLs, or test a crawler path.
API specifications and schema models in robots.txt Directive Counter reflect OpenAPI and JSON Schema standards. Verify generated schemas against your runtime validator.
Client-side schema processing boundary
Schema analysis and protocol mapping in robots.txt Directive Counter execute locally in the browser tab without uploading endpoint definitions.
OpenAPI Specification & JSON Schema standards
OpenAPI Specification standards and JSON Schema draft definitions:
Operating System & Terminal CLI Interoperability Protocols
For systems engineers, DevOps practitioners, and backend developers integrating this workflow into automated Continuous Integration (CI/CD) pipelines, crontabs, or shell environments, the client-side algorithms executed by robots.txt Directive Counter map directly to native operating-system toolchains:
- Linux & macOS POSIX Terminal Environments: Mirror REST and OpenAPI workflows using `curl -sSL -X POST -H 'Content-Type: application/json' -d @payload.json https://api.endpoint/v1` or `httpie POST https://api.endpoint/v1 < payload.json`. For OpenAPI contract validation, leverage `npx @stoplight/spectral-cli lint openapi.yaml` or `oasdiff -base original.yaml -revision update.yaml` to detect breaking schema mutations in git pre-commit hooks.
- Windows 11 / 10 PowerShell 7+ & Command Environments: In Windows PowerShell 7+, dispatch API requests using `Invoke-RestMethod -Uri 'https://api.endpoint/v1' -Method Post -Body $payload -ContentType 'application/json'`. Enforce modern TLS 1.3 or TLS 1.2 handshakes via `[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 -bor [Net.SecurityProtocolType]::Tls13` to avoid handshake negotiation timeouts on legacy hosts.
- Browser Runtime Sandboxing & Resource Allocation: OpenAPI-to-MCP generation and schema graph traversals execute synchronously within a client-side WebAssembly / TypeScript abstract syntax tree compiler. Cyclic references are detected through a path-indexed recursion stack, preventing stack overflow while preserving complete shared object definitions without remote schema resolution network requests.
Diagnostic Failure Modes & Edge-Case Recovery
When unexpected inputs or boundary conditions occur during data processing, consult this structured diagnostic recovery matrix:
| Fault Condition | Observed Symptom | Engineering Remediation Procedure |
|---|---|---|
| Missing required OpenAPI path operation object | Schema validation error: paths./endpoint lacks get/post definition | Provide at least one valid HTTP method (get, post, put, delete) and an associated responses object for each path item. |
| Cyclic $ref pointer reference in schema | Recursion guard triggered badge displayed | CZOA automatically terminates recursive branches while expanding legitimate shared models. Verify that recursion terminates with an optional property. |
| Invalid JSON Schema draft version declaration | Unknown schema dialect warning | Specify $schema as http://json-schema.org/draft-07/schema# or https://json-schema.org/draft/2020-12/schema. |
| Missing Bearer token or authorization header | HTTP 401 Unauthorized simulation response | Inspect the securitySchemes block in your OpenAPI document and provide a valid Authorization: Bearer <token> test credential. |
robots.txt Directive Counter verification · Audit standards
How to use it
- Paste robots.txt text.
- Count User-agent and selected directive lines and collect Sitemap URLs.
- Read counts and missing-User-agent warning in JSON.
Frequently asked questions
What does Robots.txt Tester count?+
It trims nonblank lines, counts User-agent directives and Allow, Disallow or Crawl-delay directives, extracts Sitemap values, and warns only if no User-agent directive exists.
Can you show an input and its result?+
A group for *, one Disallow, one Allow and a Sitemap URL returned one group, two directives, that sitemap value and no warnings.
Which robots semantics are not evaluated?+
The userAgentGroups field counts User-agent lines, not logical groups: adjacent User-agent lines count separately. It does not apply longest-match or agent-specific rules, validate paths or Sitemap URLs, fetch a site, enforce crawl delay, parse inline comments, or predict crawler behavior.
Does this tool block a crawler?+
No. It summarizes text only; live robots delivery, cache, bot support and crawler policy are outside the page.