Robots Exclusion Protocol

The protocol language consists of rule(s) and group(s) that the service makes available in a file named 'robots.txt' as described in section 2.3: Rule: A line with a key-value pair that defines how a crawler may access URIs. See section 2.2.2. Group: One or more user-agent lines that is followed by one or more rules. The group is terminated by a user-agent line or end of file. See section 2.2.1. The last group may have no rules, which means it implicitly allows everything.

Below is an Augmented Backus-Naur Form (ABNF) description, as described in .

Crawlers set a product token to find relevant groups. The product token MUST contain only "a-zA-Z_-" characters. The product token SHOULD be part of the identification string that the crawler sends to the service (for example, in the case of HTTP, the product name SHOULD be in the user-agent header). The identification string SHOULD describe the purpose of the crawler. Here's an example of an HTTP header with a link pointing to a page describing the purpose of the ExampleBot crawler which appears both in the HTTP header and as a product token: HTTP header robots.txt user-agent line user-agent: Mozilla/5.0 (compatible; ExampleBot/0.1; https://www.example.com/bot.html) user-agent: ExampleBot Crawlers MUST find the group that matches the product token exactly, and then obey the rules of the group. If there is more than one group matching the user-agent, the matching groups' rules MUST be combined into one group. The matching MUST be case-insensitive. If no matching group exists, crawlers MUST obey the first group with a user-agent line with a "*" value, if present. If no group satisfies either condition, or no groups are present at all, no rules apply.

These lines indicate whether accessing a URI that matches the corresponding path is allowed or disallowed. To evaluate if access to a URI is allowed, a robot MUST match the paths in allow and disallow rules against the URI. The matching SHOULD be case sensitive. The most specific match found MUST be used. The most specific match is the match that has the most octets. If an allow and disallow rule is equivalent, the allow SHOULD be used. If no match is found amongst the rules in a group for a matching user-agent, or there are no rules in the group, the URI is allowed. The /robots.txt URI is implicitly allowed. Octets in the URI and robots.txt paths outside the range of the US-ASCII coded character set, and those in the reserved range defined by , MUST be percent-encoded as defined by prior to comparison. If a percent-encoded US-ASCII octet is encountered in the URI, it MUST be unencoded prior to comparison, unless it is a reserved character in the URI as defined by or the character is outside the unreserved character range. The match evaluates positively if and only if the end of the path from the rule is reached before a difference in octets is encountered. For example: Path Encoded Path Path to Match /foo/bar?baz=quz /foo/bar?baz=quz /foo/bar?baz=quz /foo/bar?baz=http
://foo.bar /foo/bar?baz=http%3A
%2F%2Ffoo.bar /foo/bar?baz=http%3A
%2F%2Ffoo.bar /foo/bar/U+E38384 /foo/bar/%E3%83%84 /foo/bar/%E3%83%84 /foo/bar/%E3%83%84 /foo/bar/%E3%83%84 /foo/bar/%E3%83%84 /foo/bar/%62%61%7A /foo/bar/%62%61%7A /foo/bar/baz The crawler SHOULD ignore "disallow" and "allow" rules that are not in any group (for example, any rule that precedes the first user-agent line). Implementers MAY bridge encoding mismatches if they detect that the robots.txt file is not UTF8 encoded.

Crawlers SHOULD allow the following special characters: Character Description Example "#" Designates an end of line comment. "allow: / # comment in line"

"# comment on its own line" "$" Designates the end of the match pattern. "allow: /this/path/exactly$" "*" Designates 0 or more instances of any character. "allow: /this/*/exactly" If crawlers match special characters verbatim in the URI, crawlers SHOULD use "%" encoding. For example: Percent-encoded Pattern URI /path/file-with-a-%2A.html https://www.example.com/path/file-with-a-*.html /path/foo-%24 https://www.example.com/path/foo-$

Clients MAY interpret other records that are not part of the robots.txt protocol. For example, 'sitemap' . Parsing of other records MUST NOT interfere with the parsing of explicitly defined records in section 2.

The rules MUST be accessible in a file named "/robots.txt" (all lower case) in the top level path of the service. The file MUST be UTF-8 encoded (as defined in ) and Internet Media Type "text/plain" (as defined in ). As per , the URI of the robots.txt is: "scheme:[//authority]/robots.txt" For example, in the context of HTTP or FTP, the URI is:

If the crawler successfully downloads the robots.txt, the crawler MUST follow the parseable rules.

The server may respond to a robots.txt fetch request with a redirect, such as HTTP 301 and HTTP 302. The crawlers SHOULD follow at least five consecutive redirects, even across authorities (for example, hosts in case of HTTP), as defined in . If a robots.txt file is reached within five consecutive redirects, the robots.txt file MUST be fetched, parsed, and its rules followed in the context of the initial authority. If there are more than five consecutive redirects, crawlers MAY assume that the robots.txt is unavailable.

Unavailable means the crawler tries to fetch the robots.txt, and the server responds with unavailable status codes. For example, in the context of HTTP, unavailable status codes are in the 400-499 range. If a server status code indicates that the robots.txt file is unavailable to the client, then crawlers MAY access any resources on the server.

If the robots.txt is unreachable due to server or network errors, this means the robots.txt is undefined and the crawler MUST assume complete disallow. For example, in the context of HTTP, an unreachable robots.txt has a response code in the 500-599 range. For other undefined status codes, the crawler MUST assume the robots.txt is unreachable. If the robots.txt is undefined for a reasonably long period of time (for example, 30 days), clients MAY assume the robots.txt is unavailable or continue to use a cached copy.

Crawlers SHOULD try to parse each line of the robots.txt file. Crawlers MUST use the parseable rules.

Crawlers MAY cache the fetched robots.txt file's contents. Crawlers MAY use standard cache control as defined in . Crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt is unreachable.

Crawlers MAY impose a parsing limit that MUST be at least 500 kibibytes (KiB).