Google search documentation

Google is analyzing real-world robots.txt data to expand its list of unsupported rules, helping website owners better understand which directives work.