8 Commits

Author	SHA1	Message	Date
El RIDO	878f13a643	adding an example to opt-out of the privatebin directory	2020-04-08 05:35:38 +02:00
El RIDO	50881a75e6	ensure bots are prohibited completely from indexing	2019-08-28 20:36:26 +02:00
rugk	7875d7b0a3	Some rewording	2016-07-19 13:05:07 +02:00
rugk	a5730b701c	Revert order change ... According to Wikipedia this is indeed right.. https://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive Then there are only many wrong examples out there.	2016-07-03 13:09:47 +02:00
rugk	0567a720fd	Change order of allow rule in robots.txt ... I think this is the proper order. Don't know if it matters, but it is not bad to change it in any case.	2016-07-03 13:05:26 +02:00
ic0nic	c918a9fe2e	Incorrect structure ... The structure for robots.txt is incorrect for some/most search engines.	2013-11-01 01:22:16 +01:00
ic0nic	50d4fb9bf3	Allow discovery ... To allow search engines to at least find the website.	2013-01-29 19:33:31 +00:00
Greg Knaddison	f37303d858	For server-load and privacy reasons, disallow robots.	2012-08-10 18:21:26 +02:00