El RIDO
|
878f13a643
|
adding an example to opt-out of the privatebin directory
|
2020-04-08 05:35:38 +02:00 |
|
El RIDO
|
50881a75e6
|
ensure bots are prohibited completely from indexing
|
2019-08-28 20:36:26 +02:00 |
rugk
|
7875d7b0a3
|
Some rewording
|
2016-07-19 13:05:07 +02:00 |
|
rugk
|
a5730b701c
|
Revert order change
According to Wikipedia this is indeed right.. https://en.wikipedia.org/wiki/Robots_exclusion_standard#Allow_directive
Then there are only many wrong examples out there.
|
2016-07-03 13:09:47 +02:00 |
|
rugk
|
0567a720fd
|
Change order of allow rule in robots.txt
I think this is the proper order. Don't know if it matters, but it is not bad to change it in any case.
|
2016-07-03 13:05:26 +02:00 |
ic0nic
|
c918a9fe2e
|
Incorrect structure
The structure for robots.txt is incorrect for some/most search engines.
|
2013-11-01 01:22:16 +01:00 |
|
ic0nic
|
50d4fb9bf3
|
Allow discovery
To allow search engines to at least find the website.
|
2013-01-29 19:33:31 +00:00 |
Greg Knaddison
|
f37303d858
|
For server-load and privacy reasons, disallow robots.
|
2012-08-10 18:21:26 +02:00 |