robots.txt 1.3 KB

12345678910111213141516171819202122232425262728293031
  1. # Hello if you're a human! Here is the robots.txt file for owlman.neocities.org
  2. # If you don't know what this is, the tl;dr is that it tells web crawlers
  3. # and other web robots what they can and can not view/index.
  4. # For my robots.txt I am telling all crawlers that they can't view
  5. # Who is Who in ASCII Art and a robots.txt example that I used on this page:
  6. # https://owlman.neocities.org/library/archive.html
  7. # You can see that I have also told a crawler by the name of ia_archiver that
  8. # they can view all of the site bar the robots.txt example.
  9. # The ia_archiver is a bot from The Internet Archive,
  10. # a 501(c)(3) non-profit library I hold very dear to my heart.
  11. # If you have any questions about what robots.txt are, I would view the websites
  12. # listed to help you better understand
  13. # The Web Robots Pages: http://www.robotstxt.org
  14. # Robots exclusion standard: https://en.wikipedia.org/wiki/Robots_exclusion_standard
  15. # How to Write a Robots.txt File: https://support.microsoft.com/en-us/help/217103/how-to-write-a-robots-txt-file
  16. # Last updated: 04/12/17 @ 16:20
  17. # Last updated: 10/12/17 @ 21:05
  18. User-agent: *
  19. Disallow: /Who_in_Ascii_Art.html
  20. Disallow: /ascii.html
  21. Disallow: /odds/robotstxtexample.txt
  22. User-agent: ia_archiver
  23. Allow: /
  24. Disallow: /odds/robotstxtexample.txt