First commit, proof of concept

2022-06-29 18:27:19 +02:00
commit 42f4c992fe
9 changed files with 810 additions and 0 deletions
--- a/robots.txt
+++ b/robots.txt
@@ -0,0 +1,5 @@
+# www.robotstxt.org/
+
+# Allow crawling of all content
+User-agent: *
+Disallow: