This plugin needs more documentation!
You can edit your robots.txt and add remote content to it. E.g. you have several sites and want to use a centralized robots.txt.
Features
- Include or exclude WordPress’ own robots.txt (core function)
- Include or exclude plugins – e.g. sitemap plugins – output to robots.txt (filter output)
- Include or exclude a remote text file (the common part)
- Include or exclude custom records from the settings page (the site specific part)
Where is robot.txt?
WordPress handles robots.txt as a virtual URL – just the same way as posts and pages.
So when you browse to https://example.com/robots.txt WordPress generates robots.txt on the fly.
TODO
- add more description here
- add a video too
- add an admin notice for subdir installs (robots.txt is useless in a subdir)
- ‘At least one “Disallow” field must be present in the robots.txt file.’ – check for that
Links
Development of this plugin goes on on GitHub.