Multipart robots.txt editor

This plugin needs more documentation!

You can edit your robots.txt and add remote content to it. E.g. you have several sites and want to use a centralized robots.txt.

Features

• Include or exclude WordPress’ own robots.txt (core function)
• Include or exclude plugins – e.g. sitemap plugins – output to robots.txt (filter output)
• Include or exclude a remote text file (the common part)
• Include or exclude custom records from the settings page (the site specific part)

Where is robot.txt?

WordPress handles robots.txt as a virtual URL – just the same way as posts and pages.

So when you browse to https://example.com/robots.txt WordPress generates robots.txt on the fly.

TODO

• add more description here
• add a video too
• add an admin notice for subdir installs (robots.txt is useless in a subdir)
• ‘At least one “Disallow” field must be present in the robots.txt file.’ – check for that

Links

Development of this plugin goes on on GitHub.