Server is blocking access
-
I have just had an SEO report run on my site and I have been told
your server is blocking automated access to various areas of your site, thus preventing proper crawling of the website, hence why search engines have trouble fully indexing your site
How can I rectify this please?
The blog I need help with is: (visible only to logged in users)
-
-
Sorry no, I’m not really up on all this stuff. Does it need to be more specific or do you understand the problem at all?
-
-
I’ve just received this back:
When I said “various”, I meant that it happens randomly, and thus any page can trigger this problem if crawling multiple pages at once ;) Also, since your site is based on WordPress, then it usually is a problem related to a faulty or misconfigured plugin (most of the time this happens due to a plugin called “W3 Total Cache”, or a similar caching plugin, but it can also be a plugin that is specialized in blocking automated access)”
-
I’ll tag this thread for a Staff follow-up. Please subscribe to the thread so you are notified when they respond and please be patient while waiting.
-
Hi Ruairidh,
We don’t allow any plugins at all here at WordPress.com, so it definitely wasn’t either of those which caused the spider to fail.
It is worth noting that we do block access to certain parts of your site for security reasons. Those parts we will never allow access to, such as login pages, your dashboard, our activation process, etc. Your SEO guys can see a full list of those if they view the robots.txt file right here:
http://songsfromthehowlingsea.com/robots.txt
If that is what is tripping up their crawler, they should be able to program it to ignore those and continue without too much difficulty. Otherwise, if they could get a list of the areas where they’re having problems it could help!
-
This feedback is the result of an seo report only and as such it falls to me (a novice) to fix so apologies in advance..
He said
some of the pages returned a “403 forbidden” server error.
http://songsfromthehowlingsea.com/2013/05/25/banner-2/
http://songsfromthehowlingsea.com/2013/05/25/banner/
http://songsfromthehowlingsea.com/2013/05/26/banner-3/
http://songsfromthehowlingsea.com/2013/05/26/banner-4/And then he stated
starting from page 11 of the “1_Website Structure Audit Report.pdf” file you’ll see the pages that triggered this problem
This report is here http://issuu.com/soyouwannaplayguitar/docs/1_website_structure_audit_report/0
-
It looks like the crawler is having problems with 4 specific posts: banner, banner-2, banner-3, and banner-4.
I’ve checked your site again and there is nothing which should be blocking access to these posts, so it seems like the crawler may have some sort of issue, perhaps with the file name?
-
-
Well there isn’t much that I can recommend as again, we don’t appear to be blocking the crawler, they seem to just be having problems with the pages themselves (in fact even the error states “client error.” A “client” in tech terms is usually the viewer, such as a browser or a web crawler).
With that said, you have a few options available. First off, despite the long list of URLs, it is in fact only 4 posts which are having issues according to the pdf which you provided. Given that your site has 108 posts and 14 pages, that may not matter as much to you. If it does, you can try renaming them (make sure the slug is updated as well) and have the crawler run again. If it is important that the crawler be able to finish the run without any errors but you don’t care if the posts are included in the results, you can set them to draft and then have the crawler run, that way it should not see the pages.
Those are the steps which I can think of that you can take on your end. Unfortunately there’s really not anything which we can do ourselves.
-
Hey amightywp i am having a really big problem with my blog right now and i cant fix it here’s the link to the forum https://en.forums.wordpress.com/topic/images-are-linked-from-my-old-blog-and-images-wont-show-up?replies=1
-
- The topic ‘Server is blocking access’ is closed to new replies.
WordPress.com Forums 