Indexed, though blocked by robots.txt

This is the place to post questions and comments about the FlexBanner component
Post Reply
CoHill
Posts: 11
Joined: Sat Nov 26, 2016 8:17 am

Indexed, though blocked by robots.txt

Post by CoHill » Tue May 29, 2018 8:18 am

Good Morning.

I have been informed by Google that a page is being blocked by robots.txt. Looking at the information this is the culprit:
https://www.mywebsite.nl/fr/component/flexbanners/?task=click&id=8

In my robots text is this line:
Disallow: /*?

Okay, easiest thing to do is delete this line from robots.txt, however I would not like to do this. I have also other banners with ID's of 1 to 8, however the other ones don't give an issue according to Google.
What could I do about this?

Co

Andrew998
Posts: 1384
Joined: Wed Jan 07, 2009 10:43 am

Re: Indexed, though blocked by robots.txt

Post by Andrew998 » Tue May 29, 2018 8:25 am

Are you using the latest version of FlexBanner and do you have SEF URLs turned on? Recent versions use a different router and you shouldn't be getting URLs like that.
Andrew998
Developer

CoHill
Posts: 11
Joined: Sat Nov 26, 2016 8:17 am

Re: Indexed, though blocked by robots.txt

Post by CoHill » Tue May 29, 2018 8:31 am

Hi,
Yesterday I had the one but latest version, this is the version that i got the google message from. I have since then updated it; how long does it take before Google notices it?
and sef url's are turned on, yes

CoHill
Posts: 11
Joined: Sat Nov 26, 2016 8:17 am

Re: Indexed, though blocked by robots.txt

Post by CoHill » Tue May 29, 2018 8:33 am

By the way, it only started about 4 days ago.. haven't done anything special of late

Andrew998
Posts: 1384
Joined: Wed Jan 07, 2009 10:43 am

Re: Indexed, though blocked by robots.txt

Post by Andrew998 » Tue May 29, 2018 8:47 am

Are you using a URL manager of some sort (e.g. sh404sef) because I simply don't recognise that format of click URL? With SEF turned on the link should look like https://www.mywebsite.nl/fr/component/flexbanners/click/8
Andrew998
Developer

Andrew998
Posts: 1384
Joined: Wed Jan 07, 2009 10:43 am

Re: Indexed, though blocked by robots.txt

Post by Andrew998 » Tue May 29, 2018 8:57 am

I've done some testing and that URL will only be visible if the router isn't there which suggests an installation problem. I'm going to do a new release shortly that should address the issue.
Andrew998
Developer

CoHill
Posts: 11
Joined: Sat Nov 26, 2016 8:17 am

Re: Indexed, though blocked by robots.txt

Post by CoHill » Tue May 29, 2018 9:10 am

No I am not using anything like sh404sef as far as I know, am using jsitemappro for the last 18 months

Do you suggest that I wait until the new version is released? (Google says that it is a warning not an error)

Andrew998
Posts: 1384
Joined: Wed Jan 07, 2009 10:43 am

Re: Indexed, though blocked by robots.txt

Post by Andrew998 » Tue May 29, 2018 9:12 am

I've issued 4.1.10 which should address the issue. You can force the update by going to the component and clicking on the update at the bottom of the page even if there says it isn't an update available.
Andrew998
Developer

CoHill
Posts: 11
Joined: Sat Nov 26, 2016 8:17 am

Re: Indexed, though blocked by robots.txt

Post by CoHill » Tue May 29, 2018 9:32 am

Thanks for that, I have just updated. FYI this issue is on the latest banner that I inserted about a month ago. The other banners are far older, a year or so I suppose.
I'll carry on this afternoon, and keep you informed.

Thanks again and keep up the good work. :-)

Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests