[nSLUG] Google Won't Crawl My Most Recent Webpages

Richard Bonner ak621 at chebucto.ns.ca
Fri Apr 8 19:17:52 ADT 2016


On Fri, 8 Apr 2016, Oliver Doepner wrote:

> How is the Google crawler supposed to find the page?
> http://aiel.chebucto.biz/Gallery/Archive/Theatre/2014-15.html
>
> One observation: Your page contains <Meta name="robots" content="index,
> nofollow">

***   That is correct. With that "robots" directive, I am requesting 
that search engines index the page, but not to follow any links 
within. This is the same with other pages within the "Theatre" 
subdirectory, and those can be found by Google.


> I guess that Google uses domains as starting points, i.e. something like
> http://aiel.chebucto.biz/ or even http://www.chebucto.biz/.

***   The former is correct. Google then searches through that site's 
links. It found earlier pages made in the "Theatre" directory, but 
not this and the page before it (2013-14.html).

  Richard


More information about the nSLUG mailing list