[nSLUG] Google Won't Crawl My Most Recent Webpages
ak621 at chebucto.ns.ca
Fri Apr 8 19:17:52 ADT 2016
On Fri, 8 Apr 2016, Oliver Doepner wrote:
> How is the Google crawler supposed to find the page?
> One observation: Your page contains <Meta name="robots" content="index,
*** That is correct. With that "robots" directive, I am requesting
that search engines index the page, but not to follow any links
within. This is the same with other pages within the "Theatre"
subdirectory, and those can be found by Google.
> I guess that Google uses domains as starting points, i.e. something like
> http://aiel.chebucto.biz/ or even http://www.chebucto.biz/.
*** The former is correct. Google then searches through that site's
links. It found earlier pages made in the "Theatre" directory, but
not this and the page before it (2013-14.html).
More information about the nSLUG