Idea: using sitemap for crawling

[Spone]

Hi! I'm super excited about this project, since it's pretty similar to an idea I was keeping in the back of my mind. So I'm delighted to find an existing gem that implements it!

One idea I had was using the sitemap as a starting point for crawling the site, since sitemaps are usually a pretty comprehensive list of existing URLs.

What are your thoughts on this?

Related:

• Automatically detect and parse sitemap.xml postmodern/spidr#19
• Sitemap XML support postmodern/spidr#69
• https://github.com/benbalter/sitemap-parser

[benpickles]

That's a good idea - and I can imagine how it might work within Parklife - but at the moment there's no way to hook into the inner workings so it's not possible.

In the meantime, if you have content that you want to be publicly accessible but isn't reachable via crawling I suggest adding a /sitemap.html which was quite common in the old days.