Spyglass v2022.5.15
See the assets to download this version and install.
What's Changed
- Fixed an issue w/ index writing because we were calling
commit
too often, move commits to a secondary task which also speeds up indexing as a secondary result. - By default, Spyglass will now only crawl URLs that have been specified in lenses. This setting is configurable if you want the crawler to act like a typical web crawler w/ the
crawl_external_links
setting. - The
data_directory
setting was added so users can change where the index + metadata is stored. - Better handling of crawler crashes, the crawler will now attempt to restart gracefully if something went wrong.
Full Changelog: v2022.5.11...v2022.5.15