passing custom headers to Crawler

[istavritde]

Hi,
I'm currently experimenting with Crawler using InMemoryDocumentStore, can we pass headers to crawler for Basic Authentication?

[masci]

You can't pass headers but you can specify basic auth in the URL itself, like https://<username>:<password>@www.example.com/index.html (I didn't test it)