How to run the crawler on your local machine

1. Set up a local SQL database by following the instructions in the Wiki.

2. Run the REST API by following the instructions in the Wiki.

3. With the REST API running, open a new terminal, and navigate to the root directory of selenium-optmeowt-crawler in terminal by running:

   cd selenium-optmeowt-crawler

4. Open sites.csv and enter the URLs of the sites you want to analyze in the first column. Some examples are included in the file.

5. Ensure Firefox Nightly is installed on your computer per the official Firefox documentation. Depending on where you install it, you may need to change the location of your Firefox Nightly installation in the crawler. Navigate to line 38 and change change the value following .setBinary to Nightly's executable path on your machine.

6. Install the crawler's dependencies by running from within the crawler directory:

   npm install

7. To start the crawler, run from within the crawler directory:

   node local-crawler.js

   If you receive the error WebDriverError: Process unexpectedly closed with status 0, update Firefox Nightly to the latest version.

8. To check the analysis results, open a browser and navigate to http://localhost:8080/analysis. Ports may be different depending on your local server setup. So, you would ned to adjust the URL or your configuration accordingly.