How to run the crawler on your local machine

1. Set up a local SQL database by following the instructions in the Wiki.

2. Run the REST API by following the instructions in the Wiki.

3. With the REST API running, open a new terminal, and navigate to the root directory of selenium-optmeowt-crawler in terminal by running:

   cd selenium-optmeowt-crawler

4. Open sites.csv and enter the URLs of the sites you want to analyze in the first column. Some examples are included in the file.

5. Ensure Firefox Nightly is installed on your computer per the official Firefox documentation. Depending on where you install it, you may need to change the location of your Firefox Nightly installation in the crawler. Navigate to line 38 and change change the value following .setBinary to Nightly's executable path on your machine.

6. Install the crawler's dependencies by running from within the crawler directory:

   npm install

7. To start the crawler, run from within the crawler directory:

   • For macs:

   node local-crawler.js -m

   • For Windows OS:

   node local-crawler.js -w

   • If you receive the error WebDriverError: Process unexpectedly closed with status 0, update Firefox Nightly to the latest version.
   • If you receive the error "Error: Could not locate Firefox on the current system" then find the location of the firefox binary, the default locations can be found in the official Firefox documentation, and run:

   node local-crawler.js -bin /path/to/firefox

8. To check the analysis results, open a browser and navigate to http://localhost:8080/analysis. Ports may be different depending on your local server setup. So, you would ned to adjust the URL or your configuration accordingly.