Vagrant

Note: We strongly encourage using Docker for your development environment. Due to limited resources, we can't offer support for all possible setups, so please proceed with the understanding that you'll have to do most of the troubleshooting on your own.

This guide covers Vagrant, a tool that allows us to easily create and install a fully-configured development environment.

Table of Contents System requirements First-time setup Create the virtual machine Clone the GitHub repository Start unicorns Update the virtual machine Update gems Copy Vagrant-specific settings files Update the database Restart unicorns Interact with the local archive Create user and admin accounts Load autocomplete data Update filter counts Kill unicorns Run Resque jobs Run tests Troubleshooting Elasticsearch Memcached Missing site skins Mangled JS or CSS files More on Vagrant Use snapshots Share your work Reinstall Vagrant

System requirements

• Version 6.0.0 or greater of VirtualBox
• Version 2.2.0 or greater of Vagrant

Warning: On Windows, VirtualBox requires Hyper-V to be turned off. However, Hyper-V is required by Docker, so if you are using Docker, you may find you have to turn Hyper-V off manually and reboot.

(Note: You might be okay with earlier versions of Vagrant, VirtualBox, or both, but no guarantees. If using an earlier version causes problems, you will probably have to debug them yourself, and you will need to adjust config parameters as necessary.)

First-time setup

Make a new directory for the project. All commands in the setup section will be run inside this directory. In the end, your directory structure should look like this:

otwarchive_vagrant/   <-- the project directory, you can name it whatever
├── otwarchive/       <-- a clone of your fork of the GitHub repository
│   ├── app/
│   ├── config/
│   ├── db/
│   └── ...
└── Vagrantfile       <-- created by 'vagrant init'

Create the virtual machine

We use the box aooo/ao3_1. In a terminal, create a Vagrantfile:

cd otwarchive_vagrant
vagrant init aooo/ao3_1

You can edit this file to give your development environment more computing resources, for example, 4GB and 2 cores:

Vagrant.configure(2) do |config|
  config.vm.box = "aooo/ao3_1"
  config.vm.box_version = "0.0.15"
  # default configurations...

  config.vm.provider "virtualbox" do |vb|
    vb.customize ["modifyvm", :id, "--memory", "4096"]
    vb.customize ["modifyvm", :id, "--cpus", "2"]
  end
end

Download and boot the machine:

vagrant up

Note: The download can take a long time -- make sure to set your computer not to go to sleep or the download will be interrupted.

The output will look like:

Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'aooo/ao3_1'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'aooo/ao3_1' version '0.x.x' is up to date...
==> default: Setting the name of the VM: otwarchive_vagrant_default_1552239500513_47299
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 8208 (guest) => 8080 (host) (adapter 1)
    default: 7000 (guest) => 7000 (host) (adapter 1)
    default: 9091 (guest) => 9091 (host) (adapter 1)
    default: 8201 (guest) => 8021 (host) (adapter 1)
    default: 1025 (guest) => 1025 (host) (adapter 1)
    default: 3306 (guest) => 3306 (host) (adapter 1)
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection reset. Retrying...
    default: Warning: Connection aborted. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Connection aborted. Retrying...
    default: Warning: Connection reset. Retrying...
    default:
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if its present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /vagrant => C:/Users/red/Documents/otwarchive_vagrant

Clone the GitHub repository

If you're new to Git and GitHub:

• Create a GitHub account
• Install Git on your own computer following GitHub's instructions for your operating system
  • If you're on Windows, you may want to set up your GitHub client to use Git Bash as your default shell

You need to fork the otwarchive repository:

• Make sure you are logged in to GitHub
• Go to the otwarchive repository on GitHub
• Fork the repository using the "Fork" button in the upper right corner

Clone your new fork onto your machine.

cd otwarchive_vagrant
git clone [email protected]:YOURUSERNAME/otwarchive.git

The output will look like:

Cloning into 'otwarchive'...
remote: Reusing existing pack: 78163, done.
remote: Counting objects: 15, done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 78178 (delta 5), reused 7 (delta 3)
Receiving objects: 100% (78178/78178), 41.37 MiB | 745 KiB/s, done.
Resolving deltas: 100% (56038/56038), done.

Add a reference to the original, or upstream, repository, so you can get new changes to your fork:

cd otwarchive
git remote add upstream git://github.com/otwcode/otwarchive.git

Now you can connect to your virtual machine (as the default user "vagrant") by running:

vagrant ssh

By default, Vagrant will share your project directory (the directory with the Vagrantfile) on the host machine to /vagrant on the virtual machine.

Start unicorns

Change into your clone directory:

cd /vagrant/otwarchive/

Then run:

start_my_unicorns

This script will do the following:

• Kill any old Rails processes.
• Create a symlink from /home/vagrant/app/current to your clone directory.
• Copy Vagrant-specific settings files to your clone directory.
• Update gems.
• Start the Rails server processes.

You will see:

starting new unicorn master
setting /home/vagrant/app/current to /vagrant/otwarchive

   \.
     \'.      ;.
      \ '. ,--''-.~-~-'-,
       \,-' ,-.   '.~-~-~~,
     ,-'   (###)    \-~'~=-.
 _,-'       '-'      \=~-"~~',
/o                    \~-""~=-,
\__                    \=-,~"-~,
   """===-----.         \~=-"~-.
               \         \*=~-"
                \         "=====----
                 \
                  \

The Rails server processes are started, and you should be able to browse your archive at http://127.0.0.1:7000/.

Update the virtual machine

When there are changes to the upstream repository, you need to download them to your clone, then refresh your development environment. In your clone:

git pull upstream master

Then connect to your virtual machine:

vagrant up
vagrant ssh

Inside your virtual machine, change into your clone directory:

cd /vagrant/otwarchive/

then refresh a bunch of things...

Update gems

Note: This will be done automatically as part of start_my_unicorns or reset_test.

Note: If the Ruby version of the Vagrant box is older than the version used by the project, you need to update it manually:

rvm use $(cat .ruby-version) --install --default

Once Ruby is up to date, update gems with:

bundle install

You will see:

Using rake 12.3.1
[...]
Using will_paginate 3.1.6
Bundle complete! 86 Gemfile dependencies, 192 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

Copy Vagrant-specific settings files

Note: This will be done automatically as part of start_my_unicorns or reset_test.

Vagrant-specific versions of the settings files (e.g. database.yml, redis.yml, etc.) live at ~/conf. Before running anything, you need to copy them to your clone directory:

create_links_on_install

Update the database

Run all pending migrations:

bundle exec rake db:migrate

Restart unicorns

Run the same command to start unicorns.

Interact with the local archive

Connect to the VM, run cd /vagrant/otwarchive/, then you can...

Create user and admin accounts

See Development Data for help with creating admin and user accounts in your development environment.

Load autocomplete data

If the autocomplete fields are not showing results for any existing fandoms, you can run:

bundle exec rake autocomplete:load_data

This will take some time to complete. You can continue working on the archive in the meantime.

Update filter counts

bundle exec rake Tag:reset_filter_counts

Kill unicorns

kill_my_unicorns

Run Resque jobs

start_my_workers

and:

kill_my_workers

Note: the version of the archive code being run by the Resque jobs will not automatically reload as you write. If you change any background jobs, you will have to kill the workers and restart them before your changes show up.

Run tests

Set up local settings files required to run tests, recreate the test database, and upgrade the gems:

reset_test

Once the test environment is set up by reset_test, you can run individual tests.

Running the whole test suite on your VM can take a long time. Instead, you can run only the tests relevant to what you're working on, and run tests in parallel much faster on your fork's Actions.

If you're running Cucumber tests, you can prepend the command with QT_QPA_PLATFORM=offscreen to ensure that screenshots can be generated even though Vagrant doesn't have a GUI. For example:

QT_QPA_PLATFORM=offscreen RAILS_ENV=test bundle exec cucumber features

Troubleshooting

Elasticsearch

In the VM, check if Elasticsearch is running:

systemctl status elasticsearch

Restart if it isn't:

sudo systemctl restart elasticsearch

If Elasticsearch is running, but you're getting index_not_found_exception, you can recreate your indexes (and fill them with data, provided you have background workers running) by running the tasks in search.rake:

bundle exec rake search:index_bookmarks
bundle exec rake search:index_pseuds
bundle exec rake search:index_tags
bundle exec rake search:index_works

Memcached

In the VM, check if Memcached is running:

systemctl status memcached

Restart if it isn't:

sudo systemctl restart memcached

If you need to wipe the cache:

bundle exec rails runner "Rails.cache.clear"

Missing site skins

Reload site skins from CSS files:

bundle exec rake skins:load_site_skins

You also have to cache them. Start a console with:

bundle exec rails c

In the console, run:

Skin.where('cached = true').where('id NOT IN (?)', [887, 890, 928, 929]).each(&:cache!)

Exit the console by running exit.

Mangled JS or CSS files

If you edit JS or CSS files on the host machine and you notice they are served truncated, or with bad characters, you can fix it by editing /etc/nginx/nginx.conf in the virtual machine.

Change the line sendfile on; to sendfile off;.

More on Vagrant

Use snapshots

After setting up the environment properly, you can save a snapshot of the current VM:

vagrant snapshot push

You can revert to it later, using:

vagrant snapshot pop

This saves you from recreating the entire VM, in case you ruin your working environment.

Share your work

Make sure your VM is started and responding on port 7000, then follow the Vagrant Share instructions.

Reinstall Vagrant

If you are installing a new Vagrant version, completely remove the old virtual machine and start from scratch. In the folder with your Vagrantfile, run:

vagrant destroy