README.md

NOTE Legacy build steps can be found in build-steps-Django-2.2.md

1. Install Nix (see download page for more):

    sh <(curl -L https://nixos.org/nix/install) --daemon
   

   NOTE The Determinate Systems' Nix installer works like a charm, and should switch to that:

   curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install
   source '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' # or restart the shell
   

2. (OPTIONAL) Enter a Nix shell with the most common tools

    source <(curl https://raw.githubusercontent.com/toraritte/shell.nixes/main/run.sh) -n "22.11"
   

   ^{Read more here.}

3. Clone this repo & enter the clone

    git clone https://github.com/society-for-the-blind/slate-2.git
    cd slate-2
   

   If the dev branch is needed, then use:

    git clone -b dev https://github.com/society-for-the-blind/slate-2.git
    cd slate-2
   

4. Enter ./nix/dev_shell.nix. (See comments there or use the command below.)

    nix-shell nix/dev_shell.nix --argstr "deployment_environment"  "dev"
   

   NOTE Which deployment environment? See note at the top of dev_shell.nix.

5. See Justfile on how to build (and toy-serve) the project.

   The needed command will probably be one of the following:

   just # short for `just serve_empty`
   just serve_from <path-to-postgresql-dumpfile>
   

   Or build (and toy-serve) right away with:

   nix-shell nix/dev_shell.nix --arg "deploy" "true"
   

6. To serve with NGINX, enter ./nix/nginx_shell.nix.

   NOTE: HTTPS only, both for testing and in production.

   From the project root:

   nix-shell                                                       \
     --argstr "ssl_cert"    "$( readlink -f lynx-dev-server.crt )" \
     --argstr "private_key" "$( readlink -f lynx-dev-server.key )" \
     --argstr "domain"      "lynx.dev"                             \
     nix/nginx_shell.nix
   

   (See HTTPS notes on how to create the keys and SSL certificates for testing.)

Notes

Logs

Logs are saved in _nix-shell/.

Python / Django errors

The very first error I got with this app:

  `ImportError: bad magic number in 'mysite.settings': b'\x03\xf3\r\n'`

The solution provided in the Stackoverflow thread (i.e., deleting *.pyc files) worked.

rsync

rsync sometimes hangs on one server, but not on the other, so I use it on the "good" side to push / pull files when scp is not enough:

rsync -avze "ssh -i <PATH-TO-PRIVATE-KEY>" --rsync-path=/nix/store/...-rsync-3.2.7/bin/rsync --progress --partial --backup [email protected],4:<SOURCE-PATH> <DEST-PATH>

SchemaSpy

Get the latest SchemaSpy release and PostgreSQL JDBC driver.

schemaspy -t pgsql11 -host 1.2.3.4 -port $(just s "PORT") -db $(just s "NAME") -s public -u $(just s "USER") -o output/directory/ -dp postgresql-42.6.0.jar

schemaspy -t pgsql11 -host localhost -port 5432 -db lynx -s public -u postgres -hq -dp postgresql-42.6.0.jar -imageformat svg  -I "(django|auth)_.*" -o 07
                                                                                                                               ^^^^^^^^^^^^^^^^^^^^^
                                                                                                                               this part may be necessary
                                                                                                                               for debugging

Use SSH tunneling to connect to a remote database:

1. ssh -i <cert> -L 5432:localhost:5432 [email protected]
2. Change -host in the schemaspy conmmand above to localhost.

QUESTIONS

1. Do migration files need to be checked into version control or should they be generated from scratch via makemigration?

   ANSWER: See the relevant docs below, but the concensus is that they should be checked in. There are alternatives to it, but that is not a version control problem, but a management / methodology one. See this SO thread and links in the comments.

   • https://docs.djangoproject.com/en/4.1/topics/migrations/
   • https://docs.djangoproject.com/en/4.1/topics/db/models/
   • https://realpython.com/django-migrations-a-primer/
   • https://docs.djangoproject.com/en/4.1/intro/overview/

1. How does the django-user-accounts package integrate with django-auth?

   Decided to remove django-user-accounts because it was not used (the corresponding account_* tables were virtually empty; see backup) and it caused extra work after updating everything (see this Django forum thread).

   Nonetheless, I wonder how this package was shown on the admin page, in its own section?

   Also, a couple of notes regarding removed django-user-accounts settings:

   • ACCOUNT_PASSWORD_EXPIRY = 60*60*24*14 # seconds until pw expires, this example shows 14 days

     Found this thread while doing research on how to implement this, and, as it turns out, this practice is recommended against (see articles by the NCSC and the FTC).

     See more at the next item.

   • ACCOUNT_PASSWORD_USE_HISTORY = True

     This sounds like a good idea on the surface, but this will be abused the same way as the previous option, if it is used to disallow previous passwords (or what else is this used for?).

   Will have to do more reading on thi, but instead of using these options, a better strategy would be to

   1. (periodically) scan existing password hashes in known compromised password databases, and report it to user and supervisor

   2. immediately compare the credentials of new users and report it (this is an internal website, so users will be added by administrators) => better yet, suggest non-compromised passwords automatically

   Potentional issues: It is a challenge to educate everyday users, even people in higher positions, to adopt common sense security practices (as it affects convenience, comfort zone, etc.), but the majority of Lynx users are legally blind, and some commonly used tools may be completely inaccessible (e.g., password managers).

TODOs

Done (or considered done, at least)

1. Avoid stacking subshells -> Package properly with Nix (or, at least, do it in a single shell.nix)

   (case in point: https://stackoverflow.com/questions/21976606/why-avoid-subshells)

   RESOLUTION: see dev_shell.nix.

2. Figure out how to add settings.py to version control in a safe manner.

   Just spent ca. 3 hours solving runtime errors that were resolved by (1) adjusting version numbers in requirements.txt and (2) adding missing entries to INSTALLED_APPS in settings.py. Adding dummy_settings.py as a workaround for now.

   RESOLUTION: Use KeePassXC and SOPS. See commands in dev_shell.nix and this Stackoverlow answer.

Active

1. Audit pg_hba.conf

   ... as it is quite rudimentary (and may even be insecure) at the moment. For example, the current settings won't ask for a password when logging in (see Postgresql does not prompt for password), but it is probably prudent to set it up.

2. !!! https://docs.djangoproject.com/en/4.1/intro/tutorial01/

3. Rename the current production PostgreSQL user. (Just do \du in the psql shell and you'll see why.)

   On that note, it would be prudent to implement security best practices. For example:

   • should a postgresql cluster in production have superusers
   • see tab group on phone

4. Django: How to manage development and production settings? (SO)

   Related: (loading secrets into settings.py from textfiles)

5. Save documents in database?

6. (Related to 2.) Set up database backup and/or replication.

7. Adopt decision records (DR)

   • https://github.com/joelparkerhenderson/decision-record
   • https://adr.github.io/
   • https://docs.aws.amazon.com/prescriptive-guidance/latest/architectural-decision-records/adr-process.html

8. Cull dependencies

   There are a lot of packages that may not even be needed, so audit requirements.txt.

9. Make dev environment more flexible.

   That is, don't make dev_shell.nix run until the web server is running (this would be a good candidate task for prod_shell.nix or something similar), but just add the tools needed (plus maybe add clean up commands to shellHook), but the rest (setting up the database, set up virtualenv, run migrations etc) should be done by job runners (e.g., with just).

   UPDATE (2023-04-15) This has largely been achieved with the latest set of commits (adding Justfile and parameters to dev_shell.nix), but still unsure what the ideal producion environment would look like (and how it should be deployed). See also the LXD/LXC-related TODO somewhere on this list.

10. Explore containerization

    The DRAFT-lxc-deploy.sh script is an attempt at using LXD/LXC containers, but it also adds extra complexity (see file). The current approach with Nix seems highly reproducible already, but could be wrong and there may be other benefits.

11. Fetch Python packages from Nixpkgs (or create a new package collection)

vim: set foldmethod=marker foldmarker={{-,}}- foldlevelstart=0 tabstop=2 shiftwidth=2 expandtab: