Trajectory for backscope

[gwhitney]

Returning to the numberscope project after a roughly a semester's hiatus, it seems as though the place that needs work first is the back end, in part because of wanting to get the benefits promised in #76, which won't be merged directly, through an orderly sequence of PRs.

Looking at the features mentioned in the description of that PR, here's a list of possible features/fixes to tackle, and a possible order:

• Simplify README/document Ubuntu installation steps.
  This should be pretty straightforward, and the Ubuntu installation steps should be testable simply by trying to replicate the installation. Would address docs: update README with comprehensive install steps #73, and possibly clarify docs on resetting the database #68 and Add command to get the backscope log to the server admin guide #55. [Didn't address either of those in the end, so just leaving them in the issue queue and checking this one off.)
• Add tests of some sort
  Really valuable for all future modifications, so sooner is better than later. Would address Need testing sufficient at least for dependabot PRs #11
• Update dependencies
  Take advantage of existence of tests to get past some very old dependencies with known security issues
• Update production scripts
  Right now it is impossible to keep production in sync with current backscope code, but we need them to track each other closely as we address the fundamental problems with backscope. So this is top priority once we have an updated system to install.
• Wrap calls to the OEIS in a try/except or check the status code
  I think this is the piece that attempts to address when crawling too fast, backend gets stuck on disallowed sequence even when crawling is allowed again #69, which was one of the real motivations for change here. If not, then after we extract this part of the big PR, we should then go on to when crawling too fast, backend gets stuck on disallowed sequence even when crawling is allowed again #69.
• Make sure the OEIS sequence offset is being computed and stored in the database (OEIS sequence offset is never being set #77) Note this was not in the big PR at all
• Add a linter (As Liam points out, this will catch some problems that go under the radar otherwise).

I think that's sufficient to get us back to working on the front end. Of course we will have to see if the sequence access errors in the production instance go away at that point.

Note the above possible plan does not address the following elements that were apparently part of the big PR #76; I mostly have no objection to any of them, if it was done as a single, self-contained PR. But I don't think any of them is necessary for proper operation of the backend, and none of them have any immediate payoff. They mostly lay the groundwork to ease further development. Hence the necessity of them at a time when we think it's unlikely there will be a lot of change in the backend is unclear.

• simplify files and directories
• remove unused code
• Add a code formatter (it's a matter of taste whether a project should even use such a thing; not entirely clear it adds any benefit)
• Raise errors/exceptions rather than returning them (I think the motivation for the current design is that the backend should not crash, period. So we definitely don't want any changes that raise the possibility it might.)
• Simplify the API (again, somewhat a matter of taste; not entirely clear what the problems, if any, of the current API are. And changing the API is a little annoying, in that the new API has to be merged into backscope preserving the old, then a new version of the frontscope has to be merged that uses the new API; and then finally a version of backscope that gets rid of the old API has to be merged. So it's three PRs per API change... And really, going forward, we should be updating version numbers in backscope any time we change the API, to help keep track.)
• Add Python docstring comments. These are always good, and can be added incrementally as code is worked on. There isn't really any problem with a PR adding a docstring comment to a function or other code entity that is worked on in that PR. And in fact, we should strive to do that. Of course, it would be fine for someone to go through wholesale and add docstrings to lots of things. But if so, that should be in its own PR, and it would take someone motivated to do a PR for just that.

Have I missed anything that should be done before we leave the backscope be for a bit and re-focus on the frontscope? Thanks for your thoughts and reactions.

[gwhitney]

Another item on the list of things done in the big PR #76 that we're not planning for the immediate future (especially in this case as there's nobody on team that is clear about how exactly version-controlled database migrations are supposed to work):

• Switch from reinitializing the database on schema changes to version-controlled database migrations recorded automatically by flask.

[gwhitney]

And another item along these same lines:

• Migrate from python manage.py to flask for interacting with the backscope app and database.

[gwhitney]

This will be done perforce by #94 once that's merged.

[gwhitney]

Unfortunately, work on backscope seems to be completely stymied because I can't seem to install cysignals at all anymore: see sagemath/cysignals#175. So we may not be able to make any progress on numberscope until that is resolved.

[gwhitney]

OK, I was being a bit pessimistic. By using the frozen requirements at an earlier version of Cython it should be possible to get cysignals/cypari2 to install. I will try to get that working and create a PR.

[gwhitney]

OK, I have created PR #80 that implements the first checkbox in the initial post of this discussion. Kate, when you get a chance to review it, it would be great if you could blow away your clone of backscope completely and go through the whole installation process on your laptop as described in the new documentation (except no need to re-do the PostgreSQL stuff, you should be able to use your existing PostgreSQL installation, database user, and database.) Of course I welcome your comments on the documentation changes as well. Look forward to hearing from you if we finally have a cut-and-dried, reliably working install process for backscope. I think I will wait to work on the next item in this trajectory until we get #80 (or an improved version of it if need be) merged. Thanks!

[liammulh]

I will work on adding some sort of end-to-end test, a simple unit test, and docs on testing backscope.

[gwhitney]

Great, thanks! Feel free to issue a PR when you have just one test working, or wait til you have one of each. (But given your probable availability, maybe don't wait any longer that that ;-)

[gwhitney]

This has been completed by #84.

[gwhitney]

I have updated the checklist in the beginning of this conversation in light of the successful merge of #84, finishing one of the tasks, the full realization that some of the dependencies were extremely out of date with known security vulnerabilities, and the discovery that production scripts are no longer correct and must be updated to keep production in sync with development. Also note that once #94 is merged, we will all be obliged to use the flask <COMMAND> ... interface in place of python manage.py <COMMAND> ... (but it pretty much works the same if you just replace python manage.py by flask.

[gwhitney]

OK, making progress on the next checkbox, I have approved and merged all of the dependabot PRs, and generated a new PR that updates to the current Cython 3, significantly simplifying the install script. But as it's a fairly major change, that new PR #96 definitely needs independent review before merging.

[gwhitney]

Great, thanks -- with #96 merged, note that everyone working on backscope, when they next pull from main and end up on top of #96, it's necessary to blow away your entire .venv and recreate it from scratch.

[gwhitney]

Ok have filed #98, the last PR to accomplish the next checkbox in the top message of this discussion.

[gwhitney]

Excellent, another box checked off. Next up is #93, necessary so we can put those new packages into production. This one will be a bit of a pain. Kate, I think you and I are currently the only ones with server credentials and some slice of time to work on this. So ideally one of us would update the scripts, and the other one of us would test them by putting them into production and then beating on the server and making sure it works. But Kate, not sure if you feel comfortable with either of those tasks. If not, the alternative would be for me to write the scripts and file a PR, and then prep to switch over the production server, and then sometime we are both available like at a weekly meeting, I could make the switchover and you (and Aaron if present) could beat on the server to verify all is OK with it. Let me know your thoughts of how to proceed. Thanks.

[gwhitney]

OK, let's try that. It can be a little tricky to tell if a deployment has succeeded (is it the old server running, or the new one?) Should we implement #75 (in addition to #93) before trying the next server deployment?

[katestange]

Oh yes, I'm in favour of doing #75 first, that will just make things easier!

[katestange]

I could try doing #75 if you'd like, but I'm worried I'll just slow you down, you are on a roll. I'm happy to review it quickly.

[gwhitney]

Either way. My colleague Alissa Crans and I just found out we are on a tight production schedule for our book, so i am going to be preoccupied with that for a bit until we get the next revision submitted. So it will likely go faster if you do #75. I recommend a single endpoint that returns the version number and the latest git commit id.

[katestange]

I left a question for you over on #75