WIP docs for Plaid

CHANGES.rst

@@ -24,6 +24,7 @@ Unreleased Changes
 
 * Remove ``convert_unicode`` argument from SQLAlchemy DB engine arguments per SQLAlchemy 1.3 upgrade guide / `SQLAlchemy #4393 <https://github.com/sqlalchemy/sqlalchemy/issues/4393>`_.
 * Numerous updates to fix ``tox`` tests.
+* Implement transaction downloading via `Plaid <https://plaid.com/>`__.
 
 1.0.0 (2018-07-07)
 ------------------

README.rst

@@ -39,7 +39,8 @@ many years using Google Sheets and a handful of scripts to template out budgets
 it's time to just bite the bullet and write something that isn't a pain.
 
 **Intended Audience:** This is decidedly not an end-user application. You should be familiar with Python/Flask/MySQL. If
-you're going to use the automatic transaction download functionality, you should be familiar with `Hashicorp Vault <https://www.vaultproject.io/>`_
+you're going to use the OFX-baseed automatic transaction download functionality (as opposed to Plaid), you should be
+familiar with `Hashicorp Vault <https://www.vaultproject.io/>`_
 and how to run a reasonably secure installation of it. I personally don't recommend running this on anything other than
 your own computer that you physically control, given the sensitivity of the information. I also don't recommend making the
 application available to anything other than localhost, but if you do, you need to be aware of the security implications. This
@@ -62,7 +63,7 @@ Main Features
 
 * Budgeting on a biweekly (fortnightly; every other week) basis, for those of us who are paid that way.
 * Periodic (per-pay-period) or standing budgets.
-* Optional automatic downloading of transactions/statements from your financial institutions and reconciling transactions (bank, credit, and investment accounts).
+* Optional automatic downloading of transactions/statements from your financial institutions via OFX Direct Connect, screen scraping, or `Plaid <https://plaid.com/>`__ and reconciling transactions (bank, credit, and investment accounts).
 * Scheduled transactions - specific date or recurring (date-of-month, or number of times per pay period).
 * Tracking of vehicle fuel fills (fuel log) and graphing of fuel economy.
 * Cost tracking for multiple projects, including bills-of-materials for them. Optional synchronization from Amazon Wishlists to projects.
@@ -79,7 +80,8 @@ Vault (if you choose to take advantage of the OFX downloading), which you can al
 * Python 3.6+ (currently tested with 3.6, 3.7, 3.8 and developed with 3.8). **Python 2 is not supported.**
 * Python `VirtualEnv <http://www.virtualenv.org/>`_ and ``pip`` (recommended installation method; your OS/distribution should have packages for these)
 * MySQL, or a compatible database (e.g. `MariaDB <https://mariadb.org/>`_). biweeklybudget uses `SQLAlchemy <http://www.sqlalchemy.org/>`_ for database abstraction, but currently specifies some MySQL-specific options, and is only tested with MySQL.
-* To use the automated OFX transaction downloading functionality:
+* To use the automated Plaid transaction downloading functionality, a valid `Plaid <https://plaid.com/>`__ account.
+* To use the automated OFX Direct Connect transaction downloading functionality:
 
   * A running, reachable instance of `Hashicorp Vault <https://www.vaultproject.io/>`_ with your financial institution web credentials stored in it.
   * If your bank does not support OFX remote access ("Direct Connect"), you will need to write a custom screen-scraper class using Selenium and a browser.

docs/source/biweeklybudget.plaid_updater.rst

@@ -0,0 +1,7 @@
+biweeklybudget\.plaid\_updater module
+=====================================
+
+.. automodule:: biweeklybudget.plaid_updater
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/source/biweeklybudget.rst

@@ -29,6 +29,7 @@ Submodules
    biweeklybudget.interest
    biweeklybudget.load_data
    biweeklybudget.ofxgetter
+   biweeklybudget.plaid_updater
    biweeklybudget.prime_rate
    biweeklybudget.screenscraper
    biweeklybudget.settings

docs/source/getting_started.rst

@@ -15,7 +15,8 @@ Vault (the latter only if you choose to take advantage of the OFX downloading),
 * Python 3.6+ (currently tested with 3.6, 3.7, 3.8 and developed with 3.8). **Python 2 is not supported.**
 * Python `VirtualEnv <http://www.virtualenv.org/>`_ and ``pip`` (recommended installation method; your OS/distribution should have packages for these)
 * MySQL, or a compatible database (e.g. `MariaDB <https://mariadb.org/>`_ ). biweeklybudget uses `SQLAlchemy <http://www.sqlalchemy.org/>`_ for database abstraction, but currently specifies some MySQL-specific options, and is only tested with MySQL.
-* To use the automated OFX transaction downloading functionality:
+* To use the new :ref:`Plaid <plaid>` automated transaction downloading functionality, a valid Plaid account.
+* To use the (old) automated OFX transaction downloading functionality:
 
   * A running, reachable instance of `Hashicorp Vault <https://www.vaultproject.io/>`_ with your financial institution web credentials stored in it.
   * If your bank does not support OFX remote access ("Direct Connect"), you will need to write a custom screen-scraper class using Selenium and a browser.
@@ -259,6 +260,8 @@ please cut a pull request against ``docker_build.py``.
 Running ofxgetter in Docker
 +++++++++++++++++++++++++++
 
+**Note:** ofxgetter support is tentatively being deprecated. Please see :ref:`Plaid <plaid>` for the tentative replacement.
+
 If you wish to use the :ref:`ofxgetter <ofx>` script inside the Docker container, some special
 settings are needed:
 

docs/source/ofx.rst

@@ -3,6 +3,9 @@
 OFX Transaction Downloading
 ===========================
 
+.. important::
+   OFX support is tentatively being deprecated. Please see :ref:`Plaid <plaid>` for the tentative replacement.
+
 biweeklybudget has the ability to download OFX transaction data from your
 financial institutions, either manually or automatically (via an external
 command scheduler such as ``cron``).

docs/source/plaid.rst

@@ -3,16 +3,45 @@
 Plaid
 =====
 
-Testing (Sandbox)
------------------
+`Plaid <https://plaid.com/>`__ is a company that provides API-based tools and solutions for interfacing with financial institutions. Among their products is a transaction API that handles authentication with financial instituions and provides ReST access to transaction and balance information. As of March 2020, they provide free developer accounts with access to up to 100 "items" (financial institution accounts), and production accounts with a pay-as-you-go pricing model that is quite affordable.
 
-.. code-block:: bash
+With many of my banks and credit card companies discontinuing support for OFX Direct Connect citing security concerns (mainly that most of them use OFX 1.x implementations that require your normal credentials in cleartext), or breaking the OFX protocol in ways that only Intuit seems to support, I needed an alternative to the pain of writing error-prone screen scrapers. Plaid seems to be this solution, is reasonably priced (or free for initial testing with a developer account), and supports all of my accounts. As such, I'm going to be discontinuing development on the :ref:`ofx` component and focusing on Plaid for the future. Plaid provides a simple, easy, secure way to retrieve balance and transaction information for accounts.
 
-    export PLAID_CLIENT_ID=5cf090726590ed001352c268
-    export PLAID_SECRET=4265c2b9575665adff6dff6843ec82
-    export PLAID_PUBLIC_KEY=c248f0f1788d692421ccb7a1df2126
-    export PLAID_PRODUCTS=transactions
-    export PLAID_COUNTRY_CODES=US
-    export PLAID_ENV=sandbox
+The Plaid component uses the same "OFX" models in biweeklybudget that the older OFX Direct Connect component used. Transactions retrieved via Plaid will still show up on the "OFX Transactions" view, and reconciling works the same way. Both Plaid and OFX write their data into the same models and database tables.
 
-In sandbox, there will be one and only one fake account, a bank account. The credentials for successfully authenticating are a username of ``user_good`` and a password of ``pass_good``.
+Configuration
+-------------
+
+The first step to using the Plaid transaction support is registering for an account with `Plaid <https://plaid.com/>`__. As that process may change, I recommend using their site to find out how. As of this writing, there is a large "Get API keys" button on the homepage to get your started. You will initially be given testing / sandbox keys, which can only retrieve information about fake accounts that Plaid maintains (though it is realistic data, and can be used to see how biweeklybudget integrates with Plaid). In order to retrieve your real data, you will need to request Development access. Once you have that, you can proceed.
+
+biweeklybudget needs to be configured with your Plaid credentials. I highly recommend setting these as environment variables rather than hard-coding them in your settings file (if you use one). You will need to run biweeklybudget with the following Plaid-related environment variables / settings:
+
+* ``PLAID_CLIENT_ID`` - Your Client ID credential, provided by Plaid.
+* ``PLAID_SECRET`` - Your Secret credential, provided by Plaid.
+* ``PLAID_PUBLIC_KEY`` - Your Public Key credential, provided by Plaid.
+* ``PLAID_ENV`` - The Plaid environment name to connect to. This must be one of "sandbox", "development", or "production", and must match the environment that your credentials are for.
+* ``PLAID_PRODUCTS`` - The Plaid products you are requesting access to. Right now, for biweeklybudget, this should be ``transactions``.
+* ``PLAID_COUNTRY_CODES`` - A comma-separated list of country codes that you want to be able to select institutions from. Only ``US`` has been tested.
+
+Usage
+-----
+
+Linking Accounts to Plaid
++++++++++++++++++++++++++
+
+To link an account via Plaid, open the Account modal (by clicking on any link to the account, such as those on the ``/accounts`` view) and scroll down to the bottom.
+
+Updating Transactions via UI
+++++++++++++++++++++++++++++
+
+TBD.
+
+Updating Transactions via API
++++++++++++++++++++++++++++++
+
+TBD.
+
+Troubleshooting
+---------------
+
+API responses from Plaid are logged at debug-level. The UI process of linking an account via Plaid happens mostly in client-side JavaScript, which logs pertinent information to the browser's console log. The Plaid Dashboard (accessed by logging in to plaid.com) also provides some useful debug information.