"Down for maintenance" feature

The AuthP's multi-tenant code contains features to update, delete, move (hierarchical only) and move database (sharding), which I refer to as change / move. All of these changes / moves are done while the application is running, which means that a user could alter the tenant's data during a change / move. This could cause problems to the change / move, of which the worse could be corruption of the tenant data! Also, user reading a tenant during a change might contain invalid data. In version 4.3.0 of the AuthP library a feature called "Down for maintenance" feature (sometimes shortened to "down") was added. This document explains how this feature works and what you need to do to use this optional feature.

What the "Down for maintenance" feature does?

NOTE There is an article called "How to change / move databases on a live ASP.NET Core web site" which covers this topic in more detail.

The "Down for maintenance" feature contains a way to divert a user's HTTP request to a "please wait" type page, if the data they want to access is set as "down". The figure below shows the case where "tenant 123" is changed / moved within "Down" section which will divert any users linked to "tenant 123" to a "please wait" page, while users not linked to “tenant 123” would work normally.

The four types of "downs" are described below. The first two are triggered by code, while the last two are manually applied via

Change/move tenant database “down”

This is triggered by the ISetRemoveStatus service SetTenantDownWithDelayAsync before a tenant move is run and released when the move has finished. Any tenant users linked to the tenant being moved will diverted to a "please wait" page while the "down" is in place.

NOTE: if there is a exception in the move command, then the tenant "down" will still be in place and must be manually removed via the Status controller. This gives an admin user to check the tenant data is OK before releasing the "down".

Tenant database Delete

This permanently diverts all users linked to the deleted tenant to a page saying, “the tenant is deleted”. This is a permanent divert triggered by the ISetRemoveStatus service SetTenantDownWithDelayAsync, but can only be removed manually.

In AuthP you can't delete a tenant until all of its users are deleted, but an already logged-in tenant user could try to access the tenant. This "down" version makes sure an already logged-in tenant user is caught.

Manual, tenant database “down”:

Allows an admin user to manually “down” a tenant database via a page/Web API (in the example code, a StatusController) and diverting all users linked to “downed” tenant database to a page saying, “stopped by admin”. Access to the tenant can be restored by an admin manually removing this this “down”.

Manual, application “down”

This allows an admin user to manually “down” the whole application via a page/Web API (in the example code, a StatusController). Every user apart from the admin who took the app “down” will be diverted to a page with an explanation and expected time when the app will be available.

Adding the "down to maintenance" feature to your application

There are three parts to adding "down to maintenance" feature to your application:

1. Startup: Registering the services / middleware
2. Adding a StatusController (or an equivalent Web API)
3. Using the ISetRemoveStatus service to set / remove a “down” state

1. Startup: Registering the services / middleware

There are two parts to setup the register the “down for maintenance” feature:

• Registering the “down for maintenance” services
• Adding the “down for maintenance” middleware.

_NOTE: See Example4's Program's file (hierarchical tenant design) or Example6's Program's file (single-level + sharding) for examples of the setup.

Both parts are applied in the ASP.NET Core Program / Startup code. First is the registering of the FileStore cache, which holds the various “down” statuses, and the ISetRemoveStatus service, which provide simple methods to add / remove “down” statuses. The code below is added in the startup section that registers services with the .NET dependency injection provider.

builder.Services.AddDistributedFileStoreCache(options =>
{
    options.WhichVersion = FileStoreCacheVersions.Class;
}, builder.Environment);

builder.Services.AddTransient<ISetRemoveStatus, SetRemoveStatus>(); 

The “down for maintenance” middleware is added in the “app” part of the ASP.NET Core startup code – see the highlighted line that adds the extra middleware.

var app = builder.Build();
//other app code left out

app.UseAuthentication();
app.UseAuthorization();
app.UseDownForMaintenance();

//other code left out

2. Adding a StatusController (or an equivalent Web API)

You need pages / APIs to handle the following:

• For the admin users
  • Look at all the current “downs” and have the ability to remove any
  • Manually set the app “down” (with messages for the users)
  • Manually set a tenant “down”
• For diverted users
  • Tenant down while being updated
  • Tenant down by admin
  • Tenant is deleted
  • App Down

In the Example4 web site (hierarchical tenant design) and Example6 web site (single-level + sharding) I have a controller called StatusController that contains the actions / pages listed above. Please look at the Example4's StatusController for an example of what you need to create.

NOTE: the diverted pages are hard coded into the RedirectUsersViaStatusData class, while the controller’s name can be changed. If you want to have different urls for the diverted pages, then you need to copy the code and register your version of the RedirectUsersViaStatusData class. See the Understanding / changing the “down for maintenance” middleware section for more on this.

3. Using the ISetRemoveStatus service to set / remove a “down” state

The SetRemoveStatus class contains the code to set, remove and display the “down” statues in the FileStore distributed cache. This service creates the cache key which defines the type of divert that the user should be diverted to.

The AppDown divert has its own method called SetAppDown which takes in the ManuelAppDownDto class. This allows the admin user's UserId (which means that user won't be diverted) and a Message and ExpectedTimeDownMinutes properties that the admin user can fill in to let the users what is going on and when the application will be back up. It's also up to the admin user to remove the AppDown divert by selecting that divert shown in list of all diverts provided by the StatusController Index.

For tenant diverts there is a method called SetTenantDownWithDelayAsync which takes various parameters to create the correctly named "down" entry in the FileStore database. It also returns the code to remove the "down" entry after the tenant change has finished - see the code taken from Example6's TenantController

[HttpPost]
[ValidateAntiForgeryToken]
[HasPermission(Example6Permissions.MoveTenantDatabase)]
public async Task<IActionResult> MoveDatabase(ShardingSingleLevelTenantDto input)
{
    //This a) sets the tenant down entry and delays (default: 100 ms) to make sure
    //already running requests have stopped before returning.
    //It also returns the Func to run after the change has finished. 
    //This Func will remove the tenant down entry. 
    var removeDownAsync = await _upDownService
        .SetTenantDownWithDelayAsync(TenantDownVersions.Update, input.TenantId);
    var status = await _authTenantAdmin.MoveToDifferentDatabaseAsync(
        input.TenantId, input.HasOwnDb, input.ConnectionName);
    //This removes the tenant down entry
    await removeDownAsync();

    return status.HasErrors
        ? RedirectToAction(nameof(ErrorDisplay),
            new { errorMessage = status.GetAllErrors() })
        : RedirectToAction(nameof(Index), new { message = status.Message });
}

A couple of the tenant downs are slightly different from others.

• Tenant Delete: In this case you don't don't remove the tenant down after the delete unless there is a error.
• Hierarchical Move : The hierarchical move needs to lock two parts of the tenant data - the data to be moved and the tenant the data is moved too. The SetTenantDownWithDelayAsync method has a optional parameter called parentId which set up another tenant "down" entry. See the [Example4's TenantController](https://github.com/JonPSmith/AuthPermissions.AspNetCore/blob/main/Example4.MvcWebApp.IndividualAccounts/Controllers/TenantController.cs) and look at the Post Move` action.

Understanding / changing the “down for maintenance” middleware

The middleware code is in the RedirectUsersViaStatusData class is called on every HTTP request. Its job is to quickly let through a user if there isn’t an “down” status that effects the current user.

There are three stages in this middleware to cover each part of the filter. They are:

Additional resources

• Single level Multi Tenant
• Hierarchical Multi Tenant
• Sharding: Move Tenant to database