new features: username formats, user_id lookup, application column / docs

[listerr]

• Added/updated documentation.
• New examples / config for dovecot: README_DOVECOT.md

NEW FEATURES:

• Support a variety of username formats, or same username.
• ap4rc_strict_userid_lookup: Lookup user by id / username.
• ap4rc_show_application: show/hide application column.

FIXES:

• Fix php warning "undefined variable $out"
• Various indentation / spacing

[jdsn]

Hi Rob,
thanks for the big PR.

As it combines quite some different changes, I'd ask if you would mind if I just cherry-picked some of them to get them merged quickly?

Background:
I still have to twist my head around the username-format feature. While our original code that defined our special use-case of the username format was obviously meant to be changed in the future (because it was moved to a generator function) it kind of worries me that this PR would now raise this to 4 hard coded "special use-cases".

My current thought is, that I'd prefer a really dynamic format definition and get our and your special use-cases out of the code and rather move them to be configuration data.

For another instance that I am running, I have a similar need - so lets work together on this.

[listerr]

Sure!

To be honest, 3 + 4 are mostly experimental anyway as I was looking for different ways to get it to work with Dovecot, which has a very limited ability to extract substrings of usernames. (e.g. it can only seem to extract fixed widths, or from a given position to the end etc.) but not really any "split everything before/after/between this character" or regex match/extract capability.

https://doc.dovecot.org/configuration_manual/config_file/config_variables

Doing this, I have learned the following:

1. Dovecot's authentication config does not work in the way you might think it does.
2. The documentation doesn't explain it very well.
3. Here is lots of rope to hang yourself with.
4. Dovecot is probably the best of them though...

Nice idea to make it a "template" config format. Though it would then be possible to configure something that dovecot actually cannot match, and Dovecot's documentation is very lacking examples in this area. We could provide a few examples and matching dovecot config snippets though.

If you are not very, very careful with dovecot's auth config, It's possible to configure something which "works", and looks like a nice idea, but is insecure or otherwise broken (e.g. any user's password is accepted for any other user. Or any password is accepted for ANY known user. Whoops.) Caveat emptor!

It seems the modern "convention" is that most email clients expect the username to be an email address, and try to "auto-discover" the correct settings using a variety of methods and/or just guessing from what it finds in the DNS, and trying different port numbers etc. until a login succeeds.

[darix]

well if you want we can provide you the details we use for our setups

IIRC we extra the username from the username@appname syntax using the auth_username_format option set to %Lu

the username@appname allows us easily to have multiple app passwords per user and easily map it back to the original user with the auth_username_format.

[listerr]

In my case, the usernames are email addresses. I already have auth_username_format option set to %Lu (seems to be the default anyway).

The usernames end up looking like:-

[email protected]@application

In which case, what happens is that "%u" gets set to the entire username [email protected]@application and the
"domain" i.e. "%d" is set to example.com@application, so the SQL query won't match.

auth_username_format is applied before any auth config is run, so changing anything there doesn't seem to help (e.g, if we set it to %Ln, this strips off the domain part (everything before the first "@") permanently before we get to see it.).

You can do something like:

auth_username_format = %Ln@%{domain_first}

Which turns [email protected]@application back to [email protected], then use %{original_domain} to get example.com@application, but I couldn't see a way to get just "application". So we're back where we started!

The docs don't suggest you can do much else here, other than permanently strip or add the domain part and do some simple translations.

I guess in the original ap4rc username format, %n represents the username, and %d is the application name.

I just tested with the original ap4rc code and username format. Presumably the correct query should use %n:

password_query = SELECT username,password \
  FROM application_passwords \
   WHERE username='%n' AND application='%d' \
   AND created >= NOW() - INTERVAL 2 MONTH;

It uses %u in the example, which I can't get to work. %u contains user@application, but we want to search the username field for the username without the domain. Unless in your config, something is stripping the "domain" off before we get to the passdb lookup, so that %u contains only the username. Hmmmmm.

well if you want we can provide you the details we use for our setups

IIRC we extra the username from the username@appname syntax using the auth_username_format option set to %Lu

the username@appname allows us easily to have multiple app passwords per user and easily map it back to the original user with the auth_username_format.

I'd be interested to know if you've done anything interesting with auth_username_format / SQL query.