Add migration story for LDAP UUID home directories

docs/migrations.rst

@@ -5,10 +5,107 @@ With mail server configuration best practices changing over time we might need
 to make changes that require you to complete manual migration steps before you
 can deploy a new version of NixOS mailserver.
 
-The initial `mailserver.stateVersion` value should be copied from the setup
+The initial :option:`mailserver.stateVersion` value should be copied from the
-guide that you used to initially set up your mail server. If in doubt you can
+setup guide that you used to initially set up your mail server. If in doubt you
-always initialize it at `1` and walk through all assertions, that might apply
+can always initialize it at ``1`` and walk through all assertions, that might
-to your setup.
+apply to your setup.
+
+NixOS 26.05
+-----------
+
+#4 Dovecot LDAP UUID-based home directories
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+LDAP Support in NixOS mailserver was introduced during the 23.11 release cycle
+and came with a number of flaws that we are correcting now, three years later.
+
+This particular migration is needed because up until now we were
+relying on email addresses to construct the Dovecot home directory path
+(``var/vmail/ldap/user@example.com``) which is fragile: addresses can
+change, requiring manual homedir relocation. Switching to UUID-based homedirs
+(``/var/vmail/ldap/<uuid>``) ensures stable, unique paths and applies well-known
+best practices to mailserver management.
+
+1. Copy the migration script script to your mailserver and make it executable:
+
+   .. code-block:: console
+
+      cd /tmp
+      wcurl https://gitlab.com/simple-nixos-mailserver/nixos-mailserver/-/blob/master/migrations/nixos-mailserver-migration-04.py
+      chmod +x nixos-mailserver-migration-04.py
+
+2. Stop the ``dovecot.service``.
+
+   .. code-block:: bash
+
+      systemctl stop dovecot.service
+
+3. Create a backup or snapshot of your :option:`mailserver.mailDirectory`, so
+   you can restore should anything go wrong.
+
+4. Run the migration script and pass the required arguments to enable LDAP lookups:
+
+   The script should be run under the user who owns the :option:`mailserver.mailDirectory`.
+   If run as root it will automatically switch into the appropriate user by itself.
+
+   The script will not modify your data unless called with ``--execute``.
+
+   The migration script finds all Dovecot home directories in
+   ``/var/vmail/ldap/`` (or any other :option:`mailserver.mailDirectory`),
+   for example that of bob at ``/var/vmail/ldap/bob@example.com``.
+   It then takes ``bob@example.com`` and queries the LDAP server for
+   ``mail=bob@example.com`` to retrieve the UUID attribute. Finally
+   it starts suggesting the neceessary move operations to arrive at
+   ``/var/vmail/ldap/f3b4e8ea-087f-42cc-95f0-cbfd99386092`` for bob.
+
+   Example:
+
+   .. code-block:: bash
+
+      ./nixos-mailserver-migration-04.py \
+        --ldap-uri ldaps://ldap1.example.com
+        --ldap-bind-dn cn=mail,ou=accounts,dc=example,dc=com \
+        --ldap-bind-pw-file /run/keys/ldap-bind-pw \
+        --ldap-base ou=people,ou=accounts,dc=example,dc=com \
+        --ldap-scope sub \
+        --ldap-filter "(mail=%s)" \
+        --ldap-attr-uuid entryUUID \
+         /var/vmail
+
+   For the ``--ldap-attr-uuid`` parameter we expect a long-term stable
+   identifier, ideally a UUID field. The exact attribute name depends on your
+   LDAP implementation, for example:
+
+   - Authentik: ``uid`` `[1]`_
+   - Kanidm: ``uuid`` `[2]`_
+   - Keycloak ``entryUUID``
+   - OpenLDAP: ``entryUUID`` (`RFC4530`_)
+
+   If yours LDAP provider isn't listed you can determine the correct
+   attribute by quering a user entry with ``ldapsearch``. Finally, configure
+   :option:`mailserver.ldap.attributes.uuid` accordingly.
+
+   Add ``--ldap-starttls`` if you use the the `ldap://` URI scheme and require
+   explicit TLS.
+
+   .. _[1]: https://docs.goauthentik.io/add-secure-apps/providers/ldap#users
+   .. _[2]: https://kanidm.github.io/kanidm/stable/integrations/ldap.html#data-mapping
+   .. _RFC4530: https://www.rfc-editor.org/rfc/rfc4530.html
+
+5. Review the script output.
+
+   It's primary job is to determine the UUID for an LDAP account, so that it
+   can rename the Dovecot home directory from mail address to UUID within the
+   same directory.
+
+   The script can highlight various inconsistencies and problems, that should
+   be reviewed and acted upon.
+
+   If in doubt, join our community chat for help before applying any changes.
+
+6. Rerun the command with ``--execute`` or run the proposed commands manually.
+
+7. Update the ``mailserver.stateVersion`` to ``4``.
 
 NixOS 25.11
 -----------

docs/release-notes.rst

@@ -22,6 +22,9 @@ NixOS 26.05
   established by `agenix`_/`sops-nix`_ that instead rely on encryption. This
   option prevents files from leaking in to the Nix store.
   See :option:`mailserver.loginAccounts.<name>.passwordFile`.
+- LDAP setups require a migration of Dovecot home directories to
+  `UUID based home directories`_. The exact UUID attribute can be customized
+  through :option:`mailserver.ldap.attributes.uuid`.
 - The default login username for LDAP users has changed from the ``mail`` to
   the ``uid`` attribute. This allows users to login with their account name
   rather than their email address, which is more convenient and consistent
@@ -38,6 +41,7 @@ NixOS 26.05
 .. _DKIM key management: dkim.html
 .. _agenix: https://github.com/ryantm/agenix
 .. _sops-nix: https://github.com/Mic92/sops-nix
+.. _UUID based home directories: migrations.html#dovecot-ldap-uuid-based-home-directories
 
 NixOS 25.11
 -----------

mail-server/assertions.nix

@@ -134,5 +134,15 @@ in
         '';
       }
     ]
+    ++ lib.optionals (config.mailserver.ldap.enable) [
+      {
+        assertion = config.mailserver.stateVersion != null -> config.mailserver.stateVersion >= 4;
+        message = ''
+          NixOS Mailserver requires migrating LDAP home directories to UUID scheme
+
+          Check https://nixos-mailserver.readthedocs.io/en/latest/migrations.html#dovecot-ldap-uuid-based-home-directory for required migration steps.
+        '';
+      }
+    ]
   );
 }

migrations/nixos-mailserver-migration-04.py

@@ -0,0 +1,346 @@
+#!/usr/bin/env nix-shell
+#!nix-shell -i python3 -p "python3.withPackages (ps: with ps; [ ldap3 ])"
+
+import argparse
+import os
+import sys
+from pathlib import Path
+from pwd import getpwnam
+from typing import Literal, cast
+
+from ldap3 import BASE, LEVEL, SUBTREE, Connection, Server
+from ldap3.core.exceptions import LDAPException
+
+LDAPSearchScope = Literal["BASE", "LEVEL", "SUBTREE"]
+
+EXIT_OK = 0
+EXIT_ERROR = 1
+EXIT_LDAP_STARTTLS = 2
+EXIT_LDAP_BIND = 3
+
+GREEN = "32"
+YELLOW = "33"
+RED = "31"
+BOLD = "1"
+
+NO_COLOR = "NO_COLOR" in os.environ
+
+
+def color(text, code):
+    if NO_COLOR:
+        return text
+    return f"\033[{code}m{text}\033[0m"
+
+
+def check_user(vmail_root: Path):
+    owner = vmail_root.owner()
+    owner_uid = getpwnam(owner).pw_uid
+
+    if os.geteuid() == owner_uid:
+        return
+
+    try:
+        print(f"Trying to switch effective user id to {owner_uid} ({owner})")
+        os.seteuid(owner_uid)
+        return
+    except PermissionError:
+        print(
+            f"Failed switching to virtual mail user. Please run this script under it, for example by using `sudo -u {owner}`)"
+        )
+    sys.exit(1)
+
+
+def move(*, src: Path, dst: Path, dry_run: bool = True) -> bool:
+    print(f'mv "{src}" "{dst}"')
+    if not dry_run:
+        try:
+            src.rename(dst)
+        except OSError as exc:
+            print(f"Rename failed ({src=!s}, {dst=!s}): {exc}")
+            return False
+    return True
+
+
+def main(
+    *,
+    vmail_root: Path,
+    ldap_uri: str,
+    ldap_starttls: bool,
+    ldap_bind_dn: str,
+    ldap_bind_pw: str,
+    ldap_base: str,
+    ldap_scope: LDAPSearchScope,
+    ldap_filter: str,
+    ldap_attr_uuid: str,
+    dry_run: bool = True,
+    verbose: bool = False,
+):
+    # Begin with LDAP connection for fast feedback
+    server = Server(ldap_uri)
+    conn = Connection(server, ldap_bind_dn, ldap_bind_pw)
+
+    if ldap_starttls:
+        try:
+            if ldap_starttls:
+                conn.start_tls()
+        except LDAPException as exc:
+            print(color(f"LDAP connection setup failed: {exc!r}", RED))
+            sys.exit(EXIT_LDAP_STARTTLS)
+
+    if not conn.bind():
+        err = conn.result
+        print(
+            color(
+                f"""
+LDAP bind failed for {ldap_bind_dn}@{ldap_uri}
+Result: {err.get("result")} ({err.get("description")})
+Message: {err.get("message")!r}""",
+                RED,
+            )
+        )
+        sys.exit(EXIT_LDAP_BIND)
+
+    # Find existing dovecot home directories and collect account identifier
+    print(
+        color(
+            f"\nEnumerate accounts based on existing home directories in {(vmail_root / 'ldap')!s}",
+            BOLD,
+        )
+    )
+
+    skipped = 0
+    accounts = set()
+    homedirs = vmail_root.glob("ldap/*")
+    for path in homedirs:
+        if not path.is_dir():
+            print(f"- Not a directory ({path=!s}) (skipping)")
+            skipped += 1
+            continue
+        elif not (path / "mail").is_dir():
+            print(f"- No maildir in home ({path=!s}) (skipping)")
+            skipped += 1
+            continue
+
+        account = path.name
+        accounts.add(account)
+        if verbose:
+            print(f"- Home directory found ({path=!s}, {account=})")
+
+    print(
+        color(
+            f"\nFinding matching LDAP entries to retrieve `{ldap_attr_uuid}` attribute",
+            BOLD,
+        )
+    )
+
+    no_entry = 0
+    multiple_entries = 0
+    plan = {}
+    for account in sorted(accounts):
+        filter = ldap_filter % account
+        conn.search(
+            search_base=ldap_base,
+            search_filter=filter,
+            search_scope=ldap_scope,
+            attributes=[ldap_attr_uuid],
+        )
+
+        if conn.response is None:
+            print(f"- LDAP search produced no result for {filter}")
+
+        count = len(conn.entries)
+
+        if count < 1:
+            print(f"- No LDAP entry found ({account=}, {filter=}) (skipping)")
+            no_entry += 1
+            continue
+        elif count > 1:
+            print(f"- Multiple LDAP entries found ({account=}, {filter=}) (skipping)")
+            multiple_entries += 1
+            continue
+        else:
+            entry = conn.entries[0]
+            uuid = str(entry[ldap_attr_uuid].value)
+            if verbose:
+                print(f"- LDAP entry mapped ({account=}, {uuid=})")
+            plan.update({account: uuid})
+
+    print(color("\nThe following operations will be executed:", BOLD))
+    moved = 0
+    moves_failed = 0
+    for src, dst in plan.items():
+        _src = vmail_root / "ldap" / src
+        _dst = vmail_root / "ldap" / dst
+        if not move(src=_src, dst=_dst, dry_run=dry_run):
+            moves_failed += 1
+        else:
+            moved += 1
+
+    print(
+        color(
+            "\nMigration summary",
+            BOLD,
+        )
+    )
+
+    if any([skipped, no_entry, multiple_entries, not accounts, moves_failed]):
+        print("""
+We strongly recommend reviewing and remediating all potential issues before
+running with `--execute`. Specific details can be found further up.""")
+
+    if moved:
+        print(f"""
+- {color(f"{moved} home directories were migrated succesfully.", GREEN)} {"(dry run)" if dry_run else ""}
+  This is great news, they are now UUID-based and will be immune to username changes!""")
+
+    if skipped and accounts:
+        print(f"""
+- {color(f"{skipped} paths in {(vmail_root / 'ldap')!s} were skipped.", YELLOW)}
+  These were not a directory or did not contain a maildir. They should be
+  reviewed but can most likely be deleted.""")
+
+    if no_entry:
+        print(f"""
+- {color(f"{no_entry} LDAP queries found no entry.", YELLOW)}
+  This could be a problem, because we cannot migrate home directories without
+  finding the LDAP entry and retrieving its {ldap_attr_uuid} field. In practice
+  this can happen if an LDAP account was deleted but its mail home directory
+  remained.""")
+
+    if multiple_entries:
+        print(f"""
+- {color(f"{multiple_entries} LDAP queries returned multiple entries.", RED)}
+  This is a problem, because we cannot decide which LDAP entry owns the home
+  directory.""")
+
+    if not accounts:
+        print(f"""
+- {color("No home directories were found.", RED)}
+  Make sure you are passing the correct `vmail_root` argument. It must match
+  your `mailserver.mailDirectory` setting.""")
+
+    if moves_failed:
+        print(f"""
+- {color("{moves_failed} home directores could not be renamed", RED)}
+  No reason to panic, but the script tried to rename a home directory and that
+  triggered and error. Check further up what went wrong.""")
+
+    if dry_run:
+        print(f"\n{color('No changes were made.', YELLOW)}")
+        print("Run the script with `--execute` to apply the listed changes.")
+
+    sys.exit(EXIT_OK if moves_failed == 0 else EXIT_ERROR)
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="""
+        NixOS Mailserver Migration #4: Dovecot LDAP UUID-based home directories
+        (https://nixos-mailserver.readthedocs.io/en/latest/migrations.html#dovecot-ldap-uuid-based-home-directory)
+        """
+    )
+    parser.add_argument(
+        "vmail_root", type=Path, help="Path to the `mailserver.mailDirectory`"
+    )
+    parser.add_argument(
+        "--ldap-uri",
+        type=str,
+        required=True,
+        help="URI for your LDAP server; ldaps://ldap1.example.com (TLS) or ldap://ldap1.exampe.com (Plain)",
+    )
+    parser.add_argument(
+        "--ldap-starttls",
+        action="store_true",
+        help="Enable StartTLS on plain LDAP connections",
+    )
+    parser.add_argument(
+        "--ldap-bind-dn",
+        type=str,
+        required=True,
+        help="The distinguished user allow to bind and search the LDAP server",
+    )
+    parser.add_argument(
+        "--ldap-bind-pw-file",
+        type=Path,
+        required=True,
+        help="Path to a file containing the bind password for the LDAP DN",
+    )
+    parser.add_argument(
+        "--ldap-base",
+        type=str,
+        required=True,
+        help="Base DN below which to search for LDAP accounts",
+    )
+    parser.add_argument(
+        "--ldap-scope",
+        choices=[
+            "sub",
+            "base",
+            "one",
+        ],
+        default="sub",
+        help="Scope relative to the base DN",
+    )
+    parser.add_argument(
+        "--ldap-filter",
+        default="(mail=%s)",
+        help="LDAP query that filters for an account by the name in /var/vmail/ldap/<name> field, e.g. mail=%%s or uid=%%s if the name is not an email adress.",
+    )
+    parser.add_argument(
+        "--ldap-attr-uuid",
+        default="entryUUID",
+        help="UUID attribute that uniquely identifies an LDAP account across login name changes",
+    )
+    parser.add_argument(
+        "--execute", action="store_true", help="Actually perform changes"
+    )
+    parser.add_argument("--verbose", action="store_true", help="Print more details")
+
+    args = parser.parse_args()
+
+    if args.ldap_filter.count("%s") != 1:
+        print(
+            "The --ldap-filter argument must contain exactly one '%s' as a placeholder for the primary email address.",
+        )
+        sys.exit(1)
+
+    def read_ldap_bind_pw():
+        try:
+            with open(args.ldap_bind_pw_file) as fd:
+                return fd.read().strip()
+        except OSError as exc:
+            print(f"Unable to read LDAP bind password file: {exc}")
+            sys.exit(1)
+
+    ldap_bind_pw = None
+    if os.geteuid() == 0:
+        # if we're root, read before priv drop
+        ldap_bind_pw = read_ldap_bind_pw()
+
+    check_user(args.vmail_root)
+
+    if ldap_bind_pw is None:
+        ldap_bind_pw = read_ldap_bind_pw()
+
+    ldap_scope: LDAPSearchScope = cast(
+        LDAPSearchScope,
+        {
+            "sub": SUBTREE,
+            "base": BASE,
+            "one": LEVEL,
+        }[args.ldap_scope],
+    )
+
+    main(
+        vmail_root=args.vmail_root,
+        ldap_uri=args.ldap_uri,
+        ldap_starttls=args.ldap_starttls,
+        ldap_bind_dn=args.ldap_bind_dn,
+        ldap_bind_pw=ldap_bind_pw,
+        ldap_base=args.ldap_base,
+        ldap_scope=ldap_scope,
+        ldap_filter=args.ldap_filter,
+        ldap_attr_uuid=args.ldap_attr_uuid,
+        dry_run=not args.execute,
+        verbose=args.verbose,
+    )