KCM server for SSSD

Problem statement

Over time, both libkrb5 and SSSD used different credential cache types to store Kerberos credentials - going from a simple file-based storage (FILE:) to a directory (DIR:) and most recently a kernel-keyring based cache (KEYRING:).

Each of these caches has its own set of advantages and disadvantages. The FILE ccache is very widely supported, but does not support multiple primary caches. The DIR cache does, but creating and managing the directories including proper access control can be tricky. The KEYRING cache is not well suited for cases where multiple semi-isolated environments might share the same kernel. Managing credential caches’ lifetime is not well solved in neither of these cache types automatically, only with the help of a daemon like SSSD.

With KCM, the Kerberos caches are not stored in a “passive” store, but managed by a daemon. In this setup, the Kerberos library (typically used through an application, like for example, kinit) is a KCM client and the daemon is being referred to as a KCM server.

Having the Kerberos credential caches managed by a daemon has several advantages:

  • the daemon is stateful and can perform tasks like Kerberos credential cache renewals or reaping old ccaches. Some tasks, like renewals are possible already with SSSD, but only for tickets that SSSD itself acquired (typically via a login through pam_sss.so) and tracks. Tickets acquired otherwise, most notably though kinit wouldn’t be tracked and renewed.
  • since the process runs in userspace, it is subject to UID namespacing, unlike the kernel keyring
  • unlike the kernel keyring-based cache, which is entirely dependant on UIDs of the caller and in a containerized environment is shared between all containers, the KCM server’s entry point is a UNIX socket which can be bind-mounted to only some containers

At the moment, only the Heimdal implementation of Kerberos contains a KCM server. This design page describes adding a KCM server to SSSD as a new SSSD service called sssd_kcm.

Use cases

The primary use-cases for the next SSSD upstream releases are related to containers. In particular:

  • A sysadmin needs to deploy applications in containers without worrying about applications clobbering each other’s credential caches in a kernel keyring as keyrings are not namespaced
  • An Administrator wants to initialize and ccache centrally and then have the ccache available in all relevant containerized applications, so that applications do not have deal with Kerberos authentication or have access to keytab separately
Having the KCM service might also enable us to solve tickets such as:

Overview of the solution

A new SSSD responder is added as part of this feature. While it’s of course possible to create a completely standalone daemon that would implement a KCM server, doing so in the context of SSSD has some advantages, notably:

  • We can reuse a lot of code to set up and configure the server, or parse Kerberos related data
  • SSSD already has a D-Bus API that could publish information about Kerberos tickets and for example emit signals that a graphical application can consume
  • SSSD ships with a “secrets responder” to store data at rest. It makes sense to leverage this component to store Kerberos ccaches persistently

The KCM responder is socket-activated. It should be possible to use the responder without SSSD being configured, although this feature depends on the platform - in general this would only be possible with distributions that enable the files provider. On older distributions (such as RHEL-7), an SSSD domain must be enabled in order for SSSD to even generate the configuration.

The identity of the ccache owner is read from the client socket connection. However, because in containerized environments, the same ID can often identify multiple apps, it is important to also track the SELinux label as part of the client identity. This would prevent two containers running with the same ID but different SELinux label to access each other’s credentials.

Differences between sssd-kcm and Heimdal KCM

Heimdal treats the root user special in the sense that listing all ccaches as root lists everyone’s ccaches. With sssd-kcm, this is not implemented.

Implementation details

The KCM responder implements the same subset of the KCM protocol the MIT client libraries implement. Contrary to Heimdal’s KCM server that just stores the credential caches in memory, the SSSD KCM server would store the ccaches in the secrets database through the sssd-secret’s responder public rest API.

For testing, an in-memory ccache storage, similar to Heimdal’s was also implemented, although this in-memory back end is not documented and should not be used in production. If the persistent storage in sssd-secrets (which is the default) is used, the KCM responder is allowed to idle-terminate.

For user credentials the KCM Server uses a secrets responder URI in the form of http://localhost/kcm/persistent/$uid where $uid is the user ID of the user whose credentials are being saved. In order to impersonate different users, the KCM responder runs as a trusted user (defined as KCM_PEER_UID, defaulting to root) and only this trusted user can access the /kcm hive in the sssd-secrets responder.

The peer identity consists of an UID/GID tuple that is read from the client socket using getsockopt(SO_PEERCRED) and SELinux label using SELINUX_getpeercon(). To evaluate whether the MCS category the peer is running with can access the ccache potentially created with a different category, we’ll call selinux_check_access().

Since from the point of view of the KCM responder, the operations on the Kerberos caches should be seemingly atomic, but often the operations might require several round-trips to the secrets storage, all operations towards the KCM responder by a single UID are serialized.

Internally in the secrets responder, the ccaches are stored at a new top-level anchor cn=kcm. The secret responder’s quotas on secrets also apply separately to the cn=kcm tree; separately here means that it is allowed to store max_secrets secrets and at the same time max_secrets credential caches. There is a separate ticket to make the quotas per-UID.

Currently, when the quota is reached we just fail. We should consider recovering more gracefully, such as by removing the oldest service (non-TGT) tickets.

Since the secrets responder is a key-value store at heart, but the ccaches can be addressed by both name and UID, the key of the secrets store (the secret’s name) is a concatenation of the ccache’s name and UUID. The value (the secret) is a JSON object in the following format:

{
    version: number
    kdc_offset: number
    principal : {
        "type": "number",
        "realm": "string",
        "components": [ "elem1", "elem2", ...]
    }

    creds : [
        {
           "uuid": <data>,
           "payload": <data>,
        }
        {
           ...
        }
        ...
    ]
}

All the credentials are stored in the per-UID secrets container under a container named ccaches. This container is crated when a ccache is initialized by a KCM client. There is also a secret named default which contains (as secrets value) the UUID of the default ccache, if any.

Note that the credentials themselves are not unpacked. We rather just store exactly the same blob that the KCM client sends us. If we will support renewals in a future version, we might need to parse the credentials as well.

Configuration changes

The SSSD KCM responder would use the same common options like other SSSD services such as idle timeout or debug level.

There is also an option named socket_path that lets the admin select the UNIX socket the KCM service listens on. See the sssd-kcm(8) man page for more details.

Packaging changes

The KCM responder is be packaged in its own subpackage called sssd-kcm. This subpackage will not be installed by default, in other words it would not be required by the sssd meta-package, but a user will have to install this subpackage manually. Except for the KCM responder, the systemd socket and service file and documentation, the package will also contain a krb5.conf snippet that enables the KCM ccache type, so switching to the new credentials cache should be as easy as installing the package.

How To Test

First, the KCM responder must be installed. On Fedora/RHEL, this is done by installing the sssd-kcm subpackage.

In order for the admin to start using the KCM service, the sssd-kcm socket must be enabled and started and the sssd-kcm service must be enabled:

# systemctl enable sssd-kcm.socket
# systemctl start sssd-kcm.socket
# systemctl enable sssd-kcm.service

Please note that starting the KCM socket auto-starts the sssd-secrets socket so that the persistent secrets storage is available.

Then, set the KCM credential type as the default for the system. The sssd-kcm subpackage ships with a snippet file /etc/krb5.conf.d/kcm_default_ccache where it’s enough to just uncomment the following two lines:

[libdefaults]
default_ccache_name = KCM:

Of course, the same modification can be done directly in /etc/krb5.conf. Downstreams may choose to change this include file to enable the KCM cache directly so that just installing the sssd-kcm package with its snippet enables the KCM credential type.

After that, all common operations like kinit, kdestroy, kswitch or login through pam_sss should just work and store their credentials in the KCM server. Any existing tests for other collection-aware credential caches should work the same way. The KRB5CCNAME variable is in the form of:

KCM:$NAME:$CACHE_ID

The $NAME represents the collection and is typically the UID of the client. Only root is allowed to create arbitrarily named credential caches. Please note that the names are normally selected by libkrb5, but even attempting to do something like:

KRB5CCNAME=KCM:foobar kinit

must not work unless done as root. The $CACHE_ID is just an identifier of the cache in the collection. An example list of a KCM collection with two ccache look like this:

$ klist -A
Ticket cache: KCM:10327:75404
Default principal: tuser2@IPA.TEST

Valid starting       Expires              Service principal
04/04/2017 19:12:32  04/05/2017 19:12:29  krbtgt/IPA.TEST@IPA.TEST

Ticket cache: KCM:10327
Default principal: tuser1@IPA.TEST

Valid starting       Expires              Service principal
04/04/2017 19:12:26  04/05/2017 19:12:24  krbtgt/IPA.TEST@IPA.TEST

The KCM server implements per-UID credential cache ownership and access control. Therefore accessing other user’s credential caches as an unprivileged user should not work:

$ KRB5CCNAME=KCM:10327 klist

However, root can access anyone’s ccache, so doing the above as root should allow to list arbitrary user’s ccaches.

Restarting the KCM server or rebooting the machine must persist the tickets as they are stored in sssd-secrets’ on-disk storage.

The next section illustrates several use-cases related to containers step-by-step.

Use-case: separating ccaches of root users in containers, SSSD is running on the host

In this scenario, SSSD is running on the host and an application is running in a container. However, the application in a container runs as root and we want to keep its credential caches separate from the credential caches on the host. On the other hand we want to share the kerberos credentials between the containers.

  1. Make sure the sssd-kcm package is installed and the services and sockets enabled

  2. Create a directory that will contain the KCM daemon socket:

    host # mkdir /var/run/kcm

  3. Configure sssd-kcm to spawn the KCM socket there. Add the following to /etc/sssd/sssd.conf on the host:

    [kcm]
socket_path = /var/run/kcm/kcm.sock

  4. Restart sssd on the host to pick up the changes:

    host # systemctl restart sssd.service

  5. Tune the systemd sssd-kcm socket to ensure systemd will listen on the same socket KCM listens on:

    host # mkdir /etc/systemd/system/sssd-kcm.socket.d
host # cat /etc/systemd/system/sssd-kcm.socket.d/socket_override.conf
[Socket]
ListenStream=
ListenStream=/var/run/kcm/kcm.sock

  6. Re-read the unit file and verify the sssd-kcm.socket unit file is listening to the right socket:

    host # systemctl daemon-reload
host # systemctl restart sssd-kcm.socket
host # systemctl status sssd-kcm.socket
host # systemctl cat sssd-kcm.socket

  7. In order for the root user in the container to be represented as a different UID to the host, we need to create a subordinate UID and GID ranges that the ID from the containers will be mapped to. This range takes a required argument, which must correspond to a user that exists in /etc/passwd (although domain users will be supported starting with docker 1.13). The subordinate ranges are created in /etc/subuid and /etc/subgid on the host. Please refer to the docker documentation for more details on Docker user namespaces. For example:

    host # useradd kcmtest
host # grep kcmtest /etc/subgid
kcmtest:50000:65536
host # grep kcmtest /etc/subuid
kcmtest:50000:65536

  8. Configure the docker daemon to use this subordinate ID namespace by changing this line in /etc/sysconfig/docker:

    OPTIONS='--selinux-enabled --log-driver=journald --userns-remap=kcmtest'

  9. Restart the docker service. Please note that docker stores the images under a per-user-namespace directory, so you’ll need to pull the images again:

    host # systemctl restart docker.service

  10. Start a container, bind-mounting the /var/run/kcm directory from the host to make the KCM socket accessible:

    host # docker run -t -i -h=kcmtest1 -v=/var/run/kcm:/var/run/kcm fedora /bin/bash

  11. Configure the container’s Kerberos config file to use KCM: as the credential cache. Edit /etc/krb5.conf in the container:

    [libdefaults]
default_realm = IPA.TEST
dns_lookup_realm = true
dns_lookup_kdc = true
rdns = false
default_ccache_name = KCM:
kcm_socket = /var/run/kcm/kcm.sock

[realms]
IPA.TEST = {
    pkinit_anchors = FILE:/etc/ipa/ca.crt
    kdc = unidirect.ipa.test
}

  12. Acquire Kerberos credentials for the admin IPA user. Note that despite the user’s UID value in the container is 0, the UID is translated to 50000 on the host, which is what the KCM server then uses to store the credentials at:

    [root@kcmtest1 /]# id
uid=0(root) gid=0(root) groups=0(root)

[root@kcmtest1 /]# kinit admin
Password for admin@IPA.TEST:

[root@kcmtest1 /]# klist
Ticket cache: KCM:50000
Default principal: admin@IPA.TEST

Valid starting     Expires            Service principal
11/25/16 15:29:38  11/26/16 15:29:37  krbtgt/IPA.TEST@IPA.TEST

  13. Start another container, bind-mounting the /var/run/kcm directory from the host to make the KCM socket accessible:

    host # docker run -t -i -h=kcmtest2 -v=/var/run/kcm:/var/run/kcm fedora /bin/bash

  14. Configure krb5.conf in the same manner and run klist (without kinit!) in the container. Note we can access the same ccache the first container acquired:

    [root@kcmtest2 /]# klist
Ticket cache: KCM:50000
Default principal: admin@IPA.TEST

Valid starting     Expires            Service principal
11/25/16 15:29:38  11/26/16 15:29:37  krbtgt/IPA.TEST@IPA.TEST

  15. root on the host cannot access the same cache by default. An interesting property of the KCM protocol is that UID 0 can list all ccaches or all other UIDs, though:

    host # klist
klist: Matching credential not found

Note - if the container is running as a different user (using the USER directive specified in the container’s Dockerfile), then the ID the KCM server is contacted with depends on whether ID namespaces are used. Without the ID namespaces, the host receives the UID of the container user as-is. If user namespaces are in effect, then the ID of the container user is translated into the subordinate namespace. For example, if the namespace above was still in effect, a container user running as uid=1000 would be translated into user with uid=51000 on the host.

Use-case: separating ccaches of containers from ccaches of the host

In this use-case, SSSD is running in one container and keeps track of ccaches in other containers that are completely separated from the host environment. The containers must also share the credential caches between one another.

  1. Start a container that will run an SSSD instance with the KCM service. We name the container kcmserver and assign a volume called /kcmserver to this container:

    host# docker run -t -i --name=kcmserver -h=kcmserver -v=/kcmserver fedora /bin/bash

  2. Install and configure sssd in the container. The configuration can be pretty minimal, but the important piece is the KCM socket in the Docker volume at /kcmserver/kcm.socket. Please note that depending on your version, the domain might or might not be required - on Fedora, there is an implicit domain starting with F-26. Older versions might need to define a domain even if no remote server with users is being used actually:

    kcmserver # yum -y install sssd-kcm
kcmserver # cat /etc/sssd/sssd.conf
[sssd]
domains = local

[kcm]
socket_path = /kcmserver/kcm.socket

[domain/local]
id_provider = local

  3. Tune the systemd sssd-kcm socket to ensure systemd will listen on the same socket KCM listens on:

    host # mkdir /etc/systemd/system/sssd-kcm.socket.d
host # cat /etc/systemd/system/sssd-kcm.socket.d/socket_override.conf
[Socket]
ListenStream=
ListenStream=/kcmserver/kcm.socket

  4. Re-read the unit file and verify the sssd-kcm.socket unit file is listening to the right socket:

    host # systemctl daemon-reload
host # systemctl restart sssd-kcm.socket
host # systemctl status sssd-kcm.socket
host # systemctl cat sssd-kcm.socket

  5. Start another container that will represent an application. Make sure the container mounts the volume from the kcmserver instance:

    host # docker run -t -i --name=kcmclient -h=kcmclient --volumes-from=kcmserver fedora /bin/bash

  6. Observe that the container mounted the volume and the volume includes the KCM server socket:

    kcmclient # ll /kcmserver/kcm.socket
srw-rw-rw-. 1 root root 0 Nov 29 16:21 /kcmserver/kcm.socket

  7. Configure /etc/krb5.conf to use KCM: as the credentials cache and point libkrb5 to the KCM socket:

    kcmclient # grep default_ccache_name /etc/krb5.conf
default_ccache_name = KCM:
kcmclient # grep kcm_socket /etc/krb5.conf
kcm_socket = /kcmserver/kcm.socket

  8. Acquire Kerberos credentials in the kcmclient container:

    kcmclient # kinit admin
Password for admin@IPA.TEST:
kcmclient # klist
    Ticket cache: KCM:0
    Default principal: admin@IPA.TEST

    Valid starting     Expires            Service principal
    11/29/16 16:21:28  11/30/16 16:21:26  krbtgt/IPA.TEST@IPA.TEST

  9. Observe that these credentials are not visible to the host:

    host # klist
klist: Matching credential not found

  10. Start another container as another KCM client, configure its krb5.conf configuration file in the same manner. As long as this container runs as the same UID as the first KCM client, the credentials should be visible in this container immediately without having to acquire them:

    kcmclient2 # klist
    Ticket cache: KCM:0
    Default principal: admin@IPA.TEST

    Valid starting     Expires            Service principal
    11/29/16 16:21:28  11/30/16 16:21:26  krbtgt/IPA.TEST@IPA.TEST

How To Debug

The SSSD KCM server would use the same DEBUG facility as other SSSD services. In order to debug the client side operations, setting the KRB5_TRACE variable might come handy.

The KCM protocol response-request can be logged using strace.

The admin might also inspect the SSSD secrets database to see what credential caches have been stored by the SSSD.

Authors