What to do when named with bind-dyndb-ldap cannot start

I tried to start FreeIPA but it failed because Named cannot start!

$ sudo ipactl start
Starting Directory Service
Starting KDC Service
Starting KPASSWD Service
Starting DNS Service
Job failed. See system journal and 'systemctl status' for details.
Failed to start DNS Service
Shutting down
Aborting ipactl

Examples and formatting in this guide

  • Host name of the server where named is failing: srv01.example.com
  • Kerberos realm: EXAMPLE.COM
  • Kerberos principal used by named: DNS/srv01.example.com
  • Kerberos principal used by LDAP server: ldap/srv01.example.com

0. Are you unsure what to do?

Please try to follow steps below. This guide covers most common problems and solutions.

If you encounter any problem or if something in this page is unclear, don’t hesitate to write an e-mail to freeipa-users list at freeipa-users@… or come to IRC channel #freeipa on FreeNode.

1. Gather symptoms

Log files should contain some error messages. Try to examine files like /var/log/messages and /var/named/data/named.run and look for error messages.

What did you change before it started to fail? :-)

Kerberos

It is handy to get debugging information from Kerberos libraries. This is done by configuring KRB5_TRACE environment variable before you start BIND.

On Fedora/RHEL you can add following line to /etc/sysconfig/named:

export KRB5_TRACE=/tmp/named_krb5.log

Pure `systemd-way <https://wiki.archlinux.org/index.php/systemd#Editing_provided_unit_files>`__ is:

$ mkdir /etc/systemd/system/named.service.d
$ cat > /etc/systemd/system/named.service.d/krb5_debug.conf <<EOF
[Service]
Environment=KRB5_TRACE=/tmp/named_krb5.log
EOF
$ systemctl daemon-reload
$ systemctl restart named.service

Note that /tmp can reference either real /tmp or so-called private /tmp, e.g. /tmp/systemd-named.service-*/tmp.

2. Server ldap/srv01@EXAMPLE.COM not found in Kerberos database

Symptom: A log contains an error message similar to:

named: GSSAPI Error: Unspecified GSS failure.  Minor code may provide more information
(Server `ldap/srv01@EXAMPLE.COM` not found in Kerberos database)

Usual causes:

  1. The host name is not fully qualified:

    • The output of command hostname should print srv01.example.com, not only srv01.
    • Enter fully qualified host name to /etc/sysconfig/network or /etc/hostname
    • Verify that the new host name is returned by the command hostname
  2. Configuration file /etc/hosts contains only short host names or the short names precede the fully qualified names:

    • Incorrect lines:

      192.0.2.1    srv01
      192.0.2.1    srv01    srv01.example.com
      
    • Correct line:

      192.0.2.1    srv01.example.com    srv01
      

3. Failed to init credentials or Failed to get initial credentials (Decrypt integrity check failed) or (Clients credentials have been revoked)

Usual cause: The Kerberos key stored in keytab /etc/named.keytab doesn’t match the key stored on KDC. Typically, new keys for principal DNS/srv01.example.com were generated via ipa-getkeytab or kadmin utility but the new keys were stored to wrong keytab file.

You can verify if it is this case by comparing Key Version Number (kvno) between keytab and KDC:

  • Get kvno stored in keytab /etc/named.keytab:

    $ klist -kt /etc/named.keytab
    Keytab name: FILE:/etc/named.keytab
    KVNO Timestamp         Principal
    ---- ----------------- --------------------------------------------------------
       8 02/12/13 14:19:10 DNS/srv01.example.com@EXAMPLE.COM
    
  • Kinit as arbitrary principal, e.g. admin:

    $ kinit admin
    Password for admin@EXAMPLE.COM:
    
  • Get kvno of the keys stored on KDC:

    $ kvno DNS/srv01.example.com@EXAMPLE.COM
    DNS/srv01.example.com@EXAMPLE.COM: kvno = 9
    
  • There is a clear mismatch between Key Version Numbers in keytab file (8) and KDC (9).

Remediation: The new keys have to be stored in keytab file used by named, i.e. /etc/named.keytab. The simplest way is to generate new keys via ipa-getkeytab utility:

$ ipa-getkeytab -s srv01.example.com -p DNS/srv01.example.com@EXAMPLE.COM -k /etc/named.keytab
  • Make sure that the keytab file is readable by named (i.e. usually owned by user named)

4. Invalid credentials: bind to LDAP server failed

Usual causes:

  1. LDAP credentials in /etc/named.conf are misconfigured: Check auth_method, bind_dn, password, sasl_mech, sasl_user, sasl_realm, sasl_password and krb5_principal options in /etc/named.conf.
  2. LDAP server doesn’t allow LDAP bind with configured credentials.

It is recommended to use ldapsearch utility and verify that you are able to access the database from command line. This step will reveal misconfiguration outside of the bind-dyndb-ldap.

If you use Kerberos via ``auth_method`` `SASL <http://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer>`__:

  1. Determine the Kerberos principal used by bind-dyndb-ldap (values are ordered by priority):

    1. Use value krb5_principal from /etc/named.conf
    2. Use value sasl_user from /etc/named.conf
    3. Use default value DNS/$FQDN where $FQDN is output from command hostname, e.g. DNS/srv01.example.com
  2. Obtain Ticket Granting Ticket (TGT) for the principal used by bind-dyndb-ldap:

    $ kinit -kt /etc/named.keytab DNS/srv01.example.com
    

    Any error returned from kinit usually points to misconfiguration in principal name or mismatch between Key Version Numbers.

Please continue with checks in section “No zones from LDAP are loaded” if you were able to successfully authenticate.

5. No zones from LDAP are loaded

Symptoms:

  • BIND logs contain line like 0 zones from LDAP instance 'ipa' loaded (0 zones defined, 0 inactive, 0 failed to load).
  • DNS queries return NXDOMAIN answers or results obtained from forwarders instead of authoritative data.
  • Dynamic updates and zone transfers are refused with error NOTAUTH.

Usual cause: FreeIPA upgrade destroyed Access Control Instructions in LDAP which allowed access to DNS sub-tree by DNS principal.

You can test if it is the case:

  1. Obtain Kerberos ticket as described in previous section.

  2. Run ldapsearch with GSSAPI mechanism and check the output:

    $ ldapsearch -H 'ldapi://%2fvar%2frun%2fslapd-EXAMPLE-COM.socket' -Y GSSAPI -b 'cn=dns,dc=example,dc=com'
    

    You should be able to see LDAP objects with objectClasses idnsZone and idnsRecord, not only idnsConfig. Any error in this step points to misconfiguration in LDAP server.

If you use FreeIPA

If you are unable to see those objects:

  1. Gather information about ACI using this command:

    $ ipa privilege-show 'DNS Servers' --all --raw
    
  2. Contact freeipa-users@redhat.com mailing list, we will help you.

    • Please do not forget to describe all the steps you did including results you obtained, version numbers of packages, if you did an upgrade or clean FreeIPA installation etc.

If you use bind-dyndb-ldap without FreeIPA

Change permission settings on your LDAP server so you are able to see DNS data using the ldapsearch command above.

W1. (Workaround) Use simple LDAP BIND insted of Kerberos

It is not recommended to use simple LDAP BIND if Kerberos infrastructure is available. You can use this workaround if you are desperate and you need to get it working immediately. We recommend to debug your Kerberos infrastructure as soon as you can.

Configure /etc/named.conf with name/password authentication instead of using Kerberos:

auth_method "simple";
bind_dn "uid=admin,cn=users,cn=accounts,dc=example,dc=com";
password "your_secret_password";

Any account authorized to read/write to cn=dns sub-tree in LDAP should work.