freeipa

#9789 ipa-backup doesn't start ipa service if /tmp runs out of space
Opened by jjakob. Modified

Open
Close Issue

Issue

When ipa-backup fails due to no disk space while creating its archive in /tmp it doesn't start the ipa service back up, resulting in a service outage.

Steps to Reproduce

  1. Have sufficiently low free disk space available so that the backup archive size would exceed the available free space.
  2. Run a full ipa-backup

Actual behavior

ipa-backup fails and exits without starting the ipa service, resulting in a service outage

Expected behavior

ipa-backup should (try to) start the ipa service regardless of any errors encountered while backing up

Version/Release/Distribution

ipa-server-4.9.13-12.module+el8.10.0+1845+84a5752e.x86_64
ipa-client-4.9.13-12.module+el8.10.0+1845+84a5752e.x86_64
389-ds-base-1.4.3.39-8.module+el8.10.0+1866+6d9051ff.x86_64
krb5-server-1.18.2-29.el8_10.x86_64

Additional info:

ipabackup log

2025-05-10T21:17:38Z DEBUG Logging to /var/log/ipabackup.log
2025-05-10T21:17:38Z DEBUG ipa-backup was invoked with arguments [] and options: {'verbose': False, 'quiet': False, 'log_file': None, 'gpg_keyring': None, 'gpg': True, 'data_only': False, 'logs': True, 'online': False, 'rolecheck': True}
2025-05-10T21:17:38Z DEBUG IPA version 4.9.13-12.module+el8.10.0+1845+84a5752e
2025-05-10T21:17:38Z DEBUG IPA platform rhel
2025-05-10T21:17:38Z DEBUG IPA os-release Rocky Linux 8.10 (Green Obsidian)
2025-05-10T21:17:38Z DEBUG importing all plugin modules in ipaserver.plugins...
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.aci
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.automember
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.automount
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.baseldap
2025-05-10T21:17:38Z DEBUG ipaserver.plugins.baseldap is not a valid plugin module
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.baseuser
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.batch
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.ca
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.caacl
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.cert
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.certmap
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.certprofile
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.config
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.delegation
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.dns
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.dnsserver
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.dogtag
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.domainlevel
2025-05-10T21:17:38Z DEBUG importing plugin module ipaserver.plugins.group
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.hbac
2025-05-10T21:17:39Z DEBUG ipaserver.plugins.hbac is not a valid plugin module
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.hbacrule
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.hbacsvc
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.hbacsvcgroup
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.hbactest
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.host
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.hostgroup
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.idp
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.idrange
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.idviews
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.internal
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.join
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.krbtpolicy
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.ldap2
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.location
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.migration
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.misc
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.netgroup
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.otp
2025-05-10T21:17:39Z DEBUG ipaserver.plugins.otp is not a valid plugin module
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.otpconfig
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.otptoken
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.passwd
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.permission
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.ping
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.pkinit
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.privilege
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.pwpolicy
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.rabase
2025-05-10T21:17:39Z DEBUG ipaserver.plugins.rabase is not a valid plugin module
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.radiusproxy
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.realmdomains
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.role
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.schema
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.selfservice
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.selinuxusermap
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.server
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.serverrole
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.serverroles
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.service
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.servicedelegation
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.session
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.stageuser
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.subid
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.sudo
2025-05-10T21:17:39Z DEBUG ipaserver.plugins.sudo is not a valid plugin module
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.sudocmd
2025-05-10T21:17:39Z DEBUG importing plugin module ipaserver.plugins.sudocmdgroup
2025-05-10T21:17:40Z DEBUG importing plugin module ipaserver.plugins.sudorule
2025-05-10T21:17:40Z DEBUG importing plugin module ipaserver.plugins.topology
2025-05-10T21:17:40Z DEBUG importing plugin module ipaserver.plugins.trust
2025-05-10T21:17:40Z DEBUG importing plugin module ipaserver.plugins.user
2025-05-10T21:17:40Z DEBUG importing plugin module ipaserver.plugins.vault
2025-05-10T21:17:40Z DEBUG importing plugin module ipaserver.plugins.virtual
2025-05-10T21:17:40Z DEBUG ipaserver.plugins.virtual is not a valid plugin module
2025-05-10T21:17:40Z DEBUG importing plugin module ipaserver.plugins.whoami
2025-05-10T21:17:40Z DEBUG importing plugin module ipaserver.plugins.xmlserver
2025-05-10T21:17:42Z INFO Preparing backup on ipa4.example.com
2025-05-10T21:17:42Z DEBUG Starting external process
2025-05-10T21:17:42Z DEBUG args=['/bin/systemctl', 'start', 'dirsrv@EXAMPLE-COM.service']
2025-05-10T21:17:42Z DEBUG Process finished, return code=0
2025-05-10T21:17:42Z DEBUG Starting external process
2025-05-10T21:17:42Z DEBUG args=['/bin/systemctl', 'is-active', 'dirsrv@EXAMPLE-COM.service']
2025-05-10T21:17:42Z DEBUG Process finished, return code=0
2025-05-10T21:17:42Z DEBUG stdout=active
2025-05-10T21:17:42Z DEBUG stderr=
2025-05-10T21:17:42Z DEBUG wait_for_open_ports: localhost [389] timeout 120
2025-05-10T21:17:42Z DEBUG waiting for port: 389
2025-05-10T21:17:42Z DEBUG SUCCESS: port: 389
2025-05-10T21:17:42Z DEBUG Start of dirsrv@EXAMPLE-COM.service complete
2025-05-10T21:17:43Z DEBUG retrieving schema for SchemaCache url=ldapi://%2Frun%2Fslapd-EXAMPLE-COM.socket conn=<ldap.ldapobject.SimpleLDAPObject object at 0x7fc813c02e10>
2025-05-10T21:17:43Z INFO Local roles match globally used roles, proceeding.
2025-05-10T21:17:43Z INFO Stopping IPA services
2025-05-10T21:17:43Z DEBUG Starting external process
2025-05-10T21:17:43Z DEBUG args=['/usr/sbin/ipactl', 'stop']
2025-05-10T21:17:57Z DEBUG Process finished, return code=0
2025-05-10T21:17:57Z DEBUG stdout=Stopping ipa-dnskeysyncd Service
Stopping ipa-otpd Service
Stopping pki-tomcatd Service
Stopping ipa-custodia Service
Stopping httpd Service
Stopping named Service
Stopping kadmin Service
Stopping krb5kdc Service
Stopping Directory Service
2025-05-10T21:17:57Z DEBUG stderr=ipa: INFO: The ipactl command was successful
2025-05-10T21:17:57Z INFO Backing up ipaca in EXAMPLE-COM to LDIF
2025-05-10T21:17:57Z DEBUG Starting external process
2025-05-10T21:17:57Z DEBUG args=['/usr/sbin/dsctl', 'EXAMPLE-COM', 'db2ldif', '--replication', 'ipaca', '/var/lib/dirsrv/slapd-EXAMPLE-COM/ldif/EXAMPLE-COM-ipaca.ldif']
2025-05-10T21:18:08Z DEBUG Process finished, return code=0
2025-05-10T21:18:08Z DEBUG stdout=db2ldif successful
2025-05-10T21:18:08Z DEBUG stderr=ldiffile: /var/lib/dirsrv/slapd-EXAMPLE-COM/ldif/EXAMPLE-COM-ipaca.ldif
2025-05-10T21:18:08Z INFO Backing up userRoot in EXAMPLE-COM to LDIF
2025-05-10T21:18:08Z DEBUG Starting external process
2025-05-10T21:18:08Z DEBUG args=['/usr/sbin/dsctl', 'EXAMPLE-COM', 'db2ldif', '--replication', 'userRoot', '/var/lib/dirsrv/slapd-EXAMPLE-COM/ldif/EXAMPLE-COM-userRoot.ldif']
2025-05-10T21:18:14Z DEBUG Process finished, return code=0
2025-05-10T21:18:14Z DEBUG stdout=db2ldif successful
2025-05-10T21:18:14Z DEBUG stderr=ldiffile: /var/lib/dirsrv/slapd-EXAMPLE-COM/ldif/EXAMPLE-COM-userRoot.ldif
2025-05-10T21:18:14Z INFO Backing up EXAMPLE-COM
2025-05-10T21:18:14Z DEBUG Starting external process
2025-05-10T21:18:14Z DEBUG args=['/usr/sbin/dsctl', 'EXAMPLE-COM', 'db2bak', '/var/lib/dirsrv/slapd-EXAMPLE-COM/bak/EXAMPLE-COM']
2025-05-10T21:18:27Z DEBUG Process finished, return code=0
2025-05-10T21:18:27Z DEBUG stdout=db2bak successful
2025-05-10T21:18:27Z DEBUG stderr=
2025-05-10T21:18:27Z DEBUG Starting external process
2025-05-10T21:18:27Z DEBUG args=['/usr/bin/authselect', 'current', '--raw']
2025-05-10T21:18:27Z DEBUG Process finished, return code=0
2025-05-10T21:18:27Z DEBUG stdout=sssd with-sudo
2025-05-10T21:18:27Z DEBUG stderr=
2025-05-10T21:18:27Z INFO Backing up files
2025-05-10T21:18:27Z DEBUG Starting external process
2025-05-10T21:18:27Z DEBUG args=['tar', '--exclude=/var/lib/ipa/backup', '--xattrs', '--selinux', '-cf', '/tmp/tmpptzo_nxmipa/ipa/files.tar', '/usr/share/ipa/html', '/root/.pki', '/etc/pki/pki-tomcat', '/etc/sysconfig/pki', '/var/lib/pki', '/var/lib/ipa/sysrestore', '/var/lib/ipa-client/sysrestore', '/var/lib/ipa/dnssec', '/var/lib/sss/pubconf/krb5.include.d/', '/var/lib/certmonger', '/var/lib/ipa', '/run/dirsrv', '/run/lock/dirsrv', '/etc/dirsrv/slapd-EXAMPLE-COM', '/var/lib/dirsrv/slapd-EXAMPLE-COM', '/etc/named.conf', '/etc/named/ipa-ext.conf', '/etc/named/ipa-options-ext.conf', '/etc/named/ipa-logging-ext.conf', '/etc/named.keytab', '/etc/resolv.conf', '/etc/sysconfig/pki-tomcat', '/etc/sysconfig/krb5kdc', '/etc/sysconfig/ipa-dnskeysyncd', '/etc/sysconfig/ipa-ods-exporter', '/etc/sysconfig/named', '/etc/sysconfig/ods', '/etc/ipa/nssdb/pwdfile.txt', '/etc/pki/ca-trust/source/ipa.p11-kit', '/etc/authselect/user-nsswitch.conf', '/etc/krb5.keytab', '/etc/sssd/sssd.conf', '/etc/openldap/ldap.conf', '/etc/security/limits.conf', '/var/lib/ipa/gssproxy/http.keytab', '/etc/ipa/kdcproxy/ipa-kdc-proxy.conf', '/etc/httpd/conf.d/ipa-pki-proxy.conf', '/etc/httpd/conf.d/ipa-rewrite.conf', '/etc/httpd/conf.d/ssl.conf', '/etc/httpd/conf.d/ssl.conf', '/var/lib/ipa/certs/httpd.crt', '/var/lib/ipa/private/httpd.key', '/etc/httpd/conf.d/ipa.conf', '/etc/ssh/sshd_config', '/etc/ssh/ssh_config', '/etc/krb5.conf', '/var/lib/ipa-client/pki/kdc-ca-bundle.pem', '/var/lib/ipa-client/pki/ca-bundle.pem', '/etc/ipa/ca.crt', '/etc/ipa/default.conf', '/etc/dirsrv/ds.keytab', '/etc/chrony.conf', '/etc/samba/smb.conf', '/var/lib/ipa/ra-agent.pem', '/var/lib/ipa/ra-agent.key', '/root/cacert.p12', '/var/kerberos/krb5kdc/kdc.conf', '/var/kerberos/krb5kdc/kdc.crt', '/var/kerberos/krb5kdc/kdc.key', '/var/kerberos/krb5kdc/cacert.pem', '/etc/systemd/system/multi-user.target.wants/ipa.service', '/etc/systemd/system/httpd.service.d/ipa.conf', '/etc/systemd/system/multi-user.target.wants/sssd.service', '/etc/systemd/system/multi-user.target.wants/certmonger.service', '/etc/systemd/system/pki-tomcatd.target.wants/pki-tomcatd@pki-tomcat.service', '/etc/opendnssec/conf.xml', '/etc/opendnssec/kasp.xml', '/etc/opendnssec/zonelist.xml', '/var/opendnssec/kasp.db', '/etc/ipa/dnssec/softhsm2.conf', '/etc/ipa/dnssec/softhsm_pin_so', '/etc/ipa/dnssec/ipa-dnskeysyncd.keytab', '/etc/ipa/custodia/server.keys', '/etc/ipa/custodia/custodia.conf', '/etc/gssproxy/10-ipa.conf', '/etc/hosts', '/etc/systemd/system/pki-tomcatd@pki-tomcat.service.d/ipa.conf', '/etc/NetworkManager/conf.d/zzz-ipa.conf', '/etc/ipa/nssdb/cert9.db', '/etc/ipa/nssdb/key4.db', '/etc/ipa/nssdb/pkcs11.txt', '/etc/pkcs11/modules/softhsm2.module', '/etc/tmpfiles.d/dirsrv-EXAMPLE-COM.conf', '/etc/systemd/system/dirsrv@EXAMPLE-COM.service.d/ipa-env.conf', '/var/lib/ipa/passwds/ipa4.example.com-443-RSA', '/var/log/pki/', '/var/log/httpd', '/var/log/ipareplica-install.log', '/var/log/ipareplica-conncheck.log', '/var/log/kadmind.log', '/var/log/messages', '/var/log/ipaclient-install.log', '/var/log/secure', '/var/named/data/named.run', '/var/log/dirsrv/slapd-EXAMPLE-COM']
2025-05-10T21:19:19Z DEBUG Process finished, return code=2
2025-05-10T21:19:19Z DEBUG stdout=
2025-05-10T21:19:19Z DEBUG stderr=tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
tar: /tmp/tmpptzo_nxmipa/ipa/files.tar: Wrote only 8192 of 10240 bytes
tar: Error is not recoverable: exiting now
2025-05-10T21:19:20Z DEBUG   File "/usr/lib/python3.6/site-packages/ipapython/admintool.py", line 180, in execute
    return_value = self.run()
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ipa_backup.py", line 349, in run
    self.file_backup(options)
  File "/usr/lib/python3.6/site-packages/ipaserver/install/ipa_backup.py", line 652, in file_backup
    (result.returncode, result.error_log))
2025-05-10T21:19:20Z DEBUG The ipa-backup command failed, exception: ScriptError: tar returned non-zero code 2: tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
tar: /tmp/tmpptzo_nxmipa/ipa/files.tar: Wrote only 8192 of 10240 bytes
tar: Error is not recoverable: exiting now
2025-05-10T21:19:20Z ERROR tar returned non-zero code 2: tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
tar: /tmp/tmpptzo_nxmipa/ipa/files.tar: Wrote only 8192 of 10240 bytes
tar: Error is not recoverable: exiting now
2025-05-10T21:19:20Z ERROR The ipa-backup command failed. See /var/log/ipabackup.log for more information

You can redefine TMPDIR for the ipa-backup run.

I should just expand the disk (this is a VM so it's easy) as it's 82% used (6.8G of 8.8G). I don't know why the temporary archive would fill it up, the last successful full backup tar archive is only 261M.
There's also another thing, ipa-backup uses ipactl directly, so when it stops ipa services the systemd service still says ipa.service is active and healthy. Even though 'ipactl status' says 'Directory Service: STOPPED', a 'systemctl start ipa.service' does not do anything, it needs 'restart'.
Perhaps ipa-backup should use the systemd service instead of ipactl directly.

/tmp is typically tmpfs it is RAM, not disk the limit here.

In Rocky 8 it's on the disk, only /run is tmpfs. /tmp is not a symlink to /run either.
Metadata
None
None
None
None
None
None
Static archive. pagure.io was sunset in 2026; this is a read-only snapshot.
Red Hat Logo

Fedora is sponsored by Red Hat. Learn more about the relationship between Red Hat and Fedora »

© Red Hat, Inc. and others.