Cfxtrjtrk

I've been banging my head against google for a day now and I'm no closer to a solution so I'm hoping someone out there can help out.

I'm installing 18.04 via fai and then doing some system config using puppet. ALL of the systems work for about a week and then users can no longer log back into their boxes. I installed five systems last week within a four hour time frame and today, within that same four hour window, all of the systems began exhibiting the same issue.

Users can not log into the box from gdm. ssh into the boxes works, but only after a 30s or so wait time.

I'm seeing the following in the log:

==> /var/log/syslog <==

Nov 16 15:44:15 pre043 systemd-logind[1921]: do_ypcall: clnt_call: RPC: Unable to send; errno = Operation not permitted



==> /var/log/auth.log <==

Nov 16 15:44:40 pre043 sshd[1971]: pam_systemd(sshd:session): Failed to create session: Connection timed out

And this is happening:

$ systemctl list-unit-files --user

Failed to connect to bus: No such file or directory

pam_systemd is not creating the /run/user/USERID directories and the normally set XDG_RUNTIME_DIR is not set.

If I wait a bit, I start to see possibly related stuff in the logs:

==> /var/log/apport.log <==

ERROR: apport (pid 2071) Fri Nov 16 15:46:58 2018: called for pid 1921, signal 6, core limit 0, dump mode 1

ERROR: apport (pid 2071) Fri Nov 16 15:46:58 2018: executable: /lib/systemd/systemd-logind (command line "/lib/systemd/systemd-logind")

ERROR: apport (pid 2071) Fri Nov 16 15:46:58 2018: is_closing_session(): no DBUS_SESSION_BUS_ADDRESS in environment

ERROR: apport (pid 2071) Fri Nov 16 15:46:58 2018: apport: report /var/crash/_lib_systemd_systemd-logind.0.crash already exists and unseen, doing nothing to avoid disk usage DoS



==> /var/log/auth.log <==

Nov 16 15:46:58 pre043 systemd-logind[2072]: New seat seat0.

Nov 16 15:46:58 pre043 systemd-logind[2072]: Watching system buttons on /dev/input/event2 (Power Button)

Nov 16 15:46:58 pre043 systemd-logind[2072]: Watching system buttons on /dev/input/event1 (Power Button)

Nov 16 15:46:58 pre043 systemd-logind[2072]: Watching system buttons on /dev/input/event0 (Sleep Button)

Nov 16 15:46:58 pre043 systemd-logind[2072]: Watching system buttons on /dev/input/event7 (Dell Dell USB Entry Keyboard)

Nov 16 15:46:58 pre043 systemd-logind[2072]: Watching system buttons on /dev/input/event4 (Compx 2.4G Receiver)

Nov 16 15:46:58 pre043 systemd-logind[2072]: Watching system buttons on /dev/input/event6 (Compx 2.4G Receiver)



==> /var/log/syslog <==

Nov 16 15:46:58 pre043 systemd[1]: systemd-logind.service: Watchdog timeout (limit 3min)!

Nov 16 15:46:58 pre043 systemd[1]: systemd-logind.service: Killing process 1921 (systemd-logind) with signal SIGABRT.

Nov 16 15:46:58 pre043 systemd[1]: systemd-logind.service: Main process exited, code=dumped, status=6/ABRT

Nov 16 15:46:58 pre043 systemd[1]: systemd-logind.service: Failed with result 'watchdog'.

Nov 16 15:46:58 pre043 systemd[1]: systemd-logind.service: Service has no hold-off time, scheduling restart.

Nov 16 15:46:58 pre043 systemd[1]: systemd-logind.service: Scheduled restart job, restart counter is at 2.

Nov 16 15:46:58 pre043 systemd[1]: Stopped Login Service.

Nov 16 15:46:58 pre043 systemd[1]: Starting Login Service...

Nov 16 15:46:58 pre043 systemd[1]: Started Login Service.



==> /var/log/auth.log <==

Nov 16 15:46:58 pre043 systemd-logind[2072]: New session c1 of user gdm.

Nov 16 15:46:58 pre043 systemd-logind[2072]: New session 3 of user root.



==> /var/log/syslog <==

Nov 16 15:46:59 pre043 gnome-shell[1429]: Could not open device /dev/input/event2: GDBus.Error:org.freedesktop.login1.DeviceIsTaken: Device already taken

Nov 16 15:46:59 pre043 gnome-shell[1429]: Could not open device /dev/input/event3: GDBus.Error:org.freedesktop.login1.DeviceIsTaken: Device already taken

Nov 16 15:46:59 pre043 gnome-shell[1429]: Could not open device /dev/input/event1: GDBus.Error:org.freedesktop.login1.DeviceIsTaken: Device already taken

Nov 16 15:46:59 pre043 gnome-shell[1429]: Could not open device /dev/input/event0: GDBus.Error:org.freedesktop.login1.DeviceIsTaken: Device already taken

Nov 16 15:46:59 pre043 gnome-shell[1429]: Could not open device /dev/input/event18: GDBus.Error:org.freedesktop.login1.DeviceIsTaken: Device already taken

Nov 16 15:46:59 pre043 gnome-shell[1429]: Could not open device /dev/input/event19: GDBus.Error:org.freedesktop.login1.DeviceIsTaken: Device already taken

Nov 16 15:46:59 pre043 gnome-shell[1429]: Could not open device /dev/input/event20: GDBus.Error:org.freedesktop.login1.DeviceIsTaken: Device already taken

Nov 16 15:46:59 pre043 gnome-shell[1429]: Could not open device /dev/input/event21: GDBus.Error:org.freedesktop.login1.DeviceIsTaken: Device already taken

Something seems to go south with dbus but I've got no idea what or why. Reboot does not fix it. Reinstalling dbus does not fix it.

Of possible note, the parameters for the dbus bits on the broken box are notably different than on a working box.

(BROKEN BOX)

# ps -ef | grep -i dbus | grep mparker

mparker   1916     1  0 15:29 pts/0    00:00:00 dbus-launch --autolaunch 629f8bd4627543a0a62559707dac566f --binary-syntax --close-stderr

mparker   1917     1  0 15:29 ?        00:00:00 /usr/bin/dbus-daemon --syslog-only --fork --print-pid 5 --print-address 7 --session

(WORKING BOX)

# ps -ef | grep -i dbus | grep mparker

mparker   1893  1860  0 17:08 ?        00:00:00 /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only

mparker   1994  1989  0 17:08 ?        00:00:00 /usr/bin/dbus-daemon --config-file=/usr/share/defaults/at-spi2/accessibility.conf --nofork --print-address 3

Of possible other note, I see a number of half-configured or half-installed packages listed in the dpkg.log file after reboot on the broken box.

grep half- /var/log/dpkg.log | grep '15:22'

2018-11-16 15:22:35 status half-configured dbus:amd64 1.12.2-1ubuntu1

2018-11-16 15:22:35 status half-installed dbus:amd64 1.12.2-1ubuntu1

2018-11-16 15:22:35 status half-installed dbus:amd64 1.12.2-1ubuntu1

2018-11-16 15:22:35 status half-configured ureadahead:amd64 0.100.0-20

2018-11-16 15:22:36 status half-configured systemd:amd64 237-3ubuntu10.6

2018-11-16 15:22:36 status half-configured man-db:amd64 2.8.3-2ubuntu0.1

2018-11-16 15:22:36 status half-configured dbus:amd64 1.12.2-1ubuntu1

I've done a re-install of these via

# apt-get install --reinstall $(grep half- /var/log/dpkg.log | grep '15:22' | awk '{print $5}' | sort -u | sed 's/:amd64//')

Reading package lists... Done

Building dependency tree       

Reading state information... Done

0 upgraded, 0 newly installed, 4 reinstalled, 0 to remove and 9 not upgraded.

Need to get 3932 kB/4082 kB of archives.

After this operation, 0 B of additional disk space will be used.

Get:1 http://mirror/ubuntu bionic-updates/main amd64 systemd amd64 237-3ubuntu10.6 [2894 kB]

Get:2 http://mirror/ubuntu bionic-updates/main amd64 man-db amd64 2.8.3-2ubuntu0.1 [1019 kB]

Get:3 http://mirror/ubuntu bionic/main amd64 ureadahead amd64 0.100.0-20 [19.3 kB]

Fetched 3932 kB in 0s (52.5 MB/s)    

Preconfiguring packages ...

(Reading database ... 242468 files and directories currently installed.)

Preparing to unpack .../systemd_237-3ubuntu10.6_amd64.deb ...

Unpacking systemd (237-3ubuntu10.6) over (237-3ubuntu10.6) ...

Preparing to unpack .../man-db_2.8.3-2ubuntu0.1_amd64.deb ...

Unpacking man-db (2.8.3-2ubuntu0.1) over (2.8.3-2ubuntu0.1) ...

Preparing to unpack .../dbus_1.12.2-1ubuntu1_amd64.deb ...

Unpacking dbus (1.12.2-1ubuntu1) over (1.12.2-1ubuntu1) ...

Preparing to unpack .../ureadahead_0.100.0-20_amd64.deb ...

Unpacking ureadahead (0.100.0-20) over (0.100.0-20) ...

Processing triggers for mime-support (3.60ubuntu1) ...

Setting up ureadahead (0.100.0-20) ...

Setting up systemd (237-3ubuntu10.6) ...

Setting up man-db (2.8.3-2ubuntu0.1) ...

Updating database of manual pages ...

Setting up dbus (1.12.2-1ubuntu1) ...

A reboot is required to replace the running dbus-daemon.

Please reboot the system when convenient.

These packages remain in a half state, although dpkg -s doesn't list any issues.

# dpkg -s dbus

Package: dbus

Status: install ok installed

Priority: standard

Section: admin

Installed-Size: 559

Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>

Architecture: amd64

Multi-Arch: foreign

Version: 1.12.2-1ubuntu1

Depends: adduser, lsb-base, libapparmor1 (>= 2.8.94-0ubuntu1), libaudit1 (>= 1:2.2.1), libc6 (>= 2.14), libcap-ng0, libdbus-1-3 (= 1.12.2-1ubuntu1), libexpat1 (>= 2.1~beta3), libselinux1 (>= 2.0.65), libsystemd0

Suggests: default-dbus-session-bus | dbus-session-bus

Conffiles:

 /etc/default/dbus 0d0f25a2f993509c857eb262f6e22015

 /etc/init.d/dbus ec9a7d183ec50837a12aca3f9c95cc27

I am COMPLETELY out of ideas. Any help is greatly appreciated!