Log files sometimes don't get cleared by cleanup script

[m90]

I'm trying to package an application for DO using packer.

After creating my image I do run the (top-level) cleanup script provided in this repository and the (top-level) image check script.

Doing so I am running in intermittent failures, having the image check complain about un-cleared log files (either /var/log/ufw.log or /var/log/auth.log or sometimes both of them). I would assume that 75% of the build succeed, the rest will fail for the above reason.

I have a hard time understanding how these log files cannot be caught here:

marketplace-partners/scripts/cleanup.sh

Lines 11 to 12 in 28849bb

	find /var/log -mtime -1 -type f -exec truncate -s 0 {} \;
	rm -rf /var/log/*.gz /var/log/*.[0-9] /var/log/*-????????

and it's even stranger that manually adding:

rm -f /var/log/auth.log /var/log/ufw.log

does not resolve the issue either.

Is there some race condition going on here? I also added set -eo pipefail to the cleanup script so I can be sure that it does not error on something unexpected here before trying to delete the log files.

This is the order of scripts in my Packer config

    {
      "type": "shell",
      "environment_vars": [
        "OFFEN_VERSION={{user `offen_version`}}"
      ],
      "scripts": [
        "scripts/10-install",
        "scripts/20-configure_service",
        "scripts/30-configure_firewall",
        "scripts/40-add_firstrun",
        "scripts/90-cleanup",
        "scripts/99-img_check"
      ]
    }

and for the sake of completeness this is the error output:

    digitalocean: Checking for log files in /var/log                                                                                                                                                        
    digitalocean:                                                                                                                                                                                           
    digitalocean: [WARN] un-cleared log file, /var/log/ufw.log found       

[m90]

So I debugged this further by creating an image that does not run the check and creating a droplet from that image. It seems like someone is writing to /var/log/auth.log after it has been cleared by the cleanup script or it is never cleared:

Apr  3 13:54:43 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20845]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
Apr  3 13:54:43 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20845]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Apr  3 13:54:43 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20845]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Apr  3 13:54:43 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20845]: fatal: No supported key exchange algorithms [preauth]
Apr  3 13:55:53 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20873]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
Apr  3 13:55:53 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20873]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Apr  3 13:55:53 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20873]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Apr  3 13:55:53 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20873]: fatal: No supported key exchange algorithms [preauth]
Apr  3 13:56:33 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20883]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
Apr  3 13:56:33 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20883]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Apr  3 13:56:33 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20883]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Apr  3 13:56:33 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20883]: fatal: No supported key exchange algorithms [preauth]
Apr  3 13:57:04 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20891]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
Apr  3 13:57:04 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20891]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Apr  3 13:57:04 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20891]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Apr  3 13:57:04 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[20891]: fatal: No supported key exchange algorithms [preauth]
Apr  3 13:57:09 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 systemd-logind[825]: Power key pressed.
Apr  3 13:57:09 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 systemd-logind[825]: Powering Off...
Apr  3 13:57:09 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 systemd-logind[825]: System is powering down.
Apr  3 13:57:09 packer-5e873f75-52b1-b812-72a0-51d57e3237b2 sshd[1055]: Exiting on signal 15

This behavior seems to be tied to adding ufw like this:

#!/bin/bash
set -eo pipefail

configure_firewall () {
  ufw default deny incoming
  ufw default allow outgoing
  ufw allow ssh
  ufw allow http
  ufw allow https
  ufw --force enable
}

echo "---> Configuring UFW firewall..."
configure_firewall
echo "---> Successfully configured firewall."

as I never see the errors about non-cleared logs when I skip that step. Yet, it will create a warning about no firewall being configured.

[m90]

This seems to be a duplicate of #90

[scott]

I finally had to patch img_check.sh line 634 to get it to pass and get Packer to build the image.

sudo rm /var/log/auth.log  /var/log/kern.log /var/log/ufw.log && checkLogs

Seems less than ideal.

[m90]

We worked around it by deleting the logs as the very last step of the cleanup step. This means disk space occupied by them will not be zeroed out though. The image check passes reliably when doing that though. See: https://github.com/offen/digitalocean/blob/f3d73aa1b525c6282b073df8898ea9cfb100237c/scripts/90-cleanup#L38-L41

[m90]

The most reliable way to handle this is probably: #90 (comment) - although this will be very hard when done from a CI environment or similar.

[ximon18]

If the imaging tool, eg Packer, can create and destroy the cloud firewall, then this is easy. Packer CAN do this for AWS, but not for DO.

…

On 13 May 2020, at 19:24, Frederik Ring ***@***.***> wrote:  The most reliable way to handle this is probably: #90 (comment) - although this will be very hard when done from a CI environment or similar. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.