For host admins#

Host admins have root access to the host machine. They are responsible for everything, especially for security.

In this chapter, we sketch how to set up a (hopefully) secure host publicly accessible from the outside world, and we provide information on maintenance tasks, general ones and JupyterHub related ones.

Considerations and steps described here by no means cover all relevant situations and settings. They originate from setting up public test systems for the Ananke project and, thus, focus on hardware and network features available to the project team during time of development.

We assume that the reader has basic Linux administration and network knowledge.

Installing the host system#

Warning

Setting up a publically accessible server is highly nontrivial. It’s a job for an experienced administrator. The Ananke project team at the moment does not have an experienced administrator (just some mathematicians and other scientists). Instructions written down below may contain rookie mistakes and potentially may harm your machine and your network! Ensure that you understand every single step you go and be aware of the consequences.

Ananke is mainly developed and tested on Debian GNU/Linux, but should work on all other major Linux distributions, too.

Major sources for the install and configuration instructions:

Base system#

If you choose Debian for the host machine, follow the usual install instructions. In some cases you might be interested in a cross-install.

Choose a minimal installation (no GUI required).

E-Mail notifications#

We would like to receive an email if something important happens on the server. Daily or weekly usage reports via email are a good idea, too.

To make email notifications work, we need a world-facing SMTP server. Setting one up on our host machine is not trivial and should be avoided (usually all mails coming from ‘small’ SMTP servers are classified as spam). A better idea is to use an external SMTP server (a free mailer or your institution’s mail server).

Important

Don’t use your standard mail account (at your institution, for instance) for sending because we have to store credentials on your server. Although only admin users have access to the credentials file, it’s not a good idea. Better create a new mail account solely used for sending notifications from your server to the outside world.

Here we use the small and simple DragonFly Mail Agent (dma). Install:

sudo apt install dma

For system mail name use your machine’s hostname and domain name (host.domain.org). Leave the smarthost field blank.

Install standard tools for mail management (send mail to other users aso.):

sudo apt install mailutils

To configure dma modify /etc/dma/dma.conf as follows

SMARTHOST your_mail_accounts_smtp_server
PORT your_mail_accounts_smtp_port
AUTHPATH /etc/dma/auth.conf
SECURETRANSFER
MAILNAME /etc/mailname
MASQUERADE your_mail_accounts_mail_address

Comment all other lines. Now write your credentials to /etc/dma/auth.conf:

your_mail_accounts_username|your_mail_accounts_smtp_server:your_mail_accounts_password

All emails sent from root or your admin user shall be forwarded to your standard (institutional) email account. Emails generated by other accounts shall be ignored. Modify /etc/aliases as follows

root: your_username
your_username: where.to.send@notifications.to
*: mail_trash_user

Then run

sudo newaliases

We may remove the admin user’s local inbox if it has been created:

rm /var/spool/mail/your_username

To prevent other users (container admins) from sending mails run

sudo chmod o-rwx /sbin/dma

and add your admin user to the mail group:

sudo adduser your_username mail

Logout and login to activate the new group membership.

Hardening the system#

Consider the following steps as suggestions. Some may be appropriate for your setting, some not. Some may be missing, some may be considered overly paranoid.

Unused network services#

If you are in an IPv4-only network, you may disable all the IPv6 stuff. There are several possibilities to accomplish this, but the simplest and most trustworthy one is to set a kernel option. Edit following lines in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="ipv6.disable=1 quiet"
GRUB_CMDLINE_LINUX="ipv6.disable=1"

Then

sudo update-grub
sudo reboot

In some settings, it may be appropriate to disable LLMNR. To /etc/systemd/resolved.conf add the line

LLMNR=no

Then

sudo systemctl restart systemd-resolved

To list all open TCP and UDP ports, run

ss -tulpan

SSH#

We use a simple form of two-factor authentication: password and cryptographic key.

Important

When modifying SSH config always keep one connection open until the new config works. Else you may get locked out from your machine.

The following /etc/ssh/sshd_config file is rather restrictive:

Include /etc/ssh/sshd_config.d/*.conf

Port 986
AddressFamily inet
ListenAddress 123.456.789.012

PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive
UsePAM yes

LoginGraceTime 1m
X11Forwarding no
PrintMotd no
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

AllowUsers your_username

Choose a non-standard port for SSH to become invisible to attackers scanning only for port 22. The ListenAddress is the IP address of the host’s network interface used for internet connectivity. Only SSH connections coming in via that IP address are allowed. See sshd_config man page for details.

For crypto key authentication, your admin user has to provide a public key:

cd ~
mkdir .ssh
chmod 700 .ssh
cd .ssh
touch authorized_keys
chmod 600 authorized_keys

Place the public key in authorized_keys file. On a Linux machine a user’s standard public key is in ~/.ssh/id_rsa.pub. If you do not have a key pair on your client machine (the one used to SSH into the server), create one with

ssh-keygen

Now restart sshd:

sudo systemctl restart sshd.service

If you see the message ‘syslogin_perform_logout: logout() returned an error’ in your logs after logout, create the file /etc/tmpfiles.d/run-utmp.conf containing

f /run/utmp 0644 root root

and then run sudo systemd-tmpfiles --create. See Leap-16: openssh misses /run/utmp? for some background on this problem.

Fail2ban#

To prevent brute-force attacks against SSH, install and enable ‘Fail2ban’:

sudo apt install fail2ban
sudo systemctl enable fail2ban
sudo systemctl start fail2ban

Place the following in /etc/fail2ban/jail.local:

[DEFAULT]
destemail = where.to.send@notifications.to
sender = your_mail_accounts_mail_address
sendername = Fail2ban
banaction = nftables-multiport
banaction_allports = nftables-allports
action = %(action_mwl)s

[sshd]
enabled = false

[sshd-aggressive]
enabled = true
filter = sshd[mode=aggressive]
port = ssh
backend = systemd
journalmatch = _SYSTEMD_UNIT=sshd
action = %(action_mw)s

Then

sudo systemctl restart fail2ban

Fail2ban allows only for a limited number of failed login attempts (default: 3 attempts in 10 minutes). IP addresses with to many attempts are blocked for a certain time (default: 10 minutes) via temporary firewall rules and an email is sent to you. To list all currently blocked IP addresses run

sudo fail2ban-client status sshd-aggressive

To unban an IP address, manually run

sudo fail2ban-client set sshd-aggressive unbanip 123.456.789.012

Firewall#

Debian’s standard firewall is nftables. To enable it run

sudo systemctl enable nftables.service
sudo systemctl start nftables.service

Currently active filter rules are shown with

sudo nft list ruleset

Create the file /etc/nftables.conf with content

#!/usr/sbin/nft -f

flush ruleset

table ip filter {
    chain in_ipv4 {
        type filter hook input priority 0; policy drop;
        ct state vmap { established : accept, related : accept, invalid : drop }
        iifname lo accept
        icmp type echo-request limit rate 5/second accept
        tcp dport 986 accept
        log prefix "[nftables] Inbound Denied: " counter drop
    }
    chain for_ipv4 {
        type filter hook forward priority 0; policy drop;
    }
    chain out_ipv4 {
        type filter hook output priority 0; policy accept;
    }
}

and run

sudo nft -f /etc/nftables.conf

to get a rather restrictive firewall (only outgoing connections and SSH allowed, limited ICMP echos).

To restrict SSH access to certain IP ranges replace tcp dport 986 acceptby ip saddr {12.34.56.0/8, 123.45.0.0/16} tcp dport 986 accept.

Configuration details can be found in the nftables wiki.

Log reports#

We would like to get daily log summaries via email. This can be done with logwatch. Install

sudo apt install logwatch net-tools

and create /etc/logwatch/conf/logwatch.conf with content

Detail = 5
Service = All

Then create a dummy log file /var/log/logwatch_journald.log with arbitrary content (not empty!):

This file exists to make logwatch work with journald.
Logwatch wants to have some log file, but messages to process will come from journalctl.

Create the file /etc/logwatch/conf/logfiles/journald.conf with content

LogFile = logwatch_journald.log

and a file /etc/logwatch/conf/services/journald.conf with content

LogFile =
LogFile = journald
*JournalCtl = "--output=cat"
Title = "journald messages"

We also need a file /etc/logwatch/scripts/services/journald with content

#!/usr/bin/bash
egrep 'error|warning|critical'

This must be executable:

sudo chmod a+x /etc/logwatch/scripts/services/journald

Logwatch will send a daily report to the root user (which will be forwarded to your mail account). Time for daily tasks can be adjusted in /etc/crontab.

If reports are a bit lengthy because of lots of HTTP errors, you may adjust the detail level of webserver related messages by creating a the file /etc/logwatch/conf/services/http.conf containing

$http_rc_detail_rep_400 = 20
$http_rc_detail_rep_403 = 20
$http_rc_detail_rep_404 = 20
$http_rc_detail_rep_405 = 20

This shows only a summary of HTTP errors in the reports. Default values and available options can be found in /usr/share/logwatch/default.conf/services/http.conf.

Restrict file access#

Debian’s standard file access permission should be tightened somewhat. Access rights for newly created files usually are 755 (or 644). We change this to 750 (or 640). Then only the file’s owner has full access, and its group has read access. This prevents users from looking into home directories of other users. For this purpose create a file /etc/profile.d/umask.sh with content

umask 027

Set access rights for new home directories accordingly by editing /etc/adduser.conf:

DIR_MODE=0750

and running

sudo chmod -R o-rwx /etc/skel

For already existing home directories we have to adjust access manually:

sudo chmod -R o-rwx /root
sudo chmod -R o-rwx /home/your_username

Finally, we adjust access to some config files:

sudo chmod o-rwx /etc/sudoers.d

sudo chmod o-rwx /etc/ssh/sshd_config
sudo chmod o-rwx /etc/ssh/sshd_config.d

sudo chmod -R o-rwx /etc/cron.d
sudo chmod -R o-rwx /etc/cron.daily
sudo chmod -R o-rwx /etc/cron.hourly
sudo chmod -R o-rwx /etc/cron.monthly

sudo chmod o-rwx /etc/crontab

Unused kernel modules#

Debian’s Linux kernel ships with some kernel modules for network services we do not need. Thus, we should deactivate them. Simply add some lines to /etc/modprobe.d/blacklist.conf:

install dccp /bin/true
install sctp /bin/true
install rds /bin/true
install tipc /bin/true

Core dumps#

Core dumps are used for debugging programs, but may also be used by attackers to obtain system information. We should prevent the kernel from automatically generating core dumps if something went wrong. Create /etc/sysctl.d/50-coredump.conf with content

kernel.core_pattern=/dev/null

and then run

sudo sysctl -p /etc/sysctl.d/50-coredump.conf

To /etc/security/limits.conf add the line

* hard core 0

Malware and vulnerability detection#

We should install some tools for detecting malware and unexpected modifications of the system.

To scan for rootkits install rkhunter and chkrootkit:

sudo apt install chkrootkit rkhunter

Test them with

sudo rkhunter -c --rwo
sudo chkrootkit -q

If rkhunter yields false positives, tell it that they are okay by running

sudo rkhunter --propupd

In some cases, like warnings about hidden files, you have to modify the file /etc/rkhunter.conf. A line ALLOWHIDDENFILE=/path/to/file there supresses corresponding message about a hidden file.

Vulnerable Debian packages can be found with debsecan. Install:

sudo apt install debsecan

Run it with

sudo debsecan --suite trixie --format report

to see all vulnerabilities. Add the --only-fixed argument to see only vulnerabilities fixable by updating corresponding packages.

Lynis scans the system for all kinds of standard security problems and suggests solutions. Install with

sudo apt install lynis

Run with

sudo lynis audit system

The debsums tool looks for modifications of files from installed packages. Install with

sudo apt install debsums

Run

sudo debsums -ca

to get a list of files modified since installation of the corresponding package.

Install

sudo apt install apt-listbugs apt-listchanges

to automatically get a list of bugs and changes for packages you install with apt.

With

sudo apt install needrestart

you get information about the required reboot after each package installation.

Intrusion detection#

To check for modifications to your system you may use TripWire or AIDE. To install AIDE run

sudo apt install aide

To add a list of files to ignore create /etc/aide/aide.conf.d/00_excludes with content

!/$
-/home/
-/proc/
-/var/lib/aide/
-/var/lib/systemd/timers/
-/var/log/account/
!/var/log/account
-/var/log/audit/
!/var/log/audit
-/var/log/sysstat/
!/var/log/sysstat
!/var/log/chkrootkit/chkrootkit-daily.log
!/var/log/chkrootkit/log.today
!/var/log/chkrootkit/log.today.raw
!/var/log/lynis-report.dat
!/var/log/lynis.log
!/var/log/wtmp.db
!/var/log/nginx/access.log
!/var/log/nginx/error.log
-/var/log/aide/
-/var/spool/
-/var/cache/
-/dev/disk/
-/run/
-/var/lib/nginx/
-/var/www/html/

You may adjust this to your needs, see aide.conf manpage.

Initialize the AIDE data base with

sudo aide --init --config=/etc/aide/aide.conf
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

After modifications to your system run

sudo aide --update --config=/etc/aide/aide.conf

to update the data base.

To get daily AIDE reports via email, run

sudo adduser _aide mail

and add the line MAILCMD=mail to /etc/default/aide.

System monitoring#

We would like to gather statistics on who did what on our machine. If we are facing problems, we may find the source more easily. In addition, we have to keep an eye on network and CPU load to know when to go shopping for a better machine. In this section, we install several tools for system monitoring and reporting.

User activity can be monitored with the GNU Accounting Utilities. Install:

sudo apt install acct
sudo accton on

To get user connect time statistics, commands executed, and a summary run

sudo ac
sudo lastcomm
sudo sa

See GNU Accounting Utilities documentation for details.

Performance monitoring tools are provided by the sysstat package. Install:

sudo apt install sysstat
sudo systemctl start sysstat
sudo systemctl enable sysstat

For general statistics run

sudo sar
sudo iostat

See sysstat project page for details.

The Linux Auditing System (auditd) records all kinds of events on the system (e.g., syscalls). It’s quite useful for incident analysis. Install:

sudo apt install auditd audispd-plugins
sudo systemctl start auditd
sudo systemctl enable auditd

The file /etc/audit/rules.d/audit.rules controls what to log. We may use one-size-fits-all audit.rules or more individual auditing rules like

## First rule - delete all
-D

## Increase the buffers to survive stress events.
## Make this bigger for busy systems
-b 32768

## This determine how long to wait in burst of events
--backlog_wait_time 60000

## Set failure mode to syslog
-f 1

-a always,exit -F arch=b64 -F dir=/var/log/audit/ -F key=LOG_audit
-a always,exit -F arch=b64 -F dir=/etc/audit/ -F perm=wa -F key=CFG_audit
-a always,exit -F arch=b64 -F path=/etc/libaudit.conf -F perm=wa -F key=CFG_audit

-a exit,always -F arch=b64 -S chmod -S fchmod -S chown -S fchown -S lchown -k SYSCALL_file_attributes
-a exit,always -F arch=b64 -S mount -k SYSCALL_mount
-a exit,always -F arch=b64 -S sethostname -k SYSCALL_sethostname

-a always,exit -F arch=b64 -F path=/etc/cron.allow -F perm=wa -F key=CFG_cron
-a always,exit -F arch=b64 -F path=/etc/cron.deny -F perm=wa -F key=CFG_cron
-a always,exit -F arch=b64 -F dir=/etc/cron.d/ -F perm=wa -F key=CFG_cron
-a always,exit -F arch=b64 -F dir=/etc/cron.daily/ -F perm=wa -F key=CFG_cron
-a always,exit -F arch=b64 -F dir=/etc/cron.hourly/ -F perm=wa -F key=CFG_cron
-a always,exit -F arch=b64 -F dir=/etc/cron.monthly/ -F perm=wa -F key=CFG_cron
-a always,exit -F arch=b64 -F dir=/etc/cron.weekly/ -F perm=wa -F key=CFG_cron
-a always,exit -F arch=b64 -F path=/etc/crontab -F perm=wa -F key=CFG_cron
-a always,exit -F arch=b64 -F path=/var/spool/cron/root -F key=CFG_cron

-a always,exit -F arch=b64 -F path=/etc/group -F perm=wa -F key=CFG_login
-a always,exit -F arch=b64 -F path=/etc/passwd -F perm=wa -F key=CFG_login
-a always,exit -F arch=b64 -F path=/etc/shadow -F key=CFG_login
-a always,exit -F arch=b64 -F path=/etc/login.defs -F perm=wa -F key=CFG_login
-a always,exit -F arch=b64 -F path=/var/log/lastlog -F key=CFG_login

-a always,exit -F arch=b64 -F path=/etc/hosts -F perm=wa -F key=CFG_hosts
-a always,exit -F arch=b64 -F dir=/etc/init.d/ -F key=CFG_init
-a always,exit -F arch=b64 -F path=/etc/ld.so.conf -F perm=wa -F key=CFG_ld
-a always,exit -F arch=b64 -F dir=/etc/ld.so.conf.d/ -F perm=wa -F key=CFG_ld
-a always,exit -F arch=b64 -F path=/etc/localtime -F perm=wa -F key=CFG_time
-a always,exit -F arch=b64 -F dir=/etc/sysctl.d/ -F perm=wa -F key=CFG_sysctl
-a always,exit -F arch=b64 -F path=/etc/aliases -F perm=wa -F key=CFG_aliases
-a always,exit -F arch=b64 -F path=/etc/ssh/sshd_config -F key=CFG_sshd
-a always,exit -F arch=b64 -F path=/etc/issue -F perm=wa -F key=CFG_issue
-a always,exit -F arch=b64 -F path=/etc/issue.net -F perm=wa -F key=CFG_issue

-a exit,always -F arch=b64 -S umask -k SYSCALL_umask
-a exit,always -F arch=b64 -S adjtimex -S settimeofday -k SYSCALL_time

After modifying that file run

sudo systemctl restart auditd

Logs are written to /var/log/audit/audit.log and may be inspected with ausearch and aureport.

Resource limits#

To define memory and CPU usage limits for all container admins create the file /etc/systemd/system/user-.slice.d/50-memory.conf with content

[Slice]
MemoryMax=20G
CPUQuota=200%

and run sudo systemctl daemon-reload. Memory limit is in GB and 200% CPU quota corresponds to two CPU cores.

OOM killer configuration#

With default settings for out-of-memory (OOM) handling in rare situtations the system loses SSH connectivity. For instance, spawning lots of (short-lived) Python processes eating up all memory, the OOM killer starts to kill important (long-lived) networking processes, because the OOM score for Python processes is relatively low. Preventing the Linux kernel from committing more memory than available (for performance reasons) should solve this problem. To do this, run

sudo sysctl vm.overcommit_memory=2
sudo sysctl vm.overcommit_ratio

Warning

Consider this solution experimental. The authors of this document do not fully understand the details here, but tested suggested commands on a production system. See How to Adjust Linux Out-Of-Memory Killer Settings for PostgreSQL for some background information.

Reverse proxy#

The host machine will provide (next to SSH) only one service to the outside world: an HTTP server. This server takes incoming requests and forwards them to the correct Podman container. Which container to forward to is determined by the request’s URL. Here we use nginx as HTTP server.

Connection between the world and HTTP server will be encrypted (HTTPS). Connection to containers won’t be encrypted because traffic does not leave the machine.

Install nginx:

sudo apt install nginx

Then tell the firewall to accept incoming requests on ports 80 (HTTP) and 443 (HTTPS): in /etc/nftables.conf add the line tcp dport {80, 443} accept below the SSH related line and run

sudo nft -f /etc/nftables.conf

to reload firewall rules.

To improve security, we disable some nginx modules enabled by default:

cd /etc/nginx/modules-enabled
sudo rm -r *
sudo systemctl restart nginx

To set up encrypted communication, run

sudo openssl dhparam -out /etc/nginx/dhparam.pem 4096

and modify /etc/nginx/nginx.conf to have only the following lines:

user www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
}

http {
        sendfile on;
        tcp_nopush on;
        types_hash_max_size 2048;
        server_tokens off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA;
        ssl_prefer_server_ciphers on;
        ssl_dhparam /etc/nginx/dhparam.pem;

        access_log /var/log/nginx/access.log;

        gzip off;

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

You may use a self-signed certificate (for testing only) or Let’s Encrypt or a certificate issued by your institution. To create a self-signed certificate run

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/your_server.key -out /etc/ssl/certs/your_server.crt

Now create /etc/nginx/sites-available/some_name with content

map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
}

server {
        listen 80 default_server;

        server_name your-server.org;
        return 301 https://$server_name$request_uri;
}

server {
        listen 443 ssl default_server;

        server_name your-server.org;

        ssl_certificate /etc/ssl/certs/your_server.crt;
        ssl_certificate_key /etc/ssl/private/your_server.key;

        root /var/www/html;
        index index.html index.htm index.nginx-debian.html;

        client_max_body_size 10M;
        
        location / {
                try_files $uri $uri/ =404;
        }

}

and make this configuration active (and default configuration inactive) via

cd /etc/nginx/sites-enabled
sudo ln -s ../sites-available/some_name some_name
sudo rm default

sudo nginx -t
sudo systemctl restart nginx

The nginx -t makes nginx check its configuration. Run the last line only if the check result is okay.

Hint

The client_max_body_size 10M; line in /etc/nginx/sites-available/some_name restricts the size of HTTP file uploads (JupyterLab upload button, for instance) to the server to 10 MB. Default value is 1 MB.

Podman#

Install Podman:

sudo apt install podman containers-storage

Accounts for container admins#

In this section, we add a new JupyterHub (its Podman container) to the host machine. Each container will run in a separate user account to which the container admin has SSH access.

Hint

Standard Linux disk quotas work on a per user basis and cannot be setup for not yet exiting users. But Ananke’s Podman containers use systemd’s dynamic user feature resulting in more or less random and temporary user IDs. Thus, standard Linux quotas won’t work. To prevent a container from filling the host machine’s disk, we use separate virtual file systems for each container admin.

The container admin’s file system’s image file will be a sparse file. Thus, it requires much less disk space than its size suggests. See Disk quota checks and extension for commands to check container admins’ true disk usage.

Warning

The quota setup sketched in above hint does not prevent hub users from filling a hub’s disk capacity. But it prevents users from DOSing the whole host machine.

User account#

Prepare the container admin’s home directory with

CONT_ADMIN=testhub_user

sudo truncate -s 20G /home/$CONT_ADMIN.img
sudo mkfs.ext4 /home/$CONT_ADMIN.img
sudo mkdir /home/$CONT_ADMIN

Then add the line

/home/testhub_user.img /home/testhub_user ext4 loop,rw,nosuid,nodev    0    3

to /etc/fstab and run

sudo systemctl daemon-reload
sudo mount /home/$CONT_ADMIN
sudo adduser $CONT_ADMIN
sudo cp -r /etc/skel/. /home/$CONT_ADMIN/
sudo chown -R $CONT_ADMIN:$CONT_ADMIN /home/$CONT_ADMIN
sudo chmod -R go-rwx /home/$CONT_ADMIN

(adduser will complain about already existing home directory. This can be safely ignored.)

Do not forget to tell the container admin to change his or her password at first login!

SSH access#

To activate SSH access, modify corresponding line in /etc/ssh/sshd_config:

AllowUsers your_username testhub_user

Then restart sshd:

sudo systemctl restart sshd

Run

sudo mkdir /home/testhub_user/.ssh

Get the new container admin’s public key and write it to the file /home/testhub_user/.ssh/authorized_keys. Then

sudo chown -R testhub_user:testhub_user /home/testhub_user/.ssh
sudo chmod -R go-rwx /home/testhub_user/.ssh

Run the command

sudo loginctl enable-linger testhub_user

This tells systemd not to stop the user’s Podman containers on logout.

Port and hub name#

Open /etc/nginx/sites-available/your-server and place

        location /testhub/ {
                proxy_pass http://127.0.0.1:8000;

                proxy_redirect   off;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header Host $host;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;

                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
        }

in the server block right above the location / line. Then

sudo nginx -t
sudo systemctl restart nginx.service

Provide the location (testhub) and the port (8000 in proxy_pass line) to container admin.

For each container on the machine use different location and different port.

Enlarge container admin’s home dir#

If the initial maximum size of a container admin’s home directory turns out to be too small, you may increase the size limit. This requires three steps detailled below:

Stop the container und unmount the file system.
Entlarge the file system.
Mount the file system and restart the container.

For step 2 we provide two fundamentally different variants. A simple one, that should work, but does not work in at least some cases. And more complex one, that always works.

Stop and unmount#

Tell the containter admin to stop the container by running

systemctl --user stop ananke-base-hub.service

in the container admin’s shell (SSH). Here ananke-base-hub.service is the systemd service for the Ananke container. It’s structure is ananke-CONTAINER_DEFINITION.service, where CONTAINER_DEFINITION is the name of the directory containing the containers config.py.

Identify processes accessing the container admin’s home directory:

sudo fuser -mv /home/testhub_user

Then kill all processes listed there:

sudo kill PID_OF_PROCESS

Unmount the container admin’s home directory:

sudo umount /home/testhub_user

Enlarge the file system (simple method)#

In priciple, following commands should suffice to increase file system size by 5 GB. But in some settings Ananke’s development team observed file system corruption and only partial enlargement (not all tools recognize the larger file system size) for unknown reasons. To avoid data loss you should resort to the more complex (and more reliable) procedure described in the next section.

sudo truncate -s +5G /home/testhub_user.img
sudo e2fsck -f /home/testhub_user.img
sudo resize2fs /home/testhub_user.img

Enlarge the file system (complex method)#

Following steps create a new, larger file system and copy all files to this new file system. If something goes wrong, you may restore the old, small file system.

Rename the old image file:

sudo mv /home/testhub_user.img /home/testhub_user_old.img

Create a new, empty image file:

sudo truncate -s 25G /home/testhub_user.img
sudo mkfs.ext4 /home/testhub_user.img

Create a directory for mounting the old file system:
```
sudo mkdir /home/testhub_user_old
```

Add the line

/home/testhub_user_old.img /home/testhub_user_old ext4 loop,rw,nosuid,nodev    0    3

to /etc/fstab and run

sudo systemctl daemon-reload
sudo mount /home/testhub_user_old

to mount the old file system.

Mount the new file system:
```
sudo mount /home/testhub_user
```

Transfer all files from old to new file system:

sudo rsync -av /home/testhub_user_old/ /home/testhub_user

Unmount both file systems:

sudo umount /home/testhub_user_old
sudo umount /home/testhub_user

(optional) Remove the testhub_user_old line from /etc/fstab and run
```
sudo rm /home/testhub_user_old.img
rmdir /home/testhub_user_old
```
to remove the old file system.

Mount and restart#

Mount the home directory:

sudo mount /home/testhub_user

Tell the container admin to start its container by running

systemctl --user start ananke-base-hub.service

in the container admin’s shell (SSH).

Regular maintenance work#

At least weekly we should check the system for problems und install updates. Note that following the above installation instructions, there will be no automatic updates!

Updates#

Run

sudo apt update
sudo apt list --upgradable

und check the list of available updates (else you do not know what package might have killed your system). Then run

sudo apt upgrade

You should also have a look at vulnerable packages. Problems repairable by updating packages:

debsecan --suite trixie --format report --only-fixed

All vulnerable packages (including ones not repairable by update at the moment):

debsecan --suite trixie --format report

Malware#

Run

sudo chkrootkit
sudo rkhunter --check

and check the output. For each alert, carefully check whether you are in trouble or if it’s a false positive.

Run AIDE to see modifications to system files:

sudo aide --check --config=/etc/aide/aide.conf

If there are modifications and modifications are okay (!) run

sudo aide --update --config=/etc/aide/aide.conf
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

to tell Tripwire that modifications are okay.

Hint

It’s a good idea to keep a copy of AIDE’s data base outside the machine. Else an attacker could temper the data base in a way which prevents detection of files modified by the attacker.

Login attempts#

Have a look at the list of failed logins:

sudo last -f /var/log/btmp

Is someone trying to get access with brute-force or other methods? Did someone try to log in as you?

You may also have a look at all past logins to see you are working on your machine:

sudo last

Most recent login for each user:

sudo lastlog

Services and connections#

Have a look at all active background services:

sudo systemctl list-unit-files --state=enabled

Something unusual here?

Unusual network connections?

sudo lsof -i

General security scan#

Every time you modified the system (updates, config changes), run

sudo lynis audit system

to get hints for improving security.

System load#

System load reports can be shown with sar and iostat. Try

sar -q ALL -f /var/log/sysstat/saDD

where DD is the day of the month. This shows different statistics from which we see whether the machine had been at its maximum load that day. Many other statistics are available, too. For iostat try

iostat -N --human --pretty

This prints CPU and disk usage statistics.

Disk quota check#

List disk usage with

du -h /home/*.img

Optional features#

Features described below may be relevant to some users only.

GPU support#

If the host machine’s GPUs shall be accessible inside Podman containers, the steps below should be a good starting point. Here we only cover NVIDIA GPUs. Instructions are compiled and adapted from different sources:

Things change rapidly, so consult those sources, too.

Warning

Current Ananke version has not been tested with GPU support. All instructions below have been tested with Ananke 0.6 on Debian 12. For more recent Ananke and system versions you are on your own.

Install GPU drivers#

Debian’s NVIDIA driver packages require installation of some (not all) X-Server components. Maybe it’s not necessary, but we should tell the system to not boot into any graphical environment:

sudo systemctl set-default multi-user.target

GPUs visible to the system?

sudo apt install pciutils
lspci | grep -i nvidia

GPU drivers are on Debian’s non-free repo. Thus, append non-free to the relevant line in /etc/apt/sources.list. Then

apt update
apt search nvidia

Depending on your GPU you have to install nvidia-driver or nvidia-tesla-driver or maybe something different:

sudo apt install nvidia-tesla-driver nvidia-cudnn

(note that installing nvidia-cudnn may be unnecessary).

Now

nvidia-smi

should show all your GPUs and current driver version.

Podman with GPUs#

We need an NVIDIA repo for installing nvidia-container-toolkit. To make the repo available to apt run

sudo apt install curl

cd ~
curl -o nvidia.asc https://nvidia.github.io/libnvidia-container/gpgkey
cat nvidia.asc | gpg --dearmor > nvidia.gpg
sudo install -o root -g root -m 644 nvidia.gpg /usr/share/keyrings/nvidia-archive-keyring.gpg
rm nvidia.asc
rm nvidia.gpg

Then create the file /etc/apt/sources.list.d/nvidia-container-toolkit.list with content

deb [signed-by=/usr/share/keyrings/nvidia-archive-keyring.gpg] https://nvidia.github.io/libnvidia-container/stable/debian10/amd64 /

(note that although we are running Debian 12 we have to provide debian10 here until NVIDIA officially supports Debian 12).

Install nvidia-container-toolkit with

sudo apt update
sudo apt install nvidia-container-toolkit-base

and check with

nvidia-ctk --version

Now create the file /usr/local/bin/nvidia-start.sh with content

#!/bin/bash

/sbin/modprobe nvidia

if [ "$?" -eq 0 ]; then
  # Count the number of NVIDIA controllers found.
  NVDEVS=`lspci | grep -i NVIDIA`
  N3D=`echo "$NVDEVS" | grep "3D controller" | wc -l`
  NVGA=`echo "$NVDEVS" | grep "VGA compatible controller" | wc -l`

  N=`expr $N3D + $NVGA - 1`
  for i in `seq 0 $N`; do
    mknod -m 666 /dev/nvidia$i c 195 $i
  done

  mknod -m 666 /dev/nvidiactl c 195 255

else
  exit 1
fi

/sbin/modprobe nvidia-uvm

if [ "$?" -eq 0 ]; then
  # Find out the major device number used by the nvidia-uvm driver
  D=`grep nvidia-uvm /proc/devices | awk '{print $1}'`

  mknod -m 666 /dev/nvidia-uvm c $D 0
  mknod -m 666 /dev/nvidia-uvm-tools c $D 0
else
  exit 1
fi

and set permissions with

sudo chmod o+x /usr/local/bin/nvidia-start.sh

Then create /etc/systemd/system/nvidia-start.service with content

[Unit]
Description=Runs /usr/local/bin/nvidia-start.sh

[Service]
ExecStart=/usr/local/bin/nvidia-start.sh

[Install]
WantedBy=multi-user.target

and activate the systemd service with

sudo systemctl daemon-reload
sudo systemctl enable nvidia-start.service
sudo systemctl start nvidia-start.service

Container toolkit should work now, and we can generate information relevant to Podman:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
sudo chmod o+rx /etc/cdi

Running

cat /etc/cdi/nvidia.yaml | grep "name:"

should show all GPU identifiers (maybe including all).

Important

The above nvidia-ctk cdi generate step has to be rerun after each update of NVIDIA GPU drivers! Else Podman won’t work with GPU.

To make GPUs available in rootless Podman containers in /etc/nvidia-container-runtime/config.toml add/modify the line

no-cgroups = true

Test the setup by running nvidia-smi in a container:

podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L

Whenever GPUs shall be available in a Podman container, append

--device nvidia.com/gpu=all

to podman run, where all may be replaced by one of the GPU identifiers shown by the cat command above.

For host admins

Contents

For host admins#

Installing the host system#

Base system#

E-Mail notifications#

Hardening the system#

Unused network services#

SSH#

Fail2ban#

Firewall#

Log reports#

Restrict file access#

Unused kernel modules#

Core dumps#

Malware and vulnerability detection#

Intrusion detection#

System monitoring#

Resource limits#

OOM killer configuration#

Reverse proxy#

Podman#

Accounts for container admins#

User account#

SSH access#

Port and hub name#

Enlarge container admin’s home dir#

Stop and unmount#

Enlarge the file system (simple method)#

Enlarge the file system (complex method)#

Mount and restart#

Regular maintenance work#

Updates#

Malware#

Login attempts#

Services and connections#

General security scan#

System load#

Disk quota check#

Optional features#

GPU support#

Install GPU drivers#

Podman with GPUs#