The Linux commands I actually needed to run a server this year

I run two small Ubuntu servers on DigitalOcean. One runs Sendy, the app I use to send my newsletter. The other runs a self-hosted Plausible Analytics instance for this site. I connect to both from my Mac over SSH.

In 2026 they gave me a run of incidents: a process that kept dying, a disk at 99%, a newsletter that took eight hours to send, a security pass on a rebuilt machine, and a Plausible upgrade I had postponed for years. I wrote a post about each one. This post collects the commands that mattered across all of them, with the exact invocation, the real output where a post quotes it, and what I decided because of it.

Most of these were typed by a coding agent (Codex on my Mac, working over SSH) while I watched. During the full disk morning I typed them myself and pasted the output back. Either way I had to read the output and make the call.

I log in as root on these boxes, so the original sessions often skip sudo. I added it here where a normal user needs it.

1. ssh: get in

Everything below runs on the server. This is the one command that runs on the Mac:

ssh root@<IP> -i ~/.ssh/digitalocean

-i picks the private key. DigitalOcean installed the matching public key when the droplet was created, so there is no password step. That exact line is also what I paste to the agent when I want it to work on a server.

If you type it often, put the host in ~/.ssh/config and use an alias, which I covered in SSH for developers. And for long work by hand, start it inside tmux: during the Plausible upgrade the SSH session reset in the middle of a 2.2 GB backup and the agent had to reconnect.

2. df -h: how full is the disk

df reports free space per filesystem. -h prints sizes as G and M instead of blocks.

df -h

One morning the Sendy server stopped responding and I couldn’t even SSH in. After a reboot from the DigitalOcean panel, this is the line df -h printed for the root filesystem:

/dev/vda1        48G   47G  951M  99% /

951 MB left on a 48 GB disk, which is why a reader had emailed me that morning to say the unsubscribe link didn’t work. df tells you that the disk is full and nothing about what fills it.

3. du -xhd1 | sort -h: what is eating it

du measures how much space a directory tree uses. These flags make it useful on a server:

sudo du -xhd1 / | sort -h

-x stays on one filesystem, so it skips /proc and mounted volumes. -h gives human sizes. -d1 goes one level deep. Piping into sort -h sorts those human sizes correctly, so the biggest directory ends up last.

On the full disk it printed:

43G     /var
47G     /

The same command on /var gave 42G /var/lib, and on /var/lib:

42G     /var/lib/mysql
42G     /var/lib

Three runs, each one level deeper, and I had the directory.

4. ls -lhS: which files, exactly

Once du points at a directory, ls sorted by size tells you what’s inside:

sudo ls -lhS /var/lib/mysql | head -40

-S sorts by size, biggest first. This is what came back, over and over:

-rw-r----- 1 mysql mysql 111M May  5 09:47 binlog.000300
-rw-r----- 1 mysql mysql 110M May  5 09:47 binlog.000301
-rw-r----- 1 mysql mysql 105M May  5 17:11 binlog.000408

Over 400 files, about 100 MB each. MySQL binary logs, a journal of every change made to the database. The Sendy database itself was 528 MB. The logs were 41 GB.

The decision this led to was not rm. MySQL keeps an index of its binary logs and can refuse to start if they vanish, so the fix went through MySQL:

sudo mysql -e "PURGE BINARY LOGS BEFORE DATE_SUB(NOW(), INTERVAL 1 DAY);"

Then df -h again:

/dev/vda1        48G   20G   29G  41% /

From 951 MB free to 29 GB. The permanent fix was binlog_expire_logs_seconds=86400 under [mysqld] in /etc/mysql/mysql.conf.d/mysqld.cnf. Sendy writes a row for every send, open and click, and MySQL 8 turns binary logging on by default, so the logs had grown at the speed of my newsletter.

5. journalctl --vacuum-time and the other cheap cleanups

Before digging, free the easy space. The systemd journal first:

journalctl --disk-usage
sudo journalctl --vacuum-time=3d

The first prints the journal’s size, the second deletes archived journal files older than three days. sudo apt clean and sudo apt autoremove --purge do the same for the package cache. Neither was my problem that day, but they were the first two items in the triage plan Codex gave me.

If a log file is the problem, empty it instead of deleting it:

sudo truncate -s 0 /var/log/apache2/access.log

If you rm a log that a running process has open, the process keeps writing to the deleted file and the space is not freed until you restart it.

On a Docker host, docker system df shows what images, containers and volumes take. After the Plausible upgrade, docker image prune -a recovered about 2.57 GB of old images without touching the volumes.

6. journalctl -k: why did my process die

In April 2026 I installed Claude Code on the Sendy server and it got killed a few seconds after starting, every time, with no error. I pointed Codex at the server from my Mac, and this is the command it ran:

journalctl -k --since "2 hours ago" --no-pager | grep -Ei "killed process|out of memory|oom|claude|node" | tail -n 80

-k shows only kernel messages, --since limits the window, --no-pager prints everything instead of opening less. The grep filters for the words Linux uses when it kills a process for using too much memory. It found this:

Out of memory: Killed process ... (claude)
anon-rss: ~864MB

That’s the OOM killer, the part of the kernel that picks a process and kills it when RAM runs out and there is no swap. The droplet had 2 GB of RAM, MySQL was using about 786 MB, and claude grew to about 860 MB. The whole session is in Debugging a process killed on my server with Codex.

One gotcha: -k implies -b, the current boot only. If the machine rebooted since the kill, add -b -1 to read the previous boot. sudo dmesg -T reads the same kernel buffer.

7. free -h and ps --sort=-rss: where the memory went

free shows total, used, free and available memory, plus swap. -h scales the numbers:

free -h

On the Sendy droplet, during the OOM session, it reported 1.9 GiB total and 0B swap. With zero swap a tight machine has nothing to fall back on, so the kernel starts killing. The column to read is available, not free, because Linux uses spare RAM for disk cache and gives it back when a process asks.

free says memory is tight. ps says who has it:

ps -eo pid,ppid,comm,rss,%mem,%cpu,args --sort=-rss | head -n 15

-e lists every process, -o picks the columns, --sort=-rss orders by resident memory, biggest first. On the Sendy server the list showed MySQL at roughly 786 MB, several Apache workers, and claude on its way to 860 MB. Two processes wanted more than the box had.

8. swapon: give the kernel a buffer

Swap is disk space the kernel uses as overflow when RAM runs out. It’s slow, but it gives the system a buffer instead of a hard wall.

Both my servers had none. Both got a 2 GB swap file during this series, added by the agent:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

fallocate reserves the space, chmod 600 keeps other users out of it, mkswap formats it, swapon activates it. To keep it after a reboot, add /swapfile none swap sw 0 0 to /etc/fstab. swapon --show lists the active swap, and nothing printed means none.

Swap was the cheaper fix for the OOM problem, and Codex listed it first. I resized the droplet instead, which turned into a much bigger mistake.

9. systemctl status and journalctl -u: is the service running

systemd manages the services on Ubuntu, and systemctl is how you ask about them:

systemctl status mysql

It prints whether the unit is active, since when, its main PID, memory use and the last few log lines. After the binary log purge, the new expiry setting needed sudo systemctl restart mysql to apply.

The security pass also ran a check I had never thought of:

systemctl --failed

This lists units in the failed state, and an empty list is what you want. After the Sendy server’s upgrade and reboot, it confirmed Apache, MySQL, SSH, Fail2ban and cron were all active again.

Service logs live in the journal, and -u picks one unit:

sudo journalctl -u ssh --since "10 minutes ago"

On Ubuntu the SSH daemon’s unit is ssh, not sshd. This shows every connection attempt in the window, including the ones the server rejected and why. ssh -v on the client shows what your computer offered; this shows why the server said no.

10. docker compose ps and logs: the same questions on the Docker box

Plausible runs as three containers, so “is it running” becomes:

docker compose ps

Run it inside the directory with the compose.yml (mine is /opt/plausible-ce-current). It lists each service with its status and ports. After the upgrade all three showed healthy or running: Plausible CE v3.2.0, PostgreSQL 16, ClickHouse 24.12.

The first HTTP check after the upgrade got a 502. The logs explained it:

docker compose logs -f plausible

The app was still running database migrations. Once the logs showed normal startup, the check returned 200. A 502 right after a restart often means “not ready yet”.

One snag from that session: docker compose exec -T hung after the command inside had finished. Plain docker exec -i <container> worked.

11. curl -I: does it answer

curl -I sends a HEAD request and prints only the response headers. It works from the Mac as well as from the server:

curl -I https://b.flaviocopes.com

The answers I was after: HTTP/2 200 after the Plausible upgrade, 200 OK after every hardening step on the Sendy server (a security pass that takes the site down is a classic), and HTTP/1.1 101 Switching Protocols on /live/websocket once the Nginx websocket headers were fixed, which is what proved the Plausible upgrade was done.

The variant worth knowing is --resolve. During the Sendy migration the new server had to be tested before DNS pointed at it, and --resolve makes curl use a specific IP for a hostname:

curl -I --resolve list.flaviocopes.com:443:<new-ip> https://list.flaviocopes.com/

TLS still validates against the real hostname.

12. dig +short: what does DNS say

dig asks DNS a question. +short prints just the answer:

dig +short list.flaviocopes.com

Right after the migration I got ERR_CONNECTION_REFUSED in the browser. The new server was fine. My browser had the old IP cached, and the old Apache had just been stopped. dig +short printing the new IP was the sign DNS had caught up.

When the newsletter took eight hours to send and the open rate dropped from 24% to 16%, the open rate part traced back to a record I had set to “proxied” in Cloudflare. With the proxy on, dig returns Cloudflare’s IPs, not your server, and AWS SNS webhooks were not reaching Apache. A public resolver shows what the world sees:

dig +short list.flaviocopes.com @1.1.1.1

Switching the record to DNS-only fixed the webhooks.

13. ss -tulpn: what is listening

ss lists sockets. The flags that matter on a server:

sudo ss -tulpn

-t TCP, -u UDP, -l listening only, -p the process that owns each socket, -n numeric ports instead of service names. You get one line per listener: local address and port, plus the program behind it.

On the Sendy box, the security audit confirmed MySQL listened only on 127.0.0.1 and Apache exposed only 80 and 443. On the Plausible box, the app listens on 127.0.0.1:8000 behind Nginx.

Anything listening on 0.0.0.0 or * is reachable from the internet unless the firewall says otherwise. A service bound to localhost is safe even with the firewall off, so read this list before the firewall rules.

14. ufw status verbose: is the door locked

UFW is Ubuntu’s firewall front end:

sudo ufw status verbose

It prints Status: active or inactive, the default policies for incoming and outgoing traffic, and one line per rule.

On the freshly migrated Sendy server it said inactive, so every port on the machine was reachable from the internet. When I asked the agent what the biggest issue had been, this was its answer. The fix was to allow 22, 80 and 443 and turn it on. The whole audit is in Securing a server using AI.

On the Plausible box it was active, but with two extra rules: public 2375/tcp and 2376/tcp, the Docker API ports. Nothing was listening there, and they got deleted.

15. fail2ban-client status sshd: who is knocking

Fail2ban watches the logs and bans IPs that fail to log in too many times. To see it working:

sudo fail2ban-client status sshd

It prints the filter side (currently failed, total failed, which log it reads) and the action side (currently banned, total banned, the banned IP list). On a public server the total climbs steadily.

The security pass confirmed the jail was active on the Sendy server. SSH there rejects passwords, and root can log in with a key only (PermitRootLogin prohibit-password), so the bans are mostly noise reduction. If you change the SSH configuration, run sudo sshd -t before restarting, and keep your session open until a second login works.

16. apt update && apt upgrade: keep it patched

Ubuntu prints pending updates in the login banner: 124 security updates on the rebuilt Sendy server, 127 lines on the Plausible box that had gone untouched for years. Applying them:

sudo apt update && sudo apt upgrade

update refreshes the package lists. upgrade installs newer versions of what you have without removing anything. apt list --upgradable shows what’s pending first, and after the Plausible upgrade only ubuntu-advantage-tools was left on that list.

Kernel and libc updates need a reboot, and Ubuntu leaves a marker when that’s the case:

cat /var/run/reboot-required

Both servers needed it. After the reboot, uname -r tells you which kernel is running. The Sendy server came back on 6.8.0-110-generic, the Plausible one on 5.15.0-176-generic (more on uname in Linux commands: uname). The agent rebooted on purpose, to check that the firewall and the SSH policy survived a restart.

Ubuntu can also install security updates on its own, through the unattended-upgrades package. To see what it would do right now, without doing it:

sudo unattended-upgrade --dry-run -d

The security pass enabled this on the Sendy server: refresh the package lists daily, install security updates automatically, clean the cache weekly. The one thing it can’t do is the kernel reboot, which I can schedule at a quiet hour when /var/run/reboot-required shows up.

The last check the agent ran after hardening was sudo certbot renew --dry-run, a rehearsal of the certificate renewal against Let’s Encrypt’s staging server, because copied certificates are only useful if renewal works on the new box.

17. crontab -l: what runs on a schedule

Sendy sends scheduled campaigns from cron. During the migration the agent disabled that cron job on the new server until cutover, because two live servers would have sent 150,000 people every email twice. I hadn’t thought of that.

crontab -l
systemctl list-timers

crontab -l lists the current user’s cron jobs. systemctl list-timers shows the systemd timers, where newer software puts its scheduled work. Read both before cloning a server.

What I’d check first now

I still have close to zero Linux server skills, and an agent typed most of what you just read. What changed over these months is that I recognize the output: 99% from df -h, 0B in the swap row of free -h, Killed process in the kernel log, Status: inactive from ufw. Each one tells me which command to run next.

Setting up a server tomorrow, I would run five of these before installing anything: free -h to check for swap, ufw status verbose, ss -tulpn, apt list --upgradable, and df -h with a reminder to run it again in a month.

If the command line itself is new to you, the free Shell Commands course covers the basics, and the Linux Server Troubleshooting course turns a list like this one into a method.

Want me to talk about your product? You can sponsor this site.