When systemd swallows your service logs

You restart a service. Nothing happens. Or worse—it crashes silently. You check the console and find… nothing. Systemd ate your logs.

This is the 2 AM special. Your app is down, systemd says it’s “running”, but there’s not a single log message to tell you why it actually died. Here’s how to actually find them.

The Problem: Where Did stdout Go?

When you start a service directly (python app.py), you see everything printed to the terminal. When systemd starts the same service, all that stdout and stderr gets intercepted and redirected to the journal. But here’s the gotcha: systemd doesn’t display journal output in your terminal by default. You need to ask for it explicitly.

Your service isn’t broken. The logs are there. You just weren’t looking in the right place.

journalctl: Your New Best Friend

The journal is systemd’s logging daemon. Everything systemd starts gets logged here, but you have to query it. This is the most important command in your arsenal:

journalctl -u myservice.service -n 50

Breaking it down:

• -u myservice.service — follow the unit (service) named myservice.service
• -n 50 — show the last 50 lines

This shows you the most recent 50 log lines from your service. You’re looking for the crash, the error, the “exit code 1” that explains everything.

If 50 lines aren’t enough, bump it:

journalctl -u myservice.service -n 200

Real-Time Following: Tail Behavior

Want to watch logs as they happen? Use -f (follow):

journalctl -u myservice.service -f

Now restart your service in another terminal and watch the logs stream in. When it crashes, you’ll see the exact error that caused it.

Logs Since the Last Boot

Service failed to start this morning? Find all logs since the last reboot:

journalctl -u myservice.service -b -n 100

The -b flag means “since last boot”. Combine it with -n 100 to get the most recent 100 lines since reboot. This is gold for morning postmortems.

Disable the Pager (For Scripting)

By default, journalctl pipes output through less, which is great for interactive use but breaks in scripts. Add --no-pager:

journalctl -u myservice.service --no-pager | grep -i error

Now you can pipe to grep, awk, or feed it into whatever analysis you need.

systemctl status: The Quick Peek

When you need a fast diagnostic, don’t use journalctl. Use this:

systemctl status myservice.service

This shows:

• Is it running or dead?
• When did it start/stop?
• The last few log lines (10 by default)
• Exit code (if it crashed)
• PID

It’s not as detailed as journalctl, but it’s the fastest way to go from “is this thing running?” to “here’s why it’s not.”

Debugging Failed Starts

Your service won’t start at all. systemctl status shows exit code 1 but no error. This is usually an ExecStartPre problem—something in the startup hooks failed silently.

Check for a more verbose error:

journalctl -u myservice.service -n 30 --no-pager

Look for lines like:

condition not met
unable to load configuration
permission denied
socket already in use

Common culprits:

• A config file that doesn’t exist (path is relative, not absolute)
• A prerequisite service not started yet (missing After= in the unit file)
• Insufficient permissions (User= mismatch)
• A script in ExecStartPre that returns non-zero

Real-World Debugging Workflow

You’ve got a service that starts but then crashes after 5 seconds. Here’s how to find it:

# 1. Follow logs in real time
journalctl -u myservice.service -f

# 2. In another terminal, restart the service
systemctl restart myservice.service

# 3. Watch the logs. You'll see the crash immediately.
# Copy the error message, Google it if needed.

If the service is already crashed and you’re looking backward:

# Show the last 200 lines
journalctl -u myservice.service -n 200 --no-pager

# Or search for errors in the last hour
journalctl -u myservice.service --since "1 hour ago" | grep -i error

# Or just show everything since last boot
journalctl -u myservice.service -b

The Hidden Trap: stdout vs syslog

Some apps log to stdout (print statements), others to syslog. Systemd captures both and puts them in the journal, but if your app is also configured to log to a file, you’ve got logs in two places.

Always check journalctl first. If you don’t see output there, check the app’s own log files (usually in /var/log/ or /var/lib/).

One More Thing: Check Startup Type

If Type=simple in your systemd unit, the service is considered “started” as soon as the process spawns—even if it crashes a second later. This is where journalctl -f shines.

Restart the service and watch it. If it’s failing, you’ll see it die in the logs immediately.

Next time your service goes silent, stop guessing. Open journalctl, follow the logs, and restart. The answer is always in there.

Your 2 AM self will thank you.