Incident Response for Self-Hosters

It’s 2 AM. Something is wrong. Your server is doing things you didn’t ask it to do. CPU at 100%. Network traffic going somewhere weird. Maybe you got an alert from your host that your IP is hammering someone else’s infrastructure. Or worse—you notice a cron job you definitely didn’t create.

This is that article.

You’re about to experience the self-hosted equivalent of a car alarm at 2 AM in a parking garage. The good news? You can fix this. The bad news? What you do in the next hour determines whether this is a minor inconvenience or a total disaster. So take a breath, close those Reddit tabs, and follow this playbook. Your future self (the one who gets paged at 2 AM) will appreciate the discipline.

Phase 1: Don’t Panic, Don’t Reboot

I’m saying this first because the urge to reboot is overwhelming. Your instinct is correct on most problems. This is not one of them.

When you reboot a compromised machine, you destroy forensic evidence. Everything in RAM—the attacker’s running process, their network connections, their in-memory secrets—vanishes. You also give the attacker a chance to trigger persistence mechanisms. They may have left a cron job, a systemd service, or a modified SSH config that survives the reboot. You’ll think the problem is fixed, then they’ll come back in 24 hours when you’re not watching.

Instead: Leave it running. Make notes about what you’re seeing right now. Take screenshots of your monitoring dashboards, top output, anything showing the anomaly. Your forensic timeline starts now.

Phase 2: Contain Without Deleting

Isolate the compromised machine from the rest of your infrastructure immediately. This is not optional.

If this is a VM on your hypervisor, pull the network cable (or disable the NIC). If it’s a physical machine, disconnect the network. If it’s remote and you can’t physically reach it, you’re about to do some triage over SSH—be extremely careful with commands.

Do NOT delete anything yet. Do not rm, do not truncate logs, do not clear bash history. The attacker might have covered their tracks already, but you don’t know that yet. Deleting is irreversible.

Phase 3: Preserve Evidence

You have maybe 30 minutes of isolation before you need to make a decision: is this worth fixing on this host, or are you rebuilding? Either way, you need evidence.

SSH into the machine (if you can—if you’ve lost SSH access, that’s a different problem). Run these commands in order, capturing output to a file or screenshots:

# Process tree - see what's running and parent-child relationships
ps auxf > /tmp/evidence_processes.txt

# Network connections - where is this machine talking to?
ss -tulpn > /tmp/evidence_connections.txt

# Open network sockets with process names
netstat -tulpn > /tmp/evidence_netstat.txt

# Currently logged in users
w > /tmp/evidence_who.txt

# Failed login attempts (last hour usually)
lastb -t today > /tmp/evidence_lastb.txt

# All login history
last -f /var/log/wtmp > /tmp/evidence_last.txt

# Cron jobs for root and all users
for user in $(cut -f1 -d: /etc/passwd); do echo "=== $user ==="; crontab -u $user -l 2>/dev/null; done > /tmp/evidence_crons.txt

# Recently modified files (last 7 days)
find / -mtime -7 -type f ! -path '/proc/*' ! -path '/sys/*' ! -path '/dev/*' -ls > /tmp/evidence_recent_files.txt 2>/dev/null

# Check for suspicious PHP/webshell activity
find /var/www -name "*.php" -exec ls -la {} \; > /tmp/evidence_web_files.txt 2>/dev/null

Copy these evidence files off the machine to a safe location. Email them to yourself, scp them to another machine, whatever. You’re building a timeline.

Phase 4: Identify the Entry Point

Now the detective work. How did they get in?

Check your auth logs for the breakthrough moment:

# Look for successful logins from unusual IPs
grep "Accepted" /var/log/auth.log | tail -100

# Look for brute force patterns
grep "Failed password" /var/log/auth.log | cut -d' ' -f11 | sort | uniq -c | sort -rn | head -20

If you run a web service, check access logs for webshell uploads or exploitation attempts:

# Apache/Nginx logs for suspicious requests
tail -1000 /var/log/nginx/access.log | grep -E "(\.php|\.sh|cmd=|exec=|system=|shell=)"

Check for recently modified files in web-accessible directories:

find /var/www -type f -mtime -3 -ls

Look for rootkits using built-in tools (won’t catch everything, but catches obvious stuff):

chkrootkit -q
# or
rkhunter --check --skip-keypress --report-warnings-only

Did they get in via SSH? Weak password? Key exposure? A vulnerable service? A supply-chain compromise of something you installed? Each answer changes your response.

Phase 5: Eradicate the Threat

Once you know how, you close it.

Weak SSH password? Change every password on the system immediately. Disable password auth entirely if possible. Use keys only.
SSH key compromise? Rotate all keys. Assume your private keys are stolen.
Vulnerable service? Patch it. NOW. Don’t wait for maintenance windows.
Webshell? Delete it (now it’s safe—you know what to delete). Run a full malware scan.
Persistent cron/service? Disable and delete it. Remove from /etc/cron.d/, /etc/cron.daily/, .bashrc, .profile, systemd directories.

Get aggressive. Change SSH port? Maybe, but that’s security-through-obscurity and wastes time. Better to just fix the root cause and monitor closely.

Phase 6: Recover and Rebuild

Here’s the uncomfortable truth: if the attacker had admin access, you can’t be sure they’re gone. They might have rootkit-level persistence.

Best practice: Rebuild from known-good backups on a different machine. Don’t restore onto this host. If you have backups, great—restore and verify nothing was already compromised in your backup.

If you don’t have backups, you’re about to learn why self-hosters need them. Start taking them today.

Phase 7: Lessons Learned—Prevent This Next Time

This happened because something wasn’t being watched. Fix that.

Fail2ban: Auto-blocks IP addresses after failed login attempts. 5 minutes to set up.
Wazuh: Free security monitoring. Alerts on suspicious cron jobs, rootkit signatures, unexpected processes.
Aide: File integrity monitoring. Tells you immediately when system files change.
Auditd: Linux audit framework. Logs everything. Forensic gold.

Pick at least one. In 2026, monitoring isn’t optional—it’s the difference between a contained incident and a catastrophe.

The One-Page Incident Response Checklist

INCIDENT RESPONSE CHECKLIST FOR SELF-HOSTERS

1. FIRST RESPONSE
   [ ] Do NOT reboot
   [ ] Take screenshots of monitoring/alerts
   [ ] Note the time
   [ ] Take a deep breath

2. CONTAINMENT
   [ ] Isolate network (pull cable or disable NIC)
   [ ] Prevent access to other internal machines
   [ ] Keep the machine powered on

3. EVIDENCE COLLECTION
   [ ] ps auxf → file
   [ ] ss -tulpn → file
   [ ] lastb → file
   [ ] crontab for all users → file
   [ ] Find recently modified files (last 7 days) → file
   [ ] Copy evidence files off the machine

4. INVESTIGATION
   [ ] Check /var/log/auth.log for suspicious logins
   [ ] Check web server logs for webshells/exploitation
   [ ] Run chkrootkit or rkhunter
   [ ] Identify entry point (brute force? vulnerable service? webshell?)
   [ ] Document timeline

5. ERADICATION
   [ ] Close the entry point (patch, rotate keys, change passwords)
   [ ] Remove persistent backdoors (cron jobs, services, shells)
   [ ] Delete webshells
   [ ] Change all credentials

6. RECOVERY
   [ ] Restore from clean backups on different hardware (if available)
   [ ] OR rebuild from scratch
   [ ] Verify backups weren't already compromised

7. LESSONS LEARNED
   [ ] Install monitoring (Fail2ban, Wazuh, Aide, or Auditd)
   [ ] Enable log aggregation
   [ ] Set up alerting
   [ ] Document what went wrong
   [ ] Schedule regular security audits

CRITICAL: Do NOT reboot until eradication is complete. Do NOT delete until you've collected evidence.

The Part Nobody Wants to Hear

This happened because you didn’t think you’d be targeted. You run a small homelab, not a datacenter. Nobody cares about you specifically.

But they do care about your server’s resources. Compromised machines get used for mining crypto, launching DDoS attacks, relaying spam, hosting exploit kits. You were a tool, not the target.

That actually makes this fixable. You don’t need advanced threat hunting—you just need basic hygiene. Monitoring. Updates. Backups. No default passwords. No unnecessary services. Fail2ban running on anything that faces the internet.

The self-hosters who get compromised twice are the ones who don’t implement these basics after the first time.

Don’t be that person.

It’s 3 AM now. Your incident is contained. You’ve got evidence. You know how they got in. You’re going to fix this, rebuild if needed, and wake up tomorrow knowing what went wrong and how to prevent it.

Your 2 AM self just became your 3 AM hero. You’ve got this.