Incident Response for Developers: What to Do When You Get Hacked

You will not rise to the occasion. You will fall to the level of your preparation. Have a plan before you need one.

The First 15 Minutes Matter Most

When you discover a breach, panic is the enemy. Acting without a plan causes more damage than the breach itself — wiping logs, shutting down the wrong systems, or alerting the attacker that you know.

Here is the framework: Detect, Contain, Eradicate, Recover, Learn.

Phase 1: Detection

Most breaches are not discovered by the victim. The median time to detect a breach is 204 days (IBM Cost of a Data Breach Report). Set up detection so you find out fast:

What to monitor

# Unexpected outbound connections
netstat -an | grep ESTABLISHED | grep -v "expected-ips"
 
# Failed login spikes
grep "Failed password" /var/log/auth.log | tail -50
 
# New admin accounts
grep ":0:" /etc/passwd
 
# Modified system binaries
debsums -c  # Debian/Ubuntu — check integrity of installed packages

Signs you have been compromised

• Unexpected admin accounts or SSH keys
• Unusual outbound traffic (especially to unfamiliar IPs or on unusual ports)
• Modified timestamps on system binaries
• Unexplained CPU or memory usage spikes
• Customers reporting phishing emails from your domain
• Alerts from your cloud provider about unusual API calls

Phase 2: Containment

The goal is to stop the bleeding without destroying evidence.

Do this immediately

1. Isolate affected systems — do not shut them down. Remove them from the network so the attacker cannot move laterally:

# Isolate a server by blocking all traffic except your IP
iptables -I INPUT -s YOUR_IP -j ACCEPT
iptables -I INPUT -j DROP
iptables -I OUTPUT -d YOUR_IP -j ACCEPT
iptables -I OUTPUT -j DROP

2. Preserve evidence — take disk snapshots and memory dumps before doing anything destructive:

# Create a disk snapshot (AWS example)
aws ec2 create-snapshot --volume-id vol-abc123 --description "incident-$(date +%Y%m%d)"
 
# Capture memory dump
sudo dd if=/proc/kcore of=/mnt/evidence/memory.dump bs=1M

3. Revoke compromised credentials — immediately rotate:
   • API keys and tokens
   • Database passwords
   • SSH keys
   • Cloud IAM credentials
   • OAuth client secrets

Do NOT do this

• Do not shut down servers (you lose volatile memory evidence)
• Do not delete logs
• Do not alert the attacker by changing passwords on systems they are actively monitoring
• Do not communicate about the incident over compromised channels (use out-of-band communication)

Phase 3: Eradication

Find the root cause and remove the attacker's access completely.

Common root causes

Eradication checklist

• Identify all affected systems (the attacker likely moved laterally)
• Remove backdoors — check for unauthorized SSH keys, cron jobs, systemd services, and startup scripts
• Patch the vulnerability that was exploited
• Rotate ALL credentials, not just the ones you know were compromised
• Update firewall rules and access controls

# Check for unauthorized SSH keys
find /home -name "authorized_keys" -exec cat {} \;
 
# Check for suspicious cron jobs
for user in $(cut -f1 -d: /etc/passwd); do crontab -l -u $user 2>/dev/null; done
 
# Check for suspicious systemd services
systemctl list-unit-files --type=service | grep enabled

Phase 4: Recovery

Bring systems back online safely.

1. Rebuild from known-good state — do not try to "clean" a compromised server. Redeploy from version control and restore data from backups
2. Verify backups — check that your backups are not also compromised (attackers often persist in backups)
3. Monitor intensively — the attacker may try to regain access using other footholds
4. Gradual rollout — bring systems back one at a time, monitoring each for anomalies

# Verify backup integrity before restoring
sha256sum backup-2026-03-28.tar.gz
# Compare against known-good checksum

Phase 5: Communication

Internal communication

• Use an out-of-band channel (Signal, phone calls) — not Slack or email if those systems may be compromised
• Brief leadership with facts, not speculation
• Designate one spokesperson for external communication

External communication

• Legal obligation: Many jurisdictions require breach notification within 72 hours (GDPR) or 30 days (many US state laws)
• Notify affected users with specifics: what data was exposed, what you are doing about it, what they should do
• Do not downplay: Vague statements like "a small number of users may have been affected" erode trust faster than honest disclosure

Ransomware-Specific Response

If you are hit with ransomware:

1. Do not pay — there is no guarantee of decryption, and payment funds further attacks
2. Isolate immediately — ransomware spreads laterally via SMB and other protocols
3. Check for decryption tools — nomoreransom.org maintains free decryptors for known ransomware families
4. Restore from backups — this is why you test your backup restoration process regularly
5. Report to law enforcement — in the US, contact FBI IC3; in the EU, contact your national CERT

Phase 6: Post-Incident Review

Within one week of resolution, run a blameless post-mortem:

• Timeline: What happened and when?
• Root cause: How did the attacker get in?
• Detection gap: Why did it take X hours/days to detect?
• Response effectiveness: What went well? What was too slow?
• Action items: Concrete changes with owners and deadlines

Document everything. The next incident will happen. Make sure you handle it faster.

Conclusion

Every team gets breached eventually. The difference between a minor incident and a catastrophic one is preparation. Write your incident response plan now — when you are calm and thinking clearly. Know who to call, what to preserve, and what to rotate. Practice it with tabletop exercises. When the real thing happens, you will be glad you did.