If your server may have been broken into
Use this page when something on a creekX server looks wrong: files you did not put there, a service you did not start, a processor busy with no streams, or connections to places you do not know. It covers what to do in the first hour, how to keep the evidence, and how to come back on a clean machine. Read it once now, before you need it.
On this page
The one rule
Do not upgrade the server in place and call it fixed. An upgrade replaces creekX. It does not remove what an intruder added to the machine: a new user, an SSH key, a scheduled task or a background service all survive it. Rebuild on a clean machine instead, as described below.
Signs worth acting on
- Files in the server's folders that you or creekX did not put there.
-
A service, timer or scheduled task you do not recognise:
systemctl list-units --type=service --state=running,systemctl list-timersandsudo crontab -l. - High processor use while nothing is live:
top. - Outbound connections you cannot explain:
sudo ss -tunp. -
Sign-ins you did not make in
/var/log/auth.log, or a user or SSH key you did not add. - People, API keys or platforms in creekX that nobody on your team created.
One of these can have an innocent cause. If you cannot explain it within a few minutes, treat it as an incident. Being wrong costs an afternoon.
The first hour
-
Take a snapshot of the disk
In your cloud provider's control panel, take a snapshot of the server's disk before you change anything. It is the best evidence you will have, and it costs little. On your own hardware, copy the disk if you can.
-
Cut the server off, without switching it off
In the provider's firewall, allow only your own address in, and block everything out. Viewers lose the stream; the intruder loses the server. Leave it running: a reboot destroys what is in memory and can start whatever was added.
-
Save creekX's own records
sudo creekx backup mkdir evidence sudo journalctl -u creekx --since "30 days ago" > evidence/creekx.txt sudo journalctl -u caddy --since "30 days ago" > evidence/web-server.txt sudo cp /var/log/auth.log* evidence/The backup holds the database, which includes the audit log of administrative actions and the viewer log. Copy the backup folder, the journal and the logs to another machine. Treat them as private: the backup holds the key that protects your stored stream keys.
-
Write down what you saw, and when
The time you noticed, what you noticed, and each thing you did after. You will not remember it accurately a week later, and anyone helping you will ask.
Come back on a clean machine
-
Start a new server
A new Ubuntu 24.04 machine from your provider's own image, not a copy of the old one. Use new SSH keys.
-
Install the current creekX
Follow Install on a server. The installer checks what it downloads before using it.
-
Restore your data
sudo creekx restore /var/backups/perpetua/FOLDERUse a backup from before the first sign of trouble if you have one. A backup carries data and configuration, never programs, so it does not bring an intruder's software with it. It does carry your settings as they were, including any person or API key an intruder added: check both lists after restoring. See Backups and moving.
-
Change every key and password
Assume everything the old server knew is known. In this order:
-
The owner's password:
sudo creekx owner. It signs the owner out everywhere. - Every other person: remove access for anyone you do not recognise under People, and have the rest choose new passwords.
- Every API key: revoke them under Developers and make new ones.
- Each stream's key and private links, and the stream key at each platform you restream to (YouTube, Facebook and the others), changed at the platform itself.
- Webhook signing secrets and the approval secret: make them again.
- The keys for backup storage, your email server's password, camera passwords, and the single sign-on client secret.
Limit: the key that seals stored secrets on the server cannot yet be changed by a command. Entering each secret again, as above, makes the old values useless, which is what matters.
-
The owner's password:
-
Close the doors before opening to viewers
- Turn on two-step sign-in for every person: More → Account.
- List the addresses the workspace may be managed from, under Settings.
- Open only the ports you use, and restrict SSH to your own addresses. See Ports to open.
-
Keep the old server's snapshot, then delete the old server
Keep the snapshot until you are sure nobody needs it. Never put the old machine back.
What creekX does to limit the damage
-
It runs as its own unprivileged user,
creekx, with nosudorights, and its service cannot gain new privileges or write outside its own data folder. - Every management request needs a signed-in person or an API key.
- The live server's control interface listens on the machine itself only, never on the network.
- Uploaded files are stored as data and are never run by the web server.
- Administrative actions are written to an audit log that shows if it was altered.
These reduce what a break-in can reach. They are our own design and tests, not an independent review: no penetration test has been done yet. See Security and privacy.
Who to tell
- Us. If you think creekX itself was the way in, write to creekX with what you saw. Do not send keys, passwords or unedited logs; we will ask for what we need.
- Your own people, and possibly a regulator. If personal data was on the server, such as viewer addresses or recordings of people, the law where you operate may require you to report it within a short time. Ask whoever handles data protection for you.
How do I know I am ready for this?
You have a backup from this week that you have restored once on a test machine, you know where your provider's snapshot and firewall buttons are, and two-step sign-in is on for the owner. If any of those is missing, fix it today. A plan you have never tried is a guess.