July 18, 2021

Having a hard time investigating/recovering from server hack


Hi everyone. For context: I’m a sysadmin with a decent understanding of offensive hacking, and I run a few LAMP servers on DigitalOcean for personal niche websites and a few clients’ websites. A couple months ago I put all my domains on a bug bounty (openbugbounty org) to help me identify weaknesses as I continue growing. (I’ve had only minor exploits mentioned, and most of the researchers violate the rules I wrote, ie. dont attack subdomains, don’t report self-xss, etc ). Since this started, I would occasionally see my MYSQL server drop, maybe twice a month. I would reboot my server and all would be well (in retrospect I see how dumb this attitude was). I’ve also begun getting a TON of log traffic, as one would expect.

Here’s where I make a huge embarrassing mistake; I decided to use my main production server to host the development environment for a new version of a CMS I wrote in PHP/MySQL. I was working on dev(.)mysite(.)com, which was on the same server as mysite(.)com, and used the same MySQL server. I really can’t explain my thinking on this one. At my 9-5 I’m very strict about having the developers keep dev and production environments separate, so this is out of character for me. I guess my level of security-consciousness is proportional to the amount of explaining I’ll have to do.

About a week or ago I find my CMS won’t allow me to login. I checked the DB and there are hundreds and hundreds of new entries in the *users* table. Interestingly, the user account I was trying to login with looked untouched. I obviously got caught slipping with a SQLi vulnerability. No other tables were affected that I know of, as this CMS uses an un-privileged DB account. I shut the dev site down, and moved it to a local LAMP server on my home network. Better late than never. Now I’m trying to figure out what happened, and how far the breach went.

I have over 1,000 log files in /var/log/apache2, and most other directories in /var/log/ are also packed to the gills. There’s too much noise (presumably) from the bug bounty for me to analyze anything. I spent a day writing python scripts to analyze stuff. I have one that compiles all website-specific logs into respective master files, and runs scalp.py on them, outputting everything to an HTML file I can navigate. There’s so many random injection attempts that the script is almost useless to me. I wrote another script to analyze the /var/log/auth.log file and at least count and sort the types of messages. The output can be found [https://pastebin.com/grGhT0Qx](https://pastebin.com/grGhT0Qx) . Sorry if the output is confusing, I don’t understand what a lot of the messages mean, so just did a basic count for now.

I only saw on single SSH login that I didn’t recognize, and it was on the account smmsp. This also lead me to see in my mail log files that somebody had found an old forum install I forgot about, and had been using it to send massive amounts of spam emails to email addresses in Russia. I’ve removed the account, the forum, and uninstalled sendmail.

I’m at the point of the investigation where I risk going down the wrong path and wasting valuable time.

So here’s my questions:

1. Are there any programs that will do an in-depth audit of my logs? I did a bunch of searching, but almost nothing fits the bill besides scalp.py, and that’s only for website-specific log files. I’m more than happy to expand my own scripts to do this if I need to, but would request someone push me in the direction of the information I need to start studying.
2. Is there an application that could blacklist an IP address if it recognizes injection attempts? I’m sure a WAF could pull something like this off, but have never used any so I would be swinging in the dark by picking one. I can write a script to do it too, using the logs to identify IPs, but is there a specific method of blacklisting that I should be leaning towards if I need to do this on my own?

Thanks a ton for any responses/advice. I know this is a long, dry read.

Leave a Reply

Your email address will not be published. Required fields are marked *

Note: By filling this form and submitting your commen, you acknowledge, agree and comply with our terms of service. In addition you acknowledge that you are willingly sharing your email address with AiOWikis and you might receive notification emails from AiOWikis for comment notifications. AiOWiksi guarantees that your email address WILL NOT be used for advertisement or email marketting purposes.

This site uses Akismet to reduce spam. Learn how your comment data is processed.