We’re back online! Here’s why we were offline

We’re glad you’re back with us! We also appreciate your patience since we experienced some issues late Saturday night.

Here is what happened (and will try to keep this simple):

In the period of three hours in the late hours of Saturday night, we were getting slammed with notifications from our server monitoring software that two of our databases which handle the majority of the website’s content were crashing, citing that they had ran out of memory. After roughly 200 notifications, all of our content disappeared around 11:11 PM Central time, with the exception of advertisements.

As editor-in-chief, I contacted our website host, TurnKey Internet, about the issue with technical support shortly afterward. Following an hour-long discussion with Tier 1 technical support specialist Chris D., I needed to escalate the situation with a service ticket to the Tier 2 tech support team.

Knowing that it was Christmas Eve, I didn’t really expect a response today, but around 8:14 this morning, Tier 2 NOC Technician Autumn M. returned a response:

I apologize for any delays in response. I have been going over your server thoroughly to determine the full extent of what occurred last night.

From the looks of things your wp-admin dashboard was hit with some kind of brute force or denial of service attack (not to be confused with a distributed denial of service attack, which usually involves a botnet that hammers everything at once.) The website was flooded with so many requests and attempted database queries that both PHP and MySQL ran out of memory. This may have been the attacker’s intent, as out-of-bounds memory issues like buffer overflows are easily exploited. Looking over the error logs, it seems like these errors caused WordFence to take quite a hit.

MySQL did not completely crash, though, it just exited and when I accessed your server it restarted cleanly. Looking at the memory, I can confirm that swap was filling up fast, which I have remedied by restarting most of the memory-intensive services on the server. I can see that two tables in the database heartlan_sqlnewsbase crashed and are in need of repair, which I can do through cPanel.

After performing all of these tasks, heartlandnewsfeed.com is still only loading advertisements. The good news is that I found a full drive backup of your server that was taken on the 21st, which is far enough behind the attack that there’s little chance it has any exploited payloads on it, but recent enough that you won’t have to recreate a bunch of data. It will require a few hours of downtime to restore this, but given how hard your server was attacked, I think it may be the best option. If you are uncomfortable with this, I can comb through the logs more and see if I can track down any lingering malicious files or scripts, but this will be time consuming, will not carry any guarantee of effectiveness as I can only look for malware whose existence I am aware of, and will continue to leave your site vulnerable.

Once the backup is restored, I would recommend forcing an update across WordPress, all of your themes of plugins, updating cPanel/WHM with the /scripts/upcp command, and running a system update through cPanel followed by a reboot. I would also recommend looking into Sucuri – https://sucuri.net/ It’s similar to CloudFlare but tailored to WordPress sites. You should also evaluate your themes and plugins for known vulnerabilities. A good place to start is the Exploit Database, https://exploit-db.com/search One of the most valuable pieces of information you can find on here is not the vulnerabilities per say, but when they were published so you can cross reference them with release notes.

Let me know what your preferred course of action is and whether you would like for me to attempt to repair the sqlnewsbase database and/or restore from a backup from the 21st.

So from the explanation, the website and our wp-admin dashboard was hit with a brute force denial of service attack, which resulted in a flood of requests that overloaded both PHP and MySQL, pushing out notifications that both were out of memory. It also resulted in our Wordfence security software to take a serious hit as well.

MySQL shut itself down, but was able to be restarted.

Because of the issues, for most of Christmas Eve, all you saw on the front page was advertisements and no content.

So knowing the options, I opted to just revert back to the backup created on Thursday, because I wasn’t certain how long the database repairs would take.

We did lose some content in the process between the backup time and when the website content disappeared, so if there are any articles that were published that you would like republished, please let us know.

So, as of 10:24 PM Central time, the website and content from the 12/21 backup were back online.

We at Heartland Newsfeed appreciate the hard work of Turnkey Internet employees Chris D., Autumn M. and Andrew W. for their help while we were experiencing issues. They could have spent Christmas Eve dealing with other customers or spending time with their families, but they chose to do what they could to make sure they could resolve my problem. To say that I’m satisfied with the customer service, response time and professionalism of all three individuals would be an understatement.

We’re back online and we’re going to get “our house” back into order as we attempt to restore previously published content (per requests) and get new news content published as soon as possible.