1 hour agoCreated a post • 84 points @parsecs • 1 comments
"Test your backups" is so easy to say, but quite difficult for many to do. There are a lot of shops that probably don't know how to recreate a machine from scratch. How many systems are developed as balls of clay. Little bits added and smeared in over time until the ball just gets bigger, but each piece lost in the process. How many folks can go through their local config files and explain all of entries, how many can even tell which ones they have changed, or why? Especially when they were changed by Frank, but he left 2 years ago.
You'd like to think you can just restore the system from backup and it'll just light back up. But how do you test this without cratering your existing system? Like a boat in a basement, many system are built in-situ and can be very rigid.
Modern environments like cloud computing and creation scripts can mitigate this a bit organically, but how many of these systems are just a tower running Windows w/SQL Server and who knows what else? Plus whatever client software is on the client machines.
How do you test that in isolation?
At least read the media to see if it can be read (who doesn't love watching a backup tape fail halfway through the restore).
Simply, it takes a lot of engineering to make a system that can be reliably restored, much less on a frequent basis. And this is all engineering that doesn't impact the actual project -- getting features to users and empowering the business. Which can make the task even more difficult.Reply
Don't just test your backups. Make sure your automation can't clobber or tamper with your backups. This includes both local and disaster recovery sites. Give your pen-test team super-user privs on your automation and give them Amazon gift cards if they can tamper with your backups. If they can't mess with the backups, give the gift cards to whoever designed and hardened your infrastructure.Reply
If your disaster recovery process isn't tested, you actually don't have any disaster recovery. It's not only about 'how long it takes' it's also about whether or not it works at all. Can you rebuild from scratch? What happens if your entire infrastructure goes down at the same time? What happens if a datacenter you rely on just disappears? What happens if you lose access to your systems? Can you lose access to your systems? IMHO one of the only silver lining of these attacks is that organizations are starting to ask these questions more often.Reply
There is another approach. Scrub old data you don't need.
2-3 year email retention on corp email.
Paper files for sensitive client info (or don't keep it).
We can reinstall office / windows / active director etc.
Mandatory 2FA on google suite?
Git codebases on github etc for LOB apps (we can rebuild and redeploy).
We use the lock features in S3 for copies of data that must be kept. Not sure I can even unlock to delete as account owner without waiting for timeouts.Reply
There's been a lot of good advice here about backups and disaster recovery.
But there's also a lot of other stuff to consider:
Compartmentalization. Finance and Engineering and Sales only need to interact in limited ways. How about some firewalls between them, limiting types of access?
Location isolation. Why does something that happens in Peoria affect Tuscaloosa? Once a ransomware gang breaches a perimeter, why is it allowed countrywide (or worldwide) access to a company?
Monitoring. Aren't there tools that can alert on various anomalous patterns? All of a sudden, gigabytes of data start being exfiltrated? All of a sudden, processes fire up on a multitude of servers? Monitoring these things is hard to do at scale, but surely possible?
Microsoft. In 2002, Bill Gates "Finally Discovers Security". How much longer will Microsoft be given a free pass? How many more "critical" vulnerabilities will their software have? https://www.wired.com/2002/01/gates-finally-discovers-securi...
I could go on and on. But why should I? Why can't MBA-type CEOs take IT seriously? Why can't they hire competent people and fund them and listen to them?Reply
Isn't this more of a "We don't want our client / customer information released to The World At Large" question? I would think most business entities have backups of some kind (Scripps being the only exception I can think of), and will pay the ransom to keep any sensitive information off the market.
Edit: Should have added that I find it hard to believe that companies have PB of data backed up. I could believe GB, and maybe even TB, but PB is pretty hard to swallow. The past three companies I've worked for (25 year span) had, at most, a couple of gigs of sensitive information that couldn't be easily replicated.Reply
How does one learn how to do proper backups? Using my throwaway as I suspect my company doesn't do them (and even if they do, I don't know where they are or what to do with them as the main engineer left on my piece of software).Reply
3-2-1 Backup Rule:
Three copies of your data. Two "local" but on different mediums (disk/tape, disk/object storage), and at least one copy offsite.
Then yes, absolutely perform a recovery and see how long it takes. RTOs need to be really low. Recovering from object storage is going to take at least a magnitude more time than on-prem.
Also, storage snapshots/replications are not backups, stop using them as such. Replicating is good for instant failover, but if your environment is hacked they are probably going to be destroyed as well.Reply
I’m a novice and am dealing with data that isn’t too complicated, large, or important. My approach is to build restore directly into the normal workflow. I test my backups by using them each week.
A stack is spawned from a database backup and once it passes tests, replaces the previous one.
Not sure how smart this all is but my goal is to learn through application.Reply
Not really just a backup and restore. You need to be able to rebuild from zero. I think of it more as a disaster recovery exercise, and for those… you are only as good as your last _real_ rehearsal. That may mean a suitcase of tapes, a sheet of paper, and a rack of blank servers. Then you have the problem of release of confidential information. For this reason, the sweetest target for ransomware is the company who can neither recover their data, nor can they afford to have it publicly posted or monetised by the gang. Oh and you do store those backups offline dont you? Ransomware gangs have been known to loiter and observe their target for weeks to learn how to sabotage backups when the time comes.Reply
What sucks for HIPAA is that you can get fined for the breach itself, regardless of your backup management.Reply