Maintaining good-quality backups is often seen as the spine of any organization’s ability to recover from cyberattacks quickly. Naturally, given the emphasis placed on them by experts of all stripes, you’d be forgiven for thinking that prioritizing them over anything else would be the way to go.

Small businesses looking for a steer on cybersecurity may find themselves perusing the UK NCSC’s guide on that exact matter. Front and center, ahead of anything else, is the importance of backing up business-critical data and five top tips for doing it well.

To clarify, The Reg is not sitting here saying the UK’s foremost cyber chiefs have got it wrong and that backups are not important. Not in the slightest. However, what cannot be overlooked, as it is in the NCSC’s guide, is the importance of timely incident response and how influential the speed at which a threat is detected, and ousted, can be to an organization’s recovery from an attack.

Recent research [PDF] from UK cyber consultancy Bridewell, for example, which studied the cyber threat landscape affecting critical national infrastructure (CNI) organizations, found alarmingly slow responses to threat detection. The majority of respondents (69 percent) said they take up to six hours to respond to a ransomware attack, meaning nearly a third are taking even longer.

For reference, the consultancy says anything longer than an hour is a concern, and for Dray Agha, senior SOC manager at Huntress, even this is an unacceptably long time to wait before emergency protocols are enacted.

He tells The Register: “We train our analysts to say, if you’ve not resolved a critical intrusion within 30 minutes, you’ve presented a business risk to that client. So, an hour is too much.

“We had an adversary who broke in and within 17 minutes managed to make huge configuration changes that are very expensive and a pain in the butt for everyone to clean up. That’s 17 minutes, and they weren’t even that good a threat actor. They weren’t even that good a cybercriminal. 

“We have some cybercriminals that are able to essentially automate their malice when they get into an environment. So now, seconds matter. That’s the level we’re operating at. So, when I see folks are taking hours, I’m like, we’re just in a whole different dimension.”

He goes on to say that speed isn’t just part of the solution, it’s the entire solution. He would rather a defender respond within minutes and do a five-out-of-ten initial clean-up job than action a perfect plan too late.

Different situations carry different severities, of course, and we’re talking about the worst of the worst in this case, but the point still stands. Within minutes, bang-average hackers with unexceptional skill sets are able to make damaging configuration changes, such as creating a super-privileged admin account on a VPN gateway. Simple tweaks that carry a complex threat, all while often going unnoticed.

Back up a minute…

And this is where backups would ordinarily come into play. It may be that a criminal has established network persistence, but even if they were allowed to linger a little while, once they are detected the organization can just restore to a point where they know they weren’t compromised. Backups prevail again…

However, that’s not always the case and one of the main problems with this approach is that organizations tend to have a false sense of security when it comes to their backups and/or recovery plans. 

Investigators are often drafted in and find backups haven’t been made in months, or IT teams thought backups were being created continuously but later realize it was only the case for a handful of servers. We also hear of cases where disaster recovery plans have been created but not tested, or worse still: The plans were created but no one thought to print an analog copy, and ransomware has corrupted the file rendering it digitally inaccessible.

And even when backups are working perfectly, organizations and incident responders are often facing off with their competing priorities. Business leaders think they need to return to operation as quickly as possible, and if backups allow for that, then that’s the resource that should be drawn on. That’s why they pay so much money to keep them.

But then the security guys come in telling the business folk to hold their horses and leave the backups alone for now. The digital forensics specialists need time to conduct a thorough incident triage and plug the hole(s) that sent the organization into crisis mode in the first place. 

Delaying recovery is a difficult argument for the security lead to make considering the cost of business downtime. However, restoring from backups straight away often wipes away crucial digital forensic evidence needed to carry out root cause analyses of serious incidents.

“It’s a real tough balance and there’s no one right answer,” says Agha. “The best solution is obviously somebody who’s got a decent disaster recovery plan that gives step-by-step guidance of, OK, take copies of these compromised machines, extract the forensic data.

“And there are various tools that can do this for you. You don’t have to be an expert. Then you can re-image those machines, use your backups to get your client back to business as usual. But the worst mistake that we see with folks in ransomware is just perceiving that restoring from backup solves all the problems, it doesn’t.”

It could be argued, then, that incident response is perhaps just as influential on an organization’s ability to recover from a disaster as backups. So, how can organizations do incident response correctly?

Money, money, money

One thing incident response and backups have in common is money. You need lots of it to do both well.

For incident response, there are two main influences on how effective it can be: Tooling and operators. Some tools are free (most aren’t) and can be effective with the right operators, but hiring skilled cyber professionals to use those tools certainly doesn’t come cheap. They are usually in fairly short supply too, but that isn’t an excuse to palm security off to existing IT staff.

“So some folks will just outsource that, and that’s perfectly acceptable,” Agha says. “Other folks will assume that security can be just added on to the already sort of swollen responsibilities of IT. That can’t work.

“If you’re expecting your sysadmin, your help desk guys to also be able to look at your AV products, your security solutions, and your EDRs and respond, you’re doing something wrong. You need a trained security operator who knows what they’re doing, you don’t need many of them, it depends on the size of your organization, but that is just one of the clear ways that you can make sure you are giving yourself ample and fair ways to respond to an intrusion.”

Money is one of the key differentiators between good recovery and bad recovery, therefore that is what an organization needs to stay secure. If only it were that easy.

Businesses can’t afford to slash security budgets if they have a year that’s free of any major intrusions. Especially for the cash-strapped SMBs, there needs to be someone within an organization who can continually advocate for healthy security budgets and communicate that to key decision-makers in a language they understand.

For underfunded public sector organizations and operators of CNI, there is an argument to be made that central government should be playing as significant a role in funding security in these areas as it is playing in pushing forward related legislation.

The inter-departmental fights for funding will doubtless continue for years to come, but until orgs can kit out their security teams with every single capability they need, it is best to step into those cyber sprinting shoes and ensure incident responses are engaged as quickly as possible. ®