Many Information Security practices have outcomes that are difficult to quantify. How do you prove that your measure is effective at preventing whatever malicious activity is out there from being effective against your system?

Antivirus and antimalware tools can easily point to the number of attempts blocked as measures of their success, but they aren't so good at identifying the attacks that are quite effective at completely bypassing their protection. System and network hardening, an essential component of any Information Security plan, is one of the toughest mechanisms to accurately quantify without actually measuring what is trying to enter the network, but it is one of the most effective tools in the Information Security specialist's toolbox.

If you don't log, if you don't measure what is going on, then it makes the job of quantification so much harder. The problem of correctly identifying what is in those logs is difficult enough that a mini-specialisation has established itself around interpreting log files. Getting the balance right can be difficult, for example over-sensitive Snort (an Intrustion Detection System) rules can give the impression that far more is taking place than actually is. Likewise, an under-sensitive ruleset will ignore malicious activity or completely miss it if a rule has not been created.

How do you quantify the effectiveness of a mechanism when the threat is something that has not been seen before and it strikes at 2 am or at another time when there is no one directly observing the system?

Statistics delivered by Jay Beale, of Bastille-Unix, in his DefCon 14 presentation demonstrated that Bastille was able to defeat every major threat to Red Hat 6, even before the threats were known. Statistics like this are best gathered after the fact, but they do point to how effective a thorough hardening process can be for systems and networks when faced with an unknown threat.

One of the big problems that people find when they go to apply a hardening process is that they encounter usability problems as a system or network is progressively locked down. The resultant compromise between usability and security is one that is situation-dependent and should be at the core of risk and threat management assessments (if they are carried out). The introduction of a Secure Development Lifecycle and greater awareness of security as a core part of the development process is resulting in more applications that are inherently more secure and are able to be locked down without loss of significant capability, something which is going to be of more importance in the future as more devices gain networking abilities and more sensitive data is moved onto networked devices.

While the situation is improving, there is still a significant corpus of applications that do not behave properly when locked down against unwanted access and it is these that cause the greatest problems for system hardeners.

There are a range of products and system that are available, both commercially and Open Source, which can aid in the process of hardening, including Bastille and SELinux. It doesn't just stop at the Operating System, with the NSA providing a range of useful [[xref:http://www.nsa.gov/snac/index.cfm?MenuID=scg10.3.1 |guides]] for hardening everything from a web browser, through to network hardware. If you or your organisation don't have a plan implemented to mitigate the risk to your systems and hardware that is represented by non-hardened systems, then it would be a good time to consider one.