How many risk assessments is enough? If you’re in the PCI world, it’s at least one a year; but, it could be more depending on how your organization defines the term ‘significant change.’ Although examples are in the standard, what’s deemed a ‘significant change’ is left entirely up to an organization. Generally, when a governing body takes this sort of approach to defining terms it causes varying degrees of pain as organizations seek to establish compliance on that subject.
What’s in the PCI standard?
Let’s spend some time highlighting what’s in the PCI standard. PCI Data Security Standard (DSS) 3.1 says in sections 11 and 12 that penetration testing, vulnerability scanning, and risk assessments should be performed annually or after a ‘significant change.’ (Unrelated to significant changes, certain vulnerability scanning also happens quarterly.) The examples sited for a ‘significant change’ are as follows: “new system component installations, changes in network topology, firewall modifications, product upgrades.”
Other formal guidance essentially says, ‘anything that could change how card data is accessed’ should be considered a significant change. Again, this leaves ambiguity tattooed on the forehead of the elephant in the room. Regardless of the guidance, the DSS leaves defining ‘significant change’ up to each organization.
Problems with current definitions
Defining ‘significant change’ should not be taken lightly. When organizations define this too narrowly, they cause unnecessary risk assessment and penetration testing work. If defined too broadly, organizations may run afoul of their assessors or worse – they will run a greater risk of a card breach than you otherwise would have accepted. Organizations struggle using current definitions to answer questions such as:
- Which firewall changes require testing?
- Does changing this small component of the payment application need testing?
- How many of the topology changes should be tested?
The common answer to these questions is to over-analyze changes to the environment. After working with many Fortune 500 organizations governed by PCI standards, I can tell you the same three problems exist nearly everywhere. Organizations regularly:
- Struggle to align results to the organization’s definition of ‘significant change.’
- Spend too much time conducting analyses.
- Analyze too many changes.
A better definition
Let’s spend a few minutes solving the first problem. Once the first issue is resolved the other problems are easier to solve. We will look at them in future posts.
The first problem is not about fixing the analysts; it’s rooted in poor definitions. Like other risk analysis issues, this leaves cybersecurity analysts reverting to childlike behavior of cramming a square peg into a round hole. The results follow suite: indefensible green/yellow/red risk ratings.
Since the current definition of ‘significant change’ is failing organizations, where can we look to get a clearer and more valuable definition? Look toward quantitative models and methods that yield financial comparatives. Our risk model of choice is Factor Analysis of Information Risk (FAIR).
We propose to you that we should define ‘significant changes’ based on risk ratings in financial terms, dollars and cents; not stoplight colors.
When organizations ignore quantitative financial models when defining ‘significant change’ for them, we obscure changes that aren’t significant with changes that are. Our ability to determine significance is watered down to a level that continuously forces us into the three problems above. Let’s define ‘significant change’ using quantitative results that allow us to compare the changes in the environment to risk acceptance thresholds.
‘Significant change’ quantification in action
Before we conduct an analysis, we should set the tone for a ‘significant change’. One approach is to look at any change and if the forecast result is more than a 10% increase or decrease in risk it requires testing. This is a simple way of setting a financial line in the sand that’s globally applicable to PCI environment changes.
Once we know how much current risk we have we can forecast future risk by rerunning our analysis and updating the few data points that reflect the change in control landscape. Figure 1 below shows this in action, by comparing:
- The loss exposure of retails stores relying on the current Point-of-Sale (POS) installation against the loss exposure of retails stores once the new POS installation is complete.
By comparing the average exposure we can see a clear difference in risk – approximately 40% change. Using our 10% rule above we can determine this large delta should kick-off testing upon completion of the POS project to validate the risk reduction. The significance of the change is now driven by comparable financial results. These sorts of reports provided by RiskLens CRQ demystify the notion of a ‘significant change’ by distilling the changes in the control landscape into one key measurement: financial risk. We have removed the subjectivity surrounding commonly used definitions and stoplight scoring. Objective financial measurement now determines what a ‘significant change’ is.
How does your organization define ‘significant change’? Does your definition support analyses that show how much PCI environment changes move the risk needle?
Please share your experiences in the comment box below.