How we do PCI Code Reviews

Sachin wrote about the PCI requirement for code reviews some months ago. Today, I want to discuss how we do those code reviews.

Let’s take a fictitious e-Commerce application as an example, and walk through the steps. In the process I’ll also try and answer some of the most common questions clients have about PCI code reviews.

The first step in a code review is preparing the Threat Profile.

Regular readers will remember the Threat Profile - it’s the collection of all threats to an application. A “Threat” is the goal of an adversary - it’s what he wants to achieve. We are putting ourselves in the shoes of the adversary and thinking about his goals. [For more on Threat Profiles, check out “Why We Love Threat Profiles”.

For an eCommerce applicaton, the Threat Profile might start off like this:

An adversary wants to…

• … view the shopping carts of other users
• … steal the credit cards of other users
• … change the price of products in the catalog
• … change the shipping address of another user’s order
• … deny access to other users

Our Threat Profile Repository for PCI related applications helps us prepare the threat profile quickly. Once we prepare the threat profile, we bounce it off with the client to get her feedback. Have we missed something? Have we over-emphasized a threat? That helps us finalize the threat profile.

Next, we study the code layout - first the high level blocks, then the classes, the pages and the interfaces with other applications. We are trying to get a feel of “what happens where” in the code. Remember that we will not be reading every line of code in your application - that’s ineffcient and not needed either. But we do need to get a feel of where the acton happens.

After that high level understanding, we prepare the code review plan.

The code review plan first maps each threat in the threat profile to the relevant classes and pages. Then for each of them, we specify the type of check we will do. For instance, consider the threat “An adversary wants to view the shopping carts of other users”. The relevant elements for this threat might be the class that processes shopping cart details, and the pages that request and display the shopping cart contents. What are the relevant checks? In the class that processes shopping cart details, we check (1) if it verifies that the cart requested belongs to the user, (2) if it resists a SQL Injection attack expanding the range of the SQL query, etc. Similarly, we prepare the code review plan for each of the threats.

The code review plan is conceptually similar to the Test plan we use in our penetration tests; it’s just closer to the code level now, though.

The code review plan is reviewed by a senior and then finalized. The detailed code review is next. The detailed review uses a combination of manual inspection and tool-assisted inspection. Some weaknesses like SQL Injection and Cross Site Scripting can be found efficiently with automated scanning tools. Other weaknesses related to insufficient enforcement of business logic can be found with manual inspection.

The code reviewer adheres to the code review plan, and revises it during the project when we get new ideas. As the code review progresses, he ticks off the checks to ensure we don’t miss anything.

Once the detailed review is complete, we prepare the report. The Executive Summary of the report captures the strengths and weaknesses. The detailed findings section describes each finding - the impact, how it can be exploited, the complexity and the solution. We also show the insecure code that causes the vulnerability.

Penetration Testing Healthcare Applications for HIPAA

We discussed the insecurities in healthcare applications some months ago in Palisade. Today, I want to discuss how we test online healthcare sites for security.

Online healthcare sites cover a wide range of applications: from electronic prescription management systems to MIS for medical labs, from health insurance applications to hospital management systems. As part of HIPAA, these sites are expected to be penetration tested to verify their security.

The first step in our penetration testing process is creating a Threat Profile. A “Threat” is the goal of an adversary, it’s what the bad guys want to achieve. A “Threat Profile” is the list of all threats to an application. [For more about Threat Profiles, please read “Why We Love Threat Profiles”].

The Threat Profile is central to our testing methodology. For an online health insurance application, the threat profile may read like:

• views insurance claims of other users
• modifies/deletes insurance claims of others
• views medical records he is not authorized to see
• falsely changes the status of a claim to “approved”
• changes the terms of the plan

Notice these are the things an adversary might be interested in. Logically, that’s where we should start from.

It takes about half-a-day to two days to create the threat profile - that depends on the complexity of your application. We study the application, prepare a draft threat profile and then get your feedback. The Plynt Healthcare Threat Profile Repository helps accelerate this step. This repository is a collection of threats we have already seen in similar healthcare applications.

Once the Threat Profile is ready, we create the Test Plan - the specific tests to perform for checking each threat. This is the intensely technical part of our test, when we visualize in the mind’s eye the various possibilities for attack.

The Test Plan first maps each threat in the Threat Profile to specific pages on your site. For example, consider the threat, “The adversary views insurance claims of other users” might be mapped to the “View Claims” page. Next, the Test Plan identifies all the attacks to try on those pages to realize that specific threat. For example, on the “View Claims” page, we might decide to try a Variable Manipulation attack and a SQL Injection attack to see claims of other users. The Test Plan is thus prepared for all the Threats to the application. To assist our engineers, we have a master reference checklist of all attacks - they pick attacks for the Test Plan from that checklist.

Once the Test Plan is prepared, it’s reviewed and approved by a senior. The actual testing begins only after that. The tests are a combination of manual and automated checks. The penetration tester adheres to the original Test Plan. The test plan is updated when he gets new ideas during the test.

When an attack succeeds, we capture the screenshots of the attack. Our final report walks through the attack with the aid of these screenshots.

Any large application penetration test involves hundreds of test cases, so it’s important that we focus on the right set of test cases. We should, for instance, focus on whether a the terms of a plan can be modified than on generating error messages by tampering unimportant variables. The Threat Profile to Test Plan approach helps us focus our testing on the threats that matter to you and HIPAA.

What we look for in our Penetration Testers

I am often asked what qualities we look for when we recruit penetration testers. We are now recruiting 8 new team members, so this is as good a time as any to share our recruitment process.

We look for 3 qualities in our penetration testers:

1. Technical clarity
2. Analytical horsepower
3. Communication skills

What exactly do we mean?

Technical clarity is a person’s clear understanding of any technical topic he loves. We don’t necessarily require you to know penetration testing – yet. We’ll bring you up to speed on that quickly. But we do require that you know a few technical topics you like really well.

Analytical horsepower is a person’s ability to analyze a problem and make good inferences. As penetration testers we analyze an application’s behavior and try to predict how the developers have written the application. Without seeing the source code, we have to deduce possible errors in the application, and that requires strong analytical skills.

Good communication skill is a person’s ability to express ideas clearly in correct and fluent English. Our findings are presented to our clients as written reports. So the ability to write good English matters. Clients seek our advice on tricky issues they encounter when they fix holes – so the ability to explain techniques clearly over the phone, or in person matters.

How do we identify new team mates?

We recruit engineers with different backgrounds to our penetration testing team - from fresh grads to experienced security engineers. We receive a large volume of applications each year. We shortlist candidates for interviews is based on three written tests, or on the direct relevance of the experience:

1. A 40-minute test of Analytical skills
2. A 20-minute test of English
   
3. A 20-minute essay

Successful candidates appear for the round of interviews. The interview panel is usually a recent entrant into Paladion and a seasoned Paladion engineer. Over the next 1 – 4 hours, it’s their job to determine how well you’ll fit our team.  

Our interviews are usually fun, and sometimes even unsuccessful candidates tell us they enjoyed the experience. The focus in these interviews is to see how strong you are in a few technical areas of your choice. We believe that if you are really strong in your favorite areas, you will take on to penetration testing also quickly.

So, you chose the topics to discuss.

The topics don’t have to be fancy. We are as comfortable discussing the 555 timer as your mechatronics project. So don’t berate yourself if your favorite subject isn’t “cool”.

We are unlikely to be experts in your favorite area, and that’s a good thing. You can teach us all you want, and we’ll be glad to clarify our doubts. We have observed that folks, who are really clear about a technique, can explain it in simple terms. So please start from the basics, and develop the topic as we go along. The deeper we go, the stronger an impression we’ll get of your technical skills.

The interviews are of course a great place for you to learn about us. The panel will be glad to help you understand Paladion better. The seasoned engineer can tell you the history and background of our work. Newly minted team mates can answer your questions about the culture and the fun we have. Both of them are quite well-trained in pitching Paladion to candidates we like :)

That’s it in a nutshell. The interviews help us find you and they enable you to decide if we are your ideal workplace. 

Reducing Variance In Our Security Tests

How do we minimize variance in our tests? How do we ensure that if two of our team mates were to test an application, they would find the same holes? That the results would be independent of whether it was tested by a senior or a relative junior?

That seems impossible - a senior would of course find more holes -, and I admit we haven’t solved the problem yet. But, this is important to our clients. Clients rightly demand that the testing methodology and the test engineer’s skills should be good enough to minimize variances in the tests.

Today I want to discuss the ideas we have been experimenting with and hear your thoughts.

On to the options:

1. Training: Ensure all engineers are equally well-trained, that everyone knows the methodology and tests thoroughly. We do a lot of training here, but we know that variances still occur. Two engineers might understand the same technique a bit differently, and this could lead to variance. Training alone, we feel, is inadequate to minimize variance. It’s an obvious first step.
2. Quizzes: This is the simplest method, and something we do regularly. Conduct periodic quizzes to see variances between engineers. Analyze the questions where there’s maximum variance. That tells us where to focus our attention on to reduce variance. Quizzes have limitations: two engineers might just interpret the question differently and give seemingly incorrect answers. And a written quiz is no guarantee that careless errors will not occur in a live test.
3. Test the same application twice: Let two engineers work independently on a live project. Do they arrive at the same findings, or is there a difference? If our 50 engineers were split into 25 pairs and tested different apps, we would understand our variances better. The approach sounds attractive, but it’s very difficult to do in practice. We are perennially short of people, and staffing 25 projects with additional team members is a luxury we can’t afford yet.
4. Have all the engineers test the same sample app for a day: Same as above, but simpler to implement. Set up a sample app (an open source one like Wordpress for example), and have all 50 security testers go after it for one day. Check the variance. We love the sound of it, especially the whiff of “Capture the Flag”. We might do this one of these days, but we are just worried that a 1-day simulation is a far-cry from a 10-day test.

For now, quizzes and training are the techniques we use most to measure and reduce variance. We have done a bit of option 3, but not across the entire team. And we haven’t tried 4 yet. We’d love to hear from you the methods you follow to reduce variance in your team’s work.

Earlier Posts

• Application Security in the ISO27001 Environment | 22 May 2008
• Security Testing in a Flat World | 22 May 2008
• Security Testing Online Travel Sites | 09 May 2008
• How We Test Online Trading Sites For Security | 16 Apr 2008
• Helping B2B applications win more customers | 16 Apr 2008