Static analysis security tools are new enough that, to our knowledge, no formal studies have been done to measure their impact on the software built by large organizations But as part of our work at Fortify, we ve watched closely as our customers have rolled out our tools to their development teams and security organizations Here we describe

2 This section began as an article in IEEE Security & Privacy Magazine, co-authored with Pravir Chandra and John Steven [Chandra, Chess, Steven, 2006]

the results we ve seen at two large nancial services companies Because the companies don't want their names to be used, we'll call them East Coast and West Coast

A central security team is charged with doing code review Before adopting a tool, the team reviewed 10 million lines of code per year With Fortify, they are now reviewing 20 million lines of code per year As they have gained familiarity with static analysis, they have written custom rules to enforce larger portions of their security policy The result is that, as the tools do more of the review work, the human reviewers continue to become more ef cient In the coming year, they plan to increase the rate of review to 30 million lines of code per year without growing the size of the security team Development groups at the company are starting to adopt the tool, too; more than 100 programmers use the tool as part of the development process, but the organization has not yet measured the impact of developer adoption on the review process

A central security team is charged with reviewing all Internet-facing applications before they go to production In the past, it took the security team three to four weeks to perform a review Using static analysis, the security team now conducts reviews in one to two weeks The security team expects to further reduce the review cycle time by implementing a process wherein the development team can run the tool and submit the results to the security team (This requires implementing safeguards to ensure that the development team runs the analysis correctly) The target is to perform code review for most projects in one week The security team is con dent that, with the addition of source code analysis to the review process, they are now nding 100% of the issues in the categories they deem critical (such as cross-site scripting) The previous manual inspection process did not allow them to review every line of code, leaving open the possibility that some critical defects were being overlooked Development teams are also using static analysis to perform periodic checks before submitting their code to the security team Several hundred programmers have been equipped with the tool The result is that the security team now nds critical defects only rarely (In the past, nding critical defects was the norm) This has reduced the number of schedule slips and the number of risk-managed deployments in which the organization is forced to eld an application with known vulnerabilities The reduction in critical defects also signi cantly improves policy enforcement because when a security problem does surface, it now receives appropriate attention As a side bene t, development teams report that they routinely nd non-security defects as a result of their code review efforts

Adoption Anxiety All the software development organizations we ve ever seen are at least a little bit chaotic, and changing the behavior of a chaotic system is no mean feat At rst blush, adopting a static analysis tool might not seem like much of a problem Get the tool, run the tool, x the problems, and you re done Right Wrong It s unrealistic to expect attitudes about security to change just because you drop off a new tool Adoption is not as easy as leaving a screaming baby on the doorstep Dropping off the tool and waving goodbye will lead to objections like the ones in Table 31

Table 31 Commonly voiced objections to static analysis and their true meaning Objection "It takes too long to run" "It has too many false positives" "It doesn't t in to the way I work" Translation "I think security is optional, and since it requires effort, I don't want to do it" "I think security is optional, and since it requires effort, I don't want to do it" "I think security is optional, and since it requires effort, I don't want to do it"

In our experience, three big questions must be answered to adopt a tool successfully An organization s size, along with the style and maturity of its development processes, all play heavily into the answers to these questions None of them has a one-size- ts-all answer, so we consider the range of likely answers to each The three questions are: Who runs the tool When is the tool run What happens to the results Who Runs the Tool Ideally, it wouldn t matter who actually runs the tool, but a number of practical considerations make it an important question, such as access to the code Many organizations have two obvious choices: the security team or the programmers