ometimes bad things happen to good people.

For many developers, that is the perfect summation of how the Sarbanes-Oxley (SOX) Act has affected their daily lives. At first blush, the SOX Act looks like yet another set of checks and balances to ensure accurate financial reporting at public companies. As anyone who has worked in the financial department of a public company can attest, the SEC and FASB (Financial Accounting Standards Board) have been writing and rewriting the rules of the game for years. In this context, SOX is nothing new.

What makes SOX different is one small and very ambiguous section of the Act: section 404. The wonderful irony of this number (think the infamous HTTP 404 error) may be worth a snicker, but unfortunately, the wording and interpretation of this section of the Act have very deep implications for IT.

To paraphrase, section 404 says: “Management has to ensure appropriate internal controls of financial reporting.” If you follow this statement down the financial reporting rabbit hole, sooner or later you find a software system that helps generate these reports. For many corporations, this system comes in the form of a big packaged application such as PeopleSoft, Oracle Financials, or SAP. Other organizations will have a homegrown package or some combination of packaged software and heavy customization.

In any case, SOX 404 has come to be interpreted as saying: IT must make sure there are appropriate internal controls in place when making any changes to any software system that plays a significant part in financial controls or reports. To fully understand the implications of this interpretation, one needs to examine both the concept of an internal control as well as what types of software systems are covered by this statement.

What Software Systems are Included?
So what does it mean to play a significant role in financial reporting? Unfortunately section 404 can be interpreted very broadly. Take, for example, the sources of data used in financial reports: employees (paychecks, benefits, and outstanding vacation time), orders (and all associated financial transactions), inventory (asset management), and physical plants (facilities costs). Any software system that helps manage one of those sources can arguably be included in the list of systems section 404 is concerned with.

Add to that all additional software systems related to finance’s process for reporting (e.g. expense payouts, payables management, revenue booking and recognition). Activity in any of these source management and financial reporting systems rolls into the final numbers that appear in corporate profit and loss statements and balance sheets. It is easy to see why the broad wording in section 404 can start to feel like it covers almost any software system related to your core business.

What is an Internal Control?
So now that we have painted most of your software systems with a SOX bull’s-eye, let’s take a deeper look at the concept of an internal control. An internal control is really a check and balance system that ensures any change made to an application does not adversely effect your financial reports.

As it is likely you work on a software system related to financials, during the audit you may be required to produce proof of your internal development controls and how you can guarantee you follow them.

Let’s take a simple example derived from a real experience. A developer needed a critical fix to the company’s expense application. The developer found the fix he needed in a specific patch released by the vendor. After installing the patch and doing some quick testing, the expense application looked to be working just fine. The developer rolled the patch to production during the next scheduled downtime window. When the end of the month rolled around, everyone found their expense checks were not quite what they expected; they were all mysteriously a few dollars off. The developer did some research and discovered that the patch itself accidentally changed the database table holding expense information to use an integer column rather than a decimal column. All the data went into the user interface correctly but the data stored in the database was incorrect, thus the erroneous expense checks. At a large company, this glitch could add up to enough to cause a real problem in the balance sheet. An internal control would prevent such unforeseen changes.

One of the required activities related to SOX compliance is the SOX audit. This audit is performed by an external auditor, similar to the financial audit your company goes through on a yearly basis. The goal of the SOX audit is to validate the financial reporting processes (as opposed to the actual financial data which is still covered by the traditional audit). As it is likely you work on a software system related to financials, during the audit you may be required to produce proof of your internal development controls and how you can guarantee you follow them.

The time is ripe for automation. Two years ago, internal controls were a moving target; every auditor would tell you something different. Things have calmed down a bit now so a technology investment can have a pretty high probability of being accepted by the auditor.

For the balance of this article I’ll talk about these best practice internal controls and how automating them in your development cycle will help you quickly produce the reports required during an audit.

Best Practice Controls
In the two years since SOX went into effect, thousands of companies have learned the hard way?thanks to an audit?which internal controls are really the most important. These controls should be inserted into any existing development cycle. Because you work in this cycle on a daily basis, chances are you’ve probably already been confronted by some of these changes already. The five most common internal control concepts you will come across are as follows:

• Documentation
• Approvals
• Auditing
• Separation of duties
• Testing

Documentation
One of the easiest steps to ensure you have a well controlled development process is to invest up-front in documentation. Each development process related to making a change in a software system needs to be well documented. This would include how code changes are made, the process of selecting which patches to put in, the testing process, and the review process before changes are released to production. This documentation will serve as a starting point for any auditor working on evaluating if your development process has the right internal controls.

Approvals
Approvals are probably the most common and easiest to implement internal control. Everyone knows what an approval is. The key though is when to require them, who should be in the required sign off list, and what documentation is required with the approval. The three most common approval control points that have emerged in the development process are: feature selection (or prioritization of service requests such as patches), testing signoff, and rollout to production.

In most mature development environments, many of these approvals are already in place. What is changing though is the rigor applied to these approvals. No longer is it acceptable to have an email exchange with QA saying “we’re ready to go” or a verbal conversation with the production team setting aside downtime at 3 a.m. for a production rollout. These approvals have to be formalized with electronic signoffs and authentication as well as full documentation associated with why the approval was made.

As audit data is traditionally sitting in multiple silos (database logs, change management logs in your version control system, etc.), you will find it a painful process to pull together this data for audit reports if you don’t figure out how to automate it.

With the increased pressure to collect and provide historical data related to approvals, an automated mechanism can be incredibly helpful. In many companies, a change control board is being set up for final approval before rolling changes to production. This change control board is comprised of all stakeholders across the SOX compliance landscape from managers, to testers, to members of the finance department. An approval system would require each participant on the change control board to actively sign into the approval system, document the reason for acceptance or rejection, and would prevent the development process from continuing until the approval was appropriately accepted. At any point in time, an electronic paper trail of this approval could be generated.

Auditing
Auditing is the simple activity of recording who performed a change, what was affected in the change, and when that change was made. You’ve probably got some form of auditing in place now. Perhaps you have your database’s audit capability turned on. Perhaps your developers are religious about entering comments into SourceSafe, ClearCase, or whatever development version control system you use.

The key to SOX compliance though is to ensure all types of changes are monitored and audited. If there is a manual step in your build process where you execute stored procedures to create data structures, changes to those stored procedures need to be audited as well. As audit data is traditionally sitting in multiple silos (database logs, change management logs in your version control system, etc.), you will find it a painful process to pull together this data for audit reports if you don’t figure out how to automate it.

Separation of Duties
Separation of duties is the internal control that is causing the most change in development organizations. Separation of duties is the simple concept of sandboxing different users from different parts of a software system. To use our earlier expense application example, a developer that put in the expense patch would not be able to sign off on testing for that patch nor release the patch to production. A separate tester would need to run and sign off on the test cases and another user would be required to roll those changes into the production system. This separation of duties is intended to provide the checks and balances that would have prevented the expense application problem we discussed.

Of all the reactions related to new SOX-driven controls in the development process, separation of duties has incurred the most wrath and hostility. While separation of duties will make your auditors happy and will catch a few problems like the example given, in many cases you will find it a frustratingly unrealistic overhead to introduce into your development cycle. You will surely be tempted to walk down the hall, ask another developer to open their development tool and perform a series of steps that you could have just done yourself but don’t have the right to. Beware: This temptation will come back to bite you during an audit.

Testing
Testing has become a critical aspect of SOX compliance, emerging as a consistent recommendation in most recent audit reports. The value of testing as an internal control is simple: A test can validate that key financial-related business processes still work as planned. Using our example, a good test would have the tester enter a predefined set of expense data and compare a final expense payment voucher to an expected result.

An additional benefit of tackling an automated testing approach relates to a SOX concept called “reliance on the work of others.” During a typical SOX audit, auditors will need to pick and choose which internal controls they can test and what permutations of those controls are tested (e.g. do they test an approval for one approver or every approver possible?). At some point in time, the auditor will need to rely on the work done by your own IT staff to ensure the internal control is working effectively. Automated testing solutions provide a reliable, consistent, and well documented history that many auditors accept willingly.

Automated Reporting of Application Changes
SOX feels a lot like network security. We all know about it but don’t pay much attention to it until something dramatic occurs. For the developer, while noticeable changes may occur in the daily development cycle due to SOX, not until you are asked to prove your internal controls will you have to face it directly. If you are unprepared, proving you have good controls can be a laborious and time-consuming activity. In this section, I’ll give you a few ideas for how best to prepare.

The earlier you can determine a strategy to cope with the new demands that SOX places on your development process when making changes, the better.

In a SOX world, uncontrolled change is your enemy. Unfortunately the development cycle is all about change. Patches, fixes, customizations, new development etc. are all hallmarks of daily activity in development. The earlier you can determine a strategy to cope with the new demands that SOX places on your development process when making changes, the better. Your three main weapons are repeatability, automation, and rollback.

Repeatability really means you have an articulated process that everyone follows for every kind of change. The first step is to document this process. The second step is to make sure the process includes the very controls we’ve talked about: approvals, an audit trail, testing, and clear separation of duties. If your process is not performed the same way every time it is virtually impossible to successfully argue to your auditors that it is well controlled.

Your second weapon is automation. The more you can automate SOX controls in the your development process, the better, An automatic system for collecting authenticated approvals (with required documentation) will be better than an email buried in someone’s inbox. Test results from an automated testing tool will fly much better than a signature on an Excel spreadsheet printout of a few test cases. Additionally, automate as much of the capturing of audit data as possible. This data will be the backbone of any compliance reports you have to generate.

The last weapon in your SOX compliance arsenal is the ability to rollback (sometimes called restore). The beauty of a restore framework is if you discover you are out of compliance you can immediately revert to a state in the development cycle when you were confident you were still compliant. To get to a restore framework you first need to implement versioning across all assets in the development cycle. For traditional software development version control systems such as SourceSafe, PVCS, etc., have become commonplace. However, many developed software systems in IT don’t have the same facilities. For example, large packaged applications (PeopleSoft, SAP, etc.) don’t have any inherent version control system. Additionally, many database-centric applications that rely on stored procedures or external reporting tools such as Crystal, Cognos, or Informatica don’t have any of their assets versioned. A number of solutions are emerging to bring versioning to these environments. Once versioning is in place, careful organization and versioning during releases can provide a reliable rollback point to a compliant state.

While the task may seem daunting, with these three weapons at your side, you’ll be ready for the dreaded day that your SOX auditor calls up and says “show me how you make changes to your expense application.” Consider what you’ll have to do now to answer that request versus the surprised look on your auditor’s face when, with a smile and only a few clicks, you happily hand over whatever they need.