As enterprise architecture (EA) matures into an established discipline in most moderate to large scale IT organizations, EA is increasingly being challenged to demonstrate objective evidence of its contribution to the corporation. Beginning even before the current economic crisis began, the senior business management intensified their scrutiny of IT in relation to measurable benefits. Increasingly, IT divisions are being required to produce detailed metrics that provide quantitative evidence of their contributions.
This presents a particular challenge to EA. Other IT divisions tend to have well-established criteria to measure their performance: Percent uptime for systems, error rates for applications, number of calls resolved on the first contact for help centers, etc. EA, on the other hand, is rarely directly responsible for any obvious quantifiable measure. Its impact is more subtle. EA governance, for example, may significantly enhance the quality of development efforts, but is rarely held directly responsible for the success or failure of a specific project. Similarly, while EA clearly has an important oversight role with regard to infrastructure and time to recover from a planned drill or actual disaster, you cannot reasonably attribute measurable results to the quality of EA efforts. An effective EA organization, while touching on a vast array of IT activities, is solely responsible for relatively few projects. Its value is far more often incremental, strengthening system development and maintenance and avoiding unnecessary risk rather than having primary responsibility.
The development of common services for which some EA organizations assume responsibility is an exception to this rule, but the measurement of the value of these services is not simple either. Several measures are applicable:
- The number of services used by multiple applications
- Error rates
- Transaction throughput
However, this last metric—transaction throughput—may not provide a clear indication of EA contribution
that is meaningful to senior management. CEOs and CFOs are often unimpressed with these numbers. They are aware
that the real question is whether the centralization of services is cost-effective relative to each application.
They do not accept on faith that common services—which you often need to "tweak" for effective reuse—save the corporation money. While upper management is interested in these statistics, they do not, in themselves, give executives a means to measure EA's value in relation to its cost.
Beyond the limited number of metrics that are directly attributed to common services, EA's metric waters are muddy. It's success at setting, publishing, and enforcing standards may have an effect on, say, system up time, but how much of an effect? If programs crashed the system or systems that met corporate coding standards but contained buried errors, is EA at fault? Possibly, but it certainly would not be the first area investigated in seeking root causes. If EA shouldn't be blamed if server capacity was maxed out, can it claim credit when all systems perform perfectly? Similarly, quantifying EA's role in project efficiency statistics, such as on-time delivery, number of errors detected and resolved prior to user acceptance testing, requires assumptions and interpretations that defeat the goal of providing purely objective measures.