Test economics

USA: Executives rely on engineering and manufacturing operations to maintain marketplace advantage. They measure financial and business metrics such as return on invested capital (ROIC), return on assets (ROA), time to market, profit margin, and product quality to drive improvements in product development. However, methods used to measure test organizations are less standardized.

The reality is that many companies have test organizations with reactive approaches that lack a long-term strategy or financial impact metrics, as described in the “Optimizing Test Organizations” business trend of NI Automated Test Outlook 2012. This forces executives to ask the wrong question of test functions; it is not just “How do I reduce my cost of test?” but rather “Which test investments (people, process, and technology) do I need to improve business metrics?”

According to the 2012 NI test leadership survey, approximately 66 percent of an organization’s test budget is spent on maintaining the current environment with only 34 percent dedicated to providing new business capabilities to the business. This means organizations can make only incremental improvements to test operations. By contrast, design engineering teams are developing new products with step function increases in capability, and lower price points.

To meet this challenge and justify strategic investment in test, test organizations are proposing initiatives backed by financial metrics such as return on investment (ROI), cost per unit tested, annual test costs and savings, payback periods on investments, and the breakdown of capital versus noncapital costs. Proper modeling uncovers all lifetime costs of certain test assets and provides a financial framework for justifying future investments.

Total cost of ownership
A fundamental way to assess business impact of a test organization is to determine total cost of ownership (TCO). TCO is emphasized because test cost is often attributed to capital-equipment costs of test systems. While capital cost is a key component, there are other elements such as upfront development cost, deployment test cost, and operational/ labor cost. The combination of these components represents total cost of test for a product and/or company.

Upfront development cost elements are typically one-time costs such as strategy development, hardware and software tools, non-recurring engineering (NRE), and internal training. It is important to note that an organization may have additional development test costs not represented in these common elements.

Deployment costs are costs incurred every time a tester is deployed. They are the easiest to determine since most include cost of capital equipment along with software deployment costs. There are additional deployment test costs for test organizations, such as shipping, so it is important to carefully examine deployment costs to ensure consistency.

Operational test costs are a combination of personnel, maintenance, and facility costs to keep testers running. More specifically, personnel costs accumulate man-hours, while capital costs reflect cost of maintaining spares to prevent downtime in the event of a failure or need to remove instruments for calibration.

Knowing TCO at the project, department, and company level is a powerful tool for test leaders to represent actual added value of a test strategy and to help justify future investments.

Strategic test scenarios
Accurate TCO models are invaluable when determining where to invest for maximum ROI and where to reallocate resources. It can determine process improvements and increase efficiency while lowering design and manufacturing costs affected by test. However, financial metrics are not always the same. The following case studies examine transformation initiatives executed by organizations that affected different financial metrics (reducing operational cost, minimizing true cost of test, and improving cost/defect), but each measured ROI and payback on initial investment.

Production test standardization: Large organizations segmented by business units (BUs) or product lines typically have their own P&Ls. Each BU typically owns its product development and manufacturing process and resources. This leads to each BU developing testers specific to product(s), which leads to a heterogeneous mix of test equipment based on individual product lines.

Developing a common  test platform across multiple test platforms not only reduces capital costs by leveraging economies of scale but also decreases operating and maintenance personnel costs by eliminating operators and technicians. Hella KGaA Hueck & Co executed this strategy by aligning multiple product lines and standardizing common hardware and software. Hella realized a 46 percent reduction in operational test costs while increasing test throughput by 57 percent, resulting in a 37 percent ROI and a payback period of just eight months.

Scaling test throughput: As product demand increases, companies need to reevaluate testing methodologies and develop systems that can test growing volumes of product at lower margins while maintaining product quality.

Investing in a next-generation multi-unit parallel test strategy has significant impact on test throughput and test TCO per unit. Harris Corp. reduced its cost of test by 74 percent while scaling the test solution to meet increased demand. The project had an ROI of 185 percent and a payback of just 2.8 months.

Our mission is to develop innovative high-quality products that simplify our customers’ lives. By investing in our test platform team, who leveraged NI technology for automation and reuse, we reduced our cost of quality by 81 percent, while maintaining full compliance with the international regulatory standards, saving us $4.5 million annually, said Katherine dePadua, VP of Quality and Regulatory Affairs, Philips HHS.

Engineering test re-use: While the definition of quality varies by company, it is universally a high priority. Results based on analysis of 63 software development projects at companies such as IBM, GTE, and TRW Automotive show the cost of finding a product defect during production was 21 to 78 times more expensive than during design.

Engineers at companies seeking to differentiate products by adhering to the highest quality standards focus on investing more in test systems that reuse test components to capture defects earlier. Philips Healthcare saved $4.5 million annually with a test strategy that reduces cost of quality by 81 percent. This strategy identifies defects earlier in the design process, resulting in a 316 percent ROI and three-month payback period.

Measuring and tracking financial metrics are necessary complements to any test vision and strategy. Justification for investment in test relies on proving long-term ROI as well as metrics to measure success. The test TCO approach evaluates investment alternatives and makes a proactive impact on business metrics.

-- National Instruments, USA.