Best Practices in Measuring Analytic Project Performance / Success: Financial Performance Measures

In the 5th Annual Survey (2011) data miners shared their best practices in how they measure analytics project performance / success. The previous web page summarizes the most frequently mentioned measures. Since some of the richest descriptions contain measurements that cross several of the categories, a data miner's best practice description may appear in several of the verbatim lists (model performance, financial performance, outside group performance). The remaining verbatims are in the other best practice measures list.

Below is the full text of the best practice methodologies that included measures of financial performance (ROI and other financial measures).

• We measure ROI, cost, gain, model accuracy, precision, recall, ROC, AUC, lift charts, and customized metrics. The focus is on the benefit for the business and for the customer.
• Longitudinal validation based on hard, objective outcomes, preferably financial where sensible and achievable.
• We now know approximately how much it will cost to develop a workable solution. We factor that cost into the feasibility. We also continuously refine our cost/benefit analysis throughout the evolution of the project. In the end, success is what the sponsor and team says it is. It does not always end up as increased revenue or lowered costs.
• I work in Lean Six Sigma, so we routinely quantify the financial benefits of analytic projects and opportunities.
• Real-world analysis of results, almost always tied to a financial measure (i.e., something that can be expressed in dollars or readily converted to dollars).
• Cross-validation and sliding-window validation during model training and data mining process and parameter optimization. Metrics: accuracy, recall, precision, ROC, AUC, lift, confidence, support, conversion rates, churn rates, ROI, increase in sales volume and profit, cost savings, run times, etc. Continuous monitoring of model performance metrics. Use of control groups and independent test sets.
• Standard statistical measurements (KS, ROC, R-square etc.), profitability metrics, loss impact etc.
• Metrics: model prediction accuracy, saved costs, gained increase in sales volume, gained increase in customer satisfaction, reduction of churn rate, ROI, gained insights; Best practice: ask for target metrics from day one on, i.e. as soon as talking about project and application requirements; measure project success along these metrics and optimize these metrics.
• Accuracy of model predictions, ROI.
• Try to translate results/lift in terms of money.
• Test & control groups. Incremental ROI gain.
• Client satisfaction, revenue, legal outcome.
• Business metrics, incorporating costs and benefits that simulate deployment.
• Measurable improvement in patient care and cost-of-services should eventually (1 to 2 years) outpace cost of software, hardware, offices and staff time.
• Monitoring default rate, profitability of various lines of business
• Reduced default rate, return on investment.
• ROI of marketing campaigns Calls to compare data predicted with customer replies.
• ROI, comparison with previous years, cluster rank analysis.
• Subrogation and fraud detection, dollars recovered.
• We develop KPIs to monitor how the organization change behaviour after implementing new analytical products. We register how decision making is affected by new analytical products and if that change increase sales, and/or revenue.
• Have none - revenue (change in revenue) is only metric really measured.
• Actual sales return on investment is better than Expected SROI.
• Bottom-line Business $ impact.
• Clear defined business benefits case at the end of each project demonstrating ROI.
• Collection Scoring results can be measured how much money we got afterwards.
• Critical that ROI calculations are performed upfront.
• In dollars.
• Incremental profitability.
• Loss on loan portfolios.
• Matched against earned value or lean six sigma metrics.
• Measure the outcome, not the analytics. Did the company make or lose money when something is implemented?
• Measure the ROI by detecting the deviation + or -
• Money earned
• Project management with validated measurements for ROI within 12 months.
• Relate project outcome decisions to revenue enhancement or cost optimization.
• Revenue generated by utilizing the models.
• ROI
• ROI or other form of profitability and/or expense reduction.
• ROI, sales, revenue, margin
• Tendency is to tie results to impact on the bottom line.
• We run a financial impact analysis of using the model vs. having no model.

‍