Best Practices in Measuring Analytic
Project Performance / Success:
Financial Performance Measures
Best Practices in Measuring Analytic
Project Performance / Success:
Financial Performance Measures

In the 5th Annual Survey (2011) data miners shared their best practices in how
they measure analytics project performance / success.  The
previous web page
summarizes the most frequently mentioned measures.  Since some of the richest
descriptions contain measurements that cross several of the categories, a data
miner's best practice description may appear in several of the verbatim lists (
model
performance, financial performance, outside group performance).  The remaining
verbatims are in the
other best practice measures list.

Below is the full text of the best practice methodologies that included measures of
financial performance (ROI and other financial measures).  


  • We measure ROI, cost, gain, model accuracy, precision, recall, ROC,
    AUC, lift charts, and customized metrics. The focus is on the benefit for
    the business and for the customer.

  • Longitudinal validation based on hard, objective outcomes, preferably
    financial where sensible and achievable.

  • We now know approximately how much it will cost to develop a    
    workable solution. We factor that cost into the feasibility. We also
    continuously refine our cost/benefit analysis throughout the evolution of
    the project. In the end, success is what the sponsor and team says it is.
    It does not always end up as increased revenue or lowered costs.

  • I work in Lean Six Sigma, so we routinely quantify the financial benefits
    of analytic projects and opportunities.

  • Real-world analysis of results, almost always tied to a financial measure
    (i.e., something that can be expressed in dollars or readily converted to
    dollars).

  • Cross-validation and sliding-window validation during model training
    and data mining process and parameter optimization.  Metrics:
    accuracy, recall, precision, ROC, AUC, lift, confidence, support,
    conversion rates, churn rates, ROI, increase in sales volume and profit,
    cost savings, run times, etc.  Continuous monitoring of model
    performance metrics.  Use of control groups and independent test sets.

  • Standard statistical measurements (KS, ROC, R-square etc.),
    profitability metrics, loss impact etc.

  • Metrics: model prediction accuracy, saved costs, gained increase in
    sales volume, gained increase in customer satisfaction, reduction of
    churn rate, ROI, gained insights;    Best practice: ask for target metrics
    from day one on, i.e. as soon as talking about project and application
    requirements; measure project success along these metrics and
    optimize these metrics.

  • Accuracy of model predictions, ROI.

  • Try to translate results/lift in terms of money.

  • Test & control groups. Incremental ROI gain.

  • Client satisfaction, revenue, legal outcome.

  • Business metrics, incorporating costs and benefits that simulate
    deployment.

  • Measurable improvement in patient care and cost-of-services should
    eventually (1 to 2 years) outpace cost of software, hardware, offices     
    and staff time.

  • Monitoring default rate, profitability of various lines of business

  • Reduced default rate, return on investment.

  • ROI of marketing campaigns  Calls to compare data predicted with
    customer replies.

  • ROI, comparison with previous years, cluster rank analysis.

  • Subrogation and fraud detection, dollars recovered.

  • We develop KPIs to monitor how the organization change behaviour
    after implementing new analytical products.  We register how decision
    making is affected by new analytical products and if that change
    increase sales, and/or revenue.

  • Have none - revenue (change in revenue) is only metric really
    measured.

  • Actual sales return on investment is better than Expected SROI.

  • Bottom-line Business $ impact.

  • Clear defined business benefits case at the end of each project
    demonstrating ROI.

  • Collection Scoring results can be measured how much money we got
    afterwards.

  • Critical that ROI calculations are performed upfront.

  • In dollars.

  • Incremental profitability.

  • Loss on loan portfolios.

  • Matched against earned value or lean six sigma metrics.

  • Measure the outcome, not the analytics.  Did the company make or
    lose money when something is implemented?

  • Measure the ROI by detecting the deviation + or -

  • Money earned

  • Project management with validated measurements for ROI within 12
    months.

  • Relate project outcome decisions to revenue enhancement or cost
    optimization.

  • Revenue generated by utilizing the models.

  • ROI

  • ROI

  • ROI

  • ROI or other form of profitability and/or expense reduction.

  • ROI, sales, revenue, margin

  • Tendency is to tie results to impact on the bottom line.

  • We run a financial impact analysis of using the model vs. having no
    model.
Copyright (c) 2012 Rexer Analytics, All rights reserved