Lesson 3 — Impact Analysis and Financial Evaluation

Module 3, Unit 3 | Lesson 3 of 3

By the end of this lesson, you will be able to:

• Distinguish qualitative impact analysis from quantitative financial evaluation, and explain why both are needed (K7, K24)
• Apply Cost–Benefit Analysis (CBA) and the three core financial metrics — NPV, BCR, and Payback Period — to an AI initiative (K13, S3, S4)
• Translate technical capability into measurable operational change and broader organisational value through an impact story (S5)
• Recognise the AI-specific failure modes — overstated benefits, missing lifecycle costs, time-savings that don't convert to value (B1, B2)

From "can it be built?" to "should it be built?"

Once a delivery architecture is in place, a different question emerges: is the initiative worth doing?

Technical feasibility shows that a project can be implemented. Impact and financial evaluation determine whether it should be. Decision-makers rarely approve initiatives because they are technically possible. They approve them because the expected value justifies the time, money, and organisational disruption required to deliver them — and because that value compares favourably with what could be done instead.

Evaluation usually happens in two complementary stages. The first is qualitative impact analysis: examining the operational, strategic, and governance consequences of the initiative — what improves, what gets harder, and whether the net change is positive. The second is quantitative financial evaluation: translating expected benefits and costs into economic terms so they can be compared using consistent metrics.

Neither stage removes uncertainty. What they do is force assumptions to be made explicit. Optimistic projections, missing cost categories, and undocumented "improvements that will obviously translate to value" can all hide inside a confident-sounding business case. Structured evaluation makes those weak points visible early enough to fix them.

🔑 Key term: Impact analysis — the structured assessment, conducted before financial modelling, of how an initiative is expected to change the organisation. It examines benefits, risks, governance implications, and operational adjustments together, so that the financial case is built on a credible operational story rather than on optimism.

Business Impact Analysis (BIA)

Impact analysis asks the question financial models cannot: what difference will this intervention actually make to the way the organisation works?

It is broader than financial evaluation. It examines operational performance, decision quality, user experience, governance implications, and strategic value — and it does so before benefits are translated into pounds. This sequencing matters. A financial model can determine whether an investment is economically viable, but it cannot determine whether the underlying change is meaningful or strategically desirable. That is the prior question, and skipping it is one of the most common ways a polished business case ends up funding the wrong thing.

The discipline of BIA is to examine benefits and risks together. Benefits might include faster processing, improved decision accuracy, reduced manual workload, or enhanced service quality. Risks might include implementation disruption, workflow change, new governance obligations, or shifts in accountability when an automated system enters a previously human-only decision. The objective is not to eliminate risk — most meaningful initiatives carry some — but to determine whether expected benefits outweigh exposure once both are properly seen.

Two biases this protects against. Optimism bias is the well-documented tendency to overestimate benefits and underestimate implementation cost. Excessive risk aversion is its opposite — rejecting valuable initiatives because their benefits have not been clearly articulated. Looking at both sides of the ledger forces a more balanced judgement than either bias produces alone.

For AI initiatives, the timing of impact analysis matters particularly. Intelligent systems often change how decisions are made, not just how fast. They alter who is accountable, what evidence supports a decision, and where governance attaches. These broader consequences need to be examined before deployment, not retrofitted afterwards. Responsible AI guidance treats this kind of pre-deployment assessment as a lifecycle obligation, not optional good practice.

Impact analysis should also be revisited at transition points: pilot to full deployment, expansion to new user groups, integration of additional datasets. These are exactly the moments when a system's risk profile changes and the original benefit/risk balance needs reassessing.

Cost–Benefit Analysis (CBA)

Once impact analysis has shown the initiative is worth evaluating financially, Cost–Benefit Analysis translates expected outcomes into economic terms. The objective is not to remove uncertainty — it is to make the assumptions about value transparent enough that they can be examined, challenged, and tested.

The discipline of a credible CBA is whole-life costing. Costs are not limited to development. They include implementation labour, software licences, infrastructure, integration, training — and equally, the operational costs that arise after deployment: maintenance, monitoring, model retraining, governance review, compliance activity, support. For AI systems specifically, ongoing monitoring and oversight are not optional extras; they are conditions of responsible operation, and they have to be in the budget.

Benefits are evaluated over the same time horizon. Some are straightforward to quantify — labour savings from automation, reduced rework from fewer errors. Others involve productivity improvements, increased capacity, or better decision quality, which can usually be monetised once the operational pathway is clear. Some — improved employee satisfaction, organisational reputation, customer trust — may resist precise financial estimation. Document those qualitatively and keep the financial model focused on what can defensibly be quantified.

The output of CBA is not a single number. It is a structured financial model that documents assumptions, calculations, and projected outcomes, and that allows decision-makers to examine the economic logic — not just the bottom line.

Did you know?

The first formal cost-benefit analysis was published in 1848 by Jules Dupuit, a French civil engineer asking whether a public bridge was worth building. His paper De la mesure de l'utilite des travaux publics argued that the social value of infrastructure could and should be measured by what users would willingly pay for it, not by construction cost alone. Modern CBA, including in HM Treasury's Green Book, still rests on the framework Dupuit sketched out nearly 180 years ago.

Doing this with AI

The model will happily produce a CBA with all the right-looking categories. The trap is that it tends to fill in plausible-sounding numbers without flagging that they are guesses. After a draft, ask: "For each benefit and cost line, mark whether the figure is grounded in evidence, an industry benchmark, or an assumption you have invented. List the assumptions in priority order of how much they affect the result." The first column it produces is usually long and revealing.

Three financial evaluation metrics

Three metrics summarise a CBA in ways decision-makers can interpret quickly. Each views the investment from a slightly different angle, so used together they triangulate.

Net Present Value (NPV) measures the total economic value created by an investment, after accounting for the time value of money. Money received in the future is worth less than money received today — because money today can be invested to earn returns. NPV discounts each future cash flow back to its present value, then subtracts the initial investment. A positive NPV means the project creates value relative to the organisation's required return; a negative NPV means it destroys it. NPV is widely considered the most analytically robust metric because it captures both the scale and the timing of returns.

Benefit–Cost Ratio (BCR) divides the total value of benefits by the total value of costs. A ratio above 1.0 means benefits exceed costs; below 1.0 means they don't. The strength of BCR is interpretability — sponsors can grasp instantly that a BCR of 2.0 means £2 of value for every £1 invested. The weakness is that it doesn't capture timing the way NPV does, so two projects with identical BCRs can be very different investments.

Payback Period measures how long the initiative takes to recover its initial investment from the benefits it generates. A project costing £120,000 with annual net benefits of £40,000 has a three-year payback. The metric is simple and intuitive, and it answers the question many sponsors actually ask first: how long are we funding this before it pays for itself? Its limitation is that it ignores everything that happens after recovery — a project that pays back fast but produces little long-term value can look stronger than one that pays back slowly but generates substantial total returns.

In practice, professional appraisal uses NPV as the primary metric and includes BCR and payback period as supporting indicators. Each one alone can mislead; together they provide a balanced view of scale, efficiency, and timing.

💬 Reflection

Which metric does your sponsor reach for first? In most organisations the answer reveals a governance preference: payback-first cultures tend to be cost-sensitive and short-horizon; NPV-first cultures tend to be more strategic and willing to fund longer build cycles; BCR-first cultures are usually trying to compare a portfolio of competing initiatives. Knowing which lens your sponsor uses tells you which lens to lead the case with.

The impact story: a narrative that holds the case together

Numbers rarely persuade on their own. Sponsors and governance reviewers do not just want to know that an initiative has a positive NPV — they want to understand, in clear terms, what will actually change, why that change matters, and how the organisation will know whether value has been created. That is the role of the impact story.

An impact story is a structured narrative that connects the original problem to the intervention, the intervention to a measurable change, and the change to broader organisational value. It is the bridge between technical capability and outcomes that the organisation cares about — and for AI projects in particular, that bridge is where most value claims quietly fall apart. An AI system may classify or predict effectively, but if that capability does not translate into faster decisions, fewer errors, or released staff capacity, no real value has been created.

A strong impact story has four elements:

1. The problem. What is happening now, in specific operational terms? What does it cost the organisation — in time, error, risk, or capacity? The problem statement should be evidence-based, not aspirational.

2. The intervention. What is being introduced, described in proportionate and precise terms? For AI work this matters: avoid abstractions about "intelligence" or "innovation" and describe what the system will actually do in the operational context.

3. The measurable change. What specific improvement is expected if the intervention works? Linked to a baseline where possible, or at minimum the indicators that will be used to determine whether change has occurred. This is where the story moves from intention to evidence.

4. The broader value. Why does the measurable change matter to the organisation? Strategic, financial, regulatory, human — the larger outcome that the operational improvement enables.

What makes this structure effective is not the labels but the discipline. The story has to show a plausible causal chain — if the measurable change does not logically follow from the intervention, or if the broader value does not depend on the measurable change, the narrative is weak and the financial case sitting on top of it will not hold up under questioning.

In practice, building the impact story often reveals weaknesses in the underlying analysis. A team may realise it cannot explain how the proposed solution actually creates value, or that its measurable outcomes are too vague, or that the broader value claim depends on organisational changes nobody has planned for. That is not a failure of the framework — it is one of its strengths. The story is a pressure test for coherence.

Doing this with AI

Once you have a draft impact story, paste it back with this prompt: "Stress-test the causal chain. Is there a plausible mechanism by which the intervention produces the measurable change? Is the broader value clearly dependent on the measurable change, or is it asserted independently? Identify any gap where the logic jumps." The model is reliably better at finding holes than at writing seamless prose — use it for the audit, not the first draft.

Project Activity — Complete section 4.5: impact analysis

Open the Module 3 Project workbook and complete section 4.5 Impact analysis. This is where the investment case has to show how the initiative creates value.

1. Complete the Business Impact Analysis by naming the processes, roles, systems, dependencies, and controls that change.
2. Complete the Cost-Benefit Analysis with quantified benefits, quantified costs, and the net position.
3. Write the impact pathway as a chain from technical capability to operational change to measurable economic value.
4. Run the sensitivity tests in the workbook: lower benefits, timeline slip, and lower adoption.
5. Write the one-paragraph impact story an executive could remember and repeat.

Project Checklist

• Section 4.5 names the operational processes, roles, systems, and dependencies affected.
• My CBA includes quantified benefits and quantified costs from section 4.4.
• The impact pathway has no missing causal step between intervention, behaviour change, measurable outcome, and value.
• Sensitivity analysis tests lower benefit, timeline delay, and weaker adoption.
• The financial case does not depend on unplanned organisational changes.
• The impact story is memorable but still evidence-based.
• Assumptions are visible enough for a reviewer to challenge.
• The case would still make sense to someone sceptical of the technology.

Quick Check

Unit 3 KSB summary

By the end of Unit 3, you can identify and respond to project risks, build a credible whole-life budget, and model impact and ROI in a way that survives sponsor and finance scrutiny.

Knowledge: K3 (risk frameworks), K4 (financial appraisal), K7 (impact modelling), K13 (whole-life budgeting), K21 (sensitivity analysis), K24 (cost-benefit analysis for AI)

Skills: S3 (risk identification and scoring), S4 (response strategy design), S5 (budget construction), S15 (financial evaluation), S22 (assumption logging), S24 (scenario testing)

Behaviours: B1 (financial honesty), B2 (recognising under-budget as a planning failure, not a virtue), B3 (treating risk as a register, not a checkbox), B4 (transparency about uncertainty)

⏭️ Up next — Unit 4: Unit 3 has built the analytical case for the project. Unit 4 turns to the people side of the work — stakeholder analysis, change implementation, and structured communication — because even a technically sound, financially viable initiative can still fail if the organisational change required to realise it is not actively managed.