As problems around the world are mounting, it is important to use resources in a way that maximises outcomes and minimises costs. How we assess that can be tricky. Thankfully, we can use the value for investment (VFI) framework to assess the value for money (VFM) of a program or policy. As a research assistant at the University of Melbourne, I was first exposed to the VFI framework while reading a resource on the framework. I was lucky enough to attend a half-day workshop on the VFI framework presented by Julian King, the creator of the framework, during the 2023 AES conference. I did a bit of further research on the VFI framework in preparation for this blog post.
In this blog post, I would like to give an overview of what I have learnt about the VFI framework, and apply it to a hypothetical evaluation of a reform that is currently happening.
Defining value for money (VFM)
Value for money (VFM) is commonly used when designing and assessing programs to measure how much value is produced and how much resources are consumed. However, different organisations have various ways of defining VFM. In the context of the VFI framework, VFM measures good resource use in terms of how well resources are used to produce outcomes. Essentially, VFM boils down to three key questions:
- How well are we using resources?
- Is the resource use creating enough value?
- How can we create more value from available resources?
Value in VFM does not just encompass money. It can also cover other aspects of a program such as social, environmental and economic; social objectives such as equity, sustainability and human rights; and resources invested such as people, power and inspiration.
An overview of the value for investment framework
The value for investment (VFI) framework is an evaluation framework that can help evaluators assess the VFM of a currently running program or policy. Rather than being an economic evaluation framework, the VFI framework is a process in which evaluators use evidence generated from mixed methods research and evaluative reasoning to judge the VFM of a program. Central to the VFI framework is its participatory nature, where various stakeholders, including end-users, are involved in defining how a program or policy will be judged. This involves meeting stakeholders frequently to hear and understand what they have to say.
The VFI framework consists of 8 steps: the first four relate to formulating the evaluation design while the final four relate to conducting the evaluation itself.
- Step 1 is background research behind the program which often involves collecting and reading documents of the program.
- Steps 2 and 3 engages stakeholders to co-design rubrics that can be used to judge a program or policy. Each rubric consists of two elements:
- Criteria: What stakeholders value; and
- Standards: What evidence is needed to meet a certain level of performance for each criterion.
- Step 4 identifies what evidence needs to be collected to judge a program or policy and how that evidence will be collected.
- Steps 5 and 6 collate evidence behind a program. This involves data collection and analysis respectively, often employing both qualitative and quantitative research methods to understand what the evidence is saying.
- Step 7 makes sense of the evidence to render a judgement for each criterion, leading to an overall assessment of the VFM of a program or policy.
- Step 8 reports on the evaluation results and disseminates it to relevant stakeholders.
The result of using the VFI framework is an overall judgement of the VFM of a program or policy based on reasoning and evidence.
The background behind the NDIS reforms
In this blog post, I would like to apply the VFI framework to plan a hypothetical evaluation of the National Disability Insurance Scheme (NDIS) reforms, a policy that is starting to occur. Before doing so, I would like to offer some background information on the NDIS and its reforms.
The National Disability Insurance Scheme (NDIS) is a social insurance scheme run by the National Disability Insurance Agency (NDIA) for people with disability in Australia. Launched on 1st July 2013, people with significant and permanent disability receive individual funding from governments to purchase supports that meet their reasonable and necessary needs. Although the NDIS has made a difference to the lives of people with disability and their families and carers, a number of problems emerged while implementing the NDIS such as cost blowouts, inconsistency in support and funding decisions made among different NDIS participants and accompanying community and mainstream supports not being developed for people with disability, irrespective of their eligibility of the NDIS.
Hence, on 18th October 2022, the Hon Bill Shorten MP, Commonwealth Minister for the NDIS, launched an Independent Review of the National Disability Insurance Scheme (NDIS Review). The NDIS Review panel was asked to examine the design, operations and sustainability of the NDIS and report back to federal and state disability ministers. What distinguished the NDIS Review from other government inquiries was its deep engagement with the disability community to hear their views on the NDIS and its consideration of a number of challenges the NDIS was facing.
In tabling its final report on October 2023, the NDIS Review panel put forward 26 recommendations and 139 supporting actions to put people with disability back at the centre of the NDIS and to bring federal and state governments together to make the NDIS and disability services accessible and inclusive for people with disability. One of the key issues explored in the NDIS Review was the sustainability of the NDIS which is being cast in doubt due to rising costs associated with the scheme. In its final report, the NDIS Review panel defined sustainability as follows:
“Where the NDIS provides supports that are reasonable and necessary, demonstrably net-beneficial, and cost-effective. Governance arrangements provide clear accountabilities for managing lifecycle costs and financial risks. Scheme expenditure is predictable and provides benefits to participants, carers and the broader community, ensuring that Australians remain willing to contribute to it in an enduring manner.” (emphasis added)
In other words, the panel stated that sustainability does not just cover the costs, but also the benefits of the NDIS. Improving sustainability of the NDIS is important to provide certainty that the NDIS will be available to present and future generations and to maintain the trust that Australians have towards the NDIS. Given that governments will be held accountable for achieving this goal, sustainability of the NDIS is something that can be evaluated using the VFI framework.
Applying the VFI framework to NDIS reform
There are a number of tools that we can use to evaluate the sustainability of the NDIS. Cost-effectiveness analysis involves presenting the changes in costs and benefits as a result of NDIS reform before calculating the net change in cost per unit of benefit. In contrast, cost-benefit analysis converts all costs and benefits to monetary values before subtracting benefits by costs to measure how much is gained from implementing the NDIS reforms. The limitation of these techniques is that they are reductionist and oversimplistic, providing an incomplete picture of the key benefits of NDIS reform and how costs are being reduced.
This is where the VFI framework can be used to evaluate NDIS reforms. By evaluating NDIS reform over a number of criteria, we can assess whether the NDIS is becoming more sustainable not only in terms of reduced financial costs and improved efficiencies, but also deliver increased benefits to all stakeholders. The participatory nature of the VFI framework also allows all stakeholders, particularly people with disability, to participate in the evaluation and to identify what they value the most. This is important given the call from the NDIS Review panel to put people with disability at the centre of the NDIS.
The VFI framework could be applied to evaluate the sustainability of the NDIS over the following steps:
Step 1 (Understand the program): Initially, documents relating to the NDIS before and during the NDIS review would be collected. This would not only include the final reports of the NDIS review and other government inquiries, but also documents that are publicly available such as webpages and newspaper articles. Information from these documents would provide a birds-eye view of what needs to be considered when planning the evaluation such as the context around the NDIS reforms and who we need to engage.
Step 2 (Criteria and standards): Numerous criteria can be used to evaluate NDIS reform. Relevant criteria would be raised by engaging stakeholders who are involved in or are affected by the NDIS. Respectively, these would include policymakers and public servants in the NDIA, as well as people with disability who are eligible or ineligible for the NDIS and their families and carers. It is important that the views of both groups are considered when planning and conducting an evaluation of NDIS reform.
These groups would meet separately over two engagement sessions. For each group, the first engagement session would involve identifying criteria to rate the NDIS reforms, while the second engagement session would come up with standards to assess each criterion. The end-result of these engagement sessions would be a set of rubrics that would allow NDIS reforms to be judged over a number of areas. For example, here are two rubrics on cost-effectiveness and equity that could be developed as a result of consultations with stakeholders.
| Cost-effectiveness | |
| Excellent | The costs of running the NDIS are only increasing by 8% per annum or below. All people who are on the NDIS are benefiting from being on the scheme. |
| Good | The costs of running the NDIS are starting to decrease towards 8% per annum. Most people who are on the NDIS are benefiting from being on the scheme. |
| Adequate | The costs of running the NDIS are continuing to increase unsustainably at more than 8% per annum. The benefits of the NDIS are increasing for more eligible people with disability. |
| Poor | The costs of running the NDIS are continuing to increase unsustainably at more than 8% per annum. The benefits of the NDIS for eligible people with disability have not increased or have even decreased. |
| Equity | |
| Excellent | The NDIS is accessible to all people with disability eligible for the scheme, including everyone from underrepresented groups (for example, Aboriginal and Torres Strait Islanders, rural and regional areas). Mainstream and community disability services are accessible to all people with disability, regardless of their NDIS eligibility and location. |
| Good | Most people with disability can access the NDIS, with only a small number of eligible people with disability finding it difficult to access the scheme. Most people with disability can access mainstream and community disability services, regardless of their NDIS eligibility, though there are areas where these services remain inaccessible. |
| Adequate | People with disability can access the NDIS, but it remains inaccessible for people with disability from underrepresented groups (for example, Aboriginal and Torres Strait Islanders, rural and regional areas). Some people with disability can access mainstream and community disability services, but they remain inaccessible for most people with disability. |
| Poor | The NDIS remains inaccessible to all people with disability. Mainstream and community disability services remain inaccessible to all people with disability ineligible for the NDIS. |
Step 3 (Evidence needed): Once rubrics are developed for each criterion, both groups would be engaged to brainstorm evidence that needs to be collected. This evidence would encompass both qualitative and quantitative data that directly assesses the impacts of the NDIS reforms and the experiences of implementing them. The end-result would be a data collection matrix that would guide data collection and analysis during the evaluation.
Step 4 (Gather and analyse evidence): Over a number of years, a variety of evidence would be collected on different parts of NDIS reform. This would include, but is not limited to, surveys to end-users, interviews and focus groups with NDIA staff and people with disability and observations of NDIS reform in action. These pieces of evidence would be analysed to describe what is happening in the NDIS reforms.
Step 5 (Synthesis and judgement): From the evidence collected, evaluators, ideally in consultation with different stakeholders, would use the rubrics to reach an evaluative judgement on each criterion, as well as the overall VFM of the NDIS reforms.
Step 6 (Reporting): An evaluation report on the NDIS reforms would be produced summarising the VFM and criteria judgements that are reached, and the evidence and reasoning behind these judgements. Additionally, a one-page infographic would be produced summarising the evaluation results for busy policymakers and people with disability and their families and carers.
Conclusion
The VFI framework is a simple but effective tool for making evaluative judgements on a program or policy. It provides a number of steps that can be followed to come up with a set of relevant criteria, and then to collect evidence to judge a program or policy based on evaluative reasoning. As someone who normally dislikes judging a program or policy, the VFI framework is helpful in giving a step-by-step guide of making judgements on a program. Additionally, the VFI framework stipulates the participatory nature of the evaluation to ensure that everyone’s views are considered. In an age where underrepresented voices need to be considered when planning and implementing programs, running an evaluation based on what people value the most is paramount to ensuring the evaluation is relevant and useful in improving the implementation and outcomes of programs and policies.
References
King, J., Crocket, A., & Field, A. (2023). Value for investment: Application and insights. Dovetail Consulting. https://www.julianking.co.nz/wp-content/uploads/2023/09/YPMHA-exemplar-report-230901-1.pdf
For more resources on the VFI framework, see the “Value for Investment Resources” from Julian King & Associates at https://www.julianking.co.nz/vfi/resources/ and Julian King’s blog on https://juliankingnz.substack.com.