I finally got my pre-approval and a place in the Victorian Homebuyer Fund last week! Pre-approval took a long time as I had to answer a lot of enquiries and document requests from the bank so that they could fully understand my financial situation. From sending the home loan application to getting pre-approval took around 5.5 weeks. I thought that it would take another 4-6 weeks for me to get a spot in the Victorian Homebuyer Fund. Fortunately, it only took a day between the bank forwarding my pre-approval to the Victorian Government and getting a spot in the Victorian Homebuyer Fund. I only had to respond to a brief phone call from the Victorian Government to confirm some details before I was given a place in the Fund. I guess most of the 4-6 week period the bank cited was them understanding my financial situation in a lot of detail so that they could proceed with the pre-approval process and get me a spot in the Victorian Homebuyer Fund. I have already received documents from the Victorian Government for me to read such as the conditions of being in the Fund and details on the equity scheme. The conditions looked simple enough to me. Nevertheless, I am seeking independent advice from a mortgage broker to make sure I have not missed anything.
While I was waiting to get pre-approval and a place in the Victorian Homebuyer Fund, I was eyeing a few properties that I wanted to live in. Unfortunately, those properties were sold before I received my pre-approval. I could have submitted an offer for the property I wanted. Unfortunately, not only would that invalidate my Victorian Homebuyer Fund application, but I would be in a mad rush to find a bank that would quickly grant unconditional approval for the property I wanted. Hence, I had to resist the temptation and wait for my pre-approval to pass before I started looking at property seriously.
I also want to make sure that I do the due diligence necessary for me to buy the property I want to live in with no regrets. That includes getting the okay from the bank that they will give me a home loan for the property, the contract and vendor statement coming out okay from the conveyancer and the strata records showing no significant issues. There was an apartment that would have been ideal for me to live in. Unfortunately, the auction for that property was this Saturday, so I only had a week to do all of my due diligence. That is not enough time for me to do the detailed due diligence necessary for me to make an informed decision. I don’t think I would have coped with the stress associated with trying to get everything done and prepare myself for an auction at short notice. Instead, I want to take the time to inspect the apartment, do my due diligence and make sure that I am happy to live in the apartment and to abide by the by-laws of the apartment complex.
This might take a lot more effort from my end now, but I know that it will be worth it. By doing the due diligence, I will be happy to buy a property that I can live in for a long time, where I can live harmoniously with the people who live in the same apartment complex as me. Plus I anticipate that a lot more apartments will open up soon that might be suitable to me. I just have to keep a close eye on the property market, select properties that I want to focus on and do the work necessary to uncover everything about the property to inform my decision to pursue the property seriously.
Coming back to the previous blog post where the bank wanted a university letter from me. I have been trying to hold back on providing the university letter to the bank until my full time job was confirmed for next year. In my emails with the bank; though, apparently they only wanted the university letter to show that I am working in the university full-time. They were not going to use the university letter against me to show that I did not disclose one of my fixed-term contracts ending soon. Instead, they asked me whether I thought my contract would not be extended into next year. I confirmed that my contract would likely be extended into next year as I am working on a project that will continue into next year. With the crisis averted for now, I sent the latest set of documents to the bank. Hopefully I pass the bank’s pre-approval process so that my home loan application can be passed onto the Victorian Government to assess my eligibility for the Victorian Homebuyer Fund.
Attending more inspections and observing auctions
While waiting to get into the Victorian Homebuyer Fund, I have continued to do some research on the suburbs and properties I want to live in. On most Saturdays, I have been attending inspections to see firsthand what the properties are like and auctions to see how they are run. On Saturday 19th October; though, I took a break from going to auctions and inspections as there were not many auctions going on. Instead, I spent the morning touring some suburbs in Melbourne’s south-east, taking notes on what the streets were like and where the shops were located.
On Saturday 26th October; though, I managed to watch an auction where the property was sold during the auction instead of being passed in. That property was unique in that, despite being near the train line, it is surrounded by green space and trees that shield the apartment from outside noise in the train line and main road. There was also plenty of outdoor space for one to walk around in and get some fresh air. Consequently, around seven bidders came to the auction. However, only two bidders made bids during the auction, initially raising the price at $10,000 increments before varying increments from $3,000 to $7,000. After one bidder pulled out, the other bidder won the property outright without it being passed in. That auction gave me a good taste of what it is like to bid in an auction. In particular, I see the balance between knowing in advance how much I would be prepared to buy the property and being tactical in bidding to ensure that I win the auction for the property.
My view on Melbourne’s activity centres
Something that has been on the news in Melbourne recently is the announcement of activity centres being planned across Melbourne. These activity centres are suburbs in Melbourne where more houses will be built to provide more people access to public transport and town centres. These are associated with high-rise buildings (up to 12 storeys) close to train stations and town centres and medium-rise buildings (up to 6 storeys) within walking distance of them. Ten suburbs were previously earmarked to become activity centres, but now 25 more suburbs will become activity centres, with more still to come. While some groups such as Property Council of Australia and YIMBY (Yes in my Backyard) Melbourne have welcomed the move as more homes would be built in areas where people want to live, others have lambasted the move, with local residents and the Victorian Opposition feeling concerned that high-rise buildings would change the dynamics of the suburbs.
The activity centres being proposed in Melbourne. Source
Personally, I am fearful of how the announcement of activity centres will affect my property search in the suburbs I want to live in. Plans have not yet been announced on the activity centres that have recently been announced, so I am not sure what plans will be in the suburbs I want to live in. I am avoiding high-rise apartments in my property search as they are not only uncomfortable for me to live in, they are also bad investments as the numerous similar apartments would cause prices to stagnate or fall. Instead, I would like to live in a low-rise apartment or unit with not too many people in the building, where I have plenty of indoor and outdoor space to walk around. If houses and apartment blocks have to be repossessed and demolished to pave the way for more high-rise apartments, it would rob me of my home that I plan to live in for over 10 years.
The announcement of activity centres will not deter me from searching for the ideal property that I can live in for 10 years or more. Hopefully plans for activity centres will be released before I start seriously looking for my first home. That way I can mark out areas where I do not want to live in, refining my property search to areas and properties that would not be affected too much by the activity centres.
In the last blog post, I briefly explained what I CAN Network’s peer mentoring programs were aiming to achieve and described how I have presented internal evaluation results in I CAN Network’s 2023 Social Impact Report. I represented changes in outcomes during the peer mentoring programs by comparing the distribution of responses before and after the program and using 100% stacked bar charts. From these results, we can calculate the difference in the proportion of positive responses before and after the program and test whether the result is significant and relevant. This is important for assessing whether the changes we are seeing in the peer mentoring programs are real or have arisen through chance.
In this blog post, I would like to provide some information on how to interpret the statistical results in the 2023 Social Impact Report so that you have a better understanding of how we have drawn the conclusions of the report.
Why we conduct statistical analyses
I CAN Network runs surveys and polls with Autistic young people (aged 5-20 years) before and after participating in the peer mentoring programs. In these surveys and polls, mentees rate how much they agree with statements relating to the program outcomes. We then compile these survey responses to look at how they are distributed before and after the program and calculate percentage changes in positive responses. We use these results to explain changes in outcomes among Autistic young people to different stakeholders.
A limitation of these surveys and polls is that they only sample a proportion of Autistic young people who have come to I CAN Network’s peer mentoring programs and were willing to provide feedback. It does not encompass all Autistic young people attending these peer mentoring programs, let alone across Australia. Statistics allows us to extend these survey results to describe what the general effect of the peer mentoring programs would be to all Autistic young people across Australia. It does this by testing how significant and relevant improvements in outcomes due to the peer mentoring programs are among Autistic young people.
The below figure summarises the purpose of statistics.
How statistics can be used to generalise survey responses from the sample (Autistic young people surveyed) to the population (Autistic young people across Australia)
There are a range of statistical techniques that can be used to test the significance and relevance of the findings. The 2023 Social Impact Report uses three statistical techniques to assess the significance and relevance of the findings:
Hypothesis testing: Hypothesis testing allows us to see whether we should accept a statement about a population based on whether the result would have arisen through chance. This is represented by the p-value. If the difference did not arise through chance, then we say that the peer mentoring programs have a statistically significant effect on the outcome.
95% confidence intervals: Hypothesis testing does not give us the range of possible values that could be experienced by the general population. 95% confidence intervals not only give us that range, but also describe how confident we can be of the result.
Effect size: The effect size gives us a measure of how relevant our results are. This is done by standardising the changes in outcomes.
These statistical techniques produce numerical outputs that accompany the percentage increases. As an example, in the I CAN Imagination Club® mentoring program (the peer mentoring program for primary schools), we reported an 11% increase in positive responses towards the self-confidence statement. These are accompanied by a p-value of 0.006, a 95% confidence interval of [3%, 18%] and an effect size of 0.22.
What do these numbers mean, and how are they calculated? The next few sections will explain how these statistical outputs are calculated and interpreted.
Hypothesis testing
Distributing surveys to Autistic young people and analysing their responses allows us to calculate percentage increases in positive responses that indicate how much they have changed during the program. However, we are unsure of whether this result would have arisen through chance (i.e., we get a different result if we run the survey again), or if it is an effect that can be generalised to other Autistic young people across Australia. Hypothesis testing allows us to answer this question.
In hypothesis testing, we want to see whether there is a higher proportion of positive responses after the program compared to before the program. In other words, does pafter > pbefore? To see whether that is the case, I used a two-proportion z-test. This statistical technique allows us to convert the difference in proportions into a standard z-value that can be used to determine the p-value. The p-value can be used to see whether the difference in proportions would have arisen through chance or not.
To calculate the z-value, we need two things: the percentage change in positive responses and the standard error (SE). The percentage change in positive responses is derived by calculating the proportion of positive responses before and after the program and taking the difference between the two. We can take the difference between the two proportions as we can rearrange the inequality from pafter > pbefore to pafter – pbefore > 0. If the difference between the two proportions is positive, it means there is a higher proportion of positive responses after the program compared to before.
At the same time, we calculate the standard error of the difference between two proportions. The standard error describes the spread of calculated differences from running multiple hypothetical tests to see whether we would get similar results. From there, we divide the difference between two proportions by its standard error to get the z-value.
We match the z-value to the z-distribution curve to get the area underneath the tail on one side of the curve. The area underneath one side of the curve is our one-tailed p-value, the probability that we would produce a difference between two proportions that is just as big, if not bigger, than what we would get by chance. If we multiply the p-value by 2, we get a two-tailed p-value which describes the probability that we would get a difference (positive or negative) that is just as big, if not bigger, than what we would get by chance. We use the two-tailed p-value in our hypothesis testing to account for both increases and decreases in the proportion of positive responses after the program.
One-tailed and two-tailed p-values in a standard normal distribution
If the two-tailed p-value is 0.05 or below, the result is considered to be statistically significant. In other words, it is unlikely that the increased proportion of positive responses arose randomly. This result indicates that the peer mentoring programs may have contributed to positive outcomes among Autistic young people, something that is evident in our analyses of comments and mentee creations for the 2023 Social Impact Report.
Going back to our sample self-confidence result, we reported a p-value of 0.006. This means we have a 0.6% chance that we would get an increase in self-confidence that is 11% or more randomly. Given the small chance of it happening, we would likely conclude that the I CAN Imagination Club® mentoring program had an effect on students’ self-confidence.
Confidence intervals
A hypothesis test might indicate that a specific result is statistically significant as indicated by a low p-value. However, it does not give us the range of possibilities that could be experienced by the general population as a result of the program. This is where the 95% confidence interval comes in. The 95% confidence interval indicates a range of possible values where we are 95% sure the true percentage increase lies. The size of the confidence interval can tell us how confident we can be of the result:
A small 95% confidence interval indicates a small margin of error, indicating a high level of confidence in the result.
In contrast, a large 95% confidence interval makes us less certain of the result as the true percentage increase could take a wide range of values.
To calculate the lower and upper bounds of the 95% confidence interval, we multiply the standard error by 1.96 and subtract or add it to the observed percentage increase respectively. The lower and upper bounds of the 95% confidence interval are encased in square brackets to indicate the range of possible values of the true percentage increase.
A visual of the 95% confidence interval for increased self-confidence
In our sample self-confidence results, we had a 95% confidence interval of [3%, 18%]. This means we are 95% confident that the true increase in self-confidence among students attending the I CAN Imagination Club® mentoring program could be as low as 3%, or as high as 18%. This confidence interval gives us a range of possibilities that the I CAN Imagination Club® mentoring program could take to boost self-confidence among Autistic young people in Australia.
Effect size
A result could be statistically significant as indicated by a low p-value. However, this result might not be relevant in real life as the change is too small to have an impact on the population of interest. This is particularly true when we survey hundreds or even thousands of people.
This is where effect size comes in. The effect size describes whether the increase in the proportion of positive responses has an impact on real life. To do this, we use Cohen’s h which describes how far off the proportion of positive responses after the program is from before the program. A larger h value indicates bigger changes in outcomes. We use cut-offs of h = 0.2, h = 0.5 and h = 0.8 to indicate small, medium and large effect sizes respectively.
In the 2023 Social Impact report, Cohen’s h in most outcome statements range from 0.15 to 0.35, representing insignificant to small effect sizes. Given that we survey around 300 to 500 students per time point; though, getting a small effect size is still good in a policy setting as it describes a sizeable change that is happening in a large population.
The effect sizes, along with thresholds of Cohen’s h
Going back to our sample self-confidence result one last time, the effect size (Cohen’s h) is 0.22. As it falls between the Cohen’s h thresholds of 0.2 and 0.5, the effect size is considered to be small. Given our 11% increase in self-confidence over 300 students; though, we still have a notable improvement in self-confidence that would be relevant in the real world.
Conclusion
This blog post provides an overview of the statistical techniques that were used in the 2023 Social Impact Report. The result is a series of numbers that accompany the percentages changes to underline the significance and relevance of the findings. These analyses are important in providing an evidence base behind the effectiveness of I CAN Network’s peer mentoring programs to Autistic young people across Australia.
Personal disclaimer
This blog post was written by James Ong in his personal capacity. The content, views and opinions represented in this blog post are solely my own and do not reflect those of I CAN Network Ltd.
Since graduating from a PhD in immunology in 2019, I have been moving away from a career in science and heading more towards program evaluation. I am motivated to do this by the desire to see a more immediate impact from my work. Program evaluation, the art of assessing whether a program works or not, fulfils that desire. This career change allows me to use my existing science and research skills to provide an independent voice to organisations who pursue lofty goals to improve society. With my advice and evaluation results, organisations can make changes to their programs to enhance their impact and promote their programs to other people. At the moment, I am finishing my Masters of Public Health degree where I have been training in program evaluation while working two jobs on program evaluation. Soon after I graduate from my Masters, I will start my career in program evaluation full-time from 2024.
This career change to program evaluation necessitates a small change to the name of my blog. From now on, my blog and my personal brand will be called The Active Evaluator. My blog will be focused on providing information about evaluation and policy frameworks that can be applied to real-life projects. I will also be building a portfolio of work in program evaluation and program and policy analysis to show you what I can do.
At the same time, I am keeping the ‘Active’ part of my brand. I will still be keeping old blog posts that I have written as The Active Scientist, sorted under different categories (you’ll see them when you hover above the ‘side projects’ tab). Additionally, there are a lot of interesting topics that I like to explore in my blog. I feel that I have some advice, information and personal experiences that would help people. To that effect, I will be doing some side-projects on areas that interest me and that will be published on The Active Evaluator. Note that I will still be keeping the logo and colours that I have been using as The Active Scientist.
I am in the process of getting everything ready for the change in the website name and brand. I will make an announcement when the transition to my new brand is complete.
With that, I would like to thank you for your support over the past few years while I sorted out what I want to do in my life. I hope that you will keep supporting me as I embark on a career in program evaluation and to write blog posts as The Active Evaluator. If you have not subscribed yet, consider putting your name on the mailing list. You will be notified when I post something new on my blog so that you can read what I am up to.
![Showing the 95% confidence interval of [3%, 18%], with end-points shaded, on a number line.](https://i0.wp.com/activeevaluator.com/wp-content/uploads/2024/07/95-ci.jpg?resize=1024%2C206&ssl=1)
