I CAN Network is an Autistic-led social enterprise that aims “to prove what Autistics CAN do”. It is working towards this aim by running a variety of programs that engage Autistic people and their supporters (for example, parents, carers and teachers). In particular, I CAN Network runs peer mentoring programs in school and online environments for Autistic young people aged 5-20 years to come together and build their social connections, improve their self-esteem and develop life skills. It also runs professional development programs for schools to create safe environments for Autistic young people to be themselves and engage in school.

Over the past 10 years, I have been working on internal evaluations for I CAN Network. I planned and conducted evaluations measuring outcomes of I CAN Network’s programs and wrote numerous reports describing the results of these evaluations. These reports describe the outcomes I CAN Network has achieved in its programs, particularly peer mentoring programs for Autistic young people. While most reports are delivered to state governments and funders, I also wrote Social Impact Reports that are publicly available on I CAN Network’s website, with the 2023 edition being recently released.

With the 2023 Social Impact Report being the last evaluation report I will write for I CAN Network, in this blog post I will describe why I have designed the graphs in a particular way to make them easy for the reader to interpret and understand. 

What are the aims of I CAN Network’s programs?

I CAN Network runs school and online peer mentoring programs for Autistic young people aged 5-20 years. At school, I CAN Network runs I CAN Imagination Club^® and I CAN School^® mentoring programs for Autistic young people in primary and secondary schools respectively. These programs are run during school hours. In comparison, I CAN Network runs I CAN Online, delivering online mentoring programs after school for Autistic young people across Australia.

All peer mentoring programs aim to achieve outcomes in the following three areas:

• Self-esteem: The peer mentoring programs aim to improve Autistic young people’s views of themselves and their Autism.
• Social connection: Autistic young people in the peer mentoring programs interact with each other to establish social connections and friendships, as well as their parents, teachers and mentors to develop support networks.
• Skill development: These relate to Autistic young people acquiring and improving life skills such as communication and stress management so that they are more likely to advocate for themselves and manage their own lives.

These outcome areas are supported by a positive program environment, where Autistic young people feel safe to be themselves without feeling judged. This allows them to open up and interact with other people.

The below figure summarises the outcomes that I CAN Network’s peer mentoring programs aim to achieve.

These aims provide a structure in which the peer mentoring programs can be assessed. One way this can be done is by running internal evaluations within the organisation.

Why I have not used mean scores in the report

I CAN Network runs internal evaluations by distributing surveys and polls to Autistic young people before and after the program. Each survey or poll contains statements relating to the outcomes of the program, where mentees rate on a 3- or 5-point Likert scale how much they agree with the statements. We then compare responses before and after the program to see whether mentees have changed during the program. These can be shown both numerically and graphically.

One way to compare responses is to convert them into a score, calculate the mean score before and after the program and compare them to see whether that has changed. For I CAN Imagination Club^® mentoring programs, I can convert the yes, maybe and no responses into scores of 1, 0.5 and 0 respectively and calculate a mean score out of 1. Similarly, for I CAN School^® mentoring programs, I can convert the responses into scores ranging from 1 (for strongly disagree) to 5 (for strongly agree) and calculate a mean score out of 5.

We can compare the mean scores before and after the program in a table along with the standard deviation (SD which measures the spread of scores) and the total number of responses received.

Statement	Before			After
	Mean score	SD	Total # responses	Mean score	SD	Total # responses
I can try new things	0.74	0.29	331	0.80	0.27	303
I know what makes me special	0.70	0.37	323	0.79	0.34	302
I think my brain is awesome	0.69	0.36	327	0.76	0.34	298

We can also visualise the scores in a bar chart, placing the mean scores before and after the program alongside each other. We can rule black solid lines to separate the statements, making it easier to compare mean scores before and after the program within a statement.

There are two main problems with using mean scores to compare outcomes. Numerically, calculating mean scores simplifies the responses too much. This makes it difficult to explain a specific mean score (“What does a mean score of 0.74 mean?”) or changes in a mean score (“What does a mean increase of X units mean?”) and relate them to the outcome. Furthermore, because we design our own statements instead of using a standardised tool, it is not possible to compare the changes to a specific standard to explain how relevant they are. The only way to explain the changes is to use statistics to assess whether the changes are statistically significant and relevant. This makes it difficult to explain to a lay audience how large outcomes have changed.

Graphically, it is hard to accurately visualise changes in mean scores. Scaling the scores from 0 makes it very hard to differentiate mean scores before and after the program. In contrast, scaling the scores from a higher base value distorts the differences before and after the program, deceiving the reader. For example, in the ‘I can try new things’ statement, the mean score only increased by 0.06 after the program. That difference is quite small when scaling the scores from 0, but it is magnified when scaling the scores from a higher base value (specifically 0.62), distorting the difference. 

For the above reasons, I decided not to describe changes in outcomes using mean scores. Instead, I represented the changes in a different way. 

Why I have used 100% stacked bar charts in the report

To represent changes in outcomes among mentees attending the peer mentoring programs, I decided to compare the distribution of responses before and after the program. For each timepoint, I calculate the proportion of responses belonging to a particular category out of the total number of responses received (excluding not sure and missing responses). I then bring the categories together to form 100% stacked bar charts. I use black solid lines to separate the statements and place the distribution of responses before and after the program next to each other to make it easier to compare them. 

Using the distribution of responses is a better way to represent changes in outcomes compared to using mean scores. Numerically, it is easier to explain changes in outcomes by saying how much responses in a particular category have increased or decreased after the program. For instance, in the I CAN Imagination Club^® graph above, we can compare the proportion of ‘yes’ responses before and after the program and calculate how much it has increased. We can see that compared to before the program, mentees coming into the program show an:

1. 11% increase in trying new things (‘self-confidence’); 
2. 14% increase in knowing what makes them feel special (‘self-acceptance’); and
3. 9% increase in thinking their brain is awesome (‘neurodiversity acceptance’).

Representing outcomes as percentage changes makes it easy to explain to the reader how Autistic young people have changed in the peer mentoring program.

Graphically, showing the distribution of responses makes it easier to visualise how different responses shift over time. For instance, in the I CAN School^® graph above, we can see that the proportions of ‘strongly disagree’ and ‘disagree’ responses have fallen, while the proportions of ‘agree’ and ‘strongly agree’ responses have risen. Visualising the distribution of responses in this way makes it easy for the reader to interpret the result to see whether Autistic young people are improving as a result of the program. It also does not deceive the reader as the graph is scaled from 0% to 100%, ensuring that the changes are properly represented.

Hence, using 100% stacked bar charts to show the distribution of responses allows us to retain all the information we have collected from our surveys and makes it easier to explain changes in outcomes to a lay audience. 

Visualising mentee demographics using different graphs

In the 2023 Social Impact Report, I have also visualised some demographic data to describe who participates in I CAN Network’s peer mentoring programs. I used two different graphs to visualise demographic data: pie charts and bar charts.

There has been some talk as to why pie charts are bad visualisation tools and that bar charts should always be used when looking at proportions. In my opinion, pie charts are still useful for visualising proportions. However, there are some conditions that need to be met:

1. It should show as few categories as possible (five at most).
2. It should be as visually simple as possible, with the parts clearly signposted.
3. The pie chart should be in 2-D instead of 3-D to prevent distortion of the data.

For gender, I used a pie chart as it only has three categories (male, female and trans and gender diverse), making it easy to show the sizes of each gender as a whole. I have also kept the pie chart as visually simple as possible. 

1. I have earmarked each section with a solid black outline and different colours representing each gender.
2. I have also included some data labels indicating what each section represents, along with the number of mentees and the proportion out of the total number of mentees.

Using a pie chart provides a clear picture of how mentees are split up over the three gender categories.

In contrast, I used a bar chart instead of a pie chart to represent Australian states and territories where I CAN Online mentees live. A pie chart is not appropriate as there are eight states and territories in Australia, resulting in numerous sections being crammed in one space. Using different colours to represent each section and data labels to identify different sections further increase the visual complexity of the graph. Hence, it is hard for the reader to comprehend what proportion of mentees live in a specific state or territory.

In comparison, using a bar chart provides more room to show different categories. It is also easier to look at the length of each bar and align it to the y-axis to see what proportion of mentees live in a specific state or territory. The limitation of a bar chart is that it is hard to see how different states and territories can be pieced together to form a whole like what can be done in a pie chart.

Hence, pie charts have their uses, but their benefits and drawbacks compared to bar charts need to be considered when deciding how to visualise the results.

Conclusion

In this blog post, I explained how I have designed the graphs that appear in the 2023 Social Impact Report. People may have different opinions of how data should be analysed and visualised. From my perspective, I place great importance on presenting the results as simply as possible so that everyone can easily grasp what they are seeing. This facilitates their understanding of how I CAN Network’s programs are making an impact to Autistic young people attending them. 

Personal disclaimer

This blog post was written by James Ong in his personal capacity. The content, views and opinions represented in this blog post are solely my own and do not reflect those of I CAN Network Ltd.