From the Australian Bureau of Statistics:
On 17 July 2025 the Australian Bureau of Statistics (ABS) announced that it will not release statistics from the 2023-24 Survey of Income and Housing (SIH), noting that the data did not meet the ABS’ high standards for official statistics.
Wow!
Here’s the background:
The ABS has conducted surveys of household income in various forms since 1974, and the SIH has been conducted in its current form since 2003-04. It gathers data on household income sources and amounts, net worth, housing situations, and both household and personal characteristics.
While SIH has been conducted on a 2 yearly basis since 2003-04, the last annual results were released from the 2019-20 cycle. The scheduled 2021-22 cycle was cancelled in December 2021 due to disruption and statistical impacts associated with the COVID-19 pandemic. . . .
Across its survey program the ABS evolves its collection approach, wherever possible, to improve efficiency and reduce cost, reduce demands on the Australian public to provide their data, improve respondent experiences and take advantage of methodological and technological innovations. While these innovations are not always visible to data users, they are critical to maintaining the quality of official statistics and have been a feature of the SIH program since its inception.
They continue:
Given an increasing public preference for digital engagement, ABS made changes to the SIH 2023-24 sample design with the intention of delivering high quality data while making it easier for households to complete the survey. A new approach was implemented whereby the sample was divided into two groups to reduce any potential statistical impact arising from households not responding to a digital survey.
One group was randomly selected for ‘self-enumeration only’ using a digital (web or phone) approach, with no interview follow-up of non-responding households. In the second group, those households that did not complete a web or phone interview were assigned for field follow-up by an interviewer. This sample design allows the ABS to apply statistical adjustments to address potential under-representation of groups that were less likely to provide a digital survey response.
Unfortunately, when implemented, system and business process limitations resulted in some households in the second group not receiving the field follow-up required to support this design. This left the survey results open to bias arising from systematic differences between those who responded to the survey and those who did not. . . .
The ABS had planned to process SIH 2023-24 data progressively from the commencement of data collection, allowing an early assessment of data quality. Unfortunately, initial data extraction was significantly delayed by system problems, which in turn delayed the commencement of data processing. SIH 2023-24 data collection ceased on 30 June 2024. The inability to transfer the collected data from the data collection system to the statistical compilation system was not resolved until December 2024. . . . Detailed, aggregated data comparisons were made against previous iterations of the SIH, the Australian System of National Accounts (ASNA), the Person Level Integrated Data Asset (PLIDA), and non-ABS sources such as the Household, Income and Labour Dynamics (HILDA) survey.
These assessments identified inconsistencies, including:
Estimates of personal and household income were much lower than expected when compared to previous iterations of SIH or information in the National Accounts and other sources.
Aggregate liabilities and loan values were lower than expected, and exhibited lower coverage compared to the National Accounts than previous SIH cycles.
An observed decrease in superannuation income since SIH 2019-20 was not consistent with other sources and population trends.
Falls in outstanding mortgage values since SIH 2019-20 were also inconsistent with trends observed in other sources.
So:
Taken together, these results suggested the SIH 2023-24 results were not fit-for-purpose and led ABS to initiate an internal Quality Incident Response Plan (QIRP) process. . . .
Following review, the QIRP process confirmed that the observed data quality issues stemmed from two key statistical limitations discussed above, namely:
• A pattern of survey non-response that reduced the representativeness of the results.
• Pervasive question level non-response and response error, largely caused by ‘skippable’ questions in the survey form.
The QIRP panel found that the combined effect of these errors rendered the data unsuitable for its intended purpose.
That’s a big step. The report discusses the plans of the Australian Bureau of Statistics for this survey going forward.
Agreed that they are doing the responsible thing. But I still wonder – given the graph and table – whether they could do a bit more analysis regarding whether the change in trends (e.g. in home ownership, income, or housing finances) is likely due to the problems in survey design and response. The issues they identify suggest that the latest data is not reliable. That doesn’t necessarily mean that the direction of the changes is also unreliable. For instance, it is possible that weekly real income has declined and this would be important to know. Is the noted decline likely the result of the problems they have identified? It could well be, but it is also possible that some of the changes aren’t likely to result from these problems. I think that is a somewhat different question than the general question of whether the survey meets the desired quality standards. They do note that they compared these results with some other data sources (for example, income data from national accounts), which is good. But given the numerous discussions on this blog about issues associated with real income measurement, inconsistent estimates from two data sources may not be as damning as it seems.
I’m curious about the system and business process limitations that thwarted the survey.
The Australian Federal Government routinely apply an ‘efficiency dividend’ to it’s agencies. This means that an agency doing the same amount of work will have its budget cut every year, with the presumption that they will find better and cheaper ways to do the same work. My understanding (from industry gossip) is that they made experienced field managers redundant, and hired contractors to do the same work for cheaper. Clearly, they got what they paid for.