Self-monitoring and data capturing systems support accountability, openness, and transparency. A high level of accountability is especially important for agencies that hold responsibility to care for our and . The require agencies to monitor their compliance with the regulations.
Through its self-monitoring, Oranga Tamariki has assessed itself as ‘partially compliant’ across the full suite of NCS Regulations. Oranga Tamariki has also stated that it has improved the extent to which it is compliant across a number of areas of practice, although work remains to ensure all tamariki and rangatahi in care are achieving a consistent level of support that meets the full range of expected standards.
For 2022/2023, Oranga Tamariki found that, out of the six core lead indicators that apply to almost all tamariki and rangatahi, all six were met for only 45 percent of tamariki and rangatahi. This is an improvement on 31 percent for 2021/2022 but shows that Oranga Tamariki still has a long way to go to ensure all tamariki and rangatahi receive an appropriate standard of care.
Despite the progress that Oranga Tamariki has made in maturing its self-monitoring framework, there has been little change in the coverage of the data provided in response to our data request, and gaps remain.
For the past three years, Oranga Tamariki has been unable to provide data to measure compliance in key areas of the NCS Regulations. For example, support provided to meet assessed needs of tamariki and rangatahi, monitoring of education progress and attendance, and the provision of a record of tamariki life events.
This year, Oranga Tamariki also did not provide information that had previously been provided about supports and assessments for tamariki and rangatahi at risk of self-harm or substance abuse.
We had anticipated that Oranga Tamariki would have been able to provide greater structured data about the experience of caregivers owing to the implementation of the Oranga Tamariki Caregiver Information System (CGIS) case management system. This went live in July 2022. CGIS is used to administer recruitment, approval, review and support processes for caregivers.
However, in its response to our data request for the 2022/2023 reporting period, Oranga Tamariki advised us that CGIS would take “two full reporting periods to allow the information captured and reported to be reflective of not only practice but how the system is used and what reporting will tell us about our performance”.
In 2023, Oranga Tamariki commissioned an independent review of its case file analysis sampling approach. The review found that case file analysis is an essential component of self-monitoring and a necessary part of any continuous-improvement programme. The review was generally positive about the stratified random sample design, the questionnaire and the ability of Oranga Tamariki to understand and control inter-rater variability.
The review recommended that Oranga Tamariki consider producing confidence intervals for estimates in reports, which it has now done. The review also recommended that Oranga Tamariki consider a further stratification by length of stay in care and explore the possibility of an adaptive sampling scheme to provide further insights into quality improvement.
It is reassuring that the independent review shows that the sampling design used in Oranga Tamariki case file analysis is robust, and that Oranga Tamariki has already provided more detailed estimates of the accuracy of its measures.
Case file analysis is recognised as an important component of self-monitoring, however there are some areas where systematic (structured) and timely data collection is more important. These include areas of experience that are infrequent and therefore have low occurrence in the sample (such as care transitions, transition to adulthood, and caregiver approvals); and areas where it’s important that operational data is complete, accurate and up- to-date. An example of this is data on registrations with primary health organisations, an area of ongoing weakness in record keeping highlighted in our recent review into access to primary health services and dental care.
Structured data is essential for self-monitoring and assurance purposes as well as informing immediate operational responses to the needs of tamariki and rangatahi in care.
Throughout this report, we refer to an Oranga Tamariki survey of 10 – 17-year-olds in care, Te Tohu o te Ora. In 2023, Oranga Tamariki released Te Mātātaki 2023, the second report based on the Te Tohu o te Ora survey.
Results from Te Tohu o te Ora show how 10 – 17-year-olds in care feel about aspects of their lives and the services and supports they receive from Oranga Tamariki. In this report, we have used results from Te Tohu o te Ora to complement or support themes we have heard from our engagements with tamariki and rangatahi in care.
In 2021/2022, the Te Tohu o te Ora survey moved from a paper-based questionnaire to an online questionnaire given to tamariki and rangatahi on Oranga Tamariki devices by their social workers. In its methodology report, Oranga Tamariki has assessed the potential of social desirability bias, the likelihood that participating tamariki and rangatahi would change their answers to match what they thought would please the social workers who were offering them the survey. Oranga Tamariki concluded that the move to a digital platform may have reduced the risk of social desirability bias. It also found that the questions where social desirability bias could be most expected (such as relationships with social workers) were answered less positively than other questions in the survey.
Te Tohu o te Ora uses a census approach. It is intended to be offered to all 10 – 17-year-olds in care (under a Care and Protection order) for at least one month at the time of the survey. However, there was a decline in the proportion of eligible tamariki and rangatahi who were offered the opportunity to complete the survey (down from 79 percent in year one, to 43 percent in year two). Oranga Tamariki identified the Omicron outbreak of the COVID-19 pandemic and significant organisational change as contributing factors to the decrease in offer rate.
For Te Tohu o te Ora to continue to play its role, it is important that Oranga Tamariki focuses on offering the survey to as many 10 – 17-year-olds in care as possible during future rounds. It is also important for Oranga Tamariki to continue to assess whether the mode of delivery of the survey is a good fit for its participants.
Throughout this report, we also refer to an Oranga Tamariki survey of its caregivers. This survey was conducted quarterly and is now offered once per year to all Oranga Tamariki caregivers with a child in their care (or who have had a child in their care within the past two years). Unlike Te Tohu o te Ora, the caregiver survey is offered directly to participants and can be completed online, by phone or on paper questionnaire. The overall response rate for the caregiver survey in 2022 was 23 percent.This response rate is moderate compared to the benchmark of 30 percent expected for a voluntary online survey, but still high enough to ensure representativeness. Like results from Te Tohu o te Ora, in this report we have used results from recent Oranga Tamariki caregiver surveys to complement or support themes we have heard from our engagements with caregivers.
Over the course of 2022/2023, there were 75 tamariki and rangatahi in Open Home Foundation care and two in Barnardos care.1 We acknowledge that Open Home Foundation has worked hard to develop and implement its monitoring and reporting framework. Both organisations can provide data to demonstrate the level of their compliance with the NCS Regulations.
For Open Home Foundation compliance is defined as “the extent to which we are meeting the requirements of Parts 1 – 6 of the NCS Regulations; we have a set of policies, processes, and systems in place to guide and evidence compliance and, where improvement is needed, clear time-framed plans are in place to achieve it.”
Open Home Foundation has improved its compliance in most areas, however, is not fully compliant with all of the NCS Regulations. This year, it focused on continuous improvements, and where compliance is not meeting policy requirements, this is referred to the appropriate team for resolution. When teams have met the standards, an Appreciative Inquiry process is undertaken to explore how they managed to do this, what has helped them, and whether there are learnings that can assist them to repeat their success with other teams.
As we saw last year, improvements in the Open Home Foundation client management system mean that, once again, it has answered all of the applicable measures for all tamariki and rangatahi in its care. This has required effort and expense on the part of Open Home Foundation.
The Chief Executive also acknowledge that it requires ongoing investment of time to capture and organise the data, however, they are seeing the benefits for their practice and organisational knowledge and understanding.
Open Home Foundation has continued to use the graduated measures it adopted last year so we are now able to measure progress against 2021/2022. This year it has also defined the point at which each graduated scale equates to compliance. This means it can see for how many tamariki and rangatahi each compliance measure has been met, where it hasn’t been met, and whether progress has been made towards compliance. We have included its compliance tables in the appendices of this report. Open Home Foundation data for 2020/2021 has not been included in this report because improvements made to its approach to self-monitoring for 2021/2022 mean that data previously provided is not comparable.
As Barnardos has a small number of rangatahi in its custody, it can rely on peer review, supervision and case audits for self-monitoring using its self-audit tool. We have also met with Barnardos to discuss its information. It is compliant with the NCS Regulations for rangatahi in its custody.
1 Barnardos and Open Home Foundation are organisations approved “as a child and family support service” for the purposes of the Oranga Tamariki Act (Section 396).