On-Track Coaches (OTCs) are youth mentors in RISE’s Core Network schools. They work one-on-one with a caseload of up to 60 students, guiding them in their first year of high school and helping them stay on-track to promote to the tenth grade on time. We know, from years of anecdotes and great feedback, that On-Track Coaches make a difference in the lives of their students, but we wanted to know exactly how this happens and to measure that difference.
To help answer this question, RISE just wrapped up a year-long evaluation of the OTC strategy, looking at the different ways OTCs support students and measuring their impact in terms of on-track rates, GPA, and attendance. The evaluation also explored how OTCs work with teachers and families to improve communication and advocate on behalf of their students when needed. Lastly, the report shares some practical steps that schools can take to better support young people as they enter the ninth grade and work towards success.
View the executive summary or full report.
I spoke with the report’s author, Melanie Gonzalez, Senior Evaluation and Data Analytics Specialist on RISE’s Monitoring, Evaluation, and Learning team, to learn more about the evaluation process and results from her perspective.
Q: What feedback from students on an OTC caseload interested or surprised you?
A: This was the first time that we conducted formal focus groups with young people who attend our partner schools, which was a really exciting experience for us. I was able to interview eight students on an OTC caseload from three of our schools, and what stood out to me the most was the fact that OTCs were really supportive adults for their students. OTCs helped them get their priorities in order and kept them focused on the bigger picture of why their high school career was so important for their long-term goals. In talking to the students, their OTC was that person who was always there for them and who was always cheering them on.
It’s that strong relationship-building and that consistent show of support that’s so important. Students truly feel that their OTCs care, that they’re there for them, cheering them on, and that they’re there to help them accomplish their goals.
I know when I was a young person, some challenges seemed insurmountable, and I sometimes felt helpless trying to navigate them alone. It’s wonderful that there are young people who have OTCs there to coach them through those difficult challenges and help them to brainstorm solutions, weigh their options, and move forward, not only academically, but also personally.
Q: What data points from the report were the most exciting to see?
A: The biggest thing, which is really what we’re trying to achieve, is the fact that students on a caseload ended the year just as on-track as students not on a caseload. The numbers were equivalent at 85 percent for both groups. I remember when I got that finding, I double-checked my analysis to confirm that it was correct! This statistic speaks to the power of the program in helping young people who come into ninth grade with attendance and academic challenges from eighth grade to get on-track and stay there. This helps lay a strong foundation for them to succeed in high school because they are working with their OTC to build the habits that they need to be successful, which can help them succeed not only in ninth grade, but throughout their academic career.
Q: Do you remember hearing any particularly impactful student stories?
A: One of the students talked about how their OTC gave them a rubber ducky along with a motivational speech. The student talked about how she still has that rubber ducky and is reminded of her OTC’s words when she sees it. It’s those little things that the OTCs do to show that they care that go a long way for the young people whom they work with.
Generally, I remember some of the students saying that their OTC was the person they could go to when they were having a bad day. Some of the students talked to me about how their OTC was able to help them calm down after a heated experience with a teacher or a friend. I think that the OTCs being there to help them through an emotional experience was really significant.
Especially since, in a lot of our schools, the caseload-to-school counselor ratio is so high (332:1 in Connecticut during the 2022-23 academic year), it’s great that the students have someone whom they can go to when they’re in a crisis. The OTCs will give them that emotional support, so they can go about their day.
Q: When you spoke with Grade 9 administrators, what feedback stood out to you?
A: When we spoke with administrators, their message was that they love their OTCs and never want them to leave! Some were administrators before the OTCs came on board, and they talked about how critical and unique the OTC role is.
OTCs are so data-driven in their work, and they can often be the person who is leading some of the efforts of the improvement teams, because they typically have the deepest awareness of and insights as to why students on their caseload are not on-track. In their role, OTCs are regularly engaging with the RISE Data Hub and other data collection tools, because it’s important for them to know if their students are on-track and provide them with timely support.
There’s the data monitoring component, of course, but there is also the personal component. OTCs typically have strong relationships with the young people they work with and therefore have insight into what they are dealing with in their personal lives that may be impacting their academic performance. Because they have this insight, OTCs can more effectively speak on behalf of their students at these team meetings, to create a plan that’s going to be the most effective.
Q: Where did the idea for this report come from? Why did you and your team decide to focus on this topic?
A: On-track coaching has been one of our primary RISE strategies since the 2016-17 academic year. We invest a lot of time, energy, and resources into this strategy, and it’s also a strategy through which we are directly interacting with young people. We wanted to better understand the program, especially since it is implemented in various ways throughout RISE schools. We wanted to understand what the core components are that matter most and are most impactful.
This evaluation is actually part two of a two-year evaluation that we engaged in. The first part was an internal process evaluation where we worked with the OTCs to understand how they supported their students. Through that evaluation, we were able to outline the key components of the strategy and fine-tune some of the models that we use to communicate with stakeholders about the OTC strategy.
In this year’s evaluation, we wanted to look at the outcomes of the program, specifically looking at on-track rates of the students who were on a caseload. We also wanted to understand, both from the student and the administrator perspective, what they found to be most beneficial about the program, to inform and support our program development efforts.
Q: How does this report fit into the bigger picture of continuous improvement at RISE?
A: When we look at the continuous improvement process, one of the critical pieces is collecting and reviewing data to make decisions about the future of a strategy or change idea. You ask yourself, “Is it working? Does it need to change? Or should we abandon it?” This evaluation supports that work. It helped us to test whether we are actually supporting young people on an OTC caseload in ending the year on-track at high rates. The fact that they are ending the year just as on-track as their non-caseload peers is evidence that we are doing what we intend to do.
We also uncovered opportunities for improvement. There was an additional analysis included in the report where we looked at a subset of students on a caseload and off a caseload to compare their GPA and average daily attendance (ADA) in eighth and ninth grade. Those results showed that, while the gap was narrowed between students on and off a caseload in the ninth grade, it was not closed, which is what we’re striving for and continuing to work on. We want students on a caseload to have the GPAs and the ADAs that are going to set them up for success.
Q: What were some lessons learned throughout the process?
A: This was a very lengthy process, over the space of two years, and it took a long time to get this data out to our network partners. What we want to do moving forward is deliver more “bite-sized” pieces that showcase the impact of multiple strategies and activities at RISE. So now we’re pivoting towards smaller-scale learning and evaluation projects, which would allow us to share learnings from exploratory analyses in our organization and our network of networks more quickly and in a shorter format.
Q: What’s coming up next for you and the Monitoring, Evaluation, and Learning team?
A: We are working on learning briefs for two strategies that we implement in all or most of our Core Network schools: Summer Bridge, which supports the transition from eighth to ninth grade, and Summer Academy, which supports junior students in getting a head start in completing application milestones. We will be releasing those before the end of the school year.
We are also doing issue briefs where we talk about some of the pressing issues in the education sector. This past year, a lot of our schools have been working on developing change ideas to address the chronic absenteeism rates of their students. We are working on an issue brief to talk about the state of chronic absenteeism across our network of schools and highlight research around the origins of chronic absenteeism and some of the standards the State of Connecticut has in place, all to ground our network in a shared understanding of what chronic absenteeism is. We have already been learning a lot through the initial research we have conducted and are looking forward to releasing the brief.
Visit our blog and follow us on LinkedIn and Facebook to stay up-to-date with RISE publications, news, and events.