Alexandria Brasili, Sue Allen, Maggie Foster
August 2017
For more information:
Email research@mmsa.org
Or visit www.mmsa.org
Background
Beginning in 2014, the Maine Mathematics has received a series of grants from the Noyce Foundation, STEMNext, the Davis Family Foundation, and the NSF AISL program, to create a coaching system for supporting out-of-school (OST) educators such as afterschool providers who facilitate STEM activities in their programs. Implementation partners for the work include the University of Maine, Office of Extension (4-H), the National Afterschool Association, and Vermont Afterschool. While this work is ongoing (www.mmsa.org/acres), we present here an evaluation report for the first phase.
Introduction
The Afterschool Alliance (2014) reports that afterschool programs are growing rapidly, serving over 10 million children in the U.S. annually. Of these, an estimated 69% offer some kind of STEM activities (Afterschool Alliance, 2015). Despite this growing need for STEM programming, many afterschool educators have little or no background in STEM education, and often receive little or no professional development. The ACRES project is a response to that national need.
Outline of the ACRES coaching program
The ACRES (Afterschool Coaching for Reflective Educators in STEM) project provides high-quality STEM coaching for small groups of out-of-school educators, particularly afterschool providers. Participants in this professional development opportunity learn a skill during a workshop, videotape their own work with youth in their individual settings, and reflect on their teaching practice by watching and discussing their videos with other participants in their cohort and their ACRES coach.
ACRES training can have one of three formats: in-person, virtual, and blended. The in-person model allows participants to gather together at a physical site to complete the training. However, to accommodate the distance between educators in rural settings, fully virtual or blended in-person/virtual models are also available which utilize videoconferencing to bring educators together.
The full skill-based curriculum in ACRES is composed of six modules:
- Asking Purposeful Questions
- Modeling the Engineering and Design Process
- Modeling the Science Process
- Giving Youth Control
- Developing Science and Engineering Identity
- Making Authentic Assessments of STEM Learning
However, not all participants take all the modules, which are offered singly or in clusters, depending on the timing and needs of the particular educators. Each module takes approximately 6-10 hours each, in order to go beyond the “drive-by” forms of professional development and instead give the educations the opportunity to learn a skill, see it in action, try it themselves, and come back as a group to discuss how they are incorporating it into their interactions with youth.
While the ultimate goal of the ACRES project is to train coaches in existing out-of-school programs and networks, for this early phase of the project all of the coaches were our implementation partners at Vermont Afterschool, the University of Maine, Office of Extension (4-H) and 21st Century Community Learning Centers in Maine. All had extensive expertise in facilitating STEM learning by youth.
Study 2: Reflections of the Coaches
We conducted two evaluation studies for the early phase of the ACRES work. The first study, reported in a companion study, describes the impacts on the participating educators. Here, in Study 2, we report on the reflections of the coaches who implemented these coaching sessions.
Sample Selection
For this study, we conducted post-course interviews with every coach who led an ACRES cohort between 2014-2017, excluding those who were in the core development team at MMSA. In all, there were 9 coaches: six leaders of 21st Century Community Learning Centers in Maine, two faculty members from the University of Maine Office of Extension, and one project manager at Vermont Afterschool. The 11 cohorts they led included 4 fully virtual cohorts and 7 blended cohorts. Group sizes varied from 2 to 8 educators, plus the coach. In the blended cohorts, participants met face-to-face as a group with the coach for initial workshops where each skill was introduced, and then virtually for the remaining coaching sessions where videos were shared and discussed. Five of the cohorts were conducted with 4-H staff and volunteers; five were conducted with staff from 21st Century Community Learning Centers; and one cohort consisted of afterschool educators from Vermont. Other than the cohort from Vermont, the rest of the cohorts were based in Maine. The ACRES team partnered with 4-H extension professors from the University of Maine to recruit 4-H participants and pilot the ACRES modules. The ACRES team also partnered with Vermont Afterschool to recruit and train participants in the Vermont cohort.
Interview Questions and Process
Interviews with each coach were conducted three times: before, during, and after their course(s). Interviews took approximately one hour and used semi-structured and open questions in order to broadly capture the experiences and reflections of these experienced facilitators. Notes were taken by the interviewer throughout the interview. The set of question starters is given in Appendix A.
The purpose was primarily for formative evaluation purposes as the program was being developed and refined, but we have gone on to code the interviews by topic and report them here because they also provide insights into the overall experiences and learning of the coaches, which may be thought of as an impact of the program beyond the learning of the coached participants.
Analysis
Data from mid- and post- course interviews were coded and analyzed in Nvivo for themes that had already emerged from the interviews with the participating afterschool educators who had received the coaching (see Evaluation Report 1). These themes included: skill development and benefits to practice, coaching and facilitation of the course, group dynamics, and technology.
Because there were only 9 coaches, no attempt was made to quantify the results; instead, we provide a selection of quotes that typify the range of responses on each topic. The quotes provided have been lightly edited for clarity.
Detailed Results
Coach responses to the ACRES courses and feedback about the program mirrored the experiences of participants in the courses (see ACRES Evaluation Report 1), and provided a complementary perspective from which to understand the benefits and challenges of participating in a video-coaching professional development.
Interview responses fell into several major themes, and the structure of this report follows those themes. These were:
- Skill development and benefits to practice
- Coaching and facilitation of the course
- Group dynamics
- Technology
- Barriers to Participation and Suggestions
1. Skill Development and Benefits to Practice
ACRES coaches described the benefits to practice they observed in participants throughout the different modules and as they developed skills.
– “I did notice the way they communicated with youth (the way they asked, the pauses leaving time for kids to respond to what they were asking) it seemed the shift focused from being the facilitator to allowing youth to take the lead, for some of them… and that crossed over to the second skill. So that skill transferred from week to week.”
– “The engineering design I think that was another great skill to work on. Again, what was interesting was helping them see it wasn’t about the end product and necessarily getting to the end and finishing that activity. It was interesting to see them start to figure out that it is really about the process, not what they get to in the end. I think they both were really good skills to have in this series.”
– “Pushing people out of how they’ve always done something and because the way that the course was designed, and they did two videos, I was able to push after skill set 3 and then saw a change in the next skill set because they took the feedback and applied it to the next session they were doing with kids. I’ve seen growth that way. The way their language has changed and their comfort with the PQ pieces and really asking those open-ended questions has been part of their practice now instead of “I have to do this because it’s part of a course” They’ve seen that the skill sets have really benefitted them.”
– “There has been a lot of ‘aha’ moments where they’ve made really good connections and connected what we’re learning about to their practice. And I feel like a majority of them are able to improve or challenge themselves in a way that they probably wouldn’t have without this experience.”
They also described the need in the field for this type of professional development for afterschool educators.
– “We’re not focused on compliance or bad job; we’re illuminating for them a skill that exists to help them facilitate STEM. The afterschool world is hungry for afterschool skills, they’re terrified of STEM because they think they have to be a science expert. So the time is right. I love the idea of building the peer communities because this is a field of people who like people, and never get to see each other.”
Coaches described how participants gained skills in STEM instruction, but also how transferable the skills are fields other than STEM.
– “A lot of the people who did the program were not STEM providers, had never taught STEM classes. I was really surprised at how their confidence changed over time. Some of them didn’t even know what STEM meant, so to see the change in confidence, to see how drastic it was…”
– “They’ve all said they’re more comfortable doing STEM activities. There were a couple of staff who’ve never done anything in that line. After this module, they feel comfortable doing STEM activities and going on those websites to find them.”
– “I notice that all of us are using Purposeful Questions more. I asked the question of the staff. April was saying she uses it during the school day. I was talking about using it with one of the kids at my daughter’s birthday party. We are more natural using questions and trying to ask more thoughtful, engaging questions.”
– “We’re using the skills, the questions, in other avenues and activities that we do, trying to get the talk back from the kids.”
– “I appreciated that it was so much more focused on the process of professional teaching more so than on the academic of science and math. The skill of Purposeful Questions was beyond just the math and science. It can carry over in any of the skills we do. I found that extremely helpful. I think my staff did as well.”
2. Coaching and facilitation of the course
Coaches strongly felt that their role as a facilitator was to foster a group identity and build relationships within the cohort.
– “I think that that relationship piece, however you do it virtual or blended, is super important for people to be able to put that vulnerability to show videos that aren’t the best. Taking a step back, it’s amazing to look at their videos from the first time they shared and how edited they were, and now how it’s the real stuff. I’m not sure how much the participants will get from it if they don’t trust the coach that they have.”
– “I was focused on relationship building with them and helping them identify their own personal power that they could do this even though it was uncomfortable for them. I brought them together and they did the work, I chimed in and asked the right questions, but the group moved it along.”
This was especially important for the virtual cohorts, where coaches felt that they needed to be more intentional in developing a cohesive group.
– “It’s relationship building. If someone is doing an online course, even when we get on our meeting, we do a bit of personal, so we get to know each other. It leads into the richness of it. It is a little bit time heavy but it’s so worth it.”
– “I think in order for this to be successful completely virtually, the coach needs to have the ability to be able to read a group, but you’re reading a group in 9 different locations…and the relationship piece, I think the way that we built in the introduction and really getting to know people in the first module with PQ naturally builds that trust, I hope.”
Coaches described the trade-off between knowing their participants previously, which could be beneficial by adding an embedded element of trust or be a hindrance because of power dynamics.
– “I’ve always had the comfort of knowing most of my participants so I wonder would my experience be different if I didn’t have relationships with these folks. Would I not feel as comfortable pushing them. Would I not feel as comfortable challenging them on the things that are not easy?”
– “I think it went really well, allowed us to create a community, mostly not talking to teach other [during after-school programming], just working with kids, helped us grow as team. I think they would have been more intimidated if it were someone they didn’t know.”
– “As far as them knowing each other, I think that was helpful. I think they knew how to support each other with technology stuff.”
– “It may have hindered me from feeling like, from having really good feedback on how I really am as a coach. I think that we’re very collaborative to a degree, but I think that there’s probably, as much as I try and encourage it, there could be some unspoken norms that some people may not say because I am the boss.”
Coaches often personalized the experience for participants by offering “office hours” or being available to discuss issues and questions away from the larger group.
– “One of my participants does a lot of work with ESL students and she kept bringing up things that would come up around that. So we did a little work around that. I think maybe she was thinking “I don’t want to bring that up with the whole group because it doesn’t pertain to them.” So I think it was just a nice component.”
– “The one-on-one call has been the main place for me – I didn’t’ want to call out Jim in front of everyone, and we had a good conversation about how he tends to lead them to an answer. And I said “but what we’re doing is different from that.” It was great that I had an opportunity to say that to him. Because I thought “how will I say this in the group?”
Coaches felt that another major responsibility they had was to push participants to stretch outside of their comfort zone. They described how this felt more natural and comfortable as they proceeded in later modules with the same cohort because they had built up a strong foundation of trust.
– “I didn’t say as much in the first session. I was more comfortable with them and they were more comfortable with the process. I would say to them “I’m going to push you here” they were thankful for that I think. I think the other people could see it, but I wouldn’t say that the participants were quite as pushing out of their comfort zone. It’s hard to know where someone is if you’re only a couple sessions in.”
– “The video pieces of it really allow us to have a bird’s-eye view of their classroom and then as the comfort and trust builds it allows us to really go into those pieces. “I wonder what would happen if instead of lecturing to these kids, you sat with them in a circle or set up your room differently?” Pushing people out of how they’ve always done it.”
– “I think that I was primary force behind the stretch. When it came to negatives, they would skip it. I would say, ‘I love your positives, but we have to find something that we can grow in.’ I think it is facilitator’s role to find that stretch.”
In addition to recognizing the need for intentional relationship-building in virtual cohorts, coaches also made other adjustments to the structure of the course to facilitate a successful virtual experience for participants. These included utilizing Zoom technology features to vary the structure of the group throughout sessions.
– “Being behind a screen in terms of time is different from being face-to-face with people, so you want to use your time wisely. You don’t want people to be sitting behind a screen for 3 hours. Be intentional about how you present it to people, get them moving, by using the breakout rooms, the whiteboard, or other pieces of technology. You want them to experience it similar to if they were there in person.”
– “I think in order for this to be successful completely virtually, the coach needs to have the ability to be able to read a group, but you’re reading a group in 9 different locations. Different from when you’re in person and can read the group. The other piece is the technology. People can mute their microphones, reading people on their body language. I think it’s a coach that’s experienced enough to read the group and a coach that knows enough to get to know their audience and build upon the skill sets of their audience.”
Coaches experienced several challenges throughout the ACRES courses, including:
Difficulty finding an effective coaching style:
- “Most of my coaching is individual, one-on-one. It’s sometimes uncomfortable, if you don’t have a great coaching environment, I could have done better with my coaching environment, it’s not my personality to point out flaws”
- “I’m ok at coaching if I can spin it in a positive way. I had a hard time creating a coaching culture.”
- “I hadn’t had a lot of experience. I used to be super critical and I found that that was abrasive. Then I went the other way. I think I found the middle, that was good for me personally.”
Difficulty finding a balance between providing feedback and leading the group and allowing the participants to have control of the course:
- “I tried to be a much better listener. I think the first time around, I remember thinking that I talked a lot and I think I probably needed to pull back a little on that because I was…That silence sometimes it sits there. It’s just what we’re telling them…give the folks wait time…but we need to be able to do that too.”
- “The coaching piece: sometimes it doesn’t feel I’m giving a whole lot of feedback because I’ve given the group the power to do that. Sometimes the silences have been long, but if I give it some time it’ll come from the group, and that’s way more powerful than coming from me. And I often don’t know the answers.”
- “I was probably guilty of doing most of it. I always asked the question, does anyone else have any feedback. I would do one of those comprehensive answers where everyone else just nodded. For myself this is one of those areas as a coach I don’t think I was aware, then I was more self-aware that third time.”
Specifically for the virtual cohorts, a challenge in using the breakout rooms, was that the facilitator was not able to gauge the tone of whole group.
– “The only challenge is that when they’re in the breakout rooms, I can’t be in them the whole time. I have to limit myself to make sure I can stretch myself to all of them. That’s the one downfall, when they’re in the breakout rooms, you can’t see peripherally what’s going on with other groups.”
Coaches had recommendations for how to improve the coaching structure of the course to provide additional support and feedback to participants as well as to build up the virtual coaching manual.
– “We could develop a grid with skills and steps to get there, and check in on the actions they’d taken. This format didn’t allow for that, because it was too short.”
– “The individual coaching calls I wanted to do more with, but I kept them no longer than 10 mins and kept it to: how do you feel about this? What do you want to do next? And I know you can. Really short.”
– “I wish I’d had one-on-one time, maybe, I’m not sure if that was missing or not. I’m not sure if that would add to their confidence level. I didn’t email each one individually. I did talk to them each at times, but the office hours were really more troubleshooting.”
– “I feel like the way it’s [coaching manual] written right now, the virtual piece is very, very minimal in terms of what it needs to be. I feel like that needs a little bit of work. I appreciate the want to have it be only one guide, but I almost feel like there needs to be an in-person guide and a virtual guide. A lot of the pieces that come naturally to me, because I’ve done this for so long, is missing. I’m not sure that a course is going to be very successful if those intentional pieces aren’t there.”
3. Group Dynamics
Coaches felt that their cohorts were very cohesive and provided examples of how members of the cohort supported each other and advanced each other’s learning.
– “I’m seeing them push each other in a really positive way. They’re lifting each other up. I’ve seen confidence in folks that I hadn’t seen in the first couple of rounds. Especially [participant name]. I know that she’s really doing this to better herself as an educator, and she’s asked me to push her. But now other people have picked up on that and push her and cheer her in ways that she’s not expecting.”
– “Maybe it was the people that were in the group, but they just seemed to really settle in and be OK with the whole mistake thing. When people made them, it was fine, they were all really supportive. There was this richness to it. It went beyond just learning basic skills.”
– “Usually in PD there are either a lot of people around, and lots are strangers, and you have a 90min workshop, so you don’t feel cohesion. This group felt really connected, and WOULD stay connected if we gave them the platform. They have this connection and trust with each other, they don’t rely on me. I could be out of it.”
– “They like each other, get along well. The aspect of the program that was fun for them was working together.”
Coaches also described the accountability that members of the cohort had to each other throughout the modules.
– “There’s accountability – from each other, because it’s a cohort. When I did my Masters we had a cohort model and this reminded me of that experience, because when you’re part of a living breathing group as this was, there’s a lot of accountability to each other. Because if they hadn’t done their videos we wouldn’t have anything to talk about.”
– When describing a technology issue that the instructor needed to address away from the group: “I totally thought they were going to hang out and catch up about what they were doing that weekend or … I thought they’d be talking about me saying she doesn’t have a clue. I had an email message going back and forth with one of them…and she’s like “no worries we’ve got this, we’re still talking.” When I came back into it, I didn’t start talking right away because I knew I’d be interrupting them and they were deep in conversation about the activity and giving suggestions and it was like I never went away. It was cool. They just carried on.”
Coaches described the climate of providing peer-to-peer feedback as being open, honest, and respectful.
– “I was pleased with the openness of a group that only met for a small time together. They were open and honest in their feedback.”
– “I feel like the environment that we set up is a respectful and safe place. It sets up an opportunity for them [participants] to give feedback in an appropriate way, see what they like about the way other people interact with kids. Someone might watch their video and think, “I can’t believe I say ‘like’ 100 times,” or “this little person was asking me these questions and I just kept shutting them down because I wanted to move on with the activity.” They talked about things they saw in themselves, positives, and things other people needed to work on.”
Coaches also discussed how beneficial it was for members of the cohort to see each other’s videos and provide feedback.
– “So the two of them were able to look at their videos and see what was similar and what was different and challenge each other. I started the conversation but then they took it to another level and really pushed each other in a way that peer-to-peer is really different rather than me coming in as an instructor of the course. It really had a rich conversation that everybody could participate in, even though we were just viewing two different scenarios. But those scenarios were common for all 7 people watching.”
– “Sharing those skills and talking about them with each other and seeing how somebody else was doing it, maybe they could see a little bit of themselves in another person and that helped to build their confidence a little bit. They were really really good when it came to the constructive piece of the work. It never sounded or felt like they were picking apart. They were very gentle and kind.”
– “I haven’t been able to see changes in their practice because I’ve only seen one video, but in terms of feedback, I think they’re becoming more observant of each other, and able to offer more specific feedback, which means it’s starting to get into their brains.”
– “Three-quarters of my staff had no STEM background. They asked what STEM meant in the beginning. They were really grateful. Having the community to talk about how we could do better not just as a team, but as individuals.”
Coaches had recommendations on how to facilitate a cohesive group that provides supportive and helpful feedback to each other including:
Working with the same cohort for several modules
- “I think the other piece that’s happening is because my cohort are all repeat people, I’m really able now to see their growth and they’re able to reflect on what it was like 2 years ago vs. what it’s like for them now.”
Having group sizes larger than two to include more voices and experience
- “What I was missing was having more of the input and a larger group to bounce ideas so it was a richer environment before with more perspectives. But I don’t know if they would have picked up on that from their side because they haven’t seen that with a larger group. I don’t think it failed, but I don’t think I would recommend that small a group. I think it’s more powerful when you have a larger group.
Having the group watch videos before the coaching sessions to come up with intentionally thoughtful constructive feedback
- “You want to be positive. So sometimes you’re thinking how can I turn that around a little bit. Or when you’re trying to give the challenge area. You want to be thoughtful about how you say that. And that’s hard to do that for people trying to do that thinking on your feet sometimes.”
Building an online community that participants could utilize even after the ACRES course is complete
- “If we had a proprietary place online, like the Google community. Or a Facebook page or some place people could go to continue to be in conversation with each other. Or say “this speaker is coming to Maine and it’s about PQs, yay.” THAT would be great. Because they have this connection and trust with each other.”
- “Have an online community. I’ve been in projects before where they developed an online community, helped them share work. Even if you had one for staff and one for coaches. That would require web development. You might be able to use Google sheets. Say people wanted to take your model and utilize it and you wanted feedback, that would be a good model.”
Having a culminating celebration to capitalize on the cohesion they’ve built as a group and foster continued interest in maintaining their learning community.
- “I can see the interest in having a finale of sorts – that we should provide them with a certificate or “good job.” Because they’re feeling connected to each other – we’re starting this community of learners and then we drop it. I like the short-term time, but I can see why K wants more, because they don’t have those opportunities to have that kind of PLC.”
Coaches also had specific recommendations to foster a cohesive group sentiment in virtual settings. One recommendation that was mentioned by several coaches was assigning small groups to breakout rooms on Zoom
– “They were able to have these in-depth intimate conversations just one-on-one and they know I’m going to pop in and chat with them at any point. So I assigned them into breakout rooms and then I’ll give them like 90 seconds and then go in and check, answer questions, how’s it going, and visit with each one.”
– “I love breakout rooms. When we originally started, we didn’t know they existed. We thought they would have to use Google Hangout. Once we figured those out, it was awesome. In my first session, I broke them into their dyads, and I would visit. I think most powerful was with skill set 2 where they were building bridges, it was awesome.”
– “I like them very much. I think it’s a great way to have small group conversation. It’s a really nice feature that I haven’t seen on other platforms. Its powerful because it helps to change up that video webinar format. Especially if you’re working with a group for 2 hours sitting in front of the same screen, it’s not so good. It can get really, really, long. A nice way to change it up, just like in a face-to-face setting you wouldn’t just lecture, you’d get people into small groups.”
Coaches described the successes of the virtual model in keeping participants engaged.
– “I was disappointed with such a small group, I didn’t know if they’d participate, logging in from your home, having your webcam on – some people don’t do webcams – but they were happy to do it, logged in on time, were fully focused and engaged the whole time. Typically on web-based meetings people are checking email or other things. These guys were focused, there, even though they were at home and dinner was being cooked, they had full focus. I was completely thrilled that that could happen.”
– “I think about webinars that I’m on sometimes and it’s just someone talking to you all the time. So people naturally think “I have this email to write…etc.” The way that we’ve done ACRES is that we’ve designed it with the intent that they are active learners. I’ve never caught anyone multitasking because of I think the way it is built. Who knows when I’m going to ask you to respond to something? So it’s really active. It’s not synchronous where they’re just listening.”
The virtual coaches indicated that participants might feel more comfortable providing feedback in a virtual cohort because of the comfort of being behind a screen.
– “She said if I had been sitting next to him, I couldn’t have felt comfortable talking, challenging him, but because I was virtual in the safety of my home, I could do that. She felt so much more comfortable and confident in herself.”
– “I feel like the virtual model really takes away those barriers and allows people to say things with confidence that they might get embarrassed about if they are in the same room as someone else.”
4. Technology
The technology related to video-recording one’s own practice, editing and uploading videos, and sharing videos with others is an essential component of ACRES and provides the basis of the discussion for participants to provide feedback to each other. In addition, technology also allows participants to connect without having to be in the same room, through the use of Zoom videoconferencing software. Participants often describe a love-hate relationship with the technology in ACRES. When it works, it helps contribute to the effectiveness of the course. However, when there are technological glitches, it can detract from the course. Coaches mirrored participants’ sentiments about technology but described an increase in comfort with using the technology and troubleshooting issues as they gained more experience with coaching ACRES courses.
– “I think the tech piece was pretty seamless. We were able to trouble shoot. It felt more organic. Not as much of a learning curve.”
– “No one had trouble uploading videos, which was so much different from what we had in the past. I think there was flexibility in how they could upload (YouTube, Google Drive, email it to me, screen share…) That worked well because we set norms with how to take video.”
– “What was cool, some of the younger folks in the cohort who may be a little more tech savvy, in their sleep they could say “did you try this” They were very patient.”
– “Most of the ones that are still in the trainings are all good with technology. Know how to use cameras, cut and paste videos. When we watch the videos, we’re able to see some good outtakes. We did something different sometimes, and used walkie talkies if kids had to leave, so less background noise. We’ve just been using cell phones, and a go pro. Hopefully they’ll come out well this time.”
– “The challenges are as varied as the staff who would be using them. Some are seasoned with using equipment and some are not at all. Most everybody has a phone that has a video camera on it. You’ll always have a wide variety of background with equipment you use.”
– “It’s not always easy for us to share and upload videos…we use our phones, but having to go to computer to download, it hasn’t been easy for frontline staff, I think there was somebody who didn’t have a cord so we were looking at their phone, not enough storage space on phone, although I did offer alternatives, a lot of people use their phones and have lots on it.”
The heart of the coaching model is providing critique on participants’ videos of their own practice. Coaches described how initially participants were uncomfortable seeing themselves on video, but became more comfortable in later courses.
– “I think people you could tell they were a little uncomfortable at first trying to get used to that recording, but very quickly it seems like they just forgot about them and it didn’t feel like performance for the camera.”
– “From my perspective, that was the real strength in the whole activity. The staff didn’t necessarily see it that way, but they got very comfortable with it. Education should be using much more video feedback for teachers and kids. The kids like seeing themselves on video. It’s a rich opportunity for staff.”
– “Slight comfort issue initially. Took people a while to get comfortable watching videos of themselves.”
– “I think in the beginning there was an intimidation factor, but by the end it just felt natural. It picked up easy with our group, the staff using their cell phones made it comfortable, if they had been given an unfamiliar device, that would have added some hindrance, some more discomfort.”
Coaches felt the use of technology had a strong, positive impact on the group and contributed to the success of the ACRES coaching model.
– “I liked it in that you could go back and review it, replay it, look at it again, look at it. When it’s happening live you got one shot and you’re trying to take in a lot of information. I’ve done peer coaching with actual teachers back when I was a teacher and I remember taking notes like crazy. It doesn’t make you feel so on the spot… Its like when you do the replays in the sports you can go back and double check your thinking. I really liked that part.”
– “I think they’re really cool and that’s what makes ACRES and using the Zoom platform unique because of that breakout room piece. You’re not missing the small group conversations that happen in person.”
– “Videos were useful for me but also for staff in order to reflect on their own teaching skills. Discomfort early on. I even videotaped them as we were doing their meetings. I guess the one downside to videotaping is the time needed to reassess. But the quality is better than if they just thought back or used audio. They have concrete evidence of what they could do better at.”
– “I think the video was really beneficial, provides concrete evidence of their growth. Helped them be more confident. Staff had never videotaped themselves and then watched.”
Coaches identified several technological barriers that they were able to creatively overcome. These included work-arounds to deal with low Internet bandwidth issues, learning how to take good video of youth, and utilizing the technology expertise of other participants to help those who were not comfortable with technology.
– “The technology, especially Vimeo. I thought would be more user-friendly than it was. So glad we could figure out a resolution that worked for everybody.”
– “We’ve figured out with [participant name], I just send her all the links beforehand and she can watch them because she has really low internet connection at home. So she’ll stream them on her own or on her phone so there’s no delay. In terms of technology, we haven’t had any issues.
– “One volunteer was helping with technology, but she ended up staying. It was weird but it worked. She filmed Else and Donna’s videos. Because they wouldn’t have been able to figure out the technology piece without that. Else is 80. She probably could have figured it out, but it was so much easier to have Sally film her. It would have been a barrier and she probably wouldn’t have signed up for the course. She talked about how amazing the opportunity was, to see herself on camera, in her generation they never saw each other on camera, live video.
– “One person had low internet speed, so I sent her individually the link to the video, and she would watch it on her cell phone, which had a higher internet speed than her home. When she would contribute, it was 3 seconds behind. She would raise her hand, and I would cue her to talk.”
– “We did an intro video before the first session so they could get used to seeing themselves on film. Quite a bit of troubleshooting. I took charge of screens so they could see how to do it. They can grant you permission to control their screen. I did that for Cathy because she didn’t know how to turn up her volume.”
– “They’ve all shifted how their meetings happened and the settings of their meetings because they picked up on that audio and now they’ve created a way of capturing the audio differently or setting up the room really intentionally knowing that ‘I’m going to have 5 kids and they’re going to sit in a circle, and the door is shut’ getting rid of all the external stuff. I feel like if you look at the videos from this last module, with the exception of one, the audio was all really good without using those mics.”
– “Jim and Kelly had the best luck just recoding with their computers. But it’s all been fine. But they all did it with small groups, pulling aside 1-2 kids. Which seems effective. It’s a bit artificial, because they can’t normally run their programs in Ron’s office!”
Some technology issues remain unresolved. These mostly include how to get good audio from a large group or group of youth.
– “The difficulty was some of the sound pieces when it was on the Zoom platform. If they sent the video clips to me separately I didn’t have that issue as much. But the whole recording of a group setting that part was a little tricky. I think the mic idea is going to take care of that if you do the lapel mics.”
– “75% of the time I can hear OK. There’s been some scenarios where you can’t hear anything so we rely on the person sharing to set the stage for us and tell us what’s happening. So it doesn’t feel as authentic as if you were listening to the actual conversation.”
– “It is hard when you have any kind of background noise. I think the ones that work the best…one of mine had it in a library and she had the youth and there was a parent sitting there on the side. That after school program is hard if you have other kids in the room. Don’t know if there’s a way to mic kids without being too intrusive. I don’t know if you can put something in the center of the table. I don’t know. I wish I knew more about the technology. It does make it a little bit tricky to hear.”
5. Barriers and Suggestions
ACRES coaches described several barriers to participation for afterschool and youth providers. A major barrier identified was the time needed to participate in an ACFRES course.
– “Just fitting it all in. having time, flowing, especially in winter time, with cancellations and weather issues, trying to make sure that everything fits.”
– “Having time with our staff, [meetings] always had to be done after the programming, that’s a challenge if you’re working with people who have 2-3 different jobs.”
– “A lot of our staff are already working, some people do the before school stuff too, but that would happen with anything, nature of the beast.”
– “A lot of our staff are already super busy, I had to pay some staff overtime just to come to the meetings. At the end of the year I was worried that I wouldn’t have enough budgeted to fit that last module in. We had to meet Friday afternoons most of the time, not something people like to do. Working with kids is already exhausting.”
Another barrier identified was the nature of the afterschool profession.
– “There’s no career ladder, no good trajectory in afterschool, no way to use a badge. Who cares? We need the employers to care. And they don’t because they’d have to pay more, and they don’t have budgets to pay people more.”
Nonetheless, one coach suggested having a badging program to incentivize participation.
– “I think having some kind of acknowledgment for them: CEU’s, which they all cared about, and maybe some kind of digital badging. If digital badging takes off, I think it’d be worth pursuing digital competency. We’re helping them build a competency.”
Conclusions
Summarizing some of the main emergent themes of the coaches’ experiences with ACRES:
• The coaches’ experiences mirrored those of the participants, in that they found the program valuable as a combination of highly active learning, focus on relevant skills, and appropriate use of technology.
• The coaches also provided some validation of the participants’ beliefs that they had indeed changed their practices to use the skills. Specifically, several coaches reported seeing the participants using open-ended purposeful questions (the program’s central skill) to support youth learning.
• The coaches emphasized their own extensive and successful efforts to create an online community of mutual trust and safety.
• Coaches felt the use of video-based technology had a strong positive impact on the group, and were able to create work-arounds to solve most of the technology problems. At the same time, some problems were unresolved, most notably the challenge of getting good audio quality when recording a large group of youth.
• Coaches also noted a major barrier of having enough time to complete the course.
The coaches made numerous suggestions for improvements to the ACRES program, which have been passed on to the program development team, where several have already been adopted.
References
Afterschool Alliance. (2014). America After 3PM Afterschool Programs in Demand. Retrieved from http://www.afterschoolalliance.org/documents/aa3pm-2014/aa3pm_national_report.pdf
Afterschool Alliance. (2015). America After 3PM Topline Questionnaire. Retrieved from
http://www.afterschoolalliance.org/documents/AA3PM 2015/AA3PM_Topline_Questionnaire.06.09.15.pdf
Appendix A: Interview Questions (basis of semi-structured interviews)
1. How has the course been, for you?
a. What were some of its strengths?
b. What were some of its weaknesses?
2. How was it different from other kinds of PD you’ve come across?
3. What have been the biggest challenges in the course?
a. How did you work through them?
4. How many workshops did you have?
a. And how long were those?
5. How many follow-up sessions where staff reviewed videos?
a. And how long were those?
6. How many staff participated together
7. Did you have anyone else helping you coach the group, or was it just you?
8. How was the duration and spacing of the course for you?
9. How did you handle participants coming and going, with staff turnover?
10. How did you keep the staff motivated to keep going?
11. How did you deal with the other things you didn’t get to during that time your staff was during the course?
12. Do you think it helped or hindered that the participants already knew each other and that you’re their boss?
13. How was the experience of taking video-recordings as the basis of your coaching?
a. Were there any limitations or challenges?
b. Did you adapt the video recording/editing process in any way?
c. What were the most common devices that your staff used to video themselves?
d. Did they video themselves or did someone video them? Why did they do that and how well did it go?
14. After the videos were created, how did the sharing of the videos go?
a. Did you have people upload ahead of time? Why or why not?
b. How did you play them in the meetings?
c. How well could your staff hear what the youth were saying in the videos? How did you manage that? Did you struggle with lots of ambient noise from other groups not in the video?
d. Did your staff edit their videos at all? How did that part work?
15. Did you feel any pressure to troubleshoot technology when there were glitches? How was that for you?
16. Which skills do you think worked well as a focus? Where there any that didn’t work so well? Why?
17. Do you think the skills might transfer beyond STEM to other things your staff do?
18. In addition to the skills, your staff was learning how to comment on each other’s work with youth. How well did that go?
19. Have you noticed any changes in your staff, particularly in the way they interact with youth?
a. Are there any specific youth/staff interactions that stick out in your mind?
20. Have you observed any changes in your staff’s confidence or comfort level?
21. How about for you, would you say the experience of coaching this course changed your life at all? How?
22. How could the program be improved?
23. As we think about taking this broader, even nationally, to other states and other youth programs, what advice would you give us?
___
