media/files/webinars/2 web viewlaura casertano: so again, we want to welcome you to...

WorkforceGPS

Transcript of Webinar

YouthBuild Series

Aligning Program Practice & WIOA Performance Reporting

Tuesday, June 6, 2017

Transcript byNoble Transcription Services

Murrieta, CA

LAURA CASERTANO: So again, we want to welcome you to today's webinar, and if you haven't already, please introduce yourself in that chat window. Now, I'd like to turn things over to our moderator today, Mark Smith. He's a workforce analyst for the U.S. Department of Labor. Mark?

MARK SMITH: Well, thank you, and good afternoon. First, I want to go over today's objectives. The first thing we're going to do is discuss the big picture overview of WIOA performance management. We'll then do an introduction of the WIOA performance indicators. We'll talk a little bit about the program design factors and considerations. We'll talk some more about MIS reporting and how to submit the WIOA supplemental report, and lastly, we'll discuss some resources that can be shared with you and your program.

And now, I'd like to introduce today's presenters. First we have Jenn Smith. She is the national YouthBuild director from the Department of Labor, and we have Kurt Brown-Stetton. He's director of data services at YouthBuild USA. Jenn, would you like to start us off?

JENN SMITH: I would love to. Thanks, Mark.

MR. SMITH: Thank you.

MS. SMITH: So what I'm going to do is give you a brief overview of some of the – sort of the guidance that has gone out recently related to WIOA performance reporting and sort of talk a little bit about the next steps and direction that we're heading, and then I'm going to turn it over to Kurt who is really going to dive deeper into, again, showing some highlights of the field – or excuse me – highlights of the system as well as talking a little bit more about sort of how to visualize your program model in lieu of the new performance measures.

So that being said, hopefully all of you are familiar with Training and Employment Notice 8-16. This came out last August, and what this was really doing for us is signifying the intent amongst all of the programs at DOL that are authorized under the WIOA and really explaining expectations around when they would come on board with the performance reporting. It's sort of a staggered effort, and it all sort of depends on the resources available to the program, sort of how they were reporting before WIOA, etc.

So the key points that we want to make sure that you're aware of with Training and Employment 8-16 is that, as I believe most of you are aware, especially if you're a 2016 grantee, we are using what is called transition authority, which is a section that existed in the Workforce Innovation and Opportunity Act that says the secretary can use transition authority to slowly phase into some of the aspects of WIOA. So really what it was it was an acknowledgement from congress that making the necessary changes to the way things were operating and the way things were being reported under WIA is a very big list, and it's going to take us a long time to fully implement.

So in particular with YouthBuild, we're one of the unusual WIOA programs that had our own custom built web-based case management and performance reporting system, but what that also means is that, because we don't have you guys individually reporting your own data through

YouthBuild Series: Aligning Program Practice & WIOA Performance ReportingTuesday, June 6, 2017 of 27

whatever sort of system you were using, it means that we are responsible for sort of the effort of updating the system to be able to reflect and report the WIOA outcomes as they have changed from what we reported under WIA.

And so we've made it very clear through that Training and Employment Notice that it is our intent to continue using that existing system we have to – (inaudible) – basic information about the services you're providing, the demographics, and the eligibility while we work to develop a new case management system that – (inaudible) – reporting of the full six WIOA performance indicators.

Another part of the intent with that is to – really there has long been an interest in making sure that all of the many different siloed performance reporting systems that do exist at DOL are under the same umbrella as they're all reporting WIOA performance indicators.

So another part of this bolded section here is the fact that what we're really trying to do is launch a web-based case management system that can be used by a bunch of different types of grant programs that exist at the Department of Labor under WIOA because they're all reporting WIOA measures. So we're really trying to become more consistent and, again, one of the big aspects of WIOA is trying to eliminate some of the silos that exist between programs and this is yet another example of a way that we're trying to do that.

The other key – excuse me – key piece of guidance that you should be aware of is Training and Employment Guidance Letter 10-16. So this is a guidance that was developed related to the WIOA performance indicators, and what this guidance really does is lays out exactly how the performance measures are calculated within WIOA for the core programs.

And when I say core programs, what I'm really referring to there is the formula-funded programs that – the money that trickles down to the local areas through the public workforce system as well as the vocational rehab programs and adult education programs from Department of Education and the Wagner-Peyser employment service program. Those make up the four main titles of WIOA, and we call those the core programs. So this performance – this performance guidance is really focused on those core programs, but, again, the goal of WIOA is to be more consistent and to all be speaking the same language when we're talking about performance.

So in general, the guidance that's contained in this TEGL is relevant to YouthBuild participants, particularly how the measures are calculated, the types of other reasons for exit for youth. There's a lot of attachments in this TEGL that provide a lot of good information. So I think it's very important that grantees become familiar with this TEGL and not least of all because you are all one-stop partners and one of the things that we're focusing on there is accessing partner resources and potentially co-enrolling.

And that means you really want to understand how these measures are described and used across these other core programs that you may be working very closely with and you may be co-serving the same youth. So you want to understand the context there in terms of how these measures are defined and calculated. And again, one thing that YouthBuild does not have that a lot of core programs have is this idea of a reportable individual versus a participant of the program. For us,


all YouthBuild participants are just that. They're participants because they're getting a very prescribed program model, but in some of the core programs there's sort of a lesser tier of services that would make somebody reportable but not per se a participant. And it's important that you would understand, if you're tapping into some of the services of some of these other programs, those distinctions.

So as I just mentioned, some of the key things that you can get out of TEGL 10-16 that are particularly relevant are the fact that there are some revisions to the exit – other reasons for exit in terms of what those other reasons for exit are. Under TEGL 10-16 it differs from how we described other reasons for exit under WIA, and another thing that we're doing to be consistent with how the rest of the programs really define exit is once the new – and you'll note here it says "changes after transition." What that means is right now we have an existing performance reporting system that has certain rules and requirements built into it, and we're not changing those. So even though the 2016 grants are slowly starting to report on WIOA outcomes, part of that transition is that not all of these changes will be fully implemented until we have a new case management system that can reflect those changes.

So one of those to keep in mind is that in whatever new case system gets built, we will be doing away with the categories of successful versus unsuccessful exit. We'll really only have hard exit, which just means they were served and then they exited after 90 days of no services, and then a exit for other reasons, which is those category of reasons such as deceased, called up to military duty, relocated from the area. And the one that is particularly changed is we used to have a category that you had to – a youth who had to leave the program because of health or medical reasons, but we now actually have a category that is – considers incarceration to be an exit – other reason for exit.

And this I think is actually a really huge new area for YouthBuild. Generally, what we've said in the past is that participants exited for other – or excuse me – participants who were exited due to incarceration would have to be unsuccessful exits, but because we're doing away with that kind of qualification, they would – they could be considered an exit for other reasons if they were incarcerated before they finished the program and couldn't continue the program because of the incarceration. So I think that's a really interesting thing to keep in mind that that change is coming down the pike.

It also – one of the things in whatever new system gets built is that we have a lot broader allowance for what services are allowable under WIOA, and we'll be making sure to reflect all those different types of service provisions in whatever that new system is.

And then finally, I think some of you have heard this at various events recently, but just to make sure that you're all aware of this, which I think is a really, really exciting and important piece of information, although it is not going to happen right away but we are working on it, which is the fact that we're going to make sure that YouthBuild grantees are able to report the placement outcomes and the median earning outcomes through wage record matching. So what that means is that the Department of Labor engages in memorandums of understanding with each of the states to have access to their UI records solely for the purpose of being able to report these outcomes.


And that data is not given to us by participant-level data. It comes back to us in the aggregate so that the states can still be protecting their – the privacy – the high level of legal privacies that are ensured through their UI systems. So getting all those agreements in place to reflect the changes of how – what we're reporting on for WIOA takes a long time, and only once each of those individual state agreements is in place will the sort of master agreement that will allow YouthBuild access to that data occur.

And there are plenty of other programs that are in the same situation that will all be coming on board once that agreement happens. We don't see it happening until at least 2018, but when it does happen, I think it's going to be huge because it will definitely lift some of the burdens – excuse me – the burden for reporting the job placement and the median earnings outcomes off of your guys' shoulders by giving an independently verifiable source for that data.

Now, again, since the youth measures do include placement into both employment and continuing education, the continuing education piece would not be covered under those UI records. That would still be – that would still have to be verified through supplemental data sources, but still we think this is a huge boost to the program in terms of the ability to access this data and verify it. So that's something else that's coming down the line.

And so then I mentioned that TEGL 10-16 was really written for the core programs. So given that, while we're trying to be consistent with the overarching guidance that's in TEGL 10-16, there are some things that are not relevant to YouthBuild's programs as well as to some of the other programs.

And so we are putting together a separate guidance document that will reflect sort of what's different for our programs from what's in 10-16 and that guidance is currently under development and review and we hope to have it out shortly. And as soon as that specific guidance is out that has a whole section on YouthBuild and what is and isn't relevant related to 10-16, we will certainly make sure to share it with all of the contacts that we have for our grants. So you'll be the first to know. So please keep in mind that those changes are coming down the pike.

So again, this is just sort of trying to summarize where we are with the full transition that we think needs to happen in order to allow full reporting of the six WIOA performance indicators. So right now, here's where we are with the 2016 grantees who were the first grant class that was required to report WIOA performance.

And so what we have right now is our currently existing MIS system that you're using and then we have the WIOA supplemental report, which is also currently under development, and that is the report that will allow us, through our existing case management system, to report on three of those six performance indicators. And Kurt's going to go into a lot more detail on that, and he's going to show you an example of that report. So I won't spend too much time on it, but that's sort of the solution that we came up with for our transition period to be able to report on any of the performance measures.


So we're about to award the 2017 grantees later this year. This will be the second grant class that is required to report on WIOA measures, but, again, still exists in that same transition authority as we're slowly implementing these new pieces of case management and performance reporting that we need to put together. So 2017 grantees we do anticipate will certainly, at least for the beginning, still be using the existing YouthBuild MIS for their reporting as well as, again, that WIOA supplemental report.

And then you'll see I have here in italics because it's sort of a moving target and we don't know exactly when it will be complete, but I am hopeful that at some point the 2017 grantees will be able to access wage matching agreements. It's entirely possible also, depending on when that happens, that if 2016 grantees are still active, that they could be brought into that same – being able to access those records as well. I'm just not entirely certain at this point what that agreement will look like. So that's still sort of a question mark and we'll have to figure out but we certainly hope that the 2017 grantees should at some point be able to use those wage matching information.

And so then we look at 2018 grantees a little further out. You'll see everything here is in italics because this is, again, a lot of sort of unknowns in terms of what the actual timeline is to pull some of this stuff off. I mean, some of this has to do with money that's available to help implement it as well as just sort of a lot of competing priorities.

But I'm putting this information up here because I'm hopeful that perhaps by the time the 2018 grantees are awarded that we can have some sort of WIOA case management system that those grantees are using for the reporting as well as therefore being under the full WIOA performance reporting for all six indicators and, again, hopefully that wage record matching would be an integrated part of whatever that case management system is. The ultimate goal and the way that this has worked with some of the other programs is that the wage record data that would come back from that agreement would actually be put into the QPR that you guys would be submitting.

So it would all be sort of a seamless thing where it would do the calculation for the median earnings and the placement and it would just appear in the QPR. There are ways to make those systems interface, and I'm hopeful that that's one of the things that will be completed for the 2018 grantees. So again, me putting it in italics is my way of saying that you cannot hold me to it. So keep our fingers crossed, and we'll continue to update you guys as we have more information about where these moving pieces are.

OK. So finally, just some overall messaging here. Some of the things that we want you to understand is that these six performance indicators that we have under WIOA as compared to the WIA indicators I think really show sort of a shift toward really being focused on longer-term outcomes and career pathways for the people that we're serving. All of the performance indicators are very important ones for youth development, and they all are geared towards long-term success.

It's a real focus on continuing education training, career pathways into placement, growth of earnings, as well as skill development, which was a big piece of the youth measures under WIA. But another interesting thing I think to note is that for WIOA both youth and adult programs are


all reporting on the same measures, and I think that's an interesting acknowledgement of sort of how we're transitioning and focusing long term on what it means to help these youth be successful down the road.

So as I mentioned, we're giving you this information now even when we don't have all the pieces perfectly in place because we need to be preparing for that full transition down the road. Many of our – many of you are lucky to be part of YouthBuild grant programs that have been funded in the past and will be funded in the future by YouthBuild and – or excuse me – by the Department of Labor.

And what that means is that we want to make sure that you understand sort of what's coming down the pike so that your program model can evolve to sort of meet WIOA where it is. And so Kurt is going to talk a lot more about sort of what it means to visualize your program model as it impacts these revised performance indicators. So I'm going to turn it over to him in just a moment, but let me just quickly highlight for you what those measures are.

I talked about how we're going to be focused on three performance indicators with the 2016 grants, and you'll see that those are the three that are here in bold. So Kurt's going to describe in much more detail exactly what the calculations and logic are for these measures in terms of what's included in them and how we describe them. But the key things that we want you to be aware of is that under WIOA, instead of being a placement measure in the first quarter after exit, now we have that placement in the second quarter after exit.

And again, for youth that placement is both into employment and/or continuing education or training, whereas for the adult measure it's just placement into employment. And then the same thing is true for the fourth quarter measure. And then median earnings I've mentioned in the second quarter is one of the measures that we are not yet reporting on because that – there's no real way to collect that data in our current case management system, but that is something that you guys will be expected to report on once we have a new case management system and in particular once we have the wage matching that will be a much easier process and take a lot of the burden off you we hope. And then you'll see the next one in bold.

The third measure that we expect 2016 grants to report on is the credential attainment measure. And again, Kurt's going to describe that in more detail, but this one is that more nuanced measure from the way it was described under WIA, which means that if you – if the measure that the youth attains is just a secondary school diploma, they have to also be in placement for that measure to count, which is of course not true if that measure – if that credential is a industry-recognized occupational credential.

And then again, measurable skill gains. This is another one that we don't have the ability to report on through our current case management system, but it is coming down the line, as is the effectiveness in serving employers measure. So I'm going to turn it over to Kurt in a moment to talk about all of that in a lot more detail and show you some real examples of all this in our system. But first, we just wanted to see – since this is sort of a different focus from the – from this first section, we wanted to see if you had any specific questions about any of sort of the guidance and policy direction before I do turn it over to Kurt.


Oh, there's one there. Actually, and I see we already have a question. So wonderful. And anybody else to feel free to write questions and comments, if you have them. But I do see we have one question, which is that, "If the WIOA database is not in place, will we still be held to the new outcomes?"

And that's a great question. Now, again, I'm going to point back to that chart I had that had 2016, '17 and '18 and some of those wobbly italicized parts of it. Clearly, if we – I mean, at this point, as far as I know with the current guidance and what we're making sure to tell our leadership here, we do not anticipate you reporting on those other performance outcomes until such time as there is a new database that can allow it. Now, I can't tell you that there couldn't be a point at which, depending on how long some of this is taking, we don't consider other options for how to collect that information.

Those conversations are happening for some of the other programs who don't have access to a case management system like ours, but as of today and what I know, the expectation is that, if we don't have a system for you to report it into, you won't be reporting those outcomes. But it doesn't mean they don't matter, and it doesn't mean you shouldn't be focusing on how to figure out – I mean, tracking it perhaps separately to figure out how well you would do against them so that you're prepared when we do have full implementation.

And again, like I said, that guidance could change, but that's the thinking about where we are right now. I hope that helps to answer your question. Again, we'll have time for questions at the very end. So if I don't see one more – OK. Glad I answered that. I don't see any more questions popping up or anyone typing.

So Kurt has a lot of heavy content to get into. So I'm going to go ahead and turn it over to him and, if you have questions that occur to you about anything I said as well as anything Kurt is talking about, you can feel free to put it into the chat and we'll have time to address them at the end of the session as well. So Kurt?

KURT BROWN-STETTON: Great. Thank you so much, Jenn. So I am Kurt Brown-Stetton, director of data services here at YouthBuild USA, and I believe I know most of you. I'm looking through the participant list now, and I definitely see a lot of familiar faces. But I do – I know most of you through the MIS helpdesk and through us just kind of together trying to sort of data entry questions and reporting questions.

And what we're going to try to do during the remainder of this webinar is review the definitions for the three WIOA performance indicators that are implemented now, and then we're going to take a look at a general program cycle and start to outline the different data management demands for each program period or cycle period or cohort period; however, your YouthBuild program describes that. And then finally, we're going to go into the live MIS and review some of the basics about the WIOA supplemental report and other reporting resources for you in the MIS.


And for those of you that know me, I can talk a lot about data. So I'm going to really try to be concise here and kind of fall back to my position that this is the start of the conversation for many of you. So we've got many grantees on the line here that are either new or some that are re-funded or previously funded, and we're kind of shooting in the middle here to try to start some of these important conversations and rely on some of the kind of building blocks that we've come to at the MIS helpdesk for getting program staff to kind of think about these considerations and especially at this crucial time, which is when we have new performance indicators. So new outcomes are being measured and new timeframes or maybe not new outcomes but definitely new timeframes here.

So we're going to begin here with the performance indicators and just to make sure that we're all on the same page about the technical definitions for these performance indicators. And really what that means is what outcomes are being measured, and what are the timeframes for those positive attainments or accomplishments that were really being measured here?

And the first metric that we're going to look at is placement in quarter two. So placement in quarter two pretty straightforwardly is measuring the percentage of participants who are placed in employment, postsecondary education, military, or long-term occupational training in their second quarter after their exit quarter. So this is a very long-term measure. It's an exit-based measure, meaning once a young person exits your program, that the next calendar quarter and the quarter after that, that is what you're being kind of measured on here.

So your program is going to get very little feedback about this metric early on, and I think that's why our kind of discussions about how to plan for this and how to think about these timing mechanisms and how you're going to be looking at different accomplishments and different time periods, it start – it's really important to start thinking about those early on. And just to be clear also, for the DOL 2016 grantees there is no goal percentage for these metrics yet. They're still working on a baseline and establishing those goals, and I know that that question actually comes up quite a bit.

So right now, we're still establishing baseline goals here. So again, this is a placement metric that is measuring any placement in education or employment in a participant's second quarter after their exit quarter.

Let's go to the next metric here, and what we're looking at here is placement in quarter four. So again, this is very similar to the last placement metric we looked at, but it's shifting the timeframe to quarter four. So it's the percentage of participants who are placed in employment, postsecondary education, military, or long-term occupational training in the fourth quarter after exit.

So again, this is really kind of laying the groundwork for what Jenn was saying before, that these are very long-term measures and measuring kind of performance after a young person or a participant has been in your program and what trajectory they're on as far as their placements are concerned. And usually what happens or the questions that come up around this quarter four placement metric is this discussion around retention in placement. And I just want to be very


clear here. For 2000 DOL – or 2016 DOL YouthBuild grantees there is no retention of placement metric.

So these are really two isolated placement metrics, one looking at placement in employment or education in quarter two of the follow-up period and one looking at placement in education or employment in quarter four of a participant's follow-up period. So one does not exclude the other, and one is not linked to the other the way that they may have been in previous grant classes. So I just want to be extra kind of emphatic about that because we know that that question has come up quite a bit.

And so let me move on to the next performance metric here, and this is the credential attainment metric. And before I get into this at all, let me just highlight again you've got a chat window, and any questions that come up about these performance metrics, if you're with your staff right now, just start typing them in the chat window. If we don't have time to get to them during this webinar today, the MIS helpdesk will reach out to you and make sure that you've got the most information you can. But we know that there's a lot of real kind of technical questions about how these performance indicators work and what students might be eligible and which students might not be. So please don't let those questions sit. Let us take a crack at them and hopefully get you some clarity.

So credential attainment. So really what this metric is doing is kind of combining a couple different outcomes here. So it's kind of like a degree and certificate attainment metric, and it's counting the percentage of program participants who obtain a recognized postsecondary credential or secondary degree like a high school diploma or an equivalency during the program or within one year after exit from the program.

And so there is a stipulation, though, that for those participants that are getting a secondary degree like a high school equivalency or diploma, that those participants also are placed within one year of their exit. So it's kind of like a couple different outcomes being combined here, and I – at least from my perspective it seems like this dovetails pretty nicely with the two placement metrics that were set up in the earlier slides.

So your kind of three WIOA performance metrics are all dealing with follow up in some way and all dealing with placement in some way, shape, or form. So again, this is counting any participant that attains a industry-recognized credential that's approved by your federal project officer or secondary degree or a postsecondary degree. But the catch is, if the participant in your program get a secondary degree like an equivalency or a diploma, this measure stipulates that they're also should be placed within one year of their exit to be counted positively for this metric.

And so we'll definitely dig into the actual nuts and bolts of what that looks like in your reports, and I know for a lot of you that's where the rubber kind of hits the road is really just kind of understanding what this looks like in the system. But one way that we've really kind of been able to bridge the gap between what a performance indicator is and how it looks in the system is to really kind of pull apart data management practice.


And so it's really important I think that we not just think of these performance indicators as something to be measured or considered at or close to exit for your participants and really to start to think of them as the story of your young people at your program and part of your YouthBuild's program story, so really to consider everything right from the beginning and not see it as separate work.

So each performance indicator that we just reviewed, those three, that's really affected by your program design and the staff that are part of that data management process. And what that really kid of boils down to is that we have to ask that we consider the outcomes and the timing of these metrics in context and really with a kind of wide open view of all the work that really goes into that.

So while we're looking at all this work with the data management lens – and I know that most people know me through that perspective – I just really want to call out that this is not separate from your YouthBuild work. It involves your staff and tells a story about your program's success, challenges, and the young people that you're working with. So I really want to kind of encourage you to view it through that lens, and one way that we really found that's been effective to start to think about those questions is through this what we call the MIS roadmap.

So what this roadmap does is it really kind of helps you visualize the different program periods that are in a program cohort or a cycle and kind of on a very base level just gets you to visualize different types of data management demands associated in each of those periods. So what we're looking at here is kind of a bird's eye view of the MIS roadmap, and you've got the onramp in green. And the onramp is really the planning phase, and this happens before each cohort begins and lays out your policies and procedures that helps to guide how you implement your training, your data management practice, and all of those things that go on in your YouthBuild program.

Then you've got the program or training period in yellow. This is where your participants are in active education, receiving services, and starting to make some outcome attainments. And then the exit and follow-up period here is in blue, and that is really where you're starting to kind of continually have a graduate engagement policy and check in with your young people as they leave and transition into the workforce or to the next phase of their life and as well as be aware of that data management burden.

So let's dig into each of these program periods here, and let me just emphasize here that the – again, this roadmap is really just a conversation starter and we know that your YouthBuild program may have rolling admission or may have multiple cycles per year with different cycle length times and start times.

And I know that this gets very complicated and I'm generally using fake data with fake cycles and fake students and that gets complicated for me. So I know that real students and real work on a cohort or a cycle can get doubly kind of intense and involve a lot more thought. So I just wanted to flash – or flag that this is really a start of a conversation that we could help you with at the MIS helpdesk.


And that being said, the onramp period, whether it's for your first cohort or your second cohort, will really be crucial for establishing those strategies that will impact some very key things at your program. So some of these are pretty evident, but recruitment, enrollment, credential attainment, that kind of strategy and service delivery starts early on. Measurable skills gains, placement in quarter two and quarter four, a lot of programs will plan for this before the first cycle even begins because that is such long-term work and long-term follow-up support and activities and graduate engagement strategies.

Now, at the same time that these important program pieces are being planned, really you're also planning your data management practice alongside it, and that usually boils down to what staff will be involved in collecting, entering, and reviewing or checking your program data. And that also can really just simply mean are you accounting for how you're getting this data? So what forms do you use? Which staff are collecting that data from young people? Which staff are entering it into the MIS? Basic questions that can get fairly complicated at the program level when you have multiple staff members coming across different portions of data or different pieces of data.

Now, if we transition to the program period, this is really where all of your planning gets implemented. So your young people will enroll. They'll start training. They'll exit your program, and this is where data entry can be very heavy and it will be heavy at the enrollment period for sure, collecting and entering basic skills tests and eligibility information, service entry, just from a just number of minutes you're going to spend on service entry and case file maintenance of the system.

That represents real work. And then degree, certificate attainment entry, and exit. So at the same time that it's a heavy data entry period, it's also a very heavy data review period too. So you're monitoring their progress at an individual level for your participants, but you're also doing it at a programmatic level.

So when you're doing all this data entry, when you're getting this feedback from the system – and we'll look at what that really means in the system, but as you're reviewing reports and seeing this data kind of presented back to you, how are you evaluating your performance? How are you understanding when you need to readjust? If it's taking longer to get credentials, does that mean you shift midstream and figure out a new strategy to make the program cycle longer?

Are your partnerships in place for the follow-up period? I mean, I think it's a good nod to being in the evaluation mode is also pretty crucial here because you're putting data into a system, and you should definitely be using it on a regular basis to kind of inform the work that you're doing at your program. And finally, this will build in and inform how you think about the next cohort that you do. So whether or not this is your first cohort or your second cohort, this is going to influence or at least inform how you do this work the next time around.

So now, let's take a look at the follow-up period here, and really what we're looking at is from exit to the end of the full four quarters after their exit quarter. So again, I just want to call out that this is data entry and evaluation heavy. So even though your young people may be in follow up, the data entry and kind of self-assessment of program practice that comes along with this


period is substantial and should be called out. And again, you're evaluating and readjusting strategies and graduate engagement strategy is probably pretty fore in the front here, but are your young people keeping in touch with you?

Do you have staff resources needed to capture the data and maintain those relationships? And one thing that we usually call out at this point too is that for the – most programs, by the time you're done with a full year of follow up, you may be deep into the training period or even at the exit period for your next cohort. So the timing here and the overlap between cohort one and cohort two is something that we hear back from grantees quite a bit, is that that requires some planning and starts to inform how you're thinking about the timing of that final follow-up year for your DOL grant. So planning for that.

So I want to shift gears a little bit here and start to dig into MIS reports, and I think, again, that while these reports can look simple and focus on the numbers, I'm going to ask that we just continually think about all the planning, coordination, data entry, and review that goes into this work that gets kind of, in a very simple way, presented back to you in your reports and just call out that this represents a lot of work and a lot of coordination of different people at your YouthBuild program.

And so what I want to do with the remainder here is start to look at reports, and what that kind of boils down to is the very basics of this WIOA supplemental report. And this report is not live yet, but it is very important that we just start to familiarize ourselves with these metrics and understand how that information will be presented back to us and cover the technical steps for submitting the report. So what technically will you have to do when that report is due? And then as an add on to this and the other reports that we'll look at in the MIS, we have to get familiar with the report format. So how do we read this report for accuracy, and how do we use it for program evaluation well before a report due date?

So we're going to look at the WIOA supplemental report. We're also going to take a look at the quarterly performance report. So particularly, I want to look at this report and show what information is supported or in the WIOA supplemental or related to that data that's in the WIOA supplemental, and we're going to start with section B and really dig into demographic data and some of your program's baseline numbers.

And for those of you that have known me for a while, when we do a data review or when we review any report, the baseline numbers, your kind of foundational numbers that are contained in your DOL YouthBuild reports, that gets factored into a lot of other things. So they're pretty important, and we'll just kind of examine the relationship between all of it.

Then we'll look at section D and short-term indicators of success and how I commonly phrase it as your raw outcome data and how that can be used pretty effectively for you while you're trying to evaluate what you're doing, while you're waiting for those very long-term performance indicators to start counting outcomes.

So the system has a lot of different ways for you to view your data, and we'll take a look at that in section D of the quarterly performance report. Then we'll also review just a couple metrics in


section E of the quarterly performance report and what Jenn referred to as the WIA metrics, so the older metrics but ones that are nonetheless useful and can be applied to current WIOA metrics like measurable skills gain, for instance.

Then finally, if we have time, I want to take a little bit of a look at the ad hoc reports in the MIS and we might not have a full time to dig into them but I just want to kind of plug them as a valuable resource that has training videos up on WorkforceGPS on the CoP. So these are another way for you to view your outcomes and I think – outcomes and other pieces of data for that information – or for that matter. And I just want to make sure that everyone's using the system to its fullest here.

So that being said, let me transition to the live MIS, and we're going to actually look at data in a test site. And perfect. I can already see that on my end that I am continually amazed by technology because technology is not what it used to be. So what you're seeing here is the test MIS with a fake YouthBuild program and fake data, and you're seeing the basic login screen here. So what we're first going to do here is just run through the technical steps of generating the WIOA supplemental report and looking at how you would run it, check it for data, and attach it to the quarterly performance report. We'll look at the QPR and then the basic steps for submitting and checking that it was submitted successfully.

So with that being said, I'm going to click on go for the latest grant class in this test site, and I believe I will probably be logged out. I am logged out. That is pretty anticlimactic; right? Let me try one more time to log in. Hopefully, this goes smoothly, or I would – I'd be a little embarrassed. So while that is logging me in, let me just give another plug out to the MIS helpdesk and reaching out to us whenever you have questions. And perfect. We are in.

So I'm going to click go to get to the latest grant class in this test system, and I'm going to go right to the reports tab in the MIS. And this is basically a mirror of what you see in your live MIS system, and whether you're a main grantee or a support staff user, you can always run and look at this WIOA supplemental report when it is live.

So again, this report is not yet live, and let me just kind of say again, when it is live, the national office will be in communication to let you know that it is live and the MIS helpdesk will send out communications to our newsletter list to just kind of alert you that it is live and when you'll be expected to report using the WIOA supplemental report.

So you can see it down here at the bottom. Let's just click on it and begin. So whenever generating reports in the MIS, we just have to make sure that we're careful about the program year and the quarter that we're running the report for. So in the WIOA supplemental report – and some of this may look a little different once it's fully developed. In this – in the test system here this is an approximation of what the report will look like.

I think that it will be designed as well a little bit further, and I'm not sure if that includes this page as well. But whenever you're running a report in the MIS, you want to be careful about what program year and what quarter you're running the report for. So because DOL program years start in July, the next quarterly performance report that is due I want to say August 14th. I


might – it might be the 15th. The next quarterly performance report that's due is for program year 2016 and quarter four, which would cover through June 30th of 2017.

So I'm going to click on create this report. Perfect. It loaded. So this is the test version of the WIOA supplemental report. And again, just to let you know, visually we'll probably change a little bit here, but what I want to introduce you to is just the report format and how you'll look at your WIOA long-term indicators here. And so what you're seeing is a very kind of truncated report, and it's just showing you your long-term indicators, your three WIOA performance metrics, and then some breakout information of the credential attainment metric.

So this report format is very, very similar to what you will see in the quarterly performance report, and what you're seeing in section A here of any report in the MIS is your grantee identification information, the report quarter end date, and that's usually pretty important because what this is really tell you is that you ran the report for the right quarter and it's also telling you the cutoff point for this report.

So for me why that's important is because, if I'm trying to figure out why a high school equivalency or something isn't showing up in a report, it could be because it was attained in July, and this report is not capturing that data. So whenever you're submitting a quarterly performance report or this WIOA supplemental report, you're always kind of doing it after a quarter ends, and a lot of times that means that this data will seem a little bit out of date. So keeping in mind the timeframe of the report is really kind of crucial here. You'll also have your enrollment goal, the quarter that you ran it for.

So pretty standard information up at the top, and then down below you'll see all of this information related to your WIOA performance indicators. And these are exactly what we looked at on the slides. We've got the placement rate in quarter two. We've got the placement rate in quarter four, and we've got the credential attainment rate within one year of exit. But what I generally try to have grantees do, at least in the beginning when looking at the QPR or the supplemental report, is to get used to how the data is being represented back to you.

So you've got current quarter here and you've got program to date and I would just suggest that in the beginning you focus almost entirely on that program to date column. And the reason why this column is really kind of crucial is because this is what – this column represents or contains your total program numbers, and what I mean by that is it includes anyone who's eligible for a measure since the beginning of the grant up through this report quarter end date. So it's giving you your total numbers.

And if you're trying to fact check your placement in second quarter after exit, if you're trying to fact check why there might be students missing from it or why there might be students missing from the numerator, it's better to establish if there's an error in the program's date column because that's giving you your total numbers versus the current quarter which is giving you outcomes in a specific timeframe. And the current quarter would be the current quarter reflected in the current quarter of the report and that can get a little bit tricky because what you see here is really those that are eligible for that measure in that timeframe and that can get pretty tricky from metric to metric.


So again, stick to that program to date column, and what you're really seeing here, whenever you're running a report in the MIS, is you'll see in the program to date column, especially for your long-term indicators of performance, you'll see a fraction. And in that fraction you'll really see the denominator, which is anyone who would be eligible for that measure, and the numerator, which would be anyone that achieved that measure.

And where it gets really tricky and what I think as grantees and especially as maybe program directors who might be explaining this information to others or case managers who need to have the full context of what this is really representing is, when I say eligible to be in a measure, I really mean eligible based on timeframe and student type. So if I were to read this fraction here, this is telling me that 131 students have exited this YouthBuild program and made it to their second quarter after their exit quarter.

So participants that have exited the program and are only in their first quarter of follow up will not be in this denominator. So timing is really very important here. So you will only get a view in these long-term indicators of those that are eligible for these metrics once they hit the timeframe for those metrics. And that seems easy enough, but the more students you have in different exit quarters and different cohorts, it starts to get much more complex at the program level.

So again, this fraction is telling me 131 students have exited my YouthBuild program as successful or unsuccessful and made it to that second quarter after the exit quarter, and out of that 131 students, 17 of those students have found placement in that second quarter of the follow-up period. And that performance percentage is 12.98 percent.

So this information is pulling from exit information in the system, and it's pulling from follow-up information in the system. And while this is not a full data entry training that we're doing here today, it is really important to note that, when you run these reports, it's pulling directly from your data entry. So if there's a problem with the data entry, it's going to show up in these reports.

So that's looking at your placement in the second quarter of follow up, and it's really the same rules for the placement in fourth quarter after exit. And what's great about this kind of example here – and I'm going to just blow it up a teeny bit so that everyone can see a little bit better. Hopefully that's a little clearer. Let me try one more time.

What you can see here is that the denominator for the placement in the fourth quarter after exit is actually different, and that's because only 128 students have actually made it to their fourth quarter of follow up. So they are only included in that metric once they hit that fourth quarter of follow up. That's hugely important is that these denominators will look different based on the timeframes of when your participants left your program and where they are in their follow-up period.

And out of 128 students that have exited successfully and unsuccessfully and have made it to the fourth quarter of follow up, three of those participants have had placement in quarter four of their follow-up period. So on some levels pretty straightforward and simple, these two placement


metrics, but very, very limited to timeframe, and the report will only include them in those denominators once they are technically eligible to be counted in that measure.

Let me zoom back out here for one second. Now, things – (inaudible) – the credential attainment metric, and so this is another pretty great illustration I think. In the final WA performance metric – or WIOA performance metric it's saying in some of this information is still being worked on and might not look exactly right in this report, but it's only going to count those young people that have made it to a year after their exit point.

So the timeframe of this metric is from the moment that a young person enrolls up through the end of that one year from the exit date, and so this primary fraction here, the one that's dealing with your total WIOA performance metric, is only dealing with a small number of students. And in this breakout below it you're seeing the other representations of young people, and what that – what this breakout is really showing you is the number of young people that have a credential attainment being a secondary degree or an industry-recognized credential. So it's giving you different views, and I do think there were probably some logic errors that are still being worked on here.

But what you should be seeing here is your full metric up here in the credential attainment rate, meaning the full timeframe, and then breakouts of when those young people and when those participants might have attained either one of those metrics that would qualify. So that would be high school equivalency, diploma, or an industry-recognized credential, and in these two breakout lines D and E you would have greater visibility of when those things are attained outside of that full year from the exit date.

And I know that that's a lot of information. That handout, the downloadable PDF, actually has a lot of information related to that as well, and when those goes live, the MIS helpdesk will continue providing trainings specifically on this report and how to read these metrics. So when we get to the final version of what this is – and some of this will definitely change visually – we'll be able to definitively walk you through these metrics in the program to date column.

So that's it for the WIOA supplemental report, and as a workflow practice let's say you ran that report to check it and make sure that it was correct before you wanted to submit your quarterly performance report. You would actually create a PDF report here, and I'm just making sure the program year is on 2016 and quarter is four. I'm going to create a PDF report. The system is going to generate that. Download it. Now, I'll be able to – and I can see that you can see this down here that it is in my download folder. Now, I'm going to run a quarterly performance report, take a look at that report and some important parts of that report, and then attach my WIOA supplemental report to my quarterly performance report.

So I'm going to go back to the reports tab and I'm going to click on the grantee quarterly performance report link and I'm going to make sure I'm running the report for the same timeframe. So I've got program year 2016. Going to click continue, and I'm going to click quarter four, which is through June 30th of 2017, and I'm going to click build report. I was getting some errors earlier today when I was running it. So I did actually create a Word document format. Oh, perfect. Great.


So this is your quarterly performance report, and again, I just want to run through some key portions of this report and kind of link the data that's in this quarterly performance report to the WIOA supplemental report and encourage you to look at them together and to make sure that you understand the way the data from one informs the other. And a lot of the same rules apply here, although the quarterly performance report is a little bit more complex in that it's giving you a little bit more views on your program data.

But again, I think for purposes of this webinar and for your initial kind of looks at data in the system, concentrating on the program to date column is really pretty crucial up front because, again, this is really your total program numbers.

And if there's any error or something that's not being counted properly, it's definitely going to show itself in the program to date column because this column is going to give you all of your data since the beginning of the grant up through that report quarter end date. So it gets really crucial, and it's really kind of crucial for kind of seeing errors or spotting errors. And I'm just trying to make this a little bit more viewable for you here on your end.

So in section B of your quarterly performance report – and this was I believe highlighted on one of the slides – that's where we get into some of your baseline numbers for your program, and I just wanted to highlight some of these key baseline numbers and kind of link them back to the WIOA supplemental report.

So lines 1, 2 and 3 of your quarterly performance report are actually pretty important. So they're giving you a really kind of bird's eye view of some of your core numbers here. So in the program to date column we've got 154 young people that have exited your YouthBuild program, 194 students or participants that have enrolled since the beginning of the grant up through that report quarter end date, and 152 participants that have been served.

So these three numbers are really kind of key for your demographic data, for your outcome data, for your follow-up and exit data. So if this is counting in this measure or this line is counting anyone that's exited your YouthBuild program that's successful, unsuccessful, or exits for other reasons, if that number is incorrect, that is definitely affecting your WIOA long-term performance indicators because a lot of those indicators are relying on exit date and exit type.

So getting total exits correct is really kind of crucial here, and if it's wrong, it could be that some young people were not exited when they should have been or it could be that you had auto exits where students were automatically exited from the system in error and that would be because they might be missing services. So definitely a number to kind of always make sure is correct and know that, if it's incorrect, it's most likely also showing up in your WIOA performance metric report or your WIOA supplemental report. So very important number.

This next number here is the total participants enrolled, and this is counting how many participants you have brought to full active enrolled status in the system. And so this number is really crucial because it's kind of used in a lot of these other calculations. So all of or a lot of the


calculations will be considering your performance of those that are fully enrolled. So if that number is too low, you have some young people not being counted here in a lot of these metrics.

So if there are young people not being counted here, it's most likely because they're missing enrollment information, and that's really, really crucial. And usually it's because someone's missing a basic skills pretest or they're missing the employment question on their assessment and entry screen, but that's definitely something to flag if you or a case manager knows that the total enrollees should be higher.

Then you've got the total participants served, and this is measuring the number of participants that have services entered for them. So because we're looking at the program to date column, it's services entered for them at any point, and what I would kind of ask, if this were a real program with real data, I would say, is it really true that 194 young people have enrolled in your YouthBuild program but only 152 of them have had services entered for them? And most likely that would be a data entry error and something that would need addressing. So very kind of crucial numbers here.

The rest of section B is counting up a lot of demographic data for the young people at your YouthBuild program, and I guess the biggest piece of advice or the biggest piece of context I can give for the rest of these lines here is that this is all at entry demographic data. So it doesn't change once a young person is enrolled in your program. So things like the educational level at entry, that will always stay the same for your young people unless you make changes to some of that primary information. And it's really all being pulled from the case overview page in the assessment entry screen in the MIS.

So I'm going to skip through some of that, and we're not specifically going over section C but just to pause for a second and say that this is really a snapshot of your program services and a representation of the types of services that you've provided or facilitated for the young people or participants at your YouthBuild program.

And again, just looking at the program to date column, if you were – if you're providing these services and your young people are engaged in them, make sure that you are entering it and getting credit for them in this report here. And we can help you at the MIS helpdesk if some of this looks incorrect for you.

Let me skip down here to section D. So section D just in a nutshell I think is helpful to think about this section as your raw outcome data, and what I mean by this and why I think this part or this section of the report might be really helpful for you as a 2016 grantee is that this is giving you a real snapshot of many, many outcomes in real time. So section D here is showing you outcomes that occur no matter when they occur. So number of initial job placements here is giving you a number of young people that have a job placement ever.

That's two months after they enroll in your program or two quarters after they exit your program. So any job placement at all on an outcomes page or any young person with a job placement will show up here in this program to date calculation. And there's some really high-profile outcomes


here that might not show up immediately in some of your long-term indicators of performance but that you can see here.

So number obtained a certificate, number obtained a high school diploma or a GED, a lot of high-profile outcomes here. It's all pulling from the outcomes page in your MIS, or it's almost all pulling from the outcomes pages in your MIS. But it's a really good way to get quick visibility on some of these high-profile outcomes, and I usually like to point out here that also you can click on these numbers in the QPR report.

So when you click on the number –and you'll have to excuse us. These are not very imaginative test student names. So there is a Tom Brady in here somewhere I think. But basically you get the names of the young people in that count just by clicking on the number, and that might be very, very useful for a case manager or someone to spot check and let you know if someone is missing from this count here.

And if they are missing from one of these high-profile outcomes count, it's most likely that they're just missing it from their outcomes page. So really kind of great way for you to run this report whenever you need to and quickly check some of the outcomes for your young people.

We just skipped through the rest of section D and get to section E, and I just want to highlight here that as 2016 DOL YouthBuild grantees you're reporting on those WIOA long-term indicators. But since you have this view in section E of the quarterly performance report which is giving you the old WIA metrics, I just wanted to point out that there are a couple things here that might be very useful for you just in your program management and also to check on things like literacy and numeracy, which might – or sorry – which counts towards the measurable skills gain WIOA metric.

So just to flag here that even though these WIA metrics have the same names, they are measuring different timeframes. So that can get kind of confusing, but if you just look at something simple like the literacy and numeracy rate here, this is letting you know how many young people have enrolled in your program that tested basic skills deficient in one functional area and whether or not they made one educational function level increase within one year from their enrollment date.

So you're getting really kind of great visibility on the basic skills testing that you're doing at your YouthBuild program. Even though it's not in the WIOA supplemental report, this is something you may want to run frequently just to check on that work that's occurring at your program and to also flag the same kind of consideration for the recidivism rate here.

So a lot of programs will still be tracking this and it's a participant type and you're entering that data into the MIS. If you want to get feedback on it and understand how your program is performing, you can look at this recidivism rate, and this is showing you the number of young people that have enrolled in your YouthBuild program as youth or adult offenders and how many of those young people that enrolled as youth or adult offenders reoffended within one year from their enrollment date. So I just want you to get the most that you can out of these reports as far as an understanding of how you're performing.


And so if we continue on this kind of exercise of just the technical aspects of submitting the WIOA supplemental report, you would do what we just did basically. You would run the WIOA supplemental report, save it as a PDF, and you can see it down here in the bottom it's still saved for me.

You would then run a quarterly performance report for the same timeframe, and you would attach it to this quarterly performance report before you submit it. So when you submit a quarterly performance report, you're always attaching a narrative, which is your qualitative Word document or PDF about what's happened during the reporting period at your program, and you're also going to attach the WIOA supplemental report to this quarterly performance report.

So when that report is implemented and live for you, you'll be able to run it, save it, and attach it. So you would do it the same way you would do at – with an e-mail. You would basically double click, attach that report, and also do the same thing for your narrative. You would then put in your PIN, and just a reminder that only the main grantee can submit and certify the quarterly performance report. So the main grantee has a digital signature PIN they would put into that window here and click certify.

And just one thing that kind of continually comes up is, when you're submitting your reports, you've got that no later than 45 days after the end of each quarter to submit your reports. Let me just emphasize try to do that before that last week so that you have time to catch errors, so you have time to make sure that, if there's an error during submission, you have time to resubmit. But after you do submit, even if you get a confirmation page that says your report was successfully submitted, I absolutely urge you to go to the top of the page here and click on submitted QPR.

And so what's important about this is this is a running log – oh, and it logged me out. Let me log back in here nonchalantly. So what I was trying to check in that submitted QPR link is basically a time stamp of when that report was successfully submitted. So you want to make sure that that report really went through and that's it's logged in the system just in case there was an error during that final submission process. Let me just go back there and highlight that once more.

So underneath the reports tab, click on submitted QPR, and that's going to open up a log of all your successfully submitted quarterly performance reports and narratives. So this is – this will be really crucial at some point just to make sure that your reports are submitting correctly but also if you want to see narratives that were submitted before or when your previous reports were submitted. Very good resource in the MIS.

And let me just kind of do a time check here and go back out to the reports page because we're not having – we're not going to have very much time to run into or to get into the ad hoc reports. But let me just, again, kind of give an advertisement for the MIS helpdesk and the various resources we have to train you on running these reports and really understanding your program data here.


And I know based on the last couple of events I've been at and trainings with DOL YouthBuild grantees, especially the newer grant class, there's definitely a hunger and a need out there to have this data be cut up in different ways. And by customizing an ad hoc report, you really will have many different ways to look at your data and many different ways to look at your data that's not in the format of the quarterly performance report. So what's kind of great about that – and I can pull up an example here really quick.

So this is an ad hoc report that I built called the participant follow up placement history report that gives you a very kind of simple readout of who's made it to a follow-up quarter and who has placement information in each of those follow-up quarters. So I've made it a little bit prettier, but I can tell you that it did not take me much effort whatsoever to make this a little bit more readable. But this is an export from the MIS. You all have this capability to export data and to look at it in different ways.

And so what you're seeing here is a participant that's made it to their fourth quarter of follow up, and they have placement information in each of those four quarters of follow up. So if I were to look at this report and have a case manager look at it, we would be confident that this young person should be showing up positively in the second quarter after exit placement count and the fourth quarter after exit placement count. And the more and more comfortable you get with Excel and with getting data out of the system, the more and more useful this will probably be for you.

And so I will make a commitment to redistribute the training videos that we have online and know that using the ad hoc reports is something that we cover in our weekly MIS report trainings. So know that we have that as a resource and that you can basically flag for us that you need training whenever you need it, and we will facilitate that.

So with that let me go back and stop sharing my screen here and then try to just dig into some of the questions that might have come in.

MR. SMITH: All right. So thanks a lot, Kurt, and of course thanks, Jenn. If you do have questions, feel free to type them into the chat box below. We do have a few questions that came in a little bit earlier. So maybe we can get a little bit of clarification on some of this, and I think this one is a good one for Jenn.

"Does a participant need to be placed in the second quarter and fourth quarter?"

MS. SMITH: Oh, sorry. I guess I pushed mute too many times and put myself back on mute. I apologize.

MR. SMITH: No problem.

MS. SMITH: So yeah. So I think – I believe this question came in around the time that Kurt was describing the different performance measures, and so I'm going to take a stab in the dark here and say that I think that what's being asked here is related to the two placement measures. Please feel free to write in again and correct if that's not quite what you're asking.


But I think this is a question we've been getting a lot, and it's a good question because I think it's a little confusing because under WIA we had a placement measure in the first quarter after exit that was a WIA common measure. And then we had an additional measure that we tracked in YouthBuild that wasn't one of the WIA common measures, and that was what we referred to as the placement retention measure in quarter three. So in order to be in that quarter three denominator, you had to be a positive outcome in quarter one.

That is not the case with these placement measures, and I want to be really clear with everyone that these are not – the quarter four measure is not a retention measure. These are two separate and freestanding placement measures. So it's looking at for any participants you're serving are they in a placement in quarter two after exit, and then for any participants you're serving are they in quarter four – or are they in a placement in quarter four?

And those could be totally separate placements. It could be the same placement but those are both separate measures and it doesn't – and the placement – looking at the placement in those two quarters doesn't require that they have been in a placement in quarter one or quarter three either. So those are both just freestanding measures.

And then I'm also going to take this a step further because this is another question that I anticipate we might get because I've seen it before, and that is the question of whether the credential – when we talked about the credential attainment measure and the fact that, in order to have the secondary school diploma count, you have to be in a placement after exit.

That placement after exit doesn't have to be in quarter two or quarter four. That placement can happen at any point in the four quarters after exit in order for us to make that credential qualify. So those are sort of separate measures, and hopefully that helps to explain the question. But please feel free to clarify if I didn't quite get it right.

MR. BROWN-STETTON: Yes. And, Jenn, I would just echo that you're right. I really think that is a question that just keeps coming up by nature of the old performance metrics.

MS. SMITH: Yeah.

MR. BROWN-STETTON: So anything that we can do to clarify that I think would be great.

MR. SMITH: OK. Well, awesome. I see we have one here for Kurt.

"Just to be clear, degree, credential, training should not be conducted in the follow-up phase?"

MR. BROWN-STETTON: Great. That is a great question, and so the metrics are actually counting degree and certificate attainment within program or within one year of exit. So if you position your young person to make those attainments in their follow-up period, that would still count positively for your program.


So you're basically definitely still capturing that data, tracking that data. And I think the most common example we might hear at the helpdesk is a young person might be close to getting a high school equivalency by the time they finish at your program and you help position them to continue on that path and they attain that later in their follow-up period. And that would count positively for you. So no. It's not limited to just the in-program period. It's also flexible to capture those successes after exit.

MR. SMITH: OK. Great. And, Jenn, I think you might have answered this one a little bit earlier but, "Can we capture credential attainment for those participants that have exited for other reasons, i.e. incarceration?"

MS. SMITH: Yeah. So actually, we have not – we didn't – I didn't quite tap into this question. So thank you for asking it. So what I will say is exits for other reasons, while the types of exits for other reasons is slightly changed from what we had under WIA, the way it works in the system is no different.

And what that means is that, if a participant exits for other reasons, they are no longer counted in any of the credentials because what it really means to exit someone for other reasons is that they're taken out of everything except your enrollment count. They are no longer in the denominators for any measures because basically what you're signifying by exit for other reasons is that this participant, for whatever reason, was not able to successfully complete the program.

So it wouldn't make sense to then capture credential measures for a participant who never really was able to be successful in the program. So yeah.

MR. SMITH: We have another one, and it's, "Do you anticipate when the supplemental report will be live?"

MS. SMITH: Yeah. So I guess this is probably a question for me. This is the question that I was hoping no one would ask, Mindy. I'm just kidding, but no. This is a tricky question because I'll be honest. We had hoped that it would be live by now. I think when we started down this road about a year ago we had told you guys that it was our expectation that around the time any of the 2016 grants were starting to exit youth, which I imagine is probably starting to happen in a fair amount for the first grant class or the first cohort, that we would have this report for you.

And I'm sorry to say that it has not been done. Again, I'm going to go back to what I alluded to earlier about competing interests. There's a lot of technology things that are happening here at DOL in terms of different upgrades to systems, etc, and we're definitely running up against a competition for resources and programmers, etc, that has delayed the development of the supplemental report. So I am glad that we at least had it in test to show you guys.

It's close to done. We went back to programmers with a few more requests for some additional data points and also to kind of – really what needs to happen is there are still some glitches in terms of the calculations and so we're waiting for those to be implemented and then we would test it. And if it passed that testing, hopefully it would go into development.


But as I said, we are kind of up against these other delays. I am doing everything I can on this end to kind of raise the red flag of why it's not good that it's delaying our WIOA reporting, and I'm hoping that that may help us to break through some of these competing interests. But I can't promise anything for certain except that we'll continue to be honest with you about where the progress is.

I think ultimately the messaging that I want to make sure the 2016 grantees have is that as long as you're putting in the data when you need to, which means – part of the reason we wanted you to have this information now is making sure that you're putting in outcome data for not just quarter one and three like you were in WIA but putting in that data for quarter two of follow-up, putting in that data for quarter four of follow-up, being really careful that you're making sure to put in every sort of outcome that you have so that at the point when that supplemental report is available, it can generate that data and it will include all of the past outcomes that have happened for youth that you're serving now, even if at this point in time you can't look at that data. The goal is that by the time these grants end, we will be able to have that full WIOA reporting picture for those three measures for the 2016 grant.

So at this point I'd say the ultimate goal is that it happen before these grants sunset, but I hope it's certainly much sooner than that. But please do bear with us.

MR. SMITH: OK. Great. Again, if you have any questions, please feel free to type them into the main chat window. We have time for just a few more.

The next one we have is, "What does DOL consider long term?" Might need just a little bit of clarification on that one. If you could, maybe type that into the box.

MR. BROWN-STETTON: So I could just talk a teeny –

MR. SMITH: Oh, sorry.

MR. BROWN-STETTON: – bit about – I mean, maybe just if this is in reference to just timing of performance metrics, I mean, I think the full scope of what we're talking here, at least for the performance indicators, is that follow up is through four full quarters after the exit quarter for the participants in your program.

And when we were talking about the timing of things and looking at the different components of the program cycle, that will definitely cause or force the question of when the timing of follow-up periods occur throughout your cohorts and throughout your three-year grant cycle. So you're tracking placement in the fourth quarter of follow up and degree certificate attainment in that follow-up period as well. So if that gets at that question at all, I think maybe that's part of it.

MR. SMITH: OK. Thanks a lot, Kurt. I see we have one more.

"Does the enrollment page in MIS connect participants to the WIOA report?"


MR. BROWN-STETTON: And I'll take a stab at this too. I think what this is asking, is the WIOA supplemental report using or pulling from the same data that the QPR is in that the MIS is kind of giving you as data entry pages? And the answer is yes.

So whenever you enter enrollment information in a case file, whenever you enter information on an outcomes page, that very same data is being fed into your quarterly performance report and into your WIOA supplemental report. So if someone's not fully enrolled in the MIS, they're not showing up in your QPR or your WIOA supplemental.

MS. SMITH: Hi. This is Jenn. I just want to jump in here. First, I wanted to give a shout out to the fact that we have – for those of you who aren't aware, we have – excuse me – all of the region one peer-to-peer participants actually on the phone viewing this webinar from a conference space where they're in the middle of the peer to peer.

So I did want to give a shout out to all them. We thank them for being able to join. I think they're a little bit delayed with some technical issues, but I'm glad they were able to join. And now, they are heavily involved in a – in getting us some questions. So thank you. And so I guess we have one question from them here, Mark. Do you want to go ahead and – well, I guess they were asking for – oh, I see. I'm sorry. Sorry. I'll go ahead since I'm already talking here.

Just to clarify, you guys were wondering specifically what was meant in terms of long-term occupational training in the second quarter and fourth quarter. That's actually a good question, and it's probably a good time to tout the fact that we do have one piece of guidance for YouthBuild that is a – what we call a change TEGL, which is when we have an existing TEGL and then we revise it with additional guidance or clarification that becomes a change one.

And we have a TEGL that is almost through departmental clearance and will be published very soon, hopefully before the end of June, which is a revision to an existing TEGL 35-12 that we had. This TEGL was really describing what are the allowable and qualifying credentials for YouthBuild construction that are nationally recognized.

And then another part of this TEGL though really went into detail about what is considered a credential, and we revised that to reflect how WIOA defines a credential, which is slightly different than how it's defined under WIA. And I think that this also gets into a little bit of what long-term occupational training is, which is – what we mean by long-term occupational training is any sort of vocational training program that results in an industry-recognized credential.

So this is everything from registered apprenticeship would actually be a long-term training, but it's also things like vocational training – what we call proprietary schools I guess like barber school, cosmetology, some other vocational programs that could be like a culinary school. Things like those are things that we would consider long-term occupational training. They're training that results in some sort of credential that can help you enter into a specific industry career path, if that helps.

And then also while we're waiting to see if there are any more questions, I did also just want to circle back to something that one of our region five FPOs who's on this webinar, Jack


MacLennan – thank you – just cited sort of a clarification. So for those of you who didn't see it in the chat, we were talking about what credentials – so again, this is also about that TEGL – what credentials qualify and what don't.

And those of you who have been with YouthBuild for a long time are probably aware that this is always sort of the sticky issue of that we don't have a perfect list of what is a credential and what isn't. So we provide guidance around the types of – the aspects of what a credential has that make it qualifying, such as being stackable, recognized by industry, recognized by state education, etc.

And so to Jack's point, FPOs do not make that determination. Your federal project officer cannot determine for you whether a credential counts or not, but they can help to provide guidance and resources on where you can determine whether that credential counts.

There are a lot of resources that Department of Labor has developed and has online that sort of describe where you can find out about qualifying credentials, and we do have some links to those items in our revised TEGL 35-12 change one when it is published shortly. So bear that in mind. So thank you, Jack, for that clarification. That's important. We don't want to put the FPOs on the hook for something that is not their role.

MR. SMITH: Very true. Well, I'm not seeing any questions. I think we can go on to sharing some resources. So the first is the Training – well, the TEN number 08-16. We also had Training and Employment Notice 10-16. This is a resource to the Innovation and Opportunity Network and the WIOA performance indicator sheet, so sort of a cheat sheet for the performance indicators.

And of course you can always reach out to the MIS support desk at [email protected]. Their phone number is (866) 680-0855, and we just wanted to thank you again and have a great afternoon.

MR. BROWN-STETTON: Great. Thank you so much, everybody.

MR. SMITH: Thank you.

(END)


media/files/webinars/2 web viewlaura casertano: so again, we want to welcome you to...

Documents