The overarching goal of the Subminimum Wage to Competitive Integrated Employment Innovative Model Demonstration Project, known as SWTCIE Illinois, is to increase access to competitive integrated employment (CIE) jobs for people with disabilities. The SWTCIE Illinois project has taken intentional steps toward this effort in its early years, through the hiring of employment specialists across partner agencies, providing training and technical assistance to the agencies and their employees, providing benefits counseling training to community research specialists, and more. These efforts are already yielding rewards, as several SWTCIE Illinois participants have successfully gotten CIE jobs. 

One of the key components to the success of SWTCIE Illinois has been the creation of an evaluation team within the Illinois Institute for Rehabilitation and Employment Research (IIRER). The team exists to document the relationship between participant engagement with CIE practices and the strategies implemented by the project in facilitating its key outcomes. The team is composed of Dr. Bryan Austin, director of evaluation; Dr. John Kosciulek, project administrator; Dr. Stuart Rumrill, research evaluation specialist; Dr. Chun-Lung Lee, research evaluation specialist and biostatistician; and Dr. David George Strauser, interagency collaboration specialist. 

The team conducts the evaluation process through the collection of both qualitative and quantitative data from a variety of constituents, including SWTCIE Illinois participants, their parents and families, advocates, guardians, 14(c) certificate holders, state- and community-based agencies, schools, CIE employers and the Illinois Department of Human Services (IDHS) Division of Rehabilitation Services (DRS). The team solicits feedback from these partners and stakeholders and analyzes the collected data. This allows the team to holistically evaluate the SWTCIE Illinois project and the impact that it is making, including its strengths and challenges, in order to refine and implement the project with ever-increasing success. Data and findings yielded by evaluation are utilized not only by the constituents involved in the data collection process, but also by Mathematica, a research and data analytics consultancy working with the Department of Education Rehabilitation Services Administration on the evaluation of the federal initiative across all 13 states. 

The team conducts their much-needed research through both instrumental means and direct observation, such as surveys, interviews, focus groups, case studies and DRS WebCM case file reviews. Evaluation results are disseminated through an equally varied system of resources, including the SWTCIE Illinois website, social media, peer review journal articles, conference presentations and more. In addition, once the project has concluded, the team will create a SWTCIE Illinois Toolkit for use by any interested parties. 

The team’s evaluation process is conducted with six primary constructs of focus in mind: quality of life, interagency collaboration, vocational rehabilitation services, implementation, knowledge translation, and sustainability. The framework is robust in nature and is based upon the SWTCIE Notice Inviting Applications, as well as empirical, policy and practice vocational rehabilitation literature. The team evaluates and measures the project’s performance based on four specific outcomes: optimized participant CIE, increased participant community integration, enhanced participant quality of life, and a sustained innovative and collaborative partnership model. 

As the team sees it, the evaluation process offers numerous benefits. The formative evaluation component allows for continuous, time-sensitive improvements in the interventions provided by SWTCIE Illinois. “We aim to understand the effectiveness of the overall SWTCIE intervention and its components in real time,” the team said. “If our evaluations reveal less-than-desirable outcomes, the data helps us identify ‘weak spots’ and determine how to improve specific components.” 

Feedback from partners and stakeholders is particularly essential to the formative evaluation process. The evaluation team takes this feedback and uses it to make intentional change, as necessary. For example, if employment specialists receive training on job accommodation assessment provided by SWTCIE, the evaluation team would speak to those employment specialists to get their thoughts on the usefulness of the training. The team would then use that helpful feedback in determining training strategies for the future, particularly if the feedback sends the message that the current iteration of training needs improving. “We can incorporate [those] suggestions into future trainings or revise the existing ones,” the team said.  

Remaining flexible and intentional in the midst of evaluation efforts is vital in a project of this size and scope with its many interested stakeholders, and it can be difficult to track all activities throughout the project. Utilizing their established evaluation framework and seeking out input from partners has been essential in helping the team to successfully implement and evaluate the SWTCIE Illinois project. These strategies have enabled the team to “stay focused and true” in collecting data in the areas that are important to the field. 

While their work is vital to the success of the SWTCIE Illinois project, the team acknowledges that the tasks they conduct can seem a bit daunting to participants and other stakeholders. “Even titles as ‘evaluators’ can sometimes be perceived as intimidating,” the team said. They emphasize that they are there to help, not to create unease. “We always strive to emphasize that we are not enforcers or looking to highlight deficiencies. Our goal is to improve the project’s effectiveness and enhance the experience and outcomes for everyone involved … we’re nice guys!”  

By Danielle Fields, Ed.D. | contact-iirer@illinois.edu