IV. Introduction & Overview of Project Initiative

a. Project Description

Rather than putting the burden of career readiness on students, colleges need to make themselves ready to support students into, and through, the college to career transition. Improving career outcomes for graduates and making them more equitable requires a significant redesign of current campus systems as well as new solutions to address the challenges facing today's students. A significant misalignment exists in perception exists between how campus leadership and business leaders viewed student career outcomes. In 2018, 96% of chief academic officers believed they were effectively preparing students for the workforce, compared to just 11% of business leaders who agreed. In addition, according to Strada Education Network’s Institute for the Future of Work research, 85% of freshman students say their main priority for attending college is to get a good job, however, only 27% report securing a good job before graduation, and 4 in 10 college graduates are underemployed in their first job.

Over a period of three years (2018 - 2020), University Innovation Alliance (UIA) institutions utilized change management and human centered design principles to identify and deconstruct barriers to successful career readiness for first-generation and low-income students. The 2018-2020 Bridging the Gap from Education to Employment initiative aimed to improve career outcomes for the most vulnerable students by reimagining the college-to-career pathway. The initiative engaged career readiness teams on seven UIA campuses in an intensive landscape analysis process, empathy work with student and employer stakeholders, and building and piloting innovative solutions.

This initiative occurred over three phases:

As the culmination of this creative and collaborative process, all seven participating campuses developed and launched pilot tested interventions, in partnership with employers, to address the discovered needs of today’s students. These pilot projects included paid cohort-based internships, faculty programs focused on career development for historically marginalized students, incorporating career competencies into curricula, new experiential learning projects, and scalable career exploration courses. The rigorous assessment, learnings, and outcomes of these interventions provided proven recommendations for the most promising practices and career readiness solutions.

To sustain impact and progress, campus teams selected the most promising practices and interventions from all participating campuses for implementation beginning in January 2021. These selections were required to work in concert, and be thoughtfully included in a robust implementation plan.

b. Participating Campuses

Due to a higher-than-anticipated interest in the BGEE project, participating campuses were expanded from three to seven UIA institutions. In order to accommodate this expansion, a regional implementation and coordination model was developed. Each region received management, support, and day-to-day facilitation from a BGEE project fellow. The BGEE Design Lead provided oversight to all campus and project-specific activities.

• Ohio State University
• Purdue University
• Georgia State University
• University of Central Florida
• Arizona State University
• Oregon State University
• University of California, Riverside

c. Composition, Diversity of Campus Teams

Each participating campus was charged with identifying a "Team Lead", preferably a career services professional, who in turn was given the responsibility of recruiting and forming a cross-campus team of diverse professionals who spanned roles and departments within the institution.

An example of BGEE campus team composition from a participating campus:

• Director, Career Services (team lead)
• Director, Career Services, College of Arts and Sciences
• Director, Career Services, College of Engineering
• Director, Career Services, College of Business
• Faculty member
• Research Associate
• Career Services staff
• UIA Campus Fellow
• Undergraduate Student(s) (for temporary periods throughout the project)

d. Understanding Current Practices

Gap Analysis Across the Fields of Higher Education and Workforce Development

During Phase I foundational activities in 2018, the UIA completed a Gap Analysis to provide a comprehensive overview of the education-to-employment landscape, across higher education. This analysis identified three major trends:

1. A growing number of institutions struggle to help their graduates translate educational experiences into economic opportunity.
2. Campus-employer connection tools, such as VMock and Handshake provide opportunities for scaling access and information to students.
3. Most campuses are overlooking additional tools and platforms, including LinkedIn, that can be beneficial to students. Employers are using these tools to bypass higher education and career services and connect more directly with students.

The report’s findings concluded with five promising practices for engaging first-generation and low-income students in career preparation:

1. Intentionally utilize student employment on campus as a career readiness and skill-building opportunity, which is often critical to low-income students.
2. Integrate career preparation throughout students’ university experience, increasing access so that all students engage with these activities early and often.
3. Collaborate with other support services, such as academic advising and office that maintain critical student data on campus (such as first-destination surveys, Pell grant recipients, etc).
4. Expand students’ horizons by exposing them to new jobs and connecting them with networks and professionals.
5. Don’t forget about the bare essentials, such as mentorship and career closet resources for interview preparation.

e. Understanding Current Practices: Landscape Analysis & Baseline Data

Completed during Phase I activities, the Landscape Analysis and Baseline Data reports summarized key findings and promising practices on each participating campus. Each campus’ team gathered baseline data, completed introductory process mapping workshops, and conducted a career-related activity inventory. These activities complemented the broad Gap Analysis Report, providing a narrower description of the BGEE project’s baseline for participating institutions. They also served as a building block to highlight pain points captured during the foundational assessment period of the project.

Process Mapping Framework

UIA Fellows facilitated introductory process mapping workshops to understand the current landscape of career services offerings on each participating campus. These workshops documented current strengths, areas in need of improvement, promising practices, and services specifically offered to low-income, first-generation college students. They also drew on the findings of the Gap Analysis Report to provide broader context to the career readiness challenges facing students, universities, and employers.

Process Mapping Workshop Topics:

1. Defining Career Related Activities for Your Campus
2. Gathering Campus Career Activities
3. Building a Strengths Model via a Campus Landscape Analysis Summary

Key Findings

Four themes emerged from the baseline and inventory data:

1. Reimagining Career Services
2. Employer Engagement
3. Communicating Marketable Skills
4. Activities Designed Specifically for Low-Income, First-Generation Students

The Gap Analysis Report noted that campus career services have a critical role to play in closing the gap that currently exists for many students between education and employment. They stand at the forefront of the transformation that is needed to break down barriers that prevent students from finding meaningful work. Campuses must embrace new technologies and create new innovative ways to ensure students enjoy a bright and rewarding future after graduation. This call to action required a reimagining and redesign of traditional "career services" activities and experiences.

Employer Engagement

In 2018, campuses were beginning to think more broadly about creating truly robust partnerships with employers. While employer connection tools such as Handshake are utilized on many campuses, some employers are bypassing career services connections by using tools that connect them directly to students. Reimagining employer engagement beyond traditional activities such as career fairs, interviews, and networking events were necessary to provide employers with the robust relationship and reliable pipeline of skilled graduates they crave.

Students continue to struggle with communicating the non-tangible skills they gain during their college experience. This challenge is accentuated by a lack of consistency in the way universities and employers describe similar skills, or even which skills businesses are looking for in today’s ever-changing job market.

Career preparation and readiness—like higher education more broadly—has struggled with issues of access and opportunity for those from traditionally disadvantaged backgrounds, including first-generation and low-income students. Many career services experts, especially those at institutions of higher education, note that while there are a few innovations and new practices aimed at helping these populations, there has been a dearth of attention focused on, and best practices developed for, reaching these students in a robust manner. Many participating campuses called attention to this area as providing an opportunity for improvement.

Moving Beyond Understanding Current Practices

This initial assessment of college-to-career pathways across all seven participating UIA institutions was a critical step in the BGEE initiative. Institutions developed a deeper understanding of the career-related work happening on their campuses, identified several promising practices, and noted areas for both individual improvement and collective action.

Continued challenges highlighted in the landscape analysis and baseline data reporting were:

• Adopting best practices for streamlined data collection and tracking
• Developing collective metrics
• Establishing and fostering on-campus collaboration
• Collaborating directly with students to create innovative solutions

The second phase of the BGEE initiative involved iterating the project evaluation plan based upon foundational findings. Campuses solicited input from employers to strengthen the college-to-career pathway. They also identified, built, and facilitated a realistic framework to address the challenges the project faced, with a goal of building upon promising practices to advance career readiness solutions from transactional to meaningful and to progress from incremental innovation to transformational.

f. Evaluation Methodology/Approach

The initial phases of the BGEE project’s metrics of success tied to campuses’ current career readiness activities (such as career fairs, engagement at career centers, and job placement rates). Within the first year of the project, the BGEE team recognized several limitations with these metrics:

1. There was no baseline data to use for comparison. For example, it was discovered that most campuses did not have data on the number of students who participated in career services activities and services nor did they disaggregate data by student demographics. Furthermore, campus teams did not have the capacity to collect the necessary data.
2. The BGEE project aimed to generate new solutions. For these reasons, the BGEE UIA team needed metrics of success that would not presuppose solutions, but instead allow for the iteration and flexibility that would yield game-changing innovations. Specific metrics of success would depend on the innovations that the teams generated.
3. The length and focus of the project necessitated being able to measure impact in the short-term. Because the Because the BGEE project was committed to testing solutions before implementation and scale, all testing was to occur within very short time-frames and build upon itself, while interventions would not be expected to immediately show campus-wide results.
4. Transformational impact requires culture change. The original project metrics provided no structure to understand and measure the impact on the teams and institutions themselves, such as improvement in collective impact or institutional capacity to improve career readiness and student career outcomes, thus missing a key opportunity to understand and learn from the impact of the project.

For these reasons, the BGEE project created a new measurement and evaluation plan based on developmental evaluation, an emerging approach tailored to social innovation, to guide strategic learning throughout and beyond the project.¹,² Rather than traditional evaluation methods that assess adherence to planned program activities and the degree those activities lead to expected outcomes, developmental evaluation allows for the agility needed to identify and test innovative solutions in complex systems.

BGEE developmental evaluation was designed with the following goals: (1) understand the impact of the BGEE project on participating institutions; (2) identify innovative solutions to improve career preparation for first-generation and low-income students through intervention-specific metrics; and (3) guide other institutions hoping to use design thinking to develop collective impact solutions.

To achieve these three objectives, the BGEE project answered five research questions. See below for research questions and methodology used for each.

• What programs, policies, or resources have shown the greatest potential to improve career outcomes for first-generation, low-income students?

  All campuses implemented a developmental evaluation plan for their small-scale pilots, building logic models, identifying short-term outcomes, and developing data collection plans to determine the extent to which pilots yielded expected outcomes. Teams reported pilot outcomes in final reports and presentations.

  In collaboration with the design firm IDEO, the UIA developed an assessment rubric to identify which interventions demonstrated potential to improve career outcomes for first-generation and low-income students. Based on project learnings, the UIA defined “potential to improve career outcomes for first-generation and low-income students” as interventions that (1) Foster collaborative change across campus; (2) demonstrate early evidence of transformative impact on target populations; and (3) are likely to be sustainable across multiple universities

• What did we learn from the concepts prototyped, tested, incubated, and piloted on each participating BGEE campuses?

  Campus teams used human-centered design techniques including empathy interviewing and rapid prototyping and testing. Learnings were synthesized in various artifacts throughout the project, including (1) concept posters at the November 2019 Retreat; (2) pilot proposals; and (3) pilot report & presentations. These learnings are synthesized in the ‘key ingredients’ described in Section VIII.

• To what extent did the BGEE Project increase campuses’ capacity to prepare low-income, first-generation students for their careers?

  To measure the impact of the project on campuses’ institutional capacity, the BGEE team conducted surveys to measure campuses’ capacity to prepare first-generation and low-income students for their careers. Impact was measured based on comparison between surveys conducted at the middle and end of the project for ~about 30 BGEE participants involved in the project for its duration. In addition, responses from program participants involved in the last year of the project were analyzed to understand the project’s impact.

  The UIA defined institutional capacity as a core set of strategic practices and mindsets that would enable institutions to prepare first-generation and low-income students for their careers. Institutional capacity was first defined as strategic practices that aligned to five elements of institutional capacity, as defined in Achieving the Dream’s [tool]: leadership & vision, data & technology, equity, engagement & communication, and policies and practices.

  In partnership with the design firm IDEO, the UIA also articulated "creative change mindsets" during the last year of the project. While the timing of these metrics prohibited pre/post comparison, the team facilitated group reflection exercises to identify each campus’ strengths and areas of improvement.

• To what extent and in what ways do the participating campuses fulfill conditions of successful collective impact initiatives?

  Project surveys and audits of project activities were used to assess the BGEE project on a series of metrics that correspond to conditions necessary for collective impact: common agenda, backbone infrastructure, shared measurement system, mutually reinforcing activities, and continuous communication.

• What are effective principles of practice in implementing design thinking to develop collective impact solutions within and between institutions of higher education?

  Based on project activities and survey results, the UIA developed a model for Student-Centered Collaborative Change, outlined in Section 13

In alignment with the principles of developmental evaluation, the UIA adapted and updated project evaluation as needed to ensure meaningful measurement and learning throughout the course of the grant. To see the original evaluation plan, please refer to this link.

1. Patton, M.Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press; cited from Preskill, Hallie & Tanya Beer, Evaluation Social Innovation, Center for Evaluation Innovation, FSG, 2012
2. For more information about developmental evaluation, see Preskill, Hallie & Tanya Beer, Evaluation Social Innovation, Center for Evaluation Innovation, FSG, 2012

Designing for Transformative Change