UBC Metrics and Evaluation Need Attention of Its Stakeholders

At first glance, a holistic metrics development and implementation would appear to be a logical step for improving quality and extent of university business cooperation (UBC) activities. Yet, the recent research shows more work needs to be done not only at institutional, but also at the regional, and national policy levels towards generation of comprehensive qualitative and quantitative monitoring systems, addressing both short and long term UBC results. Development of such systems requires involvement of all UBC actors, which would allow improvement of the key measures, as well as increasing stakeholder engagement during the process.

Recently, a consortium led by the Science-to-Business Marketing Research Centre and involving among others UIIN has conducted the largest survey yet undertaken on the state of university-business cooperation in Europe. Here university managers participating in the study reported a high extent of development when it comes to their paper-based strategies – as first step into institutionalizing UBC – in contrast to actual monitoring and assessment practices of UBC performance, which is considerably less developed. Similar results were found in the good practices published as part of the same study, with only a limited number of good practices developing and implementing performance measures. Amongst those who adopted a certain level of monitoring and measurement system, the majority was found to neglect the long-term impact assessment, and focused on only short-term quantifiable aspects of UBC. For the universities involved, this translates as measuring the extent of R&D funding, patents/licences, spin-outs and start-ups, while for businesses, it means focusing efforts on immediate problem-solving results and revenue from new products generated through R&D instead of a longer-term approach.

Given that many of the benefits flowing from UBC take full effect in the long-term and in less quantifiable or less attributable ways, much of the impact of cooperation is difficult to capture. This factor alone makes a holistic approach to UBC less accountable to policies and strategies, and puts pressure on university and business managers to deliver shorter term, tangible and measurable benefits. More, relying on only quantifiable measures paints an incomplete and inaccurate picture of the impact of UBC activities for both university and business, which reduces the quality of the cooperation to mediocre standards.

This leads us to the question whether there is an ideal way of measuring the impact of UBC, or if, with the involvement of all stakeholders, there should be a mixture of measurements applied from varies perspectives to capture this?

Activities and measurements governed at the national level

The Higher Education Funding Council for England (HEFCE) performs monetary and non-monetary impact assessments emerging from HEFCE funding for knowledge exchange, and the use of Higher Education Innovation Funding (HEIF) via online surveys and interviews. The 2015 report “Assessing the economic impacts of the Higher Education Innovation Fund: A mixed-method quantitative assessment” provides some evidence to the policy makers whether the investments tied to the policies are fully exploited, and if re-allocation is necessary to address market and system failures.

Image 1: Mixed method approach for exploring impacts if HEIF

 The model takes the level of knowledge exchange income generated by the HEIs as proxy for economic impact, which carries the risk of favourable results obtained from research intensive universities. This challenge is overcome by the evaluation of non-monetary impact of HEIF funding. This evaluation includes collecting feedback from businesses and social and community groups on the benefits they received from working with universities. Although the measurement yields some insights into the impact of UBC, it is definitely not exhaustive and requires additional quantitative results to measure more the long-term results of UBC activities. For example, start-ups coming out of universities are a great way to extract knowledge from university. However, are those start-ups still around after 5 years, and how many jobs have they provided? Did those start-ups stay in the region, or move out to other regions or nations?

Activities and measurement at unit level

The business incubator iAccelerate of the University of Wollongong sets a good example on how to combine range of metrics diverse in nature, when it comes to UBC progress and impact measurement. Some of the quantitative assessment metrics adopted by the institution include number of event attendees who apply to iAccelerate Programmes, successful applications to the iAccelerate Start and Advanced programmes, number and background of founders, jobs created, number of employees, revenues, new investments, number of student internships, any awards or grants, and press releases.

To complement this numeric information, iAccelerate keeps feedback channels open with their clients as stakeholders via a number of modes, that include an annual client satisfaction study, feedback from iAccelerate advisors at the Monthly Advisor meetings, ad hoc discussions, review of workshops, and presentations with the CEO and GM start-up company progress that could be tracked online in incuTrack.

In the example of University of Twente, comprehensive monitoring and evaluation systems are in place to measure the impact of entrepreneurial activities, yet, the impact is mostly seen in the tangible results obtained as result of quantitative assessments. Most indicators are focused on quantifiable outcomes, such as number of spin-offs, collaborations with the industry, and percentage of third-party funding, as part of the performance agreement with the Dutch government, without any input provided for the project quality or stakeholder satisfaction.

While this being the case at the institutional level, the two individual programs TOP and Venture Lab adopt a combination of quantitative and qualitative assessment methods to foster the quality and impact of the programs. The business start-up program TOP is regularly monitored by the progress of the start-ups once per year, looking at survival, employment, regional base, need for further support. This involves per year at least three contacts/meetings between Novel-T staff and TOP graduates. Similarly, the MBA level business development program VentureLab implements an extensive monitoring and quality management tool that includes participants using a diary system and an initial intensive assessment of 3-4 hours. Every few months a follow-up survey is administered, complemented by an exit survey before graduation.

Measuring UBC and its impact is complex

Although a straightforward and comprehensive method for the measurement of UBC and its impact is yet to be created, the current methods show potential when it comes to assessing UBC on both a national and institutional level. One of the key issues in measuring the tangible and intangible impact of UBC, is the lack of access to data and a centralised approach. Going forward, there needs to be a mixed-method approach driven by regional policy makers. Through engaging the relevant stakeholders on a regional level in the collection of data, this can be aggregated on a national and international level to provide greater insights into the activities. Here, a mixture of both quantitative and qualitative metrics, as well as a short and long-term perspective in these measurements would provide policy makers, universities and business alike with greater insights into the actual benefits of UBC.