YUFA
External
Conference Report on OCUFA Conference – Accounting or Accountability in Higher Education
23-24 January 2009, Toronto, ON
By Sharon Murphy, Faculty of Education, York University
13 Feb 09 – The conference consisted of three keynote addresses and three panels of three members each. While a considerable amount of specific information was provided, I have chosen to focus on a number of themes that could be drawn from the conference in making a report. In addition, it should be noted that a number of the papers and PowerPoint presentations will be available on the OCUFA Website by the beginning of February, 2009. Conference proceedings will also be published some time in the future.
Theme 1: Universities need to be able to talk about what they do and the consequences of what they do in a manner that is clear to others both now and in the future
Underlying question: Why are we puzzled about this state of affairs?
Background issues: Our puzzlement stems from the complexity of living in contemporary knowledge-based societies characterized by the following:
· Rapid massification of post-secondary education (including expansion of number / types of institutions offering credentials),
· Globalization,
· Marketization,
· Knowledge-driven economy,
· Public’s increasing knowledge in general (aided by technology),
· Massification’s influence on the position of trust that universities have held.
Future perspectives: Institutional autonomy will mediated by webs of negotiated descriptions of activities at program, institutional, national and international levels.
Theme 2: Universities have always provided information about activities but purposes for data-gathering, the form it is delivered in, and the uses to which it is put will inevitably change
Underlying questions: Is there any sense to the investment of time and money put into data collection? If change has always occurred, what is different about these circumstances?
Background issues: Confusion is caused by several factors:
· Different terms (accountability vs. reporting, performance indicators, learning outcomes, etc.) are understood differently.
· What is currently reported varies considerably in terms of its significance. Numerous reports are generated but some reports are quite technical and not highly meaningful. Other reports are a response to legislation some of which has been overtaken by new legislation (even though the requirements to report on the old legislation have not been rescinded). The level of information – program, institution, provincial, federal etc. – affects the meaning it will have.
· There is an over-attentiveness to some consequences of data reporting and inattentiveness to others.
· Data sets that are available within institutions often do not meet the needs of external agencies.
· Purposes for data collection are quite mixed (fiscal responsibility – no embezzlement, fraud etc.; relative value for dollar to different stakeholders requires different data sets; nature of student experience; research accomplishments).
· A market mode is driving data requests.
Future perspectives: Over time, moves will be made toward standardization of types of data collected. Some examples of this are already emerging such as CUDC which will be expanded to CUDO latter this year. Other world regions are engaging in somewhat similar activities.
Theme 3: Transparency need not be feared
Underlying assumption: Universities’ reactions to requests from varied sectors is motivated by fear.
Background issues: While the issue of university autonomy is certainly one worry of universities, reasons for the reluctance to participate in data collection / reporting requests include:
· There is a lack of administrative structure devoted to data collection and reporting.
· Data are not collected on the questions of interest to some stakeholders; in essence, the traditional administrative functions of the university are required to be augmented by self-study and institutional research functions (and staff).
· There is a lack of consideration of unique contexts for each institution with the result that the aggregated data when presented are misunderstood.
Future perspectives: Administrative structures will be put in place or augmented to serve increasing informatics demands from various stakeholders. If universities have an opportunity to be involved in outlining the nature of the informatics (types of questions, context specificity, qualitative as well as quantitative data), then the process will be more palatable (and perhaps useful) to all parties.
Theme 4: Many ranking systems have had a relatively rapid rise in significance
Underlying assumptions: Metrics have substantial foundations that warrant the consequences of the interpretations made.
Background issues: No explanations are offered as to how ranking systems gained such psychological weight except perhaps an implied suggestion that a void plus good timing allowed the gap to be filled. Issues around the use of these metrics include:
· Institutions worry about being in the top 100 of the Shanghai index; yet, it lists 17,000 institutions so institutions in the top 500 are actually in the top 3%. The issue of scale and the meaning of rankings can get lost.
· The percentage of students who look at rankings is always greater than the percentage of students who use them; percentages vary from jurisdiction to jurisdiction.
· Newspapers and magazines are quick to acknowledge the flaws in their methodologies but continue to publish rankings.
· When institutions use rankings published by media and other sources to report on the measures they will take to improve the ranking, such uses are taken as evidence of the validity of the rankings.
Future perspectives: Rankings are so influential in some countries that the countries have developed their own internal systems. Therefore, one is likely to see an increase in ranking listings. One possibility not mentioned by participants is that the proliferation of rankings may well lead to the devaluation of them. Also, no substantive discussion occurred over the issue of statistical vs. practical differences inherent in rankings (e.g., Shanghai’s top 500) and the possibility that a different metric that rank ordering would offer more useful information.
Theme 5: More consideration must be given to responsible data generation / use.
Underlying assumptions: Some current methods of data generation or uses are misrepresentative.
Background issues: The ethics of data generation lie both with the institution and the stakeholder to whom the institution is reporting. On the one hand, stakeholders need to be responsible about the amount and nature of the data being requested while institutions need to be responsible about the manner of data collection.
· Because of high stakes associated with some rankings / reports in the U.S., some institutions are massaging the data reported to achieve a better result. Ranking rather than quality is the focus.
· The consequences of particular uses of data need consideration.
· Sometimes a unit of analysis (program, institution, province, country) is inappropriate to the use for which it is put.
Future perspectives: Consideration might be given to creating an external agency responsible for monitoring the quality assurance project from all sides in terms of fairness and responsible actions.
Theme 6: Einstein’s advice is instructive: “Not everything that can be counted counts and not everything that counts can be counted.”
Underlying assumptions: Institutions are being asked to provide data on silly things and some of the more important things are left undocumented.
Background issues: The fondness for quantification appears to be based on a business model. The problem is that the model is insufficient at best and inappropriate at worst.
· One example provided of a case where, because of a small drop in attendance at workshops on teaching, a unit had to demonstrate what it would do to remediate the drop from 945 to 900 in annual attendance (in part caused by reduction of faculty numbers, a shift in faculty workload due to administration and some rescheduling of workshops). The reporting system required remedial measures to be taken for failure to meet targets. The remediation was to conduct survey on faculty needs but the problem was that in order to conduct the survey workshops had to be suspended because there were insufficient staff to both do the survey and run the workshops – in short, counting surpassed explanation and the result was the suspension of the very program that was supposed to improve quality.
· Many items for which quantitative data are provided are, at best, proxies for quality. An example would be simply counting publications of faculty members rather than considering the significance of the contribution.
Future perspectives: Many presenters spoke of the need for relevant data to be collected, for fair judgments to be made on the basis of such data and for feedback to be a part of the process. The need for co-operation and collaboration between stakeholders and institutions with respect to designing data reporting schemes was reiterated as well as the need for schemes that somehow recognized that while there might be some comparability across institutions, there was a need for data to capture what made institutions distinctive as well. Institutions must move away from reactivity and be more proactive in formulating what is significant. A shift in language away from “measuring” to “documenting” and “demonstrating” might be also useful although most participants at the conference spoke about measuring which seems to align with quantitative models of evaluation.
Other Delegates from this Conference:
Julie Clark
Leslie Sanders
Richard Wellen