UCUES, the University of California Undergraduate Experience Survey, is an omnibus survey of the experience of undergraduate students enrolled at the nine University of California undergraduate campuses. Beginning in 2009, six additional public and private peer institutions outside the UC system, all members of the Association of American Universities, joined the project.
Survey results are matched with institutional data from campus records to provide a detailed portrait of students’ backgrounds, academic and co-curricular activities, goals and aspirations, experience with academic and administrative units, self-assessments of gains in academic and social skills, interactions with other students, engagement in community service and civic activities, perceived obstacles to academic success, and many other topics.
The results are used both for administrative purposes—to evaluate and improve programs and policies affecting undergraduates—as well as for scholarly research.
Since 2004, UCUES has been administered annually at Berkeley and even numbered years at the other UC undergraduate campuses. Earlier versions of UCUES date back to 1996.
The precise figure varies from year to year and by campus, but at Berkeley is has been:
50% (11,833 out of 23,904)
51% (11,957 out of 23,278)
48% (10,717 out of 22,430)
52% (11,673 out of 22,450)
The numerator is the number of undergraduate students who logged into the survey system and submitted their responses. The denominator is the population of eligible undergraduates. This formula most closely corresponds to AAPOR’s definition of RR2.
Not every student answers every item presented, and in general some items (such as parental income and open-ended items that require a typed response) are more likely to be skipped than others. One departure from common practice in reporting web survey response rates is that the denominator includes students with inactive email addresses on the assumption that they may have been contacted about the survey through other means such as advertising, direct communications from academic departments and other units, or word of mouth.
Opinions vary greatly as to what makes a “good,” “high,” or “adequate” response rate and what relationship, if any, there is between response rate and the accuracy of the results. While conventional wisdom holds that low response rates yield biased results and that increasing response rates tends to reduce bias, recent research suggests that neither of these claims is necessarily the case. See, for example:
Groves, Robert M. 2006. “Nonresponse rates and nonresponse bias in household Surveys.” Public Opinion Quarterly 70:646-675. [Also see other articles from the same special issue.]
Keeter, Scott, Carolyn Miller, Andrew Kohut, Robert M. Groves, and Stanley Presser. 2000. “Consequences of reducing nonresponse in a large national telephone survey.” Public Opinion Quarterly 64:125–48.
Krosnick, Jon A. 1999. “Survey research.” Annual Review of Psychology 50:537-67.
Visser, Penny S., Jon A. Krosnick, Jesse Marquette, and Michael Curtin. 1996. “Mail surveys for election forecasting? An evaluation of the Columbus Dispatch poll.” Public Opinion Quarterly 60: 181-227.
- It is administered to the entire population (census) of students, rather than a sample.
- It integrates institutional data from university records to provide basic demographic and educational variables such as ethnicity, age, and grade point averages.
- The modular format magnifies the scope of the survey without increasing completion time and jeopardizing response rates. Each student is asked a common core of items about academic engagement and demographics followed by a randomly assigned topical module. Consequently, administrators and researchers have a broader view of the student experience covering not just academics but also civic engagement, moral and psychological development, and experience with campus programs and services.
- The wild card module allows each campus to specify a customized set of important topical items of interest to campus constituents. At Berkeley, recent topics include undergraduates’ connection to UC Berkeley through graduation and beyond (2009), advising and mentoring (2008), diversity (2007), and technology (2006).
- It contains costs very effectively by employing a strictly web-based survey tool and relying on email as the primary form of contacting students.
- It allows analysis at the level of small groups such as majors and specific demographic groups (such as sophomores, student athletes, or student parents). This feature is critical for incorporating UCUES results into periodic assessments of academic departments (particularly academic program review).
- It enables a modular structure to ask a greater number of items about a broader array of topics.
- It yields sufficient numbers of repeated observations of the same students to create multi-year panels with repeated observations for the same individuals.
- All students have chance to participate, promoting a sense of involvement (every voice is heard) and making recruiting and participation a campuswide effort.
All undergraduates are asked a common core of questions about time use, student development, academic engagement, experience with their major (for upper division students), campus climate, and background characteristics (demographics).
Following the core, each student is asked a set of questions from one of several topical modules, determined by random assignment, on a subject such as
- academic engagement,
- civic engagement (politics and community service),
- student development (moral and psychological),
- student programs and services,
- globalization, or
- a “wild card” group of items unique to each campus.
The form of each module, except the wild card, is identical across all campuses in a given year. This arrangement allows greater breadth of topics and more discretion over content without increasing the length of the survey.
- UCUES responses about the academic experience in the major are reported each year to each department for its majors. The results are also analyzed in detail by the Office of Planning & Analysis for academic program review, the periodic in-depth evaluation of every instructional program. Students are asked to rate how satisfied they are with such aspects of their major as instruction by faculty and graduate student teaching assistants, advising, equitable treatment by faculty, and the availability and quality of courses. These responses are also compared to other majors and to prior years’ data to examine trends over time. UCUES data have helped shift academic program review from a predominant focus on faculty and graduate students to one of increasing attention to the undergraduate experience.
- UCUES results regularly inform the decisions of administrators in everyday situations, from answering specific questions (for example, what percentage of students are employed off campus or how many hours undergraduates study per week) to providing a more complete and nuanced picture of the everyday experience of a diverse population of undergraduates.
- UCUES responses are a key component of the UC system’s public accountability initiative, which is akin to the Voluntary System of Accountability template proposed by AASCU and APLU (formerly NASULGC).
- Sociology 105, Introduction to Sociological Methods (uses subset of UCUES data for computer data analysis-intensive course)
- Undergraduate Research Apprenticeship Program (undergraduates conduct original empirical research using UCUES data)
It depends on the items and the time period. Although early versions of UCUES date back to 1996,
the core set of items has remained relatively fixed only since 2006. Some items go back before that time, while others do not. It has been long established that responses to survey items are highly sensitive to even small changes in the question wording or response options. For example:
Sullivan, John L., James E. Piereson and George E. Marcus. 1978. “Ideological Constraint in the Mass Public: A Methodological Critique and Some New Findings.” American Journal of Political Science 22: 233-249.
As a result, tracking non-identical UCUES items across years is unlikely to lead to valid comparisons and is not recommended.
Currently there is no comprehensive database of which items appear in which administrations of UCUES, but the complete set of questionnaires to date
is available at the UCUES main page.
For administrative purposes (evaluating and improving programs and policies) on the Berkeley campus, please contact Berkeley UCUES manager David Radwin. Affiliates of other campuses may contact their respective campus representatives.
The SERU Survey Reporting System facilitates comparisons of individual programs (majors) or groups of programs (departments or colleges) within and between campuses. For example, one might compare English and Rhetoric majors at Berkeley, or biology majors at campus A to biology majors at campus B (see example). The survey results are condensed into a standardized scale with effect sizes and statistical (A is available.)
Currently, the Survey Reporting System is limited to selected institutional research staff at each campus.
For scholarly research, defined as “a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge” (45 CFR 46.102), please see the data access guidelines. UCUES data are collected under a research protocol governed by UC Berkeley’s Committee for the Protection of Human Subjects, and protecting the confidentiality of the research subjects is of the utmost importance.
No, not directly, since a completely different and non-overlapping subset of students is assigned to each module. Items from modules can only be directly correlated with the items from the core. However, advanced statistical methods permit analysts to impute (estimate) the results from one module to use with the results from another module. Three such approaches are described in:
Franklin, Charles H. 1989. “Estimation across Data Sets: Two-Stage Auxiliary Instrumental Variables Estimation (2SAIV).” Political Analysis 1: 1–24.
Raghunathan, Trivellore E. and James E. Grizzle. 1995. “A Split Questionnaire Survey Design.” Journal of the American Statistical Association 90: 54–63.
Gelman, Andrew, Gary King, and Chuanhai Liu. 1999. “Not Asked and Not Answered: Multiple Imputation for Multiple Surveys.” Journal of the American Statistical Association 93: 846–57; followed by comments by John Brehm, David R. Judkins, Robert L. Santos, and Joseph B. Kadane, and rejoinder by Gelman, King, and Liu.
No. UCUES responses are confidential but not anonymous, which means individual student identifiers are collected along with each set of responses, but the responses are never reported in such a way that would allow identification of individual respondents. Non-anonymous data collection also allows survey responses to be matched to individual-level institutional data from university records and across survey administrations to create multi-year panels.
Institutional data, the administrative and transactional data collected by the university, are already collected and maintained at the individual level in application, registration, and degree databases and matched to the survey results.
Among the most important institutional variables are:
- ethnicity
- age
- admission status (transfer admit or new-from-high school freshman admit)
- term and year of admission (used to calculate year in school or unofficial class level)
- units earned (used to calculate official class level)
- date and title of degrees earned (for students who have graduated)
- UC grade point average
- major(s) (e.g. Integrative Biology) and college(s) (e.g. College of Letters & Science)
- location of permanent address
- international student status
- high school grade point average and SAT scores
- high school’s Academic Performance Index ranking
- participation in selected university-sponsored pre-college outreach and preparation programs
A few items already available in institutional data, such as gender, parental income, and parental education, are repeated in the survey to achieve greater completeness and detail. Summary financial aid data are likely to be included in future releases of UCUES data.
For administrative uses, any institutional data that identify individual students by student ID number can be linked to the UCUES data, from a roster of an intramural soccer team to a list of students who graduated from a particular high school or community college to a list of scholarship winners.
There are many good reasons to participate in this research project. To list a few:
- Make a difference. UCUES is your chance to influence policy in your major, your college, throughout campus, and across the UC system. It is the only effort of its kind to systematically collect feedback from all undergraduates about what is working well, what is not working well, and what is important to you. Your responses will be used by faculty and staff in a variety of ways to help evaluate and improve the experience of current and future undergraduates.
- Be counted. As with voting, your opinions do not count unless you participate. Administrators need to hear from a broad array of students to obtain an accurate and well-rounded perspective.
- It’s easy. It only takes about twenty minutes to complete. You don’t even have to leave your computer.
- Win great prizes. Each year we give out thousands of dollars’ worth of prizes, from iPods to cash prizes to bookstore gift cards, to a lucky group of randomly selected participants. The earlier you complete the survey, the more chances you have. But you have to participate to win.
All regularly enrolled undergraduate students eighteen years or older on April 1 of the year of the survey are invited to participate. A very small number of limited enrollment/2nd degree students and minors are excluded from the eligible population. (The excluded group constitutes less than one percent of the total undergraduate population.)
UC Berkeley affiliates may contact Berkeley UCUES manager
David Radwin or Principal Researcher
Gregg Thomson in the
Office of Student Research and Campus Surveys. Affiliates of other UC campuses may contact their respective
campus representatives.
UCUES is governed by a research protocol approved by UC Berkeley’s Committee for the Protection of Human Subjects. Each of the other UC campuses submits this protocol to its respective institutional review board with an intent to rely on CPHS’s approval.
UCUES is a collaborative effort overseen by representatives of nine UC undergraduate campuses, the UC Office of the President, the Center for Studies in Higher Education at UC Berkeley, and affiliated researchers. The project is guided by four principal researchers, directed by the SERU/UCUES Project Director, and managed by the UCUES Project Manager. The survey is administered by UC Berkeley's Office of Student Research and Campus Surveys.
Conducting surveys is very easy to do but very difficult to do well. One suggestion is to look at past administrations of UCUES, SONS, and other OSR surveys to see if perhaps your topics have already been covered. If so, you may not need to conduct a survey at all, even if you are only interested in students from a certain major or other group. Instead, you may be able to use existing survey data.
If you do need to conduct a survey, you might start by consulting one or more of these sources:
- Dillman, Don. 2009. Internet, mail, and mixed-mode surveys: the tailored design method. 3rd ed. Hoboken, NJ: Wiley & Sons.
Comprehensive and research-driven book covering nearly all aspects of survey administration, including defining a universe and sample frame, sampling, constructing a questionnaire, and increasing response rates. Earlier editions may be helpful as well. Aavailable in the UC Berkeley campus libraries.
- Converse, Jean M. and Stanley Presser. 1986. Survey questions: Handcrafting the standardized questionnaire. Thousand Oaks, CA: Sage.
Thin monograph only covering questionnaire design is dated but still addresses many important issues. Available online to the UC Berkeley community.
- Krosnick, Jon A. 1999. “Survey research.” Annual Review of Psychology 50: 537-67.
More academic than practical, but offers interesting perspectives on such topics as response rates, open-ended versus closed-ended questions, and various sorts of response bias. Also contains practical advice on such topics as labeling scales and whether to include a “no opinion” response option.
- Porter, Stephen R., ed. 2004. “Special Issue: Overcoming Survey Research Problems.” New Directions for Institutional Research 121:1-97.
Contains articles by several authors on topics related specifically to surveys in colleges and universities.
- The American Association for Public Opinion Research (AAPOR) offers a concise list of best practices for surveys, a FAQ, and a more general set of resources.
Some sources require proxy server for UC Berkeley affiliates connecting from off campus.
Looking for software like SurveyMonkey? This comparison of selected web-based survey tools details their accessibility issues, features, and pricing.
The
SERU Project is a collaborative study focused on developing new types of data and innovative policy relevant scholarly analyses on the academic and civic experience of students at major research universities.
Based at the
Center for Studies in Higher Education at UC Berkeley, SERU is a collaborative project between academic and institutional researchers, and it includes the development and administration of UCUES.
The objective of the SERU project is to:
- develop new data that can assist policy development and improve the undergraduate experience; and
- create a new resource to help promote scholarly research and reflection in this field.
The project’s initial focus was on the University of California system with expansion to selected AAU member institutions beginning in 2009. SERU is also the project name used for UCUES administrations outside the UC system.
