Essay One: Defining the Meaning of the Degree and Ensuring Integrity, Quality, and Rigor
What is the “Meaning” of the Degree for UH Hilo?
He ‘aha ke kuanaike o nā kēkelē o kēia kula nui?
The “meaning” of our degrees at UH Hilo flows directly out of our Strategic Plan. The most recent iteration, drafted in 2010 for 2011 to 2015, includes our mission statement:
A‘ohe pau ka ‘ike i ka hālau ho‘okahi/One learns from many sources
The purpose of our university ‘ohana/family is to challenge students to reach their highest level of academic achievement by inspiring learning, discovery and creativity inside and outside the classroom. Our kuleana/responsibility is to improve the quality of life of the people of Hawaiʻi, the Pacific region and the world1.
This statement defines what it means for a graduate to hold a degree from our institution in accordance with the guideline stated for WASC Criterion for Review (CFR) 1.1: “The institution has a published mission statement that clearly describes its purposes. The institution’s purposes fall within recognized academic areas and/or disciplines2.” “Challenging students to reach their highest level” while also “improving the quality of life for the people of Hawaiʻi, the Pacific Region, and the world” requires an active partnership that links the University with our external constituencies, including local businesses and community projects.
Examples that demonstrate how our commitment to ensuring student learning has a direct impact on the community follow:
Under the direction of the Office of the Vice Chancellor for Academic Affairs, a group of faculty from Math, English, and History are directly working with the Hawaiʻi Department of Education (DOE) to improve the quality of secondary education and the preparation of incoming students:
A team of three UH Hilo faculty (two from the Math Department and one from the School of Education) are leading an effort to implement the Common Core State Standards (CCSS) in high school math for the entire state. Working with DOE math specialists (OCISS - Curriculum and Instruction Branch), the UH Hilo team formed two DOE author teams of top high school teachers to assist in writing new and innovative curricula for both Algebra I and Algebra II as there are currently no acceptable textbooks written for the CCSS. The team scheduled and implemented initial training sessions on three islands (O'ahu, Hawaiʻi, and Kaua'i) for 200+ Algebra I and II teachers. The training sessions introduced the entire first-quarter curriculum, lesson by lesson, for public school teachers. Similar training sessions are scheduled for each quarter throughout the 2013-14 academic year. The team expects the training efforts, combined with the widespread statewide use of the new curriculum, to have a noteworthy impact in bridging the gap between high school and college expectations in math. The team will begin a similar effort for developing geometry curriculum beginning in December of 2013, thereby completing a comprehensive Core Math implementation effort for high schools statewide.
The English Department, through its Droste Endowment, recently hosted an island-wide writing symposium that introduced 40+ secondary English teachers from around the Island – Ka‘ū (Pahala), including Kealakehe (Kona), Kea‘au, Honoka‘a, Kohala (Kapa‘au), Waiākea, and Hilo – to UH Hilo’s new General Education Rubrics and assessment techniques for writing. The secondary English teachers received free professional development credits, certified by the DOE, for this event. Several members of the department are now part of the State Network of Educators (SNE), which is developing a digital library for the Smarter Balance Assessment Consortium’s Digital Library, which will serve as a repository for CCSS-based formative assessment activities and strategies for all public schools across 22 states, including Hawaiʻi.
The History Department, which has hosted the Big Island district History Day on a yearly basis, partnered with Kilohana: the Academic Success Center and the system P-20 Office to offer a one-day workshop on aligning the National History Day Project and existing curriculum to Common Core State Standards for 18 secondary social studies teachers (from such schools as Pāhoa, Konawaena, and Waimea) and 4 teachers-in-training in the UH Hilo Education Program. Teachers were paid a $100 stipend for their efforts through a GEAR-UP Hawaiʻi Grant.
Ka Haka ‘Ula o Ke‘elikōlani or College of Hawaiian Language (KH‘UOK) is continuing their work as part of a comprehensive approach to meet the Hawaiʻi State Constitution Mandate (Article X, § 4) to “promote the study of Hawaiian culture, language and history” through its preschool to grade 12 laboratory schools, with current enrollments of 500 students statewide3. As the only P-20 indigenous language revitalization institution in the United States, KHʻUOK has developed comprehensive programs, partnerships, and other links to meet the demands evidenced in its growing teacher’s program that helps staff Hawaiian language immersion schools that currently enroll 2100 students on all the major Islands. Their work with and membership in Ku‘ikahi ‘Ōlelo Hawaiʻi – the Hawaiian Language Consortium – has been touted as a model and as a source of invaluable technical assistance to indigenous educators worldwide. Indigenous educators from more than half of the APEC economies have visited Consortium sites to study the preschool-through-doctorate level Hawaiian-medium education system in an effort to replicate the Consortium’s success4.
2. The Hawaiʻi Island Pre-Vet Club
Directed through the College of Agriculture, Forestry, and Natural Resource Management (CAFNRM), the Pre-Vet Club, which consists of roughly 25 students, maintains close ties with non-profit organizations such as the Hawaiʻi Humane Society, Rainbow Friends (Kurtistown), Hui Pono Holoholona (Mountain View), and Tasha ‘Ohana to provide for the maintenance of feral cat populations through a spay/neuter and release program. What started as a committed effort to manage the population of feral cats on campus, the Pre-Vet Club successfully implemented “best-practice” methods adopted by institutions like Stanford to achieve a threshold of 80% neutered or spayed within the first three years of operation. The program has been successful on campus and has now expanded its operation to include Wailoa Park in Hilo. This program – which includes Animal Science 490, the Animal Science Internship – is providing students with the opportunity to document the 500+ hours needed for successful application to veterinary school.
3. The Hawaiʻi Cetacean Rehabilitation Facility (HCRF)
Located at UH Hilo’s Pacific Aquaculture and Coastal Resource Center in Keaukaha, HCRF “is the only one of its kind in Hawaiʻi and the Pacific region dedicated to the humane care and treatment of sick, injured and out-of-habitat whales and dolphins. It is also one of only three such centers in the United States operated by a University5. The program partners marine science students (100+), marine biologists, veterinarians, NOAA, the Coast Guard, and a larger 220-member team called the Hilo Marine Mammal Response Network.
“Adopt-a-Beehive with Alan Wong” is a partnership between UH Hilo and noted international entrepreneur and Chef Alan Wong (owner of Alan Wong’s, The Pineapple Room, and Amasia). Much of this effort is in response to the varroa mite which began devastating local bee populations around the State. “Adopt-A-Beehive” supports undergraduate research (in Entomology and Agriculture classes such as ENT O262, ENTO 304, ENTO 350, ENTO 374, and AG 205) and the development of sustainable beehive practices in the State while also educating the community about bees and the kinds of products that can be made with honey. The program has reached out through visits to local public schools (such as E.B. deSilva Elementary in Hilo) and has also made an appearance in Kona at the Taste of the Hawaiian Range fundraiser, which aims at showcasing the agricultural products of the Island6. Much of this work is done by the approximately three dozen CAFNRM students at the Farm Laboratory in Pana‘ewa.
5. “Local First”
Sodexo and the UH Hilo Cafeteria (in partnership with HAWCC) launched what is called “Local First” in 2011, which are special days when the menus feature cuisine made from food grown on the Big Island. More importantly, “60 percent of all the food served there is locally grown and produced[,] and on the first Wednesday of each month, 100 percent is local7.” The program showcases local products and thereby directly supporting a host of small businesses on the Island, including: Hamakua Mushrooms (Laupāhoehoe); Hamakua Springs (Pepe‘ekeo); Lone Palms (Kapa‘au); Nakamoto Farms and Kekela Farms (Waimea); Berger’s Kama‘āina Farm (Mountain View); Royal Kona Coffee (Kona); Natural Pacific (Kea‘au); and Suisan Fish Market, Maebo Noodle Factory, Atebara Chips, and Kulana Foods (Hilo). General Manager Bridgette Awong’s efforts on food sustainability lead to the Chancellor’s Certificate of Recognition in 20128. Such efforts on the part of Dining Services help to educate students about the importance of local agricultural production as well as sustainable food practices.
6. The Performing Arts Center of UH Hilo
The Performing Arts Center and the Performing Arts Department hosted 68 events that drew community attendance of just under 29,000 in AY 2012-2013 alone. Those events included: Joe Turner’s Come and Gone (November 8-11, 2012), the Festival of One Act Plays (November 21, 2012), Haunt! (April 11-14, 2013), Obake Tales and Ancient Hawaiian Myths (April 13), the Great Leaps Dance Concert (April 27, 2013), Opera Day (May 1, 2013), the UH Hilo Jazz Orchestra Concert (May 2, 2013), and the Clarence Waipa Memorial Scholarship Concert (May 5, 2013). Many of these events not only showcase important local and Native Hawaiian culture, but provide community access to larger national and international material. In addition, the Department hosted six Senior Projects (required for graduation) involving a dozen majors in Spring 2013 that ranged from local Hilo performances to productions at Kilauea Military Camp Theatre in Volcano with an attendance of 1,650. More of these performances will be highlighted in Essay Three Defining and Promoting “Student Success.” Chair Jackie Johnson has maintained a close affiliation with outlying venues, including the Kona Historical Society, where she performed as the 19th century travel writer Isabella Bird in conjunction with actual tours of Hualālai in August of 20129.
7. Pacific Islander Mobile Screening Clinic (PIMSC)
The College of Pharmacy received national accolades for PIMSC “which seeks to improve public health and access to people largely from the Marshall Islands through the use of health fairs and wellness clinics. Over a hundred student volunteers (under the direction of ten core peers) conducted diabetes, cholesterol and blood pressure screenings, as well as provided wellness and lifestyle counseling and referrals to accessible health care services offered at reasonable costs10.” It was noted that “Through collaboration with the Aging and Disability Resource Center (ADRC) and networking with other community organizations island-wide, the PIMSC has screened more than 350 participants so far this year through mobile screening clinics. PIMSC participants are invited back for follow-up, and Hawaiʻi County residents are welcome to take advantage of regularly scheduled clinic office hours at the free ADRC Wellness and Safe Medication Use Clinic run by Dr. Katherine Anderson and second-year pharmacy students11.”
8. The Office of Applied Learning Experience (ALEX)
ALEX is a new program whose mission is to “to maximize the breadth and depth of applied learning opportunities for students at the University of Hawaiʻi at Hilo” though activities such as “community based projects, service learning, research, internships, practica, creative activities, and capstone projects12.” In its first term (Spring 2013), ALEX facilitated 12 independent student internships and awarded the following teachers for spearheading student projects that directly impacted the local community:
- Dr. Norman Arancon’s AG 230 featured a group of students who aided a Boys and Girls Club community garden in Haili through demonstrations on “composting, intercropping, cover cropping and organic gardening” with the harvests donated to the local community and to the garden volunteers. Dr. Arancon has also given workshops on vermicomposting in Waimea13.
- In Spring of 2013, Dr. Celia Bardwell-Jones PHIL 100 students worked with Mrs. Charmaine Haunga of Waiākea Elementary School to “read children's books to Mrs. Haunga’s second grade class and develop[ed] activities for the students that highlight the skills of the Socratic method.”
- Dr. Harald Barkoff’s KES 443 worked with students with disabilities from Waiākea High School at the UH Hilo Gym and the Student Life Center Pool to “prepare, outline, and conduct an individualized Education Program with practical implications for the Waiākea High School class groups of three to four students.”
- PSY 460, under the leadership of Dr. Dawna Coutant, worked with clients such as the County of Hawaiʻi, the Hawaiʻi Volcanoes National Park, and the UH Hilo campus to “present workable solution[s] to current problem[s].” Projects have included: “Factors, and Prevention of Illegal Dumping,” “How to Lower Electricity Costs,” “How to divert electronic waste from the landfill,” “How to Implement a Reusable Water Bottle Initiative,” “Best practices for implementing Sustainable Hand- dryers,” “How to Promote Youth Engagement at a National Park,” and “How to Reduce Paper Usage at UH Hilo14.”
As noted in the small set of detailed examples above, the applied, hand-on learning and the community outreach imbedded in many academic initiatives, co-curricular programs, and student activities ensures that their learning and their degrees are “more than the sum of its traditional parts: courses, credits, and grades” (CFR 2.2)15. It is this real-world application of learning and the accompanying value of kuleana (translated as “concern,” “responsibility, and “liability”) that adheres to our commitment of giving back to the community that we aspire to inspire in our students and that ultimately gives meaning to what we distinctly teach and offer here at UH Hilo.
UH Hilo is also committed to providing a well-rounded, liberal arts education that fulfills students’ needs for universal skills in many core areas. These skills are the mainstay of our new General Education Learning Outcomes (GLOs) which also serve as our Institutional Learning Goals (approved by Faculty Congress in Spring of 2013); they are published on our New General Education Home Page and on our online Catalog:
1. Critical Thinking
A UH Hilo graduate should be able to:
- Critique and interpret evidence (primary sources)
- Identify relevant arguments
- Analyze alternative assumptions
- Identify ethical problems and dilemmas
- Evaluate the validity of ethical arguments
- Critically reflect on value assumptions
2. Information Literacy
A UH Hilo graduate should be able to:
- Identify appropriate information required to address particular problems or issues
- Access relevant information using appropriate resources
- Evaluate different forms of data and sources
- Analyze the economic, legal, and socio-political and other issues surrounding the use of information
- Use computer technology to conduct research and find information
A UH Hilo graduate should be able to:
- Articulate and pursue a line of reasoning using oral and written forms
- Present ideas or results in a manner appropriate for college-level discourse (i.e. structure, tone, syntax, and grammar) in written form
- Present ideas or results using collegiate-level conventions (i.e. documentation, genres, and forms of presentation)
- Identify his/her audience and adapt accordingly
4. Scientific and Quantitative Reasoning
A UH Hilo graduate should be able to:
- Use systematic, empirical approaches to address questions as part of the scientific process
- Differentiate scientific and non-scientific methods of inquiry
- Conduct planned investigations including recording and analyzing data and reaching reasoned conclusions
- Solve problems using mathematical methods and relevant technology
- Represent theoretical models and data using graphs and tables
5. Human Interaction and Cultural Diversity
A UH Hilo graduate should be able to:
- Analyze multiple perspectives and articulate how perspectives based on world views differ from his/her own
- Differentiate key values, assumptions, and beliefs among cultures
- Explain how and why different racial, ethnic, religious, regional and gendered backgrounds shape experience
- Explain or predict individual and collective human behavior
6. Collaborative Skills and Civic Participation
A UH Hilo graduate should be able to:
- Formulate a rational project that contributes to the environmental, economic, social, or intellectual betterment of the local community or global forum
- Articulate how his/her activity contributes to increased awareness of local or global issues on campus
- Define what aspects of his/her group projects with peers contribute to the intellectual development of all involved16.
These Learning Outcomes not only set expectations for what students should know and be able to do at the completion of their degrees, but also serve as evidence of our compliance with Hawaiʻi Revised Statutes §304A-102, which defines the purpose of the university: “to give through instruction and conduct research in, and disseminate knowledge of, agriculture, mechanic arts, mathematics, physical, natural, economic, political, and social sciences, languages, literature, history, philosophy, and such other branches of learning as the board of regents from time to time may prescribe [ . . . ]. The standard of instruction shall be equal to that given and required in similar universities on the mainland United States17.” This is the rationale for keeping the Humanities, Social Sciences, and Natural Sciences requirements within the new GE program as a multi-disciplinary approach ensures that a student “learns from many sources” (per our mission), “sources” inclusive of multiple fields of knowledge and inquiry.
Ensuring Quality and Rigor
“Challenging students to reach their highest level” (as part of our Institutional Mission) and upholding a “standard of instruction [ . . . ] to that given and required in similar universities on the mainland United States” (per our State charter) is accomplished by setting clear goals of performance and measuring student attainment of those goals in a variety of venues. UH Hilo has met multiple challenges but has also been making strides in creating and sustaining a culture of assessment whereby faculty come together and collectively articulate levels of performance.
In the October 14-15, 2009 Final Report of the WASC Visiting Team, “university administration [was asked] to provide leadership for further development of programmatic student outcomes assessment18.” Following a series of on-site workshops by Mary Allen, the Administration (in February of 2008) solicited plans from Departments, which were paid $500 to generate this work; however, there was no financial incentive for follow-up once the Accreditation Reports were submitted to WASC. As one faculty member recently confessed in a follow-up email, “I know I was at that workshop, but failed to take it much further than cashing the check.”
Assessment in general has been eyed with suspicion by some faculty at UH Hilo because its basic purpose or methodology has not been effectively communicated. As assessment experts across the nation have pointed out, faculty resistance and use of results are major challenges to implementing ongoing assessment. At UH Hilo, faculty statements after the workshop demonstrated that UH Hilo is not immune and faculty responses prompted a shift in approach, whereby faculty leadership through the Congress began to promote the language of CFR 3.11 from the 2008 Accreditation Handbook, which states: “The institution’s faculty exercises effective academic leadership and acts consistently to ensure both academic quality and appropriate maintenance of the institution’s educational purposes and character19.”
Since AY 2008-2009, UH Hilo expended approximately $118,000 to send various administrators and approximately three dozen faculty to receive direct training on outcomes assessment. In AY 2010-2011, the Office of the Vice Chancellor of Academic Affairs sent the Chair of Assessment Support to the Assessment Leadership Academy sponsored by WASC. The Chair, in turn, helped to facilitate training by advocating for funds to send more faculty to WASC-sponsored training. The past Chair of General Education (Dr. Todd Belt) and the incoming Chair (Dr. Michael Bitter) have attended WASC workshops. The incoming Chair of Assessment (Dr. Mitchell Anderson) has also been through several training sessions. Ensuring faculty leadership with adequate training in these areas “provid[es] the kind of environment for assessment that is receptive, supportive and enabling20.”
GE Assessment: The Beginnings of a Movement
To bring us into compliance with the guidelines for CFR 2.2a, from the 2008 Handbook of Accreditation, which states “The institution has a program of General Education that is integrated throughout the curriculum, including at the upper division level [ . . . ],” UH Hilo adopted a GE certification process to promote curriculum coherence and alignment with learning outcomes. The process is overseen by the General Education Committee, a standing body that reports to the Faculty Congress. For a course to be certified, syllabi are evaluated by a faculty committee that reviews how the course meets the GLOs stated above. In the event that the syllabus is unclear, the committee requests clarification of assignments or explanations of how the class supports student learning. To address our concerns about improving student writing across the curriculum, the course must include rigorous written assignments that total, at minimum, the equivalent of five double-spaced, typed pages, or 1,250 words21.
In support of the newly adopted GE Program, the Assessment Support Committee was sent to a two-day WASC retreat in January 2010 to develop “universal” rubrics that could be used to measure student performance in General Education certified classes. The initial set of rubrics focused on the following: Critical Thinking, Communication, Information Literacy, Scientific and Quantitative Reasoning, and Cultural Diversity. (Due to the large range of projects and goals for classes that fell into the Collaborative Skills and Civic Participation Category, the Assessment Support Committee decided to explore the possibility of a survey before moving forward with an analytic rubric; assessment of this skill has been put on hold given the creation of the Office of Applied Learning).
The Assessment Support Committee piloted direct assessment using the first three of the newly developed rubrics for fourteen research-based, thesis-driven papers from a range of 100-, 200-, 300-, and 400-level classes from three colleges that contribute to General Education: the College of Business Education (CoBE), the College of Arts and Sciences (CAS), and Ka Haka ‘Ula o Ke‘elikōlani (KH‘UOK). Papers also represent the three Divisions within CAS – Natural Sciences, Social Sciences, and Humanities. The scores from the 12-13 readers (who themselves also represented these three colleges and included two student representatives) were recorded and reported back to the Congress, though the alphas (i.e. “ENG” or “BIOL”) for the courses were redacted to avoid having the data read in “punitive” ways22.
Despite calibration sessions using a 100- and a 200-level paper, there was a lack of inter-rater agreement between the readers. This result seemed to stem from disciplinary biases on what “writing” should look like, especially at the upper division level. These differences became very apparent in meetings to review committee members’ individual responses to papers. Many of the differences arose from varying expectations of mid- to upper-level student performance and whether the preference for research writing was biased against departments like Nursing or Performing Arts, which utilize different forms of written communication. The Committee also uncovered discrepancies in what constituted an actual research-based paper – namely, there were 300- and 400-level papers that incorporated highly problematic websites (such as Wikipedia) or only required the use of assigned textbooks. The “research paper” from one particular course was actually an annotated bibliography, which could not be assessed with our rubrics.
An example of the scores is given:
Paper from BBB 100
|Reader||COMM Line of Reasoning||COMM ORG/Structure||COMM Content||COMM Interpretation||COMM Style & Voice|
Paper from FFF 200
|Reader||COMM Line of Reasoning||COMM ORG/Structure||COMM Content||COMM Interpretation||COMM Style & Voice|
Paper from JJJ 300
|Reader||COMM Line of Reasoning||COMM ORG/Structure||COMM Content||COMM Interpretation||COMM Style & Voice|
Paper from MMM 400
|Reader||COMM Line of Reasoning||COMM ORG/Structure||COMM Content||COMM Interpretation||COMM Style & Voice|
Figure 2: Scores from the Assessment Support Committee, AY 2009-2010.
These finding were first shared with the members of the WASC Assessment Leadership Academy. Dr. Stephanie Juillerat, Associate Dean of Azusa Pacific University, in collaboration with Dr. Mary Allen of WASC, noted a “lack of agreement (especially on upper division papers); [though] interestingly enough, there was a relatively high degree of consensus on what faculty considered to be poor levels of skill among all levels of papers. The numbers [ . . .] reflect these differences among faculty. But a closer look at the actual numbers reveals that the gaps between ratings is often the result of one or two outliers, indicating a need for much better calibration23.”
The problem of calibration was reported back to the Faculty Congress and a resulting discussion identified key issues: first, that lower-division courses were predictably scored lower; and second, the perception of difference in “quality” at the upper division-level and how they may have reflected disciplinary biases was also discussed.
The inability to reach consensus regarding “generic” Critical Thinking skills led the Assessment Support Committee to revise the Rubrics and to imbed Critical Thinking directly into the other rubrics for other Core Skills (as was discussed in the Introduction). The results also resulted in Kilohana, in partnership with the Mookini Library, holding several workshops the following year to address the problem of credible resources, including Wikipedia. However, uneven writing standards remain a lingering problem, with some faculty reporting dissatisfaction with the Writing Intensive (WI) designation for courses as thorough assessment has not been undertaken to see if instruction in these specially designated courses are effective.
Work has been done to benchmark freshman writing over the past three years, which will be discussed in Essay Two: Achieving “Core Competencies”. To meet General Education assessment needs, the Assessment Support Committee assisted in garnering data in other 100-level courses in Summer of 2013. The College of Hawaiian Language supplied papers from its HWST 111 Hawaiian ‘Ohana Class – 43 papers from Fall 2009 and 35 papers from Spring 2013 (N = 78). These papers are drawn from sections of three different faculty who have taught the class on a regular basis. The 2012-2013 Assessment Support Committee read all 78 papers using the GE rubrics for Information Literacy and Written Communication24.
Because there were major variances in the types of papers collected, the Committee could not read the batch for Information Literacy to weigh against the scores developed for ENG 100 (see Essay Two). This highlighted the differences in expectations between teachers for the same course: some teachers assigned research papers whereas others assigned interview- or anecdotal-based oral history projects. Thus given the range of different end-of-term artifacts, the Assessment Committee recommended to the College of Hawaiian language that more faculty interaction within their core group that teaches HWST 111 can lead to better alignment of student learning objectives on syllabi and better articulation of common learning goals to students.
HWST papers were read against the General Education Written Communication Rubric25. Averages were tallied for each area on the rubric, which revealed that students exhibited the most problems with thesis construction and maintaining a linear line of reasoning:
|Line of Reasoning (Critical Thinking)||Organization and Structure||Content||Language/Prose/Syntax|
Figure 3: Averages for all categories for Written Communication for HAWST 111 papers.
The scores for all categories were averaged and formulated into a bar graph to visualize student performance, with the red line denoting the cut-off for competency according to the 4-point rubric: 1 = Beginning, 2 = Emerging, 3 = Competent, and 4 = Advanced. Averages were also calculated for each paper and plotted in a bar graph to show frequency/distribution:
The Assessment Support Committee made the following observations and recommendations:
- It may be helpful to have the HWST 111 teachers meet to address the following questions:
- Are SLOs consistent in all sections and in the assignments? And how are those expectations being communicated to students?
- Do assignments give students the opportunity to demonstrate the skills required for GE?
- Are students being held accountable for the academic format adopted by KH‘UOK?
- HAWST 111 teachers may want to work with the Instructional Librarians to develop a library module that can help students develop a better understanding of proper formatting and proper citation (including how to conduct and cite an oral history interview);
- HAWST teachers may want to work closely with writing instructors and Kilohana: The Academic Success Center;
- HAWST teachers are invited to join the Assessment Support Committee to collectively read more papers in the future.
As of September 2013, Kekoa Harmon of KH‘UOK and Kirsten Mollegaard of the English Department will be working together to collaborate on writing assignments – the hope is that the sharing of ideas will result in better aligned pedagogy and curriculum for 100-level courses, especially those certified for GE (such as HWST 111), which must include “rigorous written or quantitative assignments that assess the student learning outcomes. The assignments should total at minimum the equivalent of five double- spaced, typed pages, or 1,250 words.” History will be submitting all HIST 151 and 152 papers this academic year to help further facilitate General Education assessment. Talks are also underway to solicit 200-level artifacts.
Program Review – A New Start
The 2008 WASC Visiting Team noted in regard to Educational Effectiveness that while a new Program Review policy was implemented at UH Hilo in 2006, “the self-study is reviewed by the college dean and the Associate Vice Chancellor for Academic Affairs. However, no faculty outside of the department being reviewed are involved. With this structure program review at UH Hilo will likely become a summative exercise rather than a formative process26.” This “structural” problem came to a head in Fall of 2011, when the several departments in the College of Arts and Sciences were notified in October that their Reviews would be due in the spring. This triggered a reaction among Chairs, one of whom, as a Member of the Faculty Congress, posed serious questions during Congress sessions about the reasons for doing the review.
As a direct result of reviewing the program review process, the Chair of the Assessment Committee proposed sending a large team to receive training in assessment and to undertake a revision of the program review guidelines at the 2012 WASC Assessment 101 & Program Review Workshop in Honolulu.
That group was tasked with making the process meaningful vis-à-vis the new language associated with the Draft of the WASC Handbook of Accreditation:
Program review remains a priority for WASC. It is a natural nexus and point of integration for the collection of data and findings about the meaning of the degree, the quality of student learning, core competencies, standards of student performance, retention, graduation, and overall student success27.
New program review guidelines were drafted with the concept of “shared effort,” as encapsulated in the title of the document, Ho‘okāhi ka ‘ilau like ana: Wield the Paddles Together, which was intended to capture the spirit of our new Strategic Plan and our sense of community and shared learning by asking programs to “examine what we do and how well we do it, so we can better achieve broader university- wide goals28.”
Some departments and colleges are actively engaged in assessment, as documented by the departmental surveys and the samples of programmatic assessment we have collected and posted on our accreditation website29. But some programs scramble once every seven years with large-scale assessment. To remedy this, the new Guidelines encourage programs to undertake small assessments and yearly reviews (1-2 pages) that can be compiled into the one longer, final Program Review at the end of the cycle. This keeps the channel of communication open between all parties and allows for intervention at earlier times to avert larger crises. The Guidelines also include matrices for quantitative and qualitative data, including a large section that explains the need to undertake assessment per our ILOs and the core skills that underpin our General Education Program – Information Literacy, Communication, Quantitative and Scientific Reasoning, Human Interaction and Cultural Diversity, and Collaborative Skills and Civic Participation. The GE Rubrics (found on pages 18-22 of the new Guidelines) were also imbedded as a means of encouraging the use of “universal” measurements.
In April 2013, the Faculty Congress approved a new Program Review process. The new process includes a faculty Program Review Advisory Committee to offer assistance to programs undergoing review as well as a yearly revisiting of the MOU by the department and the administration. The Math Department volunteered to pilot the new Guidelines and their Program Review is currently awaiting a new MOU.
Another means by which UH Hilo ensures rigor and quality in its degrees is through the simultaneous maintenance of secondary accreditation for key professional programs. For example, the College of Pharmacy (CoP) responds to the needs of its specialized accrediting body (which mandate strict assessments) to further ensure that its graduating students meet the highest professional expectations upon exiting.
According to the National Association of Boards of Pharmacy, the NAPLEX, or North American Pharmacist Licensure Examination, “tests a prospective pharmacist’s ability to measure pharmacotherapy and therapeutic outcomes, prepare and dispense medications, and implement and evaluate information for optimal health care. It is just one component of the licensure process and is used by the boards of pharmacy as part of their assessment of a candidate’s competence to practice as a pharmacist30.” Thus, the NAPLEX pass rate is a measure of the rigor and integrity of the College’s program to prepare students to practice competently as pharmacists.
Per this mandate, CoP closely monitors students’ NAPLEX scores. NAPLEX pass-rates for the inaugural class of 2011 (80.25%) fell below the national average. Several measures were undertaken to address the underlying causes of the class’ poor performance. Most notably: adjustments were made to the core curriculum to add more content and examinations that mirror the NAPLEX; 4th year students were required to take the pre-NAPLEX examination before graduation; and, starting in 2014, 3rd year students will be required to take the Pharmacy Curriculum Outcomes Assessment, a nationally normed examination created by the National Association of Boards of Pharmacy. While only a few of these improvements involved the Class of 2012, performance data for this class showed marked improvement in the first time pass-rate, which is now at 92.86%. One of the four key priorities in the College’s strategic plan is to improve NAPLEX pass rates. The College is establishing a subcommittee of the Assessment Committee to monitor and analyze NAPLEX data to determine if these changes are having an impact and to make additional modifications, if necessary.
More information on other programs that are accountable to secondary accreditation – including Nursing, Education, and the English Language Institute – can be found on a public website that includes action letters31.
Distance Learning (DL) and Non-Traditional Credit Hour Compliance
UH Hilo’s online courses and enrollment have increased steadily over the years. Between AY 2008-2009 and AY 2012-2013, UH Hilo has experienced more than a doubling of classes offered as DL and of the number of registering students. Per our Office of Institutional Research:
|Semester||Number of Sections||Number of Registrations|
Figure 5: Enrollments in DL-coded classes.
The online BA in Psychology was begun in 2002 and was supported by a new DL faculty position and served students through University Centers on Kaua‘i and Maui and in West Hawaiʻi. The MA in Indigenous Language and Culture Education (MAILCE) began offering seats in courses to students across the state in 2006 in support of the College of Hawaiian Language’s commitment to prepare teachers for the state’s Hawaiian language immersion schools. In 2008, the UH Hilo Summer Session, noting that students preferred online courses in the summer, began supporting online courses with an incentive program and a course design specialist; this support was terminated in 2011 when the College administration changed. In 2009, it became possible for Communication majors to complete 50% or more of major requirements online – a “drift” that occurred as faculty responded to enrollment demands. Four new distance programs have begun since 2011: the Master of Science in Psychopharmacology, the Master of Arts in Teaching (MAT), the Doctor of Nursing Practice, and the BBA in West Hawaiʻi. The MAT is a program that offers students the option of face-to-face or DL.
Support for DL is provided at the system and campus levels. The UH system Informational Technology Services provides the Laulima online course management system as well as online training modules and instructional software32. The UH Hilo Office of Campus Technology provides technology support as well as weekly workshops33. UH Hilo also maintains a faculty resource page that includes links to key support personnel and course design guidelines34.
In Spring 2010, it was discovered that four distance learning programs had not undergone WASC substantive change review – some due to “institutional drift,” whereupon a program continually adds classes due to enrollment demand, and others to miscommunication between various parties35.
Three of these programs, as well as five new DL programs, have earned sub-change approval by December 2011. The fourth noncompliant program was approved by WASC in April 2012.
Online and DL-certified graduate programs appear to be doing rather well in terms of assessment. The MA in Indigenous Language & Culture Education (KH‘UOK) consistently gathers indirect and direct measurements of student performance per formative assessment from multiple observers (feedback on students’ action research projects) and exit surveys36. The BSN maintains careful scrutiny of licensure pass-rates and rigorous student expectations on the demonstration of skill, including key content objectives (ATI) due to secondary accreditation needs37. Newer programs such as the MAT and the DNP are set to begin collecting evidence of student performance as cohorts begin to move through their programs.
On the other hand, baccalaureate-level courses have met with challenges. The BBA in West Hawaiʻi was unable to run assessment for their DL students due to the small number of students and their unusual point of entry which fell between the established times for the administration of the CLA and the ETS. The Communication Department reports that they have not adhered to the assessment timeline that they submitted for the substantive change proposal because the larger department was unaware that such a timeline existed.
However, Psychology, which offers online and face-to-face versions of the same courses for comparison, looked at the distribution of grades, which shows that there is no remarkable difference between the two different delivery modes:
|F 2011||Psy 213||13325||A||X||2||8||5||3||4||1|
|F 2011||Psy 213||13323||B||X||1||5||7||2||6||1|
|F 2011||Psy 214||13326||C||X||11||5||3|
|F 2011||Psy 214||13327||C||X||11||4||3|
|F 2012||Psy 214||10212||X||3||6||6||2||1|
|Sp 2012||Psy 214||14552||A||X||5||5||4||1||2||1||C||1|
|F 2011||Psy 314||13329||B||X||9||19||8|
|F 2012||Psy 314||10213||B||X||15||12||4||1|
|F 2012||Psy 314||10214||B||X||10||7||6||1||3||1|
|Sp 2012||Psy 314||14556||B||X||11||11||8||1||1||3|
|Sp 2013||Psy 314||11602||B||X||7||7||10||6||2|
|F 2011||Psy 320||13334||C||X||24||5||3||1||2|
|F 2011||Psy 320||13332||E||X||9||7||8||7||1||2|
|F 2011||Psy 320||13333||E||X||8||11||8||5||3|
|F 2011||Psy 322||13337||D||X||8||12||7||5||3|
|F 2012||Psy 322||10248||D||X||8||8||7||5||5||1|
|Sp 2012||Psy 322||14768||D||X||3||5||14||9||2||C||2|
|Sp 2013||Psy 322||11605||D||X||5||18||7||5||1||1|
|F 2012||Psy 323||10249||F||X||6||6||2||3||2||1|
|Sp 2012||Psy 323||14563||F||X||7||6||4||1||3||1||B|
|F 2011||Psy 324||13339||G||X||8||13||6||5||1|
|F 2011||Psy 324||13340||H||X||15||14||2||1||1|
|F 2012||Psy 324||10250||G||X||8||10||10||3||2|
|Sp 2012||Psy 324||14564||H||X||21||5||6||2||1|
|Sp 2012||Psy 324||14565||H||X||18||7||3||1||3|
|Sp 2012||Psy 360||14769||D||X||8||6||14||3||1||1|
|Sp 2013||Psy 360||11610||D||X||7||7||11||4||3|
Figure 6: Grade distribution comparison for face-to-face versus online courses in Psychology.
In addition, Psychology also provided an example of test scores for one particular course – PSY 323 – which is taught both online and face-to-face. The results of a series of tests, which are linked to programmatic Student Learning Outcomes, also indicate that there is no statistically significant difference of grades between the modes of delivery.
PSY 323WI - Community Psychology
Sample of Student Artifacts (DL & F2F)
|Term||Course||CRN||DL||F2F||Survey Project Paper||Interview Project Paper||Exam 1||Exam 2||Exam 3||Exam 4|
|Spring 2009||PSY 323||10390||X||77.1||85.7||64.0||80.0||64.0||72.0|
|Spring 2010||PSY 323||13406||X||76.7||80.0||76.0||60.0||56.0||68.0|
|Spring 2011||PSY 323||11220||X||80.0||86.7||72.0||64.0||84.0||68.0|
|Spring 2011||PSY 323||11910||X||73.3||76.7||68.0||52.0||56.0||76.0|
|Fall 2012||PSY 323||10249||X||70.0||86.7||68.0||64.0||64.0||80.0|
|Fall 2008||PSY 323||14510||X||77.1||80.0||68.0||56.0||72.0||60.0|
|Fall 2009||PSY 323||12394||X||82.9||91.4||56.0||60.0||60.0||60.0|
|Summer 2010||PSY 323||14155||X||90.0||93.3||68.0||72.0||60.0||80.0|
|Summe 2011||PSY 323||12069||X||93.3||86.7||80.0||72.0||68.0||84.0|
|Spring 2012||PSY 323||14563||X||73.3||93.3||60.0||88.0||76.0||92.0|
|Summer 2012||PSY 323||10117||X||80.0||76.7||76.0||76.0||80.0||80.0|
|Summer 2013||PSY 323||12306||X||76.7||90.0||72.0||68.0||68.0||80.0|
~ Columns 6-11 are percentage of points earned out of total points possible.
|Program Learning Outcomes:||Survey Project Paper||Interview Project Paper||Exam 1||Exam 2||Exam 3||Exam 4|
|PO1: Demonstrate knowledge of basic concepts in statistical analysis and be able to interpret and understand both qualitative and quantitative statistical analyses.||X||X|
|PO2: Demonstrate knowledge of basic concepts and methods of psychological research: this includes defining or explaining concepts, collecting and analyzing data, presenting data in tables, conducting a scientific literature search, preparing a scientific research proposal, conducting research under the supervision of a faculty researcher.||X||X|
|PO3: Define, explain, and apply key terms and concepts in two of the following areas of Psychology: Developmental Psychology, Personality, Social Psychology, Abnormal Psychology|
|PO4: Define, explain, and apply key terms and concepts in two of the following areas of Psychology: Learning and Motivation, Sensation and Perception, Cognitive Psychology, Biopsychology|
|PO5: Define, explain, and apply key terms and concepts in areas of specialization||X||X||X||X|
Figure 7: Scores and means of student performances on key assignments linked to PLOs in Psychology 323.
It is hoped that the new Program Review Guidelines will encourage other individual programs to undertake more rigorous assessment of their programs, including both face-to-face and DL courses. The adoption of the new credit hour policy was also intended to ensure that rigor is being met in all classes:
Regardless of the type of academic activity, schedule, or method of delivery, one credit hour at UHH represents the expected amount of work a student must expend to achieve intended learning outcomes consistent with that of a traditional course (i.e. one that meets one hour per week, with a minimum of two hours additional work such as preparation, research, homework, investigation, etc. over the course of an approximate 15 week semester)38.
The new credit hour policy has led some faculty to rethink effective delivery of instruction given the changing needs of students. The Foreign Languages Department, for example, faced decreasing enrollment over the last several years due to their four-times a week, face-to-face model, which was interfering with students’ abilities to take other classes that were on either the T/TH (75-minute sessions) or M/W/F (50-minute sessions) schedule. As of Spring of 2013, several languages – including Spanish, Chinese, Filipino, and Japanese – adapted their classes to fit the T/TH or M/W/F time slots with additional activities or work to meet the fourth credit hour.
As just one example, Spanish 101 underwent intense redesign. Originally built as a four-times-a-week model with daily group practice and lectures followed by homework from the assigned textbook, Spanish 101 now employs substantial online homework that is collected and used for weekly mandatory meetings with either the lab tutor or the course instructor. Per the syllabus, the students are advised:
- This course is a four credit course that meets 3 times a week for 150 minutes with an additional 120 minutes of practice to be completed online. This homework (Tarea) will consist of grammar, structure, and vocabulary exercises that will aid the student in language acquisition.
- The homework will be diagnosed for patterns of errors. There will be mandatory advising for the student either in the lab or with the professor to understand his pattern of errors. This will present fossilization of errors and greatly improve test grades39.
Many of the online assignments for Spanish 101 are actually lengthy written assignments that better align the course to the new GE learning outcomes, which highlight written communication as a key skill in undergraduate study40.
These changes also allowed Spanish to grow in enrollment from three sections of just 56 students in Fall 2012, to three sections with 63 students in Spring of 2013, to four sections of 74 students for Fall of 2013 – an enrollment growth that is sorely needed in this department. In addition, the grade distribution doesn’t reflect statistical difference in the rate of underperformance (D, F, or W), meaning the revamping of credit hour work did not adversely impact student performance despite the change in modality and increased enrollment.
|Fall 2011||SPAN 101 001||13216||10||7||1||0||1||1||0||0||20||19|
|SPAN 101 002||13217||10||5||0||2||2||0||0||0||19||19|
|SPAN 101 003||13218||8||6||1||1||2||2||0||0||20||18|
|Spring 2012||SPAN 101 001||14169||13||6||4||0||1||0||0||0||24||24|
|SPAN 101 002||14170||8||7||2||0||0||2||0||0||19||17|
|Fall 2012||SPAN 101 001||10304||7||4||2||0||4||1||0||0||18||17|
|SPAN 101 002||10305||10||5||2||0||3||1||0||0||21||20|
|SPAN 101 003||10306||10||4||2||0||0||1||0||0||17||16|
|Spring 2013||SPAN 101 001||11142||13||2||3||4||0||0||0||0||22||22|
|SPAN 101 002||11143||7||8||4||1||2||0||0||0||22||22|
|SPAN 101 003||11144||9||5||2||1||1||2||0||0||20|
Figure 8: Grade distribution of Spanish 101 over time.
In closing, UH Hilo has struggled with the need for direct assessment of student work. We acknowledge that more needs to be done assessing undergraduate DL courses, GE certified classes, and those designated as Writing Intensive. Strides have been made in training a number of faculty as well as incentivizing assessment through the new Program Review Guidelines. It is felt that comprehensive assessment for DL and GE cannot be made independently of the gauging of the larger integrity of programs and colleges – so it is with anticipation and confidence that we move to a new timeline, a more collective effort by the faculty to gather evidence of student performance, and the use of data to improve curriculum and coursework.
University of Hawaiʻi at Hilo, Papahana Ho‘olālā Hikīaloa 2011-2015, 4, https://hilo.hawaii.edu/strategicplan/documents/2011-2015StrategicPlanWebVersionFINAL.pdf. ↩︎
Western Association of Schools and Colleges, 2013 Handbook of Accreditation: Penultimate Draft, 10, . ↩︎
Ka Papahana Lauapuni Hawaiʻi, February 19, 2008, . ↩︎
Office of Hawaiian Affairs, “Hawaiʻi and APEC economies share language ties,” Ka Wai Ola 28, no. 11 (2011), 24, . ↩︎
University of Hawaiʻi at Hilo, “UH Hilo week of welcome marked by marine mammal rescue,” August 19, 2010, https://hilo.hawaii.edu/news/press/release/954. See also “Hilo site to aid whales, dolphins,” Honolulu Advertiser, February 22, 2010. ↩︎
University of Hawaiʻi, “Local First dining at UH Hilo,” News, August 22, 2012, http://www.hawaii.edu/news/2012/08/22/local-first-dining-at-uh-hilo/. ↩︎
Western Association of Schools and Colleges, 2013 Handbook of Accreditation: Penultimate Draft, 7, . ↩︎
University of Hawaiʻi at Hilo, The General Education: Goals and Outcomes, https://hilo.hawaii.edu/academics/gened/GoalsandOutcomes.php. ↩︎
Hawaiʻi Revised Statutes, Division 1, Title 18, Chapter 304A, Part I, §304A-102, http://www.lawserver.com/law/state/hawaii/hi-statutes/hawaii_statutes_304a-102. ↩︎
Final Report of the WASC Visiting Team Special Visit Review of University of Hawaiʻi at Hilo, October 14-15, 2009, 30, https://hilo.hawaii.edu/uhh/accreditation/documents/WASC_TmRpt_2009fall_UHH_Spec-1.pdf. ↩︎
Western Associate of Schools and Colleges, Handbook of Accreditation 2008, 20, http://www.wascsenior.org/files/Handbook_of_Accreditation.pdf. ↩︎
Trudy Banta, “Characteristics of Effective Outcomes Assessment,” in Building a Scholarship of Assessment, ed. Trudy Banta et. al (San Francisco: Jossey Bass, 2002), 274. ↩︎
The data for this reading can be found in Attachment A in the April 23, 2010 Final Report of the Assessment Support Committee for AY 2009-2010, https://hilo.hawaii.edu/uhh/congress/documents/AssessmentReport23April2010rev6.23.10.pdf. ↩︎
See “Addendum to the Final Report of the Assessment Support Committee for AY 2009-2010,” https://hilo.hawaii.edu/uhh/congress/documents/AssessmentReport23April2010rev6.23.10.pdf. ↩︎
Of the 78 papers, 24 were handed out for a blind second review. The correlations ranged from a low of .10 (for Language/Prose/Syntax)) to a high of .70 (for Organization and Structure; p < .001)). Line of Reasoning was calculated at .41 (p = .05) and Content was a .40 (p < .05). Averaged correlation across “domains” was .63 (p = .001), with scores reflecting only a one-point difference. ↩︎
Please note that the GE Rubric for Written Communication was revised from five areas to four per the results of the ENG 100 Core Competency Assessment in 2012-2013. ↩︎
Report of the WASC Visiting Team, Special Review of University of Hawaiʻi at Hilo, March 18-20, 2008, 22, https://hilo.hawaii.edu/uhh/accreditation/documents/SpecialVisitTeamReportMay2008.pdf. ↩︎
2013 Handbook of Accreditation, Penultimate Draft March 2013, 32, . ↩︎
University of Hawaiʻi at Hilo, Office of the Vice Chancellor for Academic Affairs, “Ho‘okahi Ka ‘Ilau Like Ana: Wield the Paddles Together,” accessed July 15, 2013, https://hilo.hawaii.edu/uhh/vcaa/documents/PR2013wc.pdf. ↩︎
University of Hawaiʻi at Hilo, “Documents for 2013-2014 Institutional Review for Reaffirmation of Accreditation,” accessed August 13, 2013, linked from the Institutional Accreditation website . Samples of program-level assessment activities (with data) can be found under Item 8 “Samples of Programmatic Assessment Activities.” ↩︎
University of Hawaiʻi at Hilo, Accreditation, accessed July 15, 2013, https://hilo.hawaii.edu/uhh/accreditation/ProgramAccred.php. ↩︎
University of Hawaiʻi at Hilo, Distance Learning Faculty Resources “By Faculty – For Faculty,” accessed August 5, 2013, https://hilo.hawaii.edu/academics/dl/helpforfaculty.php. ↩︎
Donald O. Straney to Ralph A. Wolff, April 13, 2011, https://hilo.hawaii.edu/uhh/accreditation/documents/UHHiloResponsetoWASC4.13.2011onsubchange.pdf. ↩︎
Data from MAILCE assessment activities can be accessed at: https://hilo.hawaii.edu/uhh/accreditation/documents/MAILCEAssessmentData.pdf. ↩︎
Data on licensure pass rates and ATI competencies can be found under “Program Assessment” at the bottom of the Nursing Online Programs website: https://hilo.hawaii.edu/uhh/accreditation/DstLrngRNtoBSN.php. ↩︎
University of Hawaiʻi at Hilo, UHH Credit Hour Policy, accessed August 5, 2013, https://hilo.hawaii.edu/uhh/accreditation/documents/UHHCreditHourPolicy.pdf. ↩︎
Faith A. Mishina, “Spanish 101,” accessed August 6, 2013, https://hilo.hawaii.edu/uhh/accreditation/documents/SPAN101syllabus.pdf. ↩︎
Faith A. Mishina, “SOAN 101; tarea #3,” accessed August 6, 2013, https://hilo.hawaii.edu/uhh/accreditation/documents/SPAN101homework.pdf. ↩︎