Student Ratings Task Force Report_2010

Cover page showing the committee report letter from the Ad Hoc Committee to Investigate Student Rating Instruments

CALIFORNIA STATE UNIVERSITY, FRESNO College of Health and Human Services Department of Kinesiology 5275 N. Campus Drive MIS SG28 Fresno, CA93740-8018 559.278.2016 Fax 559.278.7010 TO: Dr. Michael Caldwell FROM: Ad Hoc Committee (of the Academic Senate Expanded Executive Committee) to Investigate Student Rating Instruments John Constable / Thomas Holyoke Dawn Lewis, Chair1(f Paula Popma . Brian Tsukimura DATE: August 5, 2010 SUBJECT: Committee Report regarding Investigation of Student Rating of Faculty Instruments The Ad Hoc committee’s charge was 1) to research student ratings of faculty instrument options for our campus including external vendors and instruments developed internally by California State University peer institutions, 2) to research and deliberate the potential for on-campus development of an instrument, and 3) to review the finding of the 2008 RFP Committee to aid in the process. Upon completion of this charge, the committee was expected to submit a summary of its investigation to the Expanded Executive Committee (EC). The following report of the committee’s findings includes A) a description of the methods used to evaluate student rating forms, B) a summary of internally developed forms used by California State University peer institutions, C) a review of forms developed by external vendors, D) whether forms and services comply with California State University Executive Order No. 926 Policy on Disability Support and Accommodation, E) a budget summary, and F) the committee’s recommendations for the selection of rating forms. The committee’s full recommendations are given in Section F which are mainly to issue a Request for Proposal (RFP) to appropriate vendors to receive full and accurate description of services and binding cost estimates, to select and implement the forme s) and services of an external vendor that meets the required and desired features outlined in Section F, and to include an expert on accessible technology and services for persons with disabilities on all subsequent committees regarding this charge. THE CALIFORNIA STATE UNNERSITY

2. How were the common questions developed? 3. Can departments add questions? Who develops these questions? How flexible is this? 4. How did you demonstrate reliability and validity? 5. Can the forms be delivered in paper, online, or both? 6. Who processes the ratings forms? 7. How are the results interpreted? How is the data normed? 8. Is there space for written comments? 9. Are the ratings used for summative and/or formative purposes? 10. How did you implement? Was there training? 11. Is the form in compliance with section 508 of the Rehabilitation Act of 1973? 12. What are the costs involved – forms, supplies, equipment, staff?

13. Questions for California State University Peer Institutions

Ad Hoc Committee to Investigate Student Rating Instruments Subcommittee of the Extended Executive Committee of the Academic Senate Summer 2010 Questions for CSU Peer Institutions using a Common Student Rating of Faculty Form 1. We understand you use a common form, is that correct?

See Appendix C External Vendors’ Contact Information IDEA The IDEA Center 211 South Seth Child Road Manhattan, KS 66502-3089 800-255-2757 or 785-320-2400 Fax: 785-320-2424 E-mail: info@theideacenter.org SIR II by ETS Diana C. McNeil Account Manager, West Region Higher Ed Division - College & Graduate Programs MS 40-L - RM# L289-R 609-683-2338 (direct line) 609-683-2040 (fax) E-mail: dmcneil@ets.org Website: www.ets.org/hea The Instructional Assessment System (IAS) Joanna Loss Manager (206) 543-9847 voice (206) 543-3961 fax jmloss@u.washington.edu CourseResponse by Digital Measures Dana Clark Senior Account Executive Digital Measures Phone: (414) 238-6212 Mobile: (262) 951-8355 Fax: (414) 918-4602 Mail: 301 N. Broadway, Floor Four, Milwaukee, WI 53202 dclark@digitalmeasures.com http://www.digitalmeasures.com The SEEQ Herbert Marsh, Ph.D. Department of Educational Studies St Cross College Oxford University herb.marsh@education.ox.ac.uk +44(0)1865274041

Partial scan of the Chico online student evaluation formPartial scan of the Chico online student evaluation formPartial scan of the Chico online student evaluation formPartial scan of the Chico online student evaluation formPartial scan of the Chico online student evaluation form

Rating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale optionRating scale option

Heading for California State University, Long Beach student rating of faculty form

California State University, Long Beach student evaluation form

Heading for California State University, East Bay student rating of faculty form

California State University, East Bay faculty course sample form

California State University, East Bay faculty course process cover sheet

Table 2 Comparison of External Vendors of Student Ratings of Teaching Performance

IDEA Long Form IDEA Short Form IAS SIR II SEEQ Questions Base questions 47 12 2 31 40 32 (developed 1970s & 1980s) Instructor written questions Yes Yes 12 10 9 Questions for Formative / Summative purposes Yes 18 formative / 4 summative (global) Yes 29 Q formative / 3 Q summative Questions about student 12 6 9 10 Questions separated into known teaching dimensions Yes Yes No Yes 8 dimensions Yes 9 dimensions Likert scale type answers Yes Yes Yes no neutral answers Yes no neutral answers Yes w/ neutral answers Questions "validated" Yes Yes Yes Yes Yes Written comments possible? Yes Yes Yes Yes Yes Short form available? Yes --- No but 13 different types of forms available No No

Table 2 title pageComparison table of external vendor features

Continuation of comparison table of external vendor featuresContinuation of comparison table of external vendor costs

Appendix J samples of IDEA forms and report title page

IDEA survey form sample page 1

IDEA survey form sample page 2

IDEA short form sample

Blank comments section with do not write in shaded area notice

IDEA faculty information form sample

IDEA contextual questions continued page

IDEA diagnostic form report summary page

IDEA diagnostic form report student ratings of learning on relevant objectives

IDEA diagnostic form report improving teaching effectiveness

IDEA diagnostic form report statistical detail

Appendix K samples of SIR II form and report title page

SIR II questionnaire sample page 1

SIR II questionnaire sample page 2

SIR II report page with course organization, communication, and faculty student interaction

SIR II report page with assignments, supplementary instructional methods, and course outcomes

SIR II report page with course difficulty, overall evaluation, student information, and supplementary questions

SIR II interpretation guide page

Appendix L samples of IAS forms and report title page

IAS form X sample

IAS form X additional items page

IAS form A sample

IAS form A additional items page

IAS form B sample

IAS form B additional items page

IAS form H sample

IAS form H additional items page

IAS report sample with challenge and engagement index and survey ID

Appendix M samples of SEEQ forms, report and additional questions bank title page

Student feedback on teaching SEEQ survey sample

SEEQ additional questions and background unit or class characteristics sample

SEEQ staff member self rating survey sample

SEEQ additional supplementary alternative questions sample

SEEQ summary report sample

SEEQ additional questions bank categories list

SEEQ additional questions bank items 1.01 through 2.10

SEEQ additional questions bank items 5.28 through 8.34

SEEQ additional questions bank items 9.01 through 12.11

SEEQ additional questions bank items 13.06 through 15.03

SEEQ additional questions bank items 15.04 through 16.04 and acknowledgments

The committee does not recommend the development of an internal instrument of student rating of faculty at California State University, Fresno.

The committee finds that IDEA satisfactorily meets many of the criteria outlined in Appendix B and should be given additional consideration by the University.

The committee finds that ETS’ SIR II and services satisfactorily meet several of the criteria outlined in Appendix B and should be given additional consideration by the University.

It is this committee’s conclusion that there is nothing to be gained by using Digital Measures CourseReponse because it requires the university to develop its own form(s). Therefore, in summary, the committee does not recommend further consideration of the CourseResponse service for student rating of faculty.

Because administration, analytical and report services for the rating form are not provided by an external vendor associated with the instrument, the committee does not recommend the SEEQ for further consideration by the University.

Download the PDF

This HTML version was generated with AI assistance and may contain errors or omissions. The original approved PDF remains the official document of record. An alternative accessible format is available upon request from the document creator in accordance with Title II of the ADA.