The validity of data from SET is further threatened by the lack of psychometrically practices often employed in constructing scales or designing questionnaires.

Rather than utilising or adapting scales that have empirically tested and verified reliability and validity items are arbitrarily combined despite no evidence that they form a coherent scale or consideration of how they correlate with any measures of learning outcome (Coffey, & Gibbs, 2001).

When students are asked to respond to questions or statements using semantic differential or Likert type scales there is frequently no criteria or labels for the ratings and little reflection from wither staff or students as to what they actually means in real terms (Edström, 2008). As a result, Edström speculates that in such cases a high rating is more indicative of the lecturer’s attitude than any element of their teaching and a ‘feel-good factor’ is rewarded.

Often in SET surveys efforts are made to include both positively and negatively worded items, to prevent a ceiling effect, where respondents fall into a pattern of ticking all responses at either the left or the right hand side of the scale. Switching from positively to negatively worded items is designed maintain the student’s interest and keep them focused on the questions. According to Roszkowski and Soven (2010) however, engaging in this practice may be adversely affecting the validity of the measure rather than enhancing it.

Rather than thinking about the survey item most respondents provide a quick, unconsidered answer to complete the survey, rather than to provide a true indicator of their attitudes or beliefs (Krosnick 1991). Consequently students are more likely to respond to the reversed item incorrectly (indicating a positive when they mean a negative) profoundly affecting the results and the validity of the evaluation.

Back to 8.4.2 Continue to Section 8.5 Back To Section 8