3 Questionnaire development and testing

Questionnaire development

The questionnaire used in the 2006 survey was based on that used for the 2001 NZNSCV, although several changes were made. The final questionnaire, showcards, and the contact sheets are shown in the Appendices, together with a few screenshots to illustrate the look and feel of the self-completion part of the questionnaire. The structure of the final questionnaire is shown in Box 3.1.

Box 3.1 Main topics covered in the 2006 NZCASS

** These were new questions.

As well as the new topics noted above, many changes were made to existing questions. These helped to improve consistency across different sections of the questionnaire, and addressed numerous other concerns with the previous questions. The development process attempted to balance desirable improvements against the need to maintain comparability with previous surveys.

In particular, one early decision was to retain a reference period stretching from the interview date back to the beginning of the previous calendar year. Along with the desire to maintain comparability, this period has other arguments in its favour:

• it starts on a date (New Year’s Day) that has resonance for many New Zealanders[10]

• it includes the entire previous calendar year

• it is long enough to provide a substantial sample of incidents

• including the period up to the interview date (instead of just the 2005 calendar year) avoids tempting respondents to give a 2005 date for a 2006 incident because they want to talk about it (Tourangeau et al., 2000).

Including the entire previous calendar year enables the survey to provide victimisation rates for that period, allowing easier comparison with Police statistics on recorded crime.

There were two particularly important questionnaire changes: first, changes to some of the victimisation screener questions, especially in the self-completion sections; second, changes to the beginning of the Victim Form. The implications of these changes are discussed in the Key Findings report.

Questionnaire preparation and testing

ACNielsen was responsible for questionnaire formatting, pre-testing and piloting of the questionnaire, and CAPI development. Most of the questionnaire development work consisted of discussion and desk review rather than pretesting with respondents.

A series of hard copy paper and laptop electronic pre-tests of the questionnaire were conducted from May through to October 2005. Twenty-one pre-tests were completed by researchers, both Māori research partners, and the National Field Manager. An interviewer completed an extra five interviews to check on the interviewer perspective (i.e. with less understanding of the background). We ensured a spread across age, sex, and ethnic groups. Pre-tests ranged in length from 15 minutes to 5 hours. The client also did a small number of pretests. Advice was taken from a SeniorNet tutor on introducing respondents to using a computer for the CASI section.

Changes to the questionnaire were made and submitted for programming on 28 September 2005. It was received back in two parts for checking programming routines and the administration of the questionnaire on laptops. The first part was received on 13 October 2005, and the remainder on 19 October. The questionnaire was fully tested before proceeding with a pilot survey.

The pilot survey

The pilot survey commenced with a two-day briefing over 9 and 10 November 2005 with fieldwork taking place during 11 to 30 November. Eighteen main sample area units and two Māori booster sample areas were surveyed for the pilot, half of which were intentionally “higher crime” to further test the survey.

As a result of feedback from the pilot, ACNielsen made a series of changes to the survey proper:

• Interviewer briefings reinforced the importance of achieving as many interviews as possible per area.

• ACN monitored interviewers who were not performing well in terms of response rates. Where under performance could not explained by the nature of the area they were working in, under-performing interviewers would be re-trained. If necessary, they would be taken off the project.

• In the pilot when two interviewers and one supervisor saw people at home during the mail drop, they hand delivered the pamphlet, and set up an appointment for a later time. (Note that this is not attempting to do an interview.) This practice was implemented in the amin survey.

• ACN thought It would be useful to offer interviewers an incentive with a monthly ‘prize draw’. This was done.

Footnote

10 There is evidence from psychological research that special events whose dates are easily remembered (such as the New Year) are important in organising autobiographical memory, and from survey methods research that reminding respondents of such 'landmark events' can improve dating of other events (Tourangeau et al., 2000, p. 134).