8 Survey weights

Weights are usually applied to sample survey data during its analysis to adjust for factors such as differential selection probabilities, non-response patterns and sample skews relative to population figures. The NZCASS is no exception.

The sample design for the NZCASS covered four levels: Nielsen Area Units, households, people, and victimisation incidents. Weights have been calculated for the NZCASS data at three of these levels; households, people, and incidents. These weights incorporate adjustments for each of the factors listed above.

Household weights

Wells (1998) gives formulae for the calculation of weights for a screening booster sample design. These only provide for person weights, however, and household weights are needed to analyse household offence in the NZCASS. Initial household weights were calculated as the reciprocal of each household’s estimated probability of inclusion in the sample, across all three samples. (Person weights will be calculated using a similar process for consistency.) The inclusion probability for household i was calculated as follows:

Since Nielsen Area Units are selected for the main sample with probability proportional to the number of households they contain, and a fixed number of occupied households are approached in each NAU, each household has approximately[25] the same probability of being approached for the main sample. This probability can be calculated as the number of NAUs selected multiplied by the number of households approached per NAU, divided by the number of households. Specifically, based on the 2001 Census data used for sampling,

Similarly, the household selection probability for the Māori booster samples was calculated as the number of dwellings selected for the relevant sample in each selected area multiplied by the number of areas selected for that sample, divided by the estimated number of eligible dwellings in the relevant sampling frame (i.e. in NAUs with a high enough incidence of Māori households). Specifically, this was

for the Māori booster sample (or zero outside the Māori booster sampling frame).

The second element of the household inclusion probability calculation is the probability that a household was eligible for each sample. All households in the dataset were eligible for the main sample, i.e. the probability of eligibility for the main sample is 1. However household eligibility for the Māori samples was not known for non-Māori respondents in the main sample (unless the household was in a low incidence NAU and thus was not in the relevant sampling frame, or the household only contained one person aged 15 or more). The probability r_i that household i was eligible for the Māori booster sample was 0 if the NAU was outside the frame for the Māori booster sample; otherwise it was 1 if the respondent was Māori; otherwise it was 0 if the number of people aged 15 or more in household was 1. For the remaining households, the desired probability was estimated as

where u is the proportion of people in the household aged 15 or more who would be eligible for the Māori booster sample, averaged over all households known to be of mixed eligibility, and v is the proportion of households in the Māori booster sampling frame that were eligible. Although this formula estimating r_i for the remaining households (i.e. those in the Māori booster sample frame where the respondent was not Māori and did not live alone) was consistent with that used in the 2001 survey, it is only an approximation and could be improved in two ways; by conditioning the calculation of u and v on the number of people in the household aged 15 or more (combining the larger household sizes, which have low sample sizes), and by replacing the units in the denominator by a factor w being the proportion of main sample households in the Māori booster sampling frame which have at least two residents aged 15 or more.[26]

The initial household weight was calculated as the reciprocal of the household inclusion probabilities described above. The resulting weights had an average value of 130.0, a coefficient of variation of 0.345, and ranged from 66.7 up to 186.5.

A non-response adjustment was made to these initial household weights. The response outcome data from each NAU was modeled using logistic regression. First a model was fitted using the following predictor variables: interviewer ethnicity (Māori or non-Māori), sample (Māori booster or main sample), interviewer experience (in years), the 2005 crime rate in that Police Station Area (on a truncated log scale), broad region, level of urbanisation, and deprivation index (NZDep2001). Then the non-response model was selected by sequentially removing predictor variables that were not statistically significant, until only significant variables remained. The final predictors were the sample, broad region, and level of urbanisation. Initial household weights were then divided by the predicted values from this model, which ranged from 0.457 to 0.692. The resulting weights had an average of 225.0 and a coefficient of variation of 0.335, and ranged from 96.4 to 354.0.

Table 8.1 Parameter estimates for non-response model

Each of the predictor variables in this model were also related to survey measures such as victimisation rates. Of these, sample membership generally has the strongest relationship. For instance, the incidence rate for any victimisation in the main sample was 109 offences per 100 people, with a 95% confidence interval of (98, 120), while the rate in the booster sample was 215 offences per 100 people, with a 95% confidence interval of (193, 237).

The household weights resulting from the non-response adjustment were then post-stratified by level of urbanisation, based on the estimated number of households in each category as at 30 June 2006.

Table 8.2 Population targets for household weight calculation

The final household weights after post-stratification ranged from 119.8 to 475.6, with an average of 287.7 and a coefficient of variation of 0.351. These weights were used for analyses of household characteristics, and in particular to calculate incidence and prevalence figures for household offences.

Person weights

Person weights were calculated in a similar way to household weights. The only differences were that the initial household weights were multiplied by the number of people living in the household who were eligible to be interviewed[27] (to adjust for only one person from each household being interviewed), and that the person weights were raked[28] by combinations of age, sex and ethnicity instead of by urbanisation. (The population targets used are shown in the following table.) Adjustment for non-response used the same non-response model as for households.

The initial inverse probability person weights ranged from 66.7 to 1119.0, with an average of 332.4 and a coefficient of variation of 0.577. After the non-response adjustment, the person weights had an average of 575.8 and a coefficient of variation of 0.585.

The final person weights after raking ranged from 53.4 to 3208.0, with an average of 588.1 and a coefficient of variation of 0.722. The lowest weights were for Māori women aged 40-59, with their final weights being slightly lower than their inverse probability weights. In contrast, the highest weights were for Asian men aged 15-24, and these weights were almost 3 times higher than inverse probability weights for these respondents.

Person weights were used in the calculation of incidence and prevalence figures for personal offences, and for the analysis of self-completion lifetime prevalence data and of most data from the main questionnaire. No further adjustments were made to account for non-response to the entire self-completion component.

Table 8.3 Population targets for person weight calculation

Incident weights

Incident weights were derived from person weights by dividing them by the selection probability for that incident (given that the current respondent had been selected). The probability that a given incident from the main questionnaire was selected for a victim form depended on whether it was a high, medium or low priority incident, as well as how many low, medium and high priority offences were experienced by that participant. It was calculated simply by enumerating the relevant parts of the probability space through a branching process and adding up the probabilities for each of the appropriate nodes. However, if the incident selection probability was less than 0.1, a value of 0.1 was used instead.[29] This avoided the high variability that would result from using the incident probabilities from heavily victimised respondents, at the cost of introducing the potential for some bias.[30] Some exploration of potential and likely levels of bias suggested that 0.1 was within the range of reasonable values to use from a statistical standpoint. Since this value was used in 2001, it was adopted in the 2006 NZCASS for consistency.

Incidents from the self-completion questionnaire had a much simpler selection probability, being just the reciprocal of the number of incidents reported at all screeners in that section.

Isolated missing values for the number of incidents (e.g. from a "Don’t wish to answer" response to a particular screener question) were imputed with the value 1, as was done in 2001. While it would be correct to derive a second set of incident weights from the household weights rather than the personal weights, and use these for analyzing household incidents, this would have introduced some bias in comparisons between 2001 and 2006 results because incident weights were derived from personal weights for all incidents in the 2001 survey. To maintain comparability with the approach used in 2001, personal weights were used for all incidents in the 2006 NZCASS. Although this will have introduced some bias in the 2006 results, all these results have been expressed in terms of percentages of incidents, which avoids the large bias that would occur if these weights were used to estimate total numbers of incidents. This method also eliminates the bias that would be introduced in comparisons of 2001 and 2006 results if household weights were used in 2006.

Figure 8.1 Densities of logarithms of the household, person and incident weights

The density plots above show that incident weights are much more variable than person weights, which in turn are much more variable than household weights. This reflects the large variation in selection probabilities at each of the last two stages of the NZCASS sample design. Since increased variation in weights can substantially decrease the effective sample size, the next comprehensive redesign of the survey should consider ways to reduce this variation.

Footnotes 

25 Some households may have a greater or lower chance of selection due to changes in the number of occupied dwellings in that NAU since the 2001 Census, but since these changes were not known when survey weights were calculated, this could not be taken into account in calculating the weights described here. There is some resulting potential for bias in the results. The magnitude of any such bias cannot be firmly established until the 2006 Census figures are released at meshblock level, but an analysis based on preliminary Census figures at area unit level indicated that the potential bias from this source would be negligible relative to sampling error.

26 A derivation of the formula with the latter improvement is given in Appendix A5.

27 If the number of eligible household members was greater than six, a value of six was used instead. This affected 14 respondents.

28 Raking (Deming and Stephan, 1940), also known as rim weighting, enables the simultaneous control of marginal distributions for several benchmark variables. It was implemented here using Lumley’s (2006) rake function.

29 This truncation of the distribution of probabilities affected 176 incidents. Although these incidents made up only 4% of sampled incidents, they would have accounted for 34% of the total of all incident weights if this truncation had not been applied. After truncation, they accounted for only 16% of the incident weight total.

30 The term "bias" is being used here in a technical sense, meaning the extent to which the average of the results would not agree with the true population figures (if these were known), supposing that the survey was conducted many times in the same circumstances but different samples were selected according to the same design.