An evaluation of the indirect cohort method to estimate the effectiveness of the pneumococcal polysaccharide vaccine

We examined the validity of the indirect cohort method as a rapid assessment tool to estimate pneumococcal polysaccharide vaccine effectiveness (VE). Using evidence from published clinical trials, we reviewed the primary assumption about the appropriateness of the control group underpinning the indirect cohort method, which is that the risk of non-vaccine type invasive pneumococcal disease is equal for vaccinated and unvaccinated participants. We found an absence of evidence to support the non-differential risk assumption for non-vaccine type invasive pneumococcal disease occurring among clinical trial participants. In those instances where the design has been utilised, we also note that these typically rely on very small numbers of non-vaccine type invasive pneumococcal disease adding to the concerns as an unreliable comparator. We do not consider the indirect cohort method to be a valid tool for rapid assessment of pneumococcal polysaccharide VE.

sterile site, were classified as cases if they were known to be due to a serotype contained in the vaccine (VT IPD) and controls if they were caused by a non-vaccine serotype (non-VT IPD).As a modified case-control methodology, the indirect cohort method compares the proportion of case subjects who are vaccinated against the proportion of control subjects who are vaccinated, that is, a comparison of the risk of vaccination between the two groups.It then follows that for an effective vaccine, the cases of VT IPD should be less likely to be vaccinated than the controls (non-VT IPD).The control group in this instance was intended to represent the population risk of vaccination.

Validity of comparator
In order for the control group to be a valid indicator of the

Open Access
NobleResearch population risk of vaccination, the key assumption of the indirect cohort method is that the risk of non-VT IPD is equal for both the vaccinated and unvaccinated individuals.That is, being vaccinated does not alter an individual's risk of acquiring non-VT disease.If this assumption is valid and provided there were no systematic differences in the ascertainment of vaccination status, then individuals with non-VT IPD are an attractive choice as controls as they likely represent the same population from which the cases of VT IPD arose (care-seeking behaviour and blood culture taking behaviours also likely to be similar).However, if the assumption (risk of non-VT IPD equal between vaccinated and non-vaccinated population) does not hold, then the estimate of VE will be biased.For example, if the risk of non-VT IPD was differentially higher in vaccinated individuals, as could occur in the presence of serotype replacement, the indirect cohort method would exaggerate VE.In contrast, if the risk of non-VT IPD was lower in vaccinated individuals, as could occur if vaccination afforded any protection against non-VT disease, then the indirect cohort method would underestimate VE.

Original evidence base
Although non-VT IPD is the comparator for the indirect cohort method, it was non-VT pneumococcal pneumonia (a much less specific outcome) that was used by Broome et al. [1] as the basis for comparison of risk between vaccinated and unvaccinated individuals.The outcome data were from three early trials [2][3], where presumptive non-vaccine type pneumonia attributed the causative serotype from a specimen collected from a non-sterile site where the pneumococcus could have colonised without causing disease.Although a seminal paper of the time, MacLeod et al. [3] diagnosed pneumonia only by clinical signs.In total, the three studies considered by Broome et al. [1] provided evidence of equal risk of presumptive non-VT pneumonia between vaccinated and unvaccinated individuals but not evidence of equal risk of non-VT IPD.An equal risk of presumptive non-VT pneumonia is perhaps not surprising given the most recent meta-analyses also suggest there is evidence of equal risk of all-cause pneumonia between vaccinated and unvaccinated individuals (i.e.no evidence of benefit against pneumonia) [5,6].

Re-assessment of evidence of equal risk of non-VT IPD in vaccinated and unvaccinated individuals
We attempted to test the hypothesis that the assumed risk of non-VT IPD was equal among vaccinated and unvaccinated individuals by utilising the data from the RCTs included in the most recent update of the Cochrane systematic review of pneumococcal polysaccharide vaccine in adults [7].However, only one included study reported data on culture confirmed non-VT IPD [4], with the remaining studies either not reporting serotype specific data, or having no cases of non-VT IPD [8][9][10].
The one study contributing data on non-VT IPD was a report from three pooled studies conducted by Austrian et al. [4] amongst South African miners in the 1970s.One study used a 6-valent PPV, the other two studies used a 13-valent PPV.In total, there were 22 cases of non-VT IPD amongst 3943 vaccinated participants compared with 63 events amongst 8024 unvaccinated participants.The rate of non-VT IPD was 5.6/1000 in vaccinated participants (95%CI 3.5 to 8.4) compared with 7.9/1000 unvaccinated participants (95%CI 6.0 to 10.0, p0.17).Given we have only one study able to contribute outcomes for non-VT IPD, we conclude that there is an absence of evidence to support the primary assumption that the risk of non-VT IPD is equal in the vaccinated and the unvaccinated.Moreover, there were no RCTs of 23vPPV that contributed data on non-VT IPD.

Studies utilising indirect cohort method
A review of studies utilising the indirect cohort method to assess the effectiveness of the 23-valent pneumococcal polysaccharide vaccine indicates an obvious limitation in the use of non-VT IPD as the comparator, namely, small numbers (Table ).Few cases of non-VT IPD results in a loss of power to detect an effect of vaccination.Broome et al. [1] had stated that 'the estimate does depend on a similar proportion of vaccine type and non-vaccine type disease in unvaccinated populations', which would have been a realistic assumption at the time that the 14-valent vaccine formulation was available.Although not a component of the formula for vaccine effectiveness under the indirect cohort method, when there are few available subjects within the comparator group this limits the power of any individual study and increases the potential risk of bias.Although estimates of vaccine effectiveness derived from studies utilising the indirect cohort method (Table ) have been similar to estimates from other case-control studies and from some meta-analyses [6], of itself, this does not provide a theoretical basis for further use of the method.

Conclusions
A novel method of its time, we do not support on going use of the indirect method for assessment of vaccine effectiveness.Our findings highlight the lack of evidence to support the methodological basis that the risk of non-VT IPD in vaccinated and unvaccinated individuals is equal.Moreover, the very small numbers of non-VT IPD available in most published studies must cast doubt on the reliability of the assumption that the comparison group adequately reflects the population from which cases arise.In our view, these are important limitations of the indirect cohort method as a rapid assessment tool, and therefore do not advocate future use.

Table
Studies utilising the indirect cohort method by total number of IPD cases, number of cases excluded, proportion of known serotypes due to a non-VT type, and estimates of vaccine effectiveness.
*Reasons for exclusions include no medical records, no vaccination records, or specimen not serotyped.