One of the most serious criticisms is the complete lack of transparency: we have no idea which eight trials were included in the final, damning, analysis. The literature references are not given, nor any information on the diagnoses, numbers of patients, etc., nor can these be deduced from the article. Prof. Egger has refused several requests to disclose the identity of the eight trials. This is not even a matter of scientific method, but of natural justice: the accused has the right to know the evidence against him.
Meta-analysis should incorporate sensitivity analysis. In this case the obvious sensitivity analysis is to look at the 21 trials of ‘higher quality’, particularly since it appears that the criterion ‘larger’ appears to have been added retrospectively to ‘higher quality’. But the result of this analysis was not published.
This meta-analysis assessed quality purely in terms of internal validity. Internal validity is the extent to which a trial measures what it purports to measure; external validity the extent to which what it measures is meaningful in the real world. This is an important concept, best explained by an example. A clinical trial of homeopathy in childhood asthma by White et al. (7), published in Thorax in 2003, was reported as showing that homeopathy is ineffective. This trial was of high internal validity because, among other things, it used a predefined primary outcome measure: quality of life as measured by the childhood asthma questionnaire.
The results showed no difference in the quality of life scores, although the secondary outcome measures (such as severity and time lost from school), consistently favored homeopathy over placebo. However, because quality of life was the primary outcome, the authors reached a negative conclusion. But, as was pointed out several times in the ensuing correspondence, this conclusion was meaningless because of a ‘ceiling effect’: the children had normal quality of life when they entered the study, and this could not have been expected to improve further! (8–11) This study was of high internal validity but very low external validity, but would have been classified ‘high quality’ in The Lancet meta-analysis.