EXTRACT Rankings and Ratings

The following explanation of the EXTRACT scoring systems is adapted from a section of a paper being submitted, with full referencing, for publication in a peer-reviewed journal. Publishing details to follow. For immediate access to the scoring tables click here.

 There are a number of systems around the world for grading levels of evidence in health research and for making healthcare recommendations. One comprehensive review has found that these are not consistent or even effective, and that there is no one that can be recommended. Most however are based on similar grading schema. In the EXTRACT database a Ranking scheme is adapted from such a standard (specifically that originally proposed by the US Agency for Health Care Policy and Research in 1992). A scale is adopted in which references are ranked from Ia through to IV. Rankings of I (a or b) and II (a, b or c) relate to research studies with human subjects. Their allocation is a reflection of the rigour in the methods used in each paper, the reliability of the conclusions drawn and particularly their relevance to clinical decisions. A formal meta-analysis may gain a Ia, or a Ib of IIa if less formal review methods are used,or if the results of the analysis are of minor clinical significance; an open uncontrolled study on human subjects may merit a IIc, or higher if the method is genuinely relevant to the material and the findings are clinically useful. Rankings of III and IV are applied to laboratory (in vivo and in vitro) studies, and to unsupported clinical recommendations or popular anecdotes. These latter Rankings cannot on their own sustain any useful clinical rating of the evidence. However they may have a role in other assessments of the evidence (see below). It is important to emphasise that these rankings are not as rigorous as for example the JADAD  ratings applied in some meta-analysis. Although as far as possible all papers are read in full and carefully assessed, such thorough assessments are impractical for the thousands of papers in this exercise.

For each indication or application in any EXTRACT monograph a Rating (from A+ to F) is also applied, based on Rankings for the relevant papers. Ratings are set by the ‘high water mark’ reached in Ranking levels, in other words a Rating level cannot be enhanced simply by increasing the number of papers at a lower Ranking. Where papers have conflicting findings the negative paper will tend to neutralize the contribution to Ratings of a positive paper at the same ranking (allowing for comparability in content) as well as overriding those at lower rankings. There are comparable Rating Schema for risk; in effect safety statements are graded to the same rigour as efficacy statements (something notably lacking in most published reviews of herb safety issues). Given the lack of solid evidence on adverse effects of herbs a large proportion of safety conclusions have the lowest ‘E’ rating  (theoretical case only).

In the case of efficacy statements (the Actions and Indications in the  EXTRACT monograph), the Rating based on the conventional scientific literature as above are listed as “Scientific Ratings”. However these are not the only Ratings provided.         Back

Distilling traditional use

The key feature of most medicinal plant use is that human experiences of their effects (the “bioassay data”) precede modern research. This is the reverse of modern pharmaceutical experience. However the recorded database of such traditional use is notoriously diverse and confusing and has generally been discounted in weighing the evidence. On the other hand it is difficult to discount the prospects for useful modern leads from the centuries of human experience, especially where useful patterns of traditional use emerge.

The information available on traditional herbal remedies may be processed to improve the ability to draw useful conclusions.

• Familiarity with traditional therapeutics in the major cultures around the world allows standardised therapeutic actions to be used as classifiers for a wide variety of traditional indications. These reduce the problems associated with translating local symptoms and physiological concepts. These classifiers are applied as standard terms within the EXTRACT database and thus critically, across both traditional and modern scientific evidence bases.

• Ethnobotanic records from around the world can be mapped according to the likelihood that reputations were transmitted from one location to the other. Where a traditional reputation can be shown to have occurred independently two or more times then the strength of that reputation can be reinforced. The EXTRACT Ethnomap charts the likelihood of transmission between cultures: orbits not in contact have less chance of direct cultural exchanges.

In the EXTRACT Tradscore table individual citations of traditional use are given a base score of zero. Scores of up to 5 can be attained however where single uses are reinforced by others around the world for the same or closely related plants, and where there are links between traditional use and pharmacological research. An absolute maximum score of 6 (leading to a Rating of B+) is possible, for example in the case of ginger, when human experience of its use is so consistent and accepted, or the properties of major constituents (like tannins or anthraquinones) are so undoubted that a discretionary extra point is merited.

In EXTRACT herb monographs the Tradscore is combined with the scientific Rating to generate a published Prospect Score for each herb activity. This is listed in each monograph. The scientific Rating alone for each activity then follows to indicate how far modern research  has supported the prospect.       Back

Why score prospects?

Clearly generating a prospect that a remedy could work does not actually establish efficacy. It can be argued that raising the score above that provided by hard evidence is an irresponsible inducement to use. However there are realities to be faced in matching hard science to real-life human need.

Helping out in human predicaments is never an exact science. The evidence base in any field of medicine cannot meet even the most rigorous and professional response to human health needs. From surgery on the one hand to the complexities of everyday health problems in primary care on the other, clinical experience and judgment are more important than simple data assessment.  It is of course widely accepted that responsible clinicians must take account of the latest evidence. However one of the originators of “evidence-based medicine” (EBM) has defined this as the integration of best evidence with clinical expertise and there have been substantial critiques of the common EBM assumption that clinical practice should be directed by theoretically satisfying hard evidence.

In the herbal sector public expectations and market realities completely overwhelm the scientific case. This is clearly demonstrated by the regulatory realities for herbal medicinal products in Europe. Since 1965 in Member States of the European Union it has been possible to license herbs as medicines (with many advantages for regulating product quality). However in almost all cases such licences have been supported by bibliographic evidence of efficacy, on the basis that these medicines have in Continental Europe at least, “well-established” use in professional medicine. Even this requirement is unattainable for most herbal remedies and it will soon be possible to “register” herbal products as medicines entirely on the basis of evidence that they have been used traditionally for the indication concerned. The requirement for this claim to be sustained is extraordinarily low, with even evidence of a 30-year old prescription being acceptable. In other jurisdictions, such as the USA, the regulator has not even attempted to provide guidance as the use of herbal remedies and has left it all to market opportunism.

If this is the position to which market realities have taken the regulators then it can readily be argued that a more rigorous assessment of traditional use is warranted. Even more significantly, only a very small proportion of plants that humans have used for medicinal purposes have any modern evidence for them at all. Without some informed assessment of their prospects many potentially beneficial medicines may be lost. More importantly is it impossible constructively to advise someone who wishes to try a herbal treatment – if there is no acceptable evidence at all it is professionally inconsistent not to actively discourage them. Weighing up the prospects, as well as the realistic safety risks, at least provides some basis for informed advice.

The EXTRACT model links all supportive evidence as a prospect, but compares this to the current scientific rating so that better informed decisions can be made. For example, where both Prospect Score and Rating is high then the claim for the activity concerned is strongly reinforced. Where the prospects are good but the evidence poor a measured judgment is invited. Each step is transparent and refutable: the schema and the actual scorings are offered for review and critique by email. In time such feedback should improve their value. The primary purpose in creating the grading systems is to display the diverse literature with maximum meaning and relevance to everyday clinical or self-medication judgments. If the “prospects” that are raised are accompanied by appropriate warnings as to their status they may both put popular use into more robust context and provide leads for future research.     Back

© SustainCare CIC