7 steps to assess scientific evidence

Microbiome Analysis is a cutting-edge discipline that requires me to stay abreast of the latest research, so I can incorporate the emerging insights into my clinical practice. With misinformation being so widespread in the area of health, and especially the gut microbiome, it's essential to know how to assess the quality of research evidence behind ideas and claims. Here are some tips to help you appraise scientific research yourself.

Caption: close-up of a laptop with a journal article investigating a novel use of probiotics as anti-viral sanitisers on screen.

1) Check the date

The date of publication is vital. Anything over 10 years old is too out of date to be relevant to microbiome health. Aim for less than 5 years, and ideally the last two years.

2) Assess journal quality

Journals differ widely in the quality of the research they publish, and some are even fraudulent. Do you recognise its name or does it sound really obscure, and how well does the subject of the article fit with the subject of the journal? Do an internet search for the journal's Impact Factor – a measure based on number of citations. Discard any articles in journals under 3 and look for those from journals with an Impact Factor of more than 10. There's a lot more to journal ranking than Impact Factor alone (for example the most prestigious journals within a particular subject area are those which occupy what's known as the first quartile or Q1) but Impact Factor is a good start.

3) Check it's peer reviewed

The best journals have panels of highly respected scientists (peers) who review the article and request improvements or further evidence where required, or reject the article if they decide the study is not robust enough for the conclusions it is drawing or ground-breaking enough for the journal.

Preprints have not yet been accepted by a journal, so you can’t have the same confidence in the information as when experts have peer-reviewed it. While preprints are great for making discoveries public quickly, so the findings can be tested, developed and used by others, we need to bear in mind these papers may never be considered good enough to be published, and most of us don’t have the same skills as a peer-review panel for assessing the quality of the study and the information it gives.

4) Appraise the study design: is it large, human, randomised, controlled and double-blind?

Is it a lab-based study or animal study? Often the findings from these simply don’t translate into humans. Is it a human clinical trial? If so, how many participants? Studies smaller than 50 people (n=50) are less reliable, while larger studies have more robust findings, but are much more rare given the costs involved in recruiting and managing large-scale trials.

Perhaps most importantly, is there a control or placebo group, or is it described as placebo-controlled? This is essential for proper comparison with untreated individuals. For those that have no control or placebo, we can’t be sure if the benefits really were down to the product being tested. "Open label" trials mean the participants know what’s being tested and that may influence results too.

The most robust clinical trials are "double-blind", meaning neither the researcher or the participants know if they are taking a placebo or the drug or supplement being tested. Ideally they should be randomised (people are assigned to the control or treatment groups randomly) and the different groups are age-matched or include an equal diversity of ethnicities and genders to reduce the potential influence of those differences on the outcomes.

Epidemiological studies are observational. The researchers may review data from thousands of people, but without a test group, they can only conclude correlation not cause. While these studies can be really useful for forming hypotheses to be tested in future research, there may be a variety of explanations for their findings, including other aspects of diet and lifestyle that have not been assessed or discussed. A number of different factors tend to contribute to health outcomes. For example, people who eat more vegetables may also lead more active lifestyles, or people who eat pulses are likely to eat wider variety of other fruits or vegetables.

Many supplements on the market only have lab-based (in-vitro, eg petri-dish or test tube) or animal studies (eg rats, mice and dogs) suggesting they may have benefits in humans. Some are even prescribed based on desk-based theory alone! As a Microbiome Analyst, I prescribe only probiotics and prebiotics that have been shown to be effective in well-designed clinical trials. However, microbiome science in clinical practice is cutting-edge, and I may occasionally prescribe supplements that have promising results in lab-based or animal studies, although only when there is good anecdotal evidence of its benefit in my own or colleagues’ clinical practice. If we are trying something new or testing a theory in practice, I always let my client know my rationale, so they can make an informed decision.

A few months ago, a new probiotic made headlines for its apparent success in treating Long Covid. However there was no control group, and we know that many people are fortunate to recover from lasting symptoms following Covid infection, especially after only a few weeks of illness (many of the participants had suffered symptoms for less than 12 weeks). I don't consider this a robust enough study to be worthy of my clients' supplement budget and so don't prescribe it in my practice, however the supplement company has now expanded massively due to the demand. I reviewed the study in a blog at the time it was being reported by the press, so that anyone who'd seen the media interviews could make an informed decision.

5) Assess the authors

Look at where the authors are based; is it a respected institution? Do they have plenty of other publications or citations suggesting their work is successful and taken seriously?

Check whether they have conflicts of interest. Each article should have details on funding, conflicts of interest, disclosures or acknowledgements at the end. You may find a study is funded by a group or company that has commercial interest in the hypothesis or product being tested. This alone is not reason to discard the study's findings, as research programmes, especially clinical trials, are resource-intensive and expensive to conduct, and independent funding is rare and highly competitive. However, it is a reason to give extra scrutiny to the other factors that help us appraise the evidence, including the authors' other sources of funding, their research history and areas of interest.

In scientific journals, the first author listed is usually the one who has done most of the work on the paper, and may not have many publications if early in their career. The last author is usually the most senior -- their name and professional standing should lend credibility to the article. Authors listed in the middle are usually involved as advisors or to contribute their experience, skills or knowledge in the field. You can usually check them all out quite easily to get a sense of their credibility and the quality of the study you're assessing, and certain authors will become familiar markers of quality if you're focusing on a particular area of research.

6) Check for citations

This can be a useful measure of how well the article is respected by the wider scientific community, ie how many other researchers have cited or referred to this evidence in papers they have published. The greater the number of citations the better – obviously this will also depend on how recently the paper was published, and it can take many months for articles to be written and pass the peer-review process.

7) Use reviews

Reviews means the findings of number of different studies and the current scientific thinking have been assessed for us, usually by several experts in that field. A "meta-analysis" or "systemic review” should therefore give us a rigorous analysis of recent evidence, including a range of studies and clinical trials. There should be a discussion of their strengths and weaknesses, including study design, data analysis, control for variables that may have influenced the findings, and any other factors that may give misleading results. The most recent reviews will include the most recent research evidence. Clinical trials and research studies can have contradictory findings, so meta-analyses help me make robust treatment decisions in my practice.

There's so much excitement around this emerging microbiome knowledge and indeed an urgency to apply the latest insights in clinical practice, especially when working with people who have health conditions that are not well-served by current medical practices. However, scientific knowledge is never set in stone and new discoveries and contextual information are always expanding our understanding. I'm grateful for the skills I learned to appraise scientific research during my original undergraduate degree in medical biosciences and in my early career as a research scientist myself. With such huge potential for a longterm impact on our health -- both positive and negative -- the individualised treatment plans I write as a Microbiome Analyst must be safe and effective and therefore grounded in strong clinical evidence. My clients deserve nothing less.

Viola Sampson BSc MCMA is a registered Microbiome Analyst based in the UK and working with clients around the world. She is not currently taking on new clients as her waiting list is more than 18 months wait, but clients can make appointments with her team of Associates, who are trained in her approach and work closely with Viola on their treatment plans. To find out more, please see www.themicrobiomegroup.com