Increase Font Size Decrease Font Size View as PDF Print

Nutrition Evidence Library Systematic Review Methodology


Evidence Synthesis, Conclusion Statements, Grading of the Evidence, & Research Recommendation

Evidence synthesis is the process by which evidence from multiple studies is compared, contrasted and combined to develop a graded conclusion statement that answers the SR question. This qualitative synthesis of the body of evidence involves identifying overarching themes or key concepts from the findings, identifying and explaining similarities and differences between studies and determining whether certain factors impact the relationships being examined. A series of probing questions designed to facilitate the review and analysis of the evidence by the TEC are developed and provided to the TEC via the Key Trends document. The TEC uses the description of the evidence, along with the full data extraction grid and full-text manuscripts, to respond to the probing questions. Feedback from the TEC is compiled and used to draft the qualitative evidence synthesis and the conclusion statement. The NEL grading rubric is then used to facilitate the grading of the body of evidence (i.e., strong, moderate, limited or grade not assignable) by the TEC, taking into consideration the quality, quantity, consistency, impact and generalizability of the evidence used to develop the conclusion statement. Finally, research recommendations are drafted based on input received on the Key Trends document, as well as discussions that occur during the process of developing the synthesis and conclusion statement, and grading.
 
A Closer Look: Conclusion Statements
The conclusion statement should be a brief summary statement that is worded as an answer to the SR question.  It must be tightly associated with the evidence and should be focused on general agreement among the studies around the independent variable(s) and outcome(s), and may acknowledge areas of disagreement where they exist. The conclusion statement reflects the evidence reviewed, and does not include information that is not addressed in the studies, or draw implications. The conclusion statement may also identify the relevant population, when appropriate (e.g., if papers cited studied only one sex, age group, ethnicity or level of health risk, this should be reflected). In addition, “key findings” (approximately 3 to 5 bulleted points found directly below the conclusion statement) may be drafted to provide context and highlight important findings which contributed to conclusion statement development (e.g., brief description of the evidence reviewed, major themes, limitations of the research reviewed or results from intermediate biomarkers). 
 
A Closer Look: Grading the Evidence
The NEL uses predefined criteria to evaluate and grade the strength of available evidence supporting each conclusion statement. The body of evidence used to develop the conclusion statement is evaluated based on five predetermined elements: risk of bias, quantity, consistency, impact and generalizability:
  • Risk of Bias assessment for studies included in a NEL SR is done using the NEL BAT. The NEL BAT assesses the internal validity of each study, or the scientific soundness of study design and execution to avoid potential bias in the findings. Also takes into consideration the strength of the study designs included in the SR.
  • Quantity involves an assessment of the number of available studies, the number of subjects studied and adequacy of statistical power to detect type I and type II errors.
  • Consistency refers to the degree of similarity in the direction and size of effect, degree of association and statistical significance across the studies available to answer the question.
  • Impact assessment evaluates the directness of the study outcomes and magnitude of effect. Directness refers to the extent to which the body of evidence was designed to address the SR question, specifically, the link between the intervention or exposure of interest and a defined health outcome. Studies are considered indirect if the outcome measured is a surrogate outcome versus a health outcome. An evaluation of the size of the effect and judgment regarding clinical significance is also involved.
  • Generalizability, or external validity to the U.S. population, is also assessed. NEL SRs are conducted to inform development of US Federal food and nutrition policy and guidance, therefore this assessment is important to decision makers. Experts must evaluate exposures and/or interventions, the comparators and outcomes measured for applicability to the US population as a whole or segments of the US population specified in the conclusion statement.
The grading rubric outlines the criteria considered when grading a NEL systematic review conclusion statement.

The grade that results from completion of this rubric communicates to decision makers and stakeholders the strength of the evidence supporting a specific conclusion statement.
 
 

Description of Grades Used by the USDA Nutrition Evidence Library

Strong

The conclusion statement is substantiated by a strong body of evidence and is unlikely to change if new evidence emerges.

Moderate

There are some methodological concerns related to the body of evidence, and new data might arise which would modify the conclusion statement.

Limited

The quality and/or quantity of evidence available to support the conclusion statement are weak, and are not strong enough to support policy recommendations.

Grade not assignable

The body of evidence is too small or has serious design flaws and a valid conclusion statement is not possible.

 

 

More information on Nutrition Evidence Library Systematic Review Methodology



Last Updated: 01/27/2014