Initial Publication Date: February 1, 2017

The Strength of Evidence Pyramid

One Approach for Characterizing the Strength of Evidence of Geoscience Education Research (GER) Community Claims

Kristen St. John, James Madison University (stjohnke@jmu.edu) and Karen McNeal, Auburn University (ksm0041@auburn.edu).

The concepts introduced here were developed into a full paper, published in the JGE Geoscience Education Research theme issue: St. John and McNeal, 2017. The Strength of Evidence Pyramid: One Approach for Characterizing the Strength of Evidence of Geoscience Education Research (GER) Community Claims. Journal of Geoscience Education, Vol. 65, No. 4, pp. 363-372. https://doi.org/10.5408/17-264.1

Introduction:

One outcome of the 2015 workshop on Synthesizing Geoscience Education Research: Where are we? What is the path forward? was general agreement that GER conclusions, and recommendations that affect the future of GER, should be evidence-based. The strengths and limitations of GER individual study claims, as well as GER community-level claims should be transparent. While the strengths and limitations of claims made in individual studies are often addressed through author attention to journal standards and peer-review feedback, GER community-level claims currently do not have a framework to evaluate their strength of evidence. Therefore to address this need, we propose a conceptual diagram (shown below) as one approach for characterizing the strength of evidence of GER community claims.

Description:

There are five levels to this GER community-claims pyramid. The first level (green) is Practitioner Wisdom/Expert Opinion. This knowledge base of "what we know" about geoscience education research is also the interface where results of geoscience education research directly connect to teaching practice. Practitioner wisdom and expert knowledge is informed by GER studies at higher levels of the pyramid, as well as personal practitioner experience. It is not, however, a level that directly involves scholarly peer-review publication or original research and analysis. At this level, wisdom about teaching and learning is often shared via professional develop workshops, and though dissemination outlets such as the Science Education Resource Center (SERC) and the NAGT On the Cutting Edge webpages, as well as articles in the NAGT practitioner magazine, In the Trenches, and sometimes in commentaries in the Journal of Geoscience Education.

The next two levels of the pyramid represent original qualitative and quantitative studies that are peer-reviewed and published as primary literature. Most common are case studies (yellow) that focus on a single course or institution that is taught by the researcher using curriculum or instructional methods that they developed and are testing in their class. The methods of analysis often rely on a single instrument appropriate to that site (e.g., course or institution). The population depends on the location and scope of the study, but may be of limited diversity.

Less common, but important for determining generalizability of study findings are cohort studies (orange). These may address some of the same research questions as case studies but are investigating a broader cross section of courses, institutions, and/or populations. The instruments therefore must be broadly applicable as well, and the research design typically uses a mixed-methods approach. By increasing the depth and/or size of the study, researchers also take steps to reduce potential bias (e.g., they are not instructors of all of the courses that are testing an intervention).

The upper most levels of the GER community claims pyramid are the least common, in part because these depend on access to data, methods, and findings from previously published research. These types of studies result in increased data aggregation, syntheses, and generalizability; and are powerful for elevating confidence in GER community claims. Both types of studies need to be done carefully to minimize bias and avoid inclusion of data from poor quality studies (e.g., weak methods used in previously published work) in their analyses (Palermo, 2012). Meta-analyses (light blue) involve application of statistical methods to look at a broad suite of existing quantitative or qualitative data. The goal is to provide a more comprehensive description and analysis of a topic or question than could be addressed by smaller-scale case or cohort studies.

Systematic reviews (dark blue) use systematic and transparent methods to identify, select, and evaluate relevant published literature on a particular topic or question (see link http://www.cochrane-handbook.org/ 'The Cochrane Database of Systematic Reviews', Higgins & Green, 2011). Reviews should encompass a significant time frame of study for a topical area of research. They should include the most up to date research, as well as historical research, in order to provide an overview of the research evolution on that topic. Reviews of this nature can help identify patterns, trends, and gaps in GER, and thereby help identify important questions and areas of need future geoscience education research. Systematic reviews may use meta-analyses when appropriate and available.

The width of the pyramid represents the breadth of the field, and the relative number of studies at each level of the pyramid (or available resources in the case of sharing practitioner wisdom). We recognize that there is a spectrum to what the community means by "geoscience education research"; it involves the development and application of new teaching innovations and curricula, as well as the development and testing of research questions/hypotheses. These can be characterized as the scholarship of teaching and learning (geo-SoTL), and discipline-based educational research (geo-DBER), respectively. Both SoTL and DBER are important for improving the geoscience teaching practice. Both are also areas for peer-review publication. For example, in the Journal of Geoscience Education, geo-SoTL typically results in Curriculum & Instruction papers, and geo-DBER typically results in Research papers.

Ways in which this conceptual diagram be used:

  • To situate or contextualize GER results from different types of studies. For example, authors of literature review papers could identify the types of studies their analysis draws from (e.g., are they largely case studies or are they cohort studies? Do they use meta-analysis as part of their review methods?)
  • To support discussions on geoscience education research needs
  • To identify gaps in GER, such as the need for literature reviews of GER topical areas
  • To consider the generalizability of study findings
  • To consider the strength and limitations of aggregate results
  • To consider the strength and limitations of claims that influence geoscience education practitioner knowledge

Ways in which this conceptual diagram should not be used:

  • To evaluate the particular impact of an original study in the primary literature
  • To judge the value of qualitative vs. quantitative research methods used in individual studies

Information and ideas that guided our thinking in designing the conceptual diagram for strength of evidence of GER community claims:

  • The DBER report (Singer et al 2012) applied a 3-tiered rubric (see Box-1-1, p 18) on levels of evidence for conclusions they drew about discipline based education research (DBER) from reviewing a wide range of science studies. An modified version of this rubric is shown to the right.
  • The DBER report (Singer et al 2012) also reported on broad challenges in DBER. These are summarized in the table to the right, and are aligned to questions that these challenges raised in our minds.
  • The heath care community has defined a strength of evidence as a pyramid, and distinguishes between filtered and unfiltered (original studies) information. The purpose of the conceptual diagram is to help ensure that medical practitioners are can situate medical finds and advice in terms of strength of evidence. This approach seems tranferable to other fields, including geoscience education research, because ultimately GER should inform geoscience education practice (teaching and learning). We also recognized that expert opinions have value, and in our field this would include practitioner wisdom that may not always appear in formal studies, but is part of the knowledge base and is informed by research at all levels.
  • The following two resources helped guide our thinking on systematic reviews:
    • Higgins, J. P. T., & Green, S. (Eds.), (2011). Cochrane handbook for systematic reviews of interventions (Version 5.1.0) The Cochrane Collaboration. Available from www.cochrane-handbook.org (updated March 2011).
    • Palermo, T., (2012). New Guidelines for Publishing Review Articles in JPP: Systematic Reviews and Topical Reviews. Journal of Pediatric Psychology pp. 1–5, 2012. doi:10.1093/jpepsy/jss124

Provide feedback »