A Field Guide to Building Your Own Departmental Course and Curriculum Matrix for Student Learning Outcomes and Program Assessment

Prepared by David Mogk, Montana State University, and Mary Savina, Carleton College for the 2014 workshop Student Learning Outcomes and Program Assessment, part of the NAGT "4-Pack", GSA Annual Meeting, Vancouver, BC

1. Review your Institutional Mission Statement and Departmental Role and Scope Documents

Curriculum design and revision should be done in the context of your institutional mission and your department's role and scope. Take a few minutes and review these documents. This will provide some high level guidance about priorities, expectations and values in designing your curriculum. Individual courses, course sequences, and student learning outcomes should be well-aligned with the overall institutional mission.

In addition, your curriculum can be designed to optimize learning opportunities in consideration of:

• the geologic and geographic setting of your institution;
• demographic profile of the students you serve;
• professional strengths and interests of the existing faculty (and plans for future growth of the faculty);
• departmental and institutional facilities and equipment;
• opportunities for collaboration across campus (e.g. with sister disciplines), with government agencies, and with the community;
• the need to optimize resources available (faculty teaching assignments, TA support, limited student credit hours that can be applied to a degree), reduce redundancies, and realize economies of scale;
• service for special groups of students (e.g. pre-service teachers).

Reflection: Given the many inputs into your departmental ecosystem, how can you optimize opportunities for students to learn, given opportunities and constraints of staffing, location, institutional resources, etc.?

Activity: (to be done as a department, perhaps at a faculty retreat) Do aSWOT (strengths, weaknesses, opportunities, threats) analysisof your department to examine the internal and external factors that help or hinder your department in achieving each of your objectives.

2. Know Your Students

Who are your students? Where do they come from? What are their aspirations? Do you have a network of feeder schools (high schools, 2YCs)? Are there demographic profiles you should know about (e.g. first generation college students)? Are there documented barriers to learning among your incoming students? Do you serve (or potentially serve) an underrepresented student population? Is Earth Science taught in your public schools, at what level, are they employing the Next Generation Science Standards? Do your students typically plan to work in a particular sector of the workforce, in the area, or will they migrate?

Consider working with your Admissions Office and with your institution's "first year" advising group to better understand who your students are as they enter your institution. Work with your Alumni Association to do some longitudinal tracking to find out where they are going. Consider also developing a pre-program survey to determine interests, motivations and concerns; develop an exit survey to determine "what worked" in their experience in your department (and use the AGI Exit Survey to compare with national responses); use a longitudinal survey to determine what preparation helped your recent graduates succeed in their careers. This foundational information can help you determine where you should start in your curriculum design, what components should be developed and reinforced.

Reflection: Take a few minutes to think about your recent cohorts of students. As they entered your program, what were their interests, where did they struggle, what were their motivations, what were there expectations? For your recent graduates, what were the attributes of the students who excelled, and what components of your department's programs contributed to their success?

3. Profile of an Ideal Student Who Graduates From Your Department

Recent graduates are the best ambassadors for your department. They will showcase to employers, other graduate programs, and to the world the scientific knowledge and skills that they learned in your program.

What is the profile of a student who successfully completed your degree program—what should students know, what should they be able to do? We advocate using the "Understanding by Design" approach advocated by Wiggins and McTighe (2005). "Backward design" allows faculty to begin to define the knowledge, skills and personal attributes that contribute to students' professional development as they prepare for the workforce or graduate school.

Activity: Here is an exercise that can be used to help define the profile of the successful graduating student from your department:

• Ideal student exercise : Write a recommendation letter for an ideal student in your program who is a new graduate, applying for graduate school or a job. In the letter, describe some of the following:
  • What kind of person will this graduate be?
  • What will they be able to do?
  • What will they know?
  • What skills will they have?
  • How will they behave?
  • What will they value?

4. Programmatic Goals: New Realities

To help prepare students for their future careers in geoscience, program-level student outcomes should anticipate external factors that shape our profession:

• Changing nature of geoscience: increasingly using an Earth System Science Approach (from the Starting Point project) that emphasizes the connections among the components of the Earth system;
• Changing nature of workforce: Understanding Workforce Needs (from the InTeGrate project) that recognizes the need for technical competencies, but also expects workers who can solve problems, work in collaborative groups, and communicate to diverse audiences;
• Changing nature of geoscience research: New Research Opportunities in the Earth Sciences (NRC, 2011), that emphasizes topics such as fossil fuel and water resources, earthquake and tsunami hazards, and profound environmental changes due to shifts in the climate system.

The work of geoscientists is increasingly multi-, inter-, and transdisciplinary. Although disciplinary content knowledge and skills, including foundational understanding of "basic" sciences (physics, chemistry, biology, mathematics), is still necessary, program-level student learning outcomes may increasingly need to shift focus towards integrative, systems thinking.

In setting the overarching learning goals for students in your department, you should be able to state: All students graduating from our department SHOULD KNOW (....e.g. the geologic time scale, the basic mechanism of plate tectonics), BE ABLE TO (...e.g. measure geologic structures, identify rocks and minerals, develop a GIS....), and BE PREPARED TO(...e.g. be life-long learners, work in groups, write a report...). The goal is to develop an integrated, coherent and comprehensive curriculum that addresses the needs of the whole student in their pre-professional training.

Activity: Do aGallery Walk Exercise with your faculty to identify a) common themes that will be reinforced across the curriculum, b) technical skills that must be developed by geoscientists, and c) professional or "soft" skills that are required for the workforce (communication, interpersonal skills, etc.). Post "starter" questions on flip charts about content knowledge and key concepts, technical (geo-specific) skills, "soft" skills (communication, interpersonal, etc.), field experiences, etc. and have the faculty write on each chart the topics that they emphasize in their classes; this can be quite revealing as faculty may have no idea about what their colleagues are teaching in related classes; the gallery walk can open a department-wide discussion about areas of strength in the department and may serve as a gap analysis to identify areas that need more attention.

Alternate Activity: Have your faculty record on sticky notes (one skill, concept, etc. per note) three major geologic concepts,three important geoscience-specific skills (e.g. using a Brunton), three geologic habits of mind, three general skills (e.g. communication), three experiences, and three values that your program expects students to develop while they are enrolled. (See outline points below, for additional definitions). Organize the sticky notes on a wall or whiteboard to see what concepts/skills are most highly valued or emphasized by your faculty. Limiting faculty members to three each of skills, experiences, etc. removes some of the pressure to try to be inclusive. Chances are, all the important things will emerge and it will be easy to see theconsensus that already exists among the faculty members in a program. Once you have articulated that consensus, then you can review your curriculum with those points in mind.

a. Essential Geologic Concepts, Content, Knowledge

What should every geoscientist know to be able to a) continue their education, b) enter the workforce, c) communicate with the public? You might want to identify a few major "themes" or "threads" that are central to your instructional program (e.g. "deep time"....).

Consider also fundamental concepts from sister disciplines (physics, chemistry, biology, math....).

b. Technical or Geoscience-Specific Skills

What should every geoscientist be able to do? (e.g. Map reading; identification of rocks and minerals; collection and interpretation of structural data....?)

c. Geologic "Habits of Mind"

What personal attributes are essential for students to succeed in the geosciences? The Earth system is open, heterogeneous, dynamic and complex. Consequently, geoscientists must develop "habits of mind" that employ a wide variety of cognitive strategies that must address vast temporal and spatial scales of observation, an incomplete geologic record, uncertainty and ambiguity, and complex system behavior. Geoscience habits of mind were explored in some detail in the InTeGrate module on Teaching Geoscientific Thinking.

Think also about training in ethics. Where and how will this training be built into your program?

d. General, Professional or "Soft Skills"

What related skills are essential for students to succeed in the workforce? (e.g. communication, quantitative literacy, interpersonal skills). What skills do students need to be "lifelong learners"? To best prepare for the workforce? (These are skills that all university students might be expected to gain; how and where do they get reinforced in your program?)

e. Experiences

What types of curricular and co-curricular experiences are key to your program? For many geoscience programs, field work (in several forms) is a key experience for students. Does your department encourage student research? Is it required? Are there opportunities for students to mentor others, to organize programs?

f. Values, including service to society and/or stewardship of Earth?

Does your departmental or institutional mission address service to society, sustainability, etc.? If so, where and how will you incorporate these goals into your program?

5. Create your Program "Matrix"

Given the background information acquired from the previous steps, you can now begin to create your program matrix. See more complete details about the Matrix Approach. There are benefits to this approach for both students and faculty members (see Savina et al., 2001):

1. Faculty members learn about each others' courses and teaching styles, building collegiality.
2. Faculty members share responsibility by understanding what skills they are responsible for teaching and which their colleagues will teach.
3. The student experience is well-coordinated, making students feel better supported and prepared.

In creating this matrix we are focusing now on program-level learning outcomes, those topics that are not "owned" by any one course, but that occur as recurring themes in your curriculum. At a finer granularity, every course will have its own specific student learning outcomes, which will hopefully be aligned with the overall programmatic goals.

Activity: If you are working with colleagues within your program, divide your faculty into small groups to explore key topics, content, concepts or skills that you want to emphasize in your instructional program. (If several programs and institutions are represented, then each participant will create their own priority list for their department, discuss those priorities and encourage other group members). Take at least about 15 minutes/topics above to discuss and record, and end with report outs from the small groups. The goal is to have a tentative list of about 10-15 skills, experiences, etc. which can be formed into your program-level student learning outcomes. Each row of your matrix will list one of these items.

6. Review Your Current Course Offerings and other Opportunities that Support Student Learning

In light of the above program-level learning goals, does your current roster of courses make sense? Commonly courses are taught year to year with little revision, and without consideration of how the course "fits" into the whole curriculum. Here are a few topics to consider:

Activity: Create a "road map" of required courses for each of your degree options (many departments have multiple ones). This will help demonstrate the optimal path for students working their way through your degree program, identifying critical classes that are prerequisites of that may present other barriers to progress towards a degree. For faculty, this will help you to demonstrate (to each other and to administrators) that you have a coherent, comprehensive, and articulated series of courses that lead to a truly integrated curriculum. An example of course progressions developed at Montana State University can be found at: A Curriculum by Design. Note that geoscience programs vary in the amount of structure and articulation among courses. For less structured programs, it's even more important to consider the "rule of threes" described below. The courses and co-curricular experiences in your road map form the columns in the program matrix.

7. Filling in Your Program Matrix

Now that you have your essential content/skill goals defined, and the sequences of courses listed, you can fill in your matrix to see the extent of how and to what extent your program goals are addressed in your curriculum.

One strategy (used by Dave Mogk at Montana State University) was to have all the faculty fill out the matrix for the courses where they have primary responsibility for instruction. All faculty contributed to this self study (yes there are problems with self-reporting, but this at least gives you an overall "snapshot" of the state of your department). Faculty provided input about their contributions to student learning outcomes according to this rating scale: 3 (red)= this topic is central and essential to the course goals, and is strongly emphasized throughout the course; 2 (yellow)= this topic is considered in this course and supports course learning goals; 1 (blue) = this topic is introduced in the course but is not covered in depth; blank= topic not covered. The results were then compiled by the Department Head to get an overall "map" of the curriculum. This is a relatively efficient way to collect the required information, and most faculty reported that it was indeed useful for them to reflect on what they actually taught and valued in their classes.

Activity: Workshop participants (or faculty at home departments) can make a first pass at filling in the matrix according to your understanding of what is taught and emphasized (or not) in each class.

8. Using Your Matrix for Program Assessment

The beauty of the program matrix is that it provides a rapid, efficient way to survey the "landscape" of your program. Your departmental matrix can be used as an effective tool for formative assessments as a guide to revise or develop the existing curriculum, and as a summative assessment to demonstrate that departmental and institutional goals are being met.

The matrix can readily be sorted by any number of attributes for deeper analysis of your overall curriculum. For example, sort by:

• Level of Offering (100, 200, 300, 400): at each level, are the concepts/skills presented appropriate for the learning abilities of the students; are these well-aligned with the cognitive skills needed for students to succeed at this level? Are the course expectations aligned with student abilities?
• Degree Option (many departments have multiple degree options: Geology, Environmental Science, etc.): for each option, does the course of study address all or most of the critical PLLO's? Do the learning sequences in the degree options fully meet the PLLO's?
• Faculty: which of the PLLO's are strongly emphasized by individual faculty members? In aggregate, do the faculty fully address the PLLO's in the breadth of their coursework? Do some faculty have particular strengths in some areas that should be supported? Or, should some faculty be directed (encouraged) to shift emphasis a bit to meet overall departmental PLLO's?
• Frequency of Offering: in your curriculum do students have regular access to courses that emphasize PLLO's at appropriate stages in meeting their degree requirements.

Consider the following questions:

a. Does this curriculum reflect the best use of your (limited) resources?

Does each course address one or more of the essential learning goals identified above? Are there redundancies in your course coverage? Or critical gaps in coverage? Is this the best use of your faculty teaching credits? Or assignment of TAs? Do you have the classrooms, laboratories , equipment or other facilities to allow you to teach these courses in an optimal way? A critical review of your courses, and what is actually taught, may reveal some inefficiencies and/or economies of scale that may help you optimize your limited human and physical teaching resources.

b. Does your curriculum support development of higher order cognitive skills?

Consider where in your curriculum students have the opportunity to develop higher order thinking skills according to Bloom's Taxonomy of Cognitive Skills (more info) : knowledge, comprehension, application, analysis, synthesis, and evaluation. These cognitive skills can be developed purposefully in a geoscience program by developing appropriate class activities at different instructional levels across the curriculum.

c. Are your courses sequenced and scaffolded?

Does your curriculum provide appropriate support for learning via scaffolding (i.e. concepts, skills, strategies, ways of thinking that must be practiced early, often and with increased levels of difficulty and from different perspectives in a baccalaureate degree program)?

Have you developed Learning sequences that reinforce essential topics throughout the curriculum? (E.g. introductory courses must prepare students for more rigorous treatment of subject matter (or advanced skill development) in higher level classes; at the same time, upper division classes can reflect upon lessons learned in lower division courses).

Consider applying the "Rule of Threes" (or fours or fives): if something is worth learning, students need to have at least three solid exposures to that topic. A sequence of learning would provide early exposure to a topic in an introductory course, followed by familiarization, competence and hopefully mastery of key concepts and skills over the four-year curriculum.

d. What important learning happens outside the classroom?: The Co-Curriculum

Learning doesn't just take place in the classroom. Informal and extracurricular activities may contribute substantially to program-level student learning outcomes. Some areas that might be emphasized in your department to support programmatic learning goals:

• Thinking Beyond the Curriculum--through mentoring, advising, internships, student research experiences, and many other activities.
• Develop and support Internship Programs,
• Encourage and support Student Research Experiences;
• Enable students to participate in Field Experiences early and often;
• Provide opportunities for students to engage international studies;
• Develop "communities of scholars"; support a student Earth Science Association in your department; work with Residence Life on your campus to establish a dorm wing dedicated to Earth (or natural/physical) science majors; support a robust departmental field trip program;
• Departmental Speakers Series; expose students to the breadth of exciting new scholarship being done by local faculty and students and by national experts; (use this as a method to reinforce and validate the activities and skills emphasized in your courses);
• Start a journal club, with informal reading from the current popular press.
• Engage in a department-wide Service Learning project.

Formative Assessment

As a guide for course and curriculum design or revision, the matrix will display the evidence to:

• Provide an overview of students' educational experience in your department; does this curricular structure provide opportunities to develop the "model student" you hope to produce?
• Define a baseline of the current breadth and scope of instruction in the department;
• Identify areas of particular strength or emphasis in your department;
• Guide course revisions, to become better aligned with the defined Program Level Learning Outcomes (PLLOs) (minor course revisions may be accommodated as faculty address the PLLOs without too much effort);
• Provide a "gap analysis" of what new courses may be needed;
• Confirm learning sequences for a given topic or theme; are key concepts/skills introduced and emphasized at appropriate points in your curriculum? Alternatively,
• Determine if some courses may be redundant, providing the opportunity to drop or combine courses to achieve efficiencies (of faculty time, distribution of departmental resources, student credit hours if there are caps to credits that may be earned by students....).
• Demonstrate to what extent key instructional approaches are being used; e.g. what courses routinely offer field exercises (how many, how long), problem-based learning, computer modeling, writing or oral presentations, application of quantitative or statistical methods, etc.?

Summative Assessment

Departmental Role and Scope documents often establish goals that must be assessed for overall departmental productivity. The departmental matrix provides ready evidence of where and to what extent stated departmental goals are being met. Some examples include:

• The department contributes to the institutional mission. For example, many institutions have a "core curriculum" (i.e. distribution requirements): specific courses can be readily identified as meeting these institutional requirements (e.g. writing intensive courses, etc.)
• The department is moving towards an Earth System approach; if that's the case, then most courses should provide a breadth of coverage of topics that show the relationships among the components of the Earth system (breadth of topics identified in the Knowledge/Concepts part of the matrix, including relations to humanity);
• The department has an integrated curriculum that is designed with learning sequences that develop higher order thinking skills?
• The department provides authentic research experiences for students, both as in-class projects and through individual research projects.
• The department prepares students who are prepared a) to go to graduate school, or b) enter the workforce. Use the matrix to demonstrate where the "professional skills" are emphasized (communication, interpersonal skills, etc.).
• The department serves society by addressing the "grand challenges" of living on Earth; courses that emphasize geohazards and natural resources are readily identified.

Each department will have its own claims, its own high level goals. The departmental matrix will help provide the evidence that supports these claims.

Final Thoughts:

The matrix approach to program design and assessment has many benefits:

• The course matrix can provide an important baseline of the current breadth and scope of our instructional efforts across the Earth Sciences curriculum.
• The process can provide a forum for faculty to begin or continue conversations about the overall curriculum and how their courses contribute to the overall instructional mission.
• Many faculty can gain a better sense of what is being taught in related parts of the curriculum, and efforts are underway to better align our identified SLOs with learning progressions in our course sequences; and
• Many faculty have independently commented on the personal value they derived from this assessment exercise, as they reflected deeply on their course goals, areas of emphasis, and overall contributions to the curriculum.
• Overall, many faculty have never had had the opportunity to see the totality of what is covered in their curriculum, and it has been a very positive thing for the department to see the depth and breadth of the scholarship that is being done in our department.
• For full disclosure, not all faculty will be happy. Adjustments may need to be made in your curriculum such as a) discontinuing some courses that are deemed to be redundant, b) some courses may need to be compressed into new hybrid courses, c) the level of instruction of some courses may need to be changed, and d) some formerly required courses may need to be moved into the "free elective" (not required) courses that are now offered on alternate years.

Other Resources

A series of postings on the Earth and Mind blog about curriculum design efforts at Montana State University can be accessed at: A Curriculum by Design and A Curriculum by Design Part II: Student Learning Outcomes and Program Assessment