examining the teaching of science, and technology and ... · the rtop (piburn & sawada, 2000)...

Journal of Technology Education Vol. 29 No. 1, Fall 2017

-45-

Examining the Teaching of Science, and Technology and Engineering Content and Practices: An Instrument

Modification Study

Tyler S. Love, John G. Wells, & Kelly A. Parkes

Abstract A modified Reformed Teaching Observation Protocol (RTOP) (Piburn &

Sawada, 2000) instrument was used to separately examine eight technology and engineering (T&E) educators’ teaching of science, and T&E content and practices, as called for by the Standards for Technological Literacy: Content for the Study of Technology (International Technology Education Association [ITEA/ITEEA], 2000/2002/2007) and the Next Generation Science Standards: For States, By States (NGSS Lead States, 2013). The modified RTOP presented in this article can help provide feedback to teachers at all grade levels concerning their reformed teaching of science content and practices, and T&E content and practices. The instrument achieved acceptable interrater reliability (> 80%) and was tested in a larger study (Love & Wells, 2017). Results revealed a significant difference among participants’ teaching of science and T&E content and practices according to a myriad of variables, such as years of teaching experience, years of experience teaching the Foundations of Technology (FoT) curriculum, length of the FoT curriculum training attended, and select preparation experiences. Research implications suggest that the instrument affords equal applicability for examining science educators’ teaching of T&E content and practices and informing pre- and in-service teacher preparation efforts by determining key factors that significantly influence educators’ teaching of these concepts. Keywords: Technology and engineering education, science education, pedagogical content knowledge (PCK), observation instrument

The main goal of this research was to examine the teaching of science and technology and engineering (T&E) content and practices embedded within Foundations of Technology (FoT), an international T&E education curriculum. To investigate this, an observation instrument was needed to separately rate instructors’ teaching of science content and practices and instructors’ teaching of T&E content and practices. As a result, a reliable and practical observation instrument was developed to quantify the level at which P–12 educators teach these concepts. This instrument, modified from the widely used Reformed Teaching Observation Protocol (RTOP) (Piburn & Sawada, 2000) and a newly created rubric (Appendix B), helps raters provide more consistent scores for observed teaching practices while providing timely and detailed feedback for instructors to enhance their teaching of science and T&E content and practices.


-46-

This research initiates a baseline for effectiveness trials that are necessary to establish the reliability of an instrument for examining the extent to which educators are adequately teaching science and T&E concepts as mandated by both the Standards for Technological Literacy: Content for the Study of Technology (STL) (International Technology Education Association [ITEA/ITEEA], 2000/2002/2007) and the Next Generation Science Standards: For States, By States (NGSS) (NGSS Lead States, 2013).

Requisite to discussing the instrumentation is establishing a clear definition of the term T&E education to alleviate commonly held misconceptions (International Technology and Engineering Educators Association [ITEEA], 2017). “Technology education . . . provides an opportunity for students to learn about the processes and knowledge related to technology that are needed to solve problems and extend human capabilities” (ITEA/ITEEA, 2000/2002/2007, p. 242). Dugger and Naik (2001) further clarified that

Technology education is concerned with the broad spectrum of technology, which encompasses, but is not limited to, such areas as: design, making, problem solving, technological systems, resources and materials, criteria and constraints, processes, controls, optimization and trade-offs, invention, and many other human topics dealing with human innovation. (p. 31)

T&E education requires the use of various technologies, materials, and tools to develop engineering solutions, which is distinctly different from educational technology (ITEEA, 2017). Understanding the difference between T&E education and educational technologies is critical for recognizing and accurately recording T&E content and practices ratings when using the instrument presented in this article.

The modified instrument in this study was developed at an important time when science education was experiencing major changes. With the NGSS mandating that science educators integrate engineering content and practices within their curricula, it posed some legitimate concerns, specifically how science educators’ would be evaluated on their teaching of engineering content and practices—an area in which they had limited exposure and were not adequately prepared to teach in alignment with the NGSS (Nadelson & Farmer, 2012). NGSS later developed the Educators Evaluating the Quality of Instructional Products (EQuIP) rubric for measuring and providing constructive criterion-based feedback to better align lessons and units with the NGSS (NGSS Lead States, 2016). However, as we present in subsequent sections, the EQuIP rubric cannot be used to adequately examine educators’ separate teaching of science and T&E content and practices. Therefore, a more accurate and reliable instrument was needed.


-47-

Review of Literature The philosophical basis for this research is grounded in Shulman’s (1987)

concept of pedagogical content knowledge (PCK). Since its conception, PCK has been the topic of many notable publications within science education (e.g., Abell, 2008; Baxter & Lederman, 1999; Gess-Newsome, 1999; Hume & Berry, 2011; Loughran, Berry, & Mulhall, 2012). It has also been a controversial research topic, with science education experts questioning its existence (Settlage, 2013) and how accurately it can be examined (Gess-Newsome & Lederman, 1999). Despite these criticisms, research has verified that teachers possess varying degrees of PCK based upon experience and training (Shulman, 2004; Williams & Lockley, 2012). Given that science educators are now expected to teach engineering content and practices, logic dictates that they must have adequate PCK to properly teach engineering concepts within the context of science education. As the science profession moves forward in preparing classroom teachers to address engineering in science education, a reliable instrument will be needed for examining the extent of science educators’ PCK in T&E separate from their PCK in science. This research presents an instrument and method to address that need.

Numerous research efforts within science, technology, engineering, and mathematics (STEM) education have examined PCK using a variety of methods, such as classroom observations, interviews, multiple choice questionnaires, assessing student work, and analyzing think aloud teaching tasks (Love, 2013). Despite these findings, there is still no unified agreement among researchers on the best method or methods to assess PCK because of its complexity (Park, Jang, Chen, & Jung, 2011). The authors therefore found it necessary to analyze a multitude of studies from various STEM disciplines in order to construct viable instrumentation for investigating PCK in this study.

Loughran, Mulhall, and Berry (2004) developed the content representation (CoRe) and pedagogical and professional-experience repertoire (PaP-eRs) instruments. The CoRe instrument captures teachers’ holistic PCK relative to a specific topic, and the PaP-eRs instrument offers a view into a teaching–learning situation in which the content shapes the pedagogy. They have been widely used in STEM education studies (e.g., Bertram & Loughran, 2012; Hume & Berry, 2011; Rollnick, Mundalamo, & Booth, 2013; Williams, Eames, Hume, & Lockley, 2012; Williams & Lockley, 2012) but are still viewed as imperfect tools due to their topic specificity and time intensive nature. A rubric with three levels of proficiency to rate science teachers’ PCK from coded interview responses was created by Lee, Brown, Luft, and Roehrig (2007). Their rubric did not accurately reflect all essential elements of PCK nor did it provide much detail about the specific teaching of content and practices. Park et al. (2011) developed a similar rubric that was used to rate teachers’ PCK based on interview responses, lesson plans, and classroom observations. Like Loughran, Berry, and Mulhall (2007), they acknowledged that PCK is topic specific, but it


-48-

is not efficient to make a rubric for each specific science topic taught. Park et al. admitted that when using a rubric to assess PCK, raters have to make inferences from what the teacher says, believes, and does. They concluded that scoring training is an important yet time intensive component needed when using rubrics to rate teachers’ PCK.

The Cognitively Activating (COACTIV) Instruction project (Kunter et al., 2007) developed instrumentation to help distinguish between content knowledge and PCK. The COACTIV instrumentation consisted of 23 paper-and-pencil questions representing scenarios that a mathematics instructor may encounter with their students. PCK ratings from this instrument were limited to participants’ paper-and-pencil responses. Furthermore, Gumbo and Williams (2014) investigated the PCK of T&E educators by recording observations every 5 minutes, which created a series of snapshots of PCK elements observed over the duration of a lesson. They triangulated these observation data with both interviews and a content analysis of course textbooks. Although effective, they acknowledged that use of this instrument may not be practical for school systems because it demands a significant amount of time to collect and analyze data.

The most notable instrument considered by the authors was the EQuIP rubric (NGSS Lead States, 2016) because it was aimed toward measuring science educators’ alignment and overall quality of lessons and units in relation to the NGSS. However, it was determined that the EQuIP rubric was not suitable for this study due to its inability to delineate between the teaching of science content and practices and the teaching of engineering content and practices. Rather, the EQuIP rubric combined science and engineering content and practices by using and/or when addressing the teaching of science and engineering concepts (NGSS Lead States, 2016). The EQuIP rubric fails to hold instructors accountable for teaching science and engineering concepts adequately. If using such an instrument to rate observed instruction of both science and engineering concepts, teachers would either rate extremely high or extremely low depending on how the term and/or is interpreted by the rater. This is a major reason that the authors chose not to utilize the EQuIP rubric for this study, because it was pertinent to investigate the differences between teaching of engineering and science content and practices. The instrument that the authors found most suitable for this study was the RTOP (Piburn & Sawada, 2000). The Reformed Teaching Observation Protocol

The RTOP (Piburn & Sawada, 2000) could be easily modified to help delineate between the teaching of science and T&E content and practices. It also examined instructors’ reformed teaching methods, which aligned well with the NGSS’s expectations for teaching scientific inquiry and engineering design. The RTOP is grounded in constructivism and was designed as an observational


-49-

instrument to measure reformed teaching of mathematics and science. Various studies (e.g., Nicholas & Lomas, 2009; Ogletree, 2007; Park et al., 2011) have used it in conjunction with other instruments to measure teachers’ PCK. Specifically, Nicholas and Lomas (2009) found that the fourth section of the RTOP was able to assess teachers’ understanding of key content and provide valuable insight about their PCK. Piburn and Sawada (2000) and Taylor et al. (2013) found the RTOP to be a reliable and valid instrument aligned with national mathematics and science standards documents (e.g., American Association for the Advancement of Science [AAAS], 1989, 1993; National Council of Teachers of Mathematics [NCTM], 1989, 1991, 1995; National Research Council [NRC], 1996). Because of its alignment with the NGSS’s recommendation for research-based instructional reform (Taylor et al., 2013), the RTOP was deemed adequate to rate observations in this study.

As previously discussed, although a variety of instruments have been used to evaluate the PCK of science and T&E teachers, none were comprehensive in their evaluation of teacher practices. Given the growing focus in P–12 education on integrating the content and practices from multiple disciplines within a single subject, an instrument robust enough to identify specific instructional areas requiring further pedagogical preparation would be an important mechanism for informing pre- and in-service teacher preparation programs on those experiences identified as necessary for preparing educators to teach cross-disciplinary STEM concepts. Hence, modifications to the RTOP instrument were warranted. The following research questions helped guide the instrument modifications and data collection.

1. How accurately can differences among educators’ teaching of science content, science practices, T&E content, and T&E practices each be quantified with a practical and reliable observation instrument?

2. To what extent does instructors’ effectiveness in teaching science content, science practices, T&E content, and T&E practices differ according to the type of teacher preparation completed and the amount of teaching experience?

Methodology

The methodology employed in this study was designed to examine the preparation factors influencing the teaching of science concepts embedded within the FoT curriculum.1 The authors decided to examine the teaching of T&E educators, as opposed to science educators who were only tasked with teaching new content and practices within the past year. T&E educators have been expected to integrate science and engineering concepts since the release of the STL (ITEA/ITEEA, 2000/2002/2007) 16 years ago, making them a more

1 Details regarding the method employed are those described in previously reported research (see Love & Wells, 2017).


-50-

viable population for testing the content and practice items of the instrument. The expectation was that T&E educators’ increased experience with teaching these concepts would provide more accurate ratings that could otherwise be impacted by recently being tasked with teaching new content and practices.

Participants

This study utilized the same pool of participants as described in Love & Wells (2017). First, an online survey collecting demographic and preparation data was sent to all FoT teachers within 12 county school systems of a mid-Atlantic state. Of the 55 survey respondents, eight were purposefully selected for the classroom observation portion, which utilized the modified RTOP. The purposeful selection ensured that a sample of teachers with varying levels of science and T&E preparation experiences were observed. The demographics and preparation experiences of the observed participants are reported in Table 1.

Table 1 Demographic Data for Observed Participants

Part. Gen. Eth. Age Years of teaching

Years teaching

FoT

Years teaching science

T&E education certified

FoT training

Inter- Disciplinary

STEM course

T 1 M C 62 30 2 0 Yes .5 day No T 2 M C 47 10 8 1 Yes 1 week Yes T 3 F C 24 2 2 0 Yes 1 week No T 4 M C 47 13 5 2 Yes .5 day Yes T 5 M C 56 33 4 0 Yes .5 day Yes T 6 M C 61 28 10 0 Yes 1 week No T 7 M C 59 21 6 0 Yes 1 week Yes T 8 M AA

25 3 3 0 No 1 week No

Notes: Part. = Participant; Gen. = Gender; Eth. = Ethnicity; T = Teacher; M = Male; F = Female; C = Caucasian; AA = African American.

The observed participants consisted of predominantly White males who were certified to teach T&E education. The mean age of the participants was 48 years, and the average number of years teaching was 18. They had very little if any experience teaching science courses and attended some form of training to learn how to teach the FoT curriculum. About half of the participants reported taking a higher education course that discussed methods for teaching interdisciplinary STEM concepts (e.g., science and engineering). The demographic and preparation data from this research and the full study (Love & Wells, 2017) were consistent with studies examining T&E educators at a national level (Love, 2015; Ernst & Williams, 2015).


-51-

Modifying the RTOP and Developing the Rubric Subscale 4 regarding content was modified to accurately examine the

content and practices (PCK) of science and T&E concepts. These modifications allowed the data to be consistent with the language and teaching strategies described in the NGSS. The RTOP was modified with help from content specialists who had expertise in teacher evaluation, science education, and T&E education. The adaptations began by duplicating Subscale 4 and creating two similar yet separate subscales: one to score teaching of T&E content and practices and one for science content and practices. The words subject matter and concepts were replaced with the term content (Appendix A) to better align with the NGSS. This made it easier for raters to distinguish between teachers’ content knowledge and pedagogical practices during observations. Additionally, a rubric for each of the Subscale 4 criteria was created to provide clarity regarding what was being observed and to help elicit more consistent ratings (Appendix B). The content specialists suggested modifications to the rubric using language and criteria similar to that provided in the training guide section of the RTOP Reference Manual (Piburn & Sawada, 2000). This rubric helped observers determine ratings more easily and consistently while also staying true to the criteria described in the original RTOP. Training and Interrater Reliability

Nicholas and Lomas (2009) established that the use of the RTOP by one trained rater consistently provided a valid assessment of teaching practice in a single classroom observation (about an hour). Due to similar time and funding limitations as experienced by Nicholas and Lomas, one trained rater conducted observations and rated participating teachers in this study. To ensure RTOP rating accuracy and reliability prior to actual use in research observations, content experts were used in establishing an acceptable interrater reliability. Establishing interrater reliability was conducted through two RTOP usage sessions with two content specialists who had expertise in Integrative STEM Education (Wells, 2016) teaching practices. All of the raters completed the online RTOP tutorials (Buffalo State University of New York, 2007). Following completion of the online modules, raters were asked to use the modified RTOP to rate two video-recorded FoT lessons from the same units as those observed later during the data collection.

In line with standard protocols for establishing interrater reliability, three rounds were needed to reach consensus with an acceptable level of reliability. The first round had each rater use 10% of the RTOP items to independently analyze the first FoT lesson video, which was followed by a second round using an additional 10% of the RTOP items. At the end of each round, arbitration among raters was conducted to compare, discuss, and justify differences in co-ratings. The same procedures were followed for analyzing the second FoT lesson video, which was followed by a third round using the remaining 80% of


-52-

the RTOP items. At this point, an acceptable level of interrater agreement, greater than 80% (Howell, 2007), had been achieved (see Table 2), resulting in a viable coding scheme for using the RTOP for scoring a FoT lesson.

Table 2 RTOP Interrater Reliability Percentage Established Among Raters

Observation 1 Observation 2

Round Rater 1

Rater 2

Rater 3

Total Rater 1

Rater 2

Rater 3

Total

1 80% 60% 40% 60% 100% 57% 57% 71% 2 67% 50% 50% 56% 100% 86% 86% 90% 3 - - - - 86% 90% 71% 83%

Note: Total was the percent agreement among all raters by round.

During the classroom observation, the researcher remained as unobtrusive as possible by taking notes on what they observed regarding the teaching of both science and T&E content and practices. To provide the most accurate RTOP ratings, it was determined that lessons should be recorded for later review. To do this, the teacher was given a lapel microphone, and the audio recording was linked to the researcher’s notes using the AudioNote software. This software allowed the researcher to click on any portion of their notes and play the corresponding audio recorded during that part of the observation. Immediately after the completion of each school visit, the researcher used the modified RTOP instrument to rate the overall teaching strategies that they observed during the lesson. To ensure the ratings were as accurate as possible, within 48 hours, the researcher reviewed the lesson audio and their corresponding notes to confirm or adjust the ratings. After all observations were completed, they were again reanalyzed all at once via the audio recordings and notes for consistency across observation ratings.

In modifying the RTOP for use in assessing FoT teaching practices, there are recognized limitations that are worthy of mention. Specifically, observer ratings could only be provided based on criterion specified by the instrument. Although Nicholas and Lomas (2009) found the RTOP acceptable for a single observer to rate one lesson, the ratings in this study only reflect a snapshot of the teacher’s full range of instructional practices. When using the RTOP, ratings are dependent upon the rater’s knowledge of teaching both science and T&E content and practices. Additionally, the sample of observed teachers consisted of primarily White males, which presents a limitation considering the diversity in the actual population of T&E educators. However, it is noteworthy that the convenience sample used in this study is actually reflective of national T&E


-53-

educator characteristics (Love, 2015; Ernst & Williams, 2015). Accounting for recognized limitations, the modified RTOP provided a viable instrument for quantifying teacher levels of science and T&E PCK, which is discussed in the following section.

Observations and Results Relationship Among Observations and Preparation Factors

The sum of all seven RTOP category ratings for each participant ranged from 18 to 118 out of a possible 140 with a mean of 59.4 (42%). Specifically in terms of science content and practices, the mean rating for teaching of science content was 9.6 (48%), and the mean rating for teaching of science practices was 5.8 (29%). Four T&E teachers scored 6 (30%) or lower on teaching of science content, and three received a score of 1 (5%) on the teaching of science practices. Conversely, these teachers scored higher in observed teaching of T&E content and practices, as demonstrated by their mean ratings of 13.6 (68%) for T&E content and 7.6 (38%) for T&E practices. When examining these scores in more detail, there were only three participants who scored 9 (45%) or lower out of a possible 20 for teaching T&E content, but three teachers scored a two (10%) or lower for teaching T&E practices (see Table 3). Table 3 Participants’ Observation Ratings: Scores According to RTOP Categories

Category and score

Part. LD&I SC T&E C SP T&E P CI S/TR Total

T 1 2 6 7 1 1 3 3 23 T 2 12 9 17 6 12 12 10 78 T 3 2 3 7 1 1 3 3 20 T 4 12 17 20 10 16 13 14 102 T 5 15 19 20 15 16 15 18 118 T 6 0 3 9 1 2 1 2 18 T 7 6 14 19 7 6 7 10 69 T 8 4 6 10 5 7 9 6 47 Mean 6.6 9.6 13.6 5.8 7.6 7.9 8.3 59.4

Notes: Part. = Participant; T = Teacher; LD&I = lesson design and implementation; SC = science content; T&E C = technology and engineering content; SP = science practices; T&E P = technology and engineering practices; CI = communicative interactions; S/TR = student–teacher relationships. Scores for each category range from 0–20, with higher scores indicating a greater rating.


-54-

When analyzing the RTOP scores according to the type of teacher preparation experiences participants had, all three groups received their highest mean ratings in T&E content and their lowest in science practices. Further analysis based on preparation experiences indicated that the participant holding an engineering degree had the highest mean RTOP rating (102), whereas teachers who completed teacher preparation programs in disciplines outside of science and T&E education received the lowest mean score (48; see Table 4). Table 4 Summary of RTOP Ratings According to Preparation

Teacher preparation

Total n(μ) SC(μ) T&E C(μ) SP(μ) T&E P(μ)

T&E teacher prep

4(58) 10.5 13 6 6

Non-T&E teacher prep

3(48) 6 9 2 3

Engineering prep

1(102) 17 20 10 16

Notes: SC = science content; T&E C = technology and engineering content; SP = science practices; T&E P = technology and engineering practices. Scores for each category range from 0–20, with higher scores indicating a greater rating.

Similar findings emerged when analyzing the data according to teaching experience categories. Novice teachers (1–5 years) had the lowest mean RTOP rating (33.5), veteran teachers (16 or more years) had the next highest rating (57), and intermediate teachers (6–15 years) had the highest rating (90). Again, all groups recorded their highest mean ratings in T&E content and their lowest in science practices (see Table 5). Table 5 Summary of RTOP Ratings According to Experience

Experience level

Total n(μ)

SC(μ) T&E C(μ) SP(μ) T&E P(μ)

Novice 2(33.5) 4.5 8.5 3 4 Intermediate 2(90) 13 18.5 8 14 Veteran 4(57) 10.5 14 6 6

Notes: SC = science content; T&E C = technology and engineering content;


-55-

SP = science practices; T&E P = technology and engineering practices. Scores for each category range from 0–20, with higher scores indicating a greater rating.

The breadth of experience levels represented by participating teachers resulted in a range of high and low ratings. Consistent across type of preparation and teaching experience were the low scores in teaching science practices and high scores in teaching T&E content (see Tables 4 and 5). Among all eight participants, only two (Teachers 4 and 5) received a perfect score on any item, which occurred in teaching T&E content. Veteran Teachers 1 and 6 earned two of the lower total cumulative ratings for all seven categories (18 and 23 respectively) among all observed participants, whereas intermediate teachers posted some of the higher summative ratings (102, 78). Furthermore, two of the lowest ratings were received by veteran instructors with over 28 years of teaching experience. It is tempting to make inferences with respect to the pedagogical expectations of veteran teachers, and caution is needed here not to draw inaccurate conclusions. The data in this study were not large enough to draw such conclusions; therefore, further analysis was done in order to identify the types and amounts of preparation experiences that had significant positive correlations with observed teaching of content and practices (Love & Wells, 2017).

Conclusions and Implications

The overarching goal of this research was to develop an instrument that could separately examine the teaching of science and T&E content and practices. Recognizing the limitations imposed by the number of participants and their demographics, our analysis would suggest that the modified RTOP, together with the criterion-based rubric, has merit for discerning the degree to which the preparation of T&E teachers provides them with sufficient science content and practices to teach the science inherent within lessons such as those for FoT. The data suggest that the instrument could serve as a reliable and feasible observation tool to help school systems better focus their professional development efforts. One of the benefits of the instrument is that it directly measures those instructional strategies observed, as opposed to test question responses, and this research provides some evidence to suggest that the modified RTOP has potential for use with other science or T&E lessons. Further research would be needed to substantiate this potential.

Although observation of T&E participants was limited to a single 1-hour lesson, their limited formal preparation in science content and practices (Love, 2015; Love & Wells, 2017), coupled with low ratings on teaching science concepts in the lessons (Table 3), suggests the need to investigate whether or not similar results would be found for science teachers attempting to teach T&E content and practices inherent within science lessons. Considering that pre-


-56-

service science teacher preparation programs prior to NGSS did not address engineering practices, such research might prove useful in efforts to enhance science educators’ effectiveness in teaching engineering concepts with the rigor needed for students to make higher level cross-disciplinary connections. Moreover, the ratings found in relation to preparation type (Table 4) and years of teaching experience (Table 5) further emphasize the importance of examining the amount and types of preparation factors that influence instructors’ teaching of science and T&E concepts. However, in spite of these findings, further research examining teacher preparation experiences that influence science educators’ teaching of T&E concepts is needed to ensure a more holistic preparation of those instructors implementing the NGSS. The instrument presented in this article provides a starting point for such research and the potential for highlighting explicit areas in which school systems should focus their professional development efforts.

Our findings have implications for examining the science and T&E PCK of teacher educators who are preparing science and T&E educators. With the recently released NGSS and mandated integration of engineering concepts, few would argue that the science educators preparing tomorrow’s science teachers may benefit from professional development on how best to integrate engineering concepts within the science curriculum. One readily available solution is collaboration between the science and T&E education programs in preparing both science and T&E teachers with the PCK that they need to teach cross-disciplinary STEM concepts (Wells, 2008).

References Abell, S. K. (2008). Twenty years later: Does pedagogical content knowledge

remain a useful idea? International Journal of Science Education, 30(10), 1405–1416. doi:10.1080/09500690802187041

American Association for the Advancement of Science (AAAS) (1989). Science for all Americans. New York, NY: Oxford University Press.

American Association for the Advancement of Science (AAAS) (1993). Benchmarks for science literacy. New York, NY: Oxford University Press.

Baxter, J. A., & Lederman, N. G. (1999). Assessment and measurement of pedagogical content knowledge. In J. Gess-Newsome & N. G. Lederman (Eds.), Examining pedagogical content knowledge: The construct and its implications for science education (pp. 147–161). Dordrecht, the Netherlands: Kluwer Academic. doi:10.1007/0-306-47217-1_6

Bertram, A., & Loughran, J. (2012). Science teachers’ views on CoRes and PaP-eRs as a framework for articulating and developing pedagogical content knowledge. Research in Science Education, 42(6), 1027–1047. doi:10.1007/s11165-011-9227-4


-57-

Buffalo State University of New York. (2007). Reformed teaching observation protocol. Retrieved from http://physicsed.buffalostate.edu/AZTEC/RTOP/RTOP_full/index.htm

Dugger, Jr., W. E., & Naik, N. (2001). Clarifying misconceptions between technology education and educational technology. The Technology Teacher, 61(1), 31–35.

Ernst, J. V., & Williams, T. O. (2015). The “who, what, and how conversation”: Characteristics and responsibilities of current in-service technology and engineering educators. The Journal of Technology Studies, 41(1), 48–56. doi:10.21061/jots.v41i1.a.6

Gess-Newsome, J. (1999). Pedagogical content knowledge: An introduction and orientation. . In J. Gess-Newsome & N. G. Lederman (Eds.), Examining pedagogical content knowledge: The construct and its implications for science education (pp. 3–17). Dordrecht, the Netherlands: Kluwer Academic. doi:10.1007/0-306-47217-1_1

Gess-Newsome, J., & Lederman, N. G. (Eds.). (1999). Examining pedagogical content knowledge: The construct and its implications for science education. Dordrecht, the Netherlands: Kluwer Academic. doi:10.1007/0-306-47217-1

Gumbo, M. T., & Williams, P. J. (2014). Discovering Grade 8 technology teachers’ pedagogical content knowledge in the Tshwane District of Gauteng Province. International Journal of Educational Sciences, 6(3), 479–488.

Howell, D. C. (2007). Statistical methods for psychology (6th ed.). Belmont, CA: Thomson Wadsworth.

Hume, A., & Berry, A. (2011). Constructing CoRes—a strategy for building PCK in pre-service science teacher education. Research in Science Education, 41(3), 341–355. doi:10.1007/s11165-010-9168-3

International Technology and Engineering Educators Association (ITEEA). (2017). Technological literacy standards: Technology education vs educational technology. Retrieved from https://www.iteea.org/Activities/2142/Technological_Literacy_Standards/45979/51801.aspx

International Technology Education Association (ITEA/ITEEA). (2000/2002/2007). Standards for technological literacy: Content for the study of technology (3rd ed.). Reston, VA: Author.

Kunter, M., Klusmann, U., Dubberke, T., Baumert, J., Blum, W., Brunner, M., Jordan, A., Krauss, S., Löwen, K., Neubrand, M., & Tsai, Y.-M. (2007). Linking aspects of teacher competence to their instruction: Results from the COACTIV project. In M. Prenzel (Ed.), Studies on the educational quality of schools: The final report on the DFG Priority Programme (pp. 32–52). Münster, Germany: Waxmann.


-58-

Lee, E., Brown, M. N., Luft, J. A., & Roehrig, G. H. (2007). Assessing beginning secondary science teachers’ PCK: Pilot year results. School Science and Mathematics, 107(2), 52–60. doi:10.1111/j.1949-8594.2007.tb17768.x

Loughran, J., Berry, A., & Mulhall, P. (2007). Pedagogical content knowledge: What does it mean to science teachers? In R. Pintó & D. Couso (Eds.), Contributions from science education research (pp. 93–105). Dordrecht, the Netherlands: Springer. doi:10.1007/978-1-4020-5032-9_7

Loughran, J., Berry, A., & Mulhall, P. (2012). Understanding and developing science teachers’ pedagogical content knowledge. Rotterdam, the Netherlands: Sense. doi:10.1007/978-94-6091-821-6

Loughran, J., Mulhall, P., & Berry, A. (2004). In search of pedagogical content knowledge in science: Developing ways of articulating and documenting professional practice. Journal of Research in Science Teaching, 41(4), 370–391. doi:10.1002/tea.20007

Love, T. S. (2013). Theoretical underpinnings toward assessing science pedagogical content knowledge (PCK) of technology educators. In J. Williams & D. Gedera (Eds.), Technology education for the future – A play on sustainability. Proceedings of the 27th Pupil's Attitude Toward Technology Conference, Christchurch, New Zealand: University of Waikato (pp. 291-296).

Love, T. S. (2015). Examining the demographics and preparation experiences of foundations of technology teachers. The Journal of Technology Studies, 41(1), 58-71. doi.org/10.21061/jots.v41i1.a.7

Love, T. S., & Wells, J. G. (2017). Examining correlations between the preparation experiences of U.S. technology and engineering educators and their teaching of science content and practices. International Journal of Technology and Design Education. doi: 10.1007/s10798-017-9395-2. Retrieved from http://rdcu.be/o61P

Nadelson, L. S., & Farmer, C. (2012, July 31). Developing standards for teaching engineering. NSTA WebNews Digest. Retrieved from http://www.nsta.org/publications/news/story.aspx?id=59528

National Council of Teachers of Mathematics (NCTM) (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author.

National Council of Teachers of Mathematics (NCTM) (1991). Professional standards for teaching mathematics. Reston, VA: Author.

National Council of Teachers of Mathematics (NCTM) (1995). Assessment standards for school mathematics. Reston, VA: Author.

National Research Council (NRC) (1996). National science education standards. Washington, DC: National Academy Press. doi:10.17226/4962

Nicholas, P., & Lomas, G. (2009). The impact of a university course focusing on PCK and MCK: Does teachers’ classroom practice reflect the professional development experience? In Findings from the New Zealand numeracy


-59-

development projects (pp. 189–197). Wellington, New Zealand: Learning Media Limited. Retrieved from https://nzmaths.co.nz/sites/default/files/Numeracy/References/Comp09/comp09_lomas_nicholas.pdf

NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: National Academies Press. doi:10.17226/18290

NGSS Lead States. (2016). EQuIP rubric for lessons & units: Science. Retrieved from http://www.nextgenscience.org/resources/equip-rubric-lessons-units-science

Ogletree, G. L. (2007). The effect of fifth grade science teachers' pedagogical content knowledge on their decision making and student learning outcomes on the concept of chemical change. (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3313734)

Park, S., Jang, J.-Y., Chen, Y.-C., & Jung, J. (2011). Is pedagogical content knowledge (PCK) necessary for reformed science teaching?: Evidence from an empirical study. Research in Science Education, 41(2), 245–260. doi:10.1007/s11165-009-9163-8

Piburn, M., & Sawada, D. (2000). Reformed Teaching Observation Protocol (RTOP): Reference manual (ACEPT Technical Report No. IN00-3). Tempe, AZ: Arizona Collaborative for Excellence in the Preparation of Teachers, Arizona State University.

Rollnick, M., Mundalamo, F., & Booth, S. (2013). Concept maps as expressions of teachers’ meaning-making while beginning to teach semiconductors. Research in Science Education, 43(4), 1435–1454. doi:10.1007/s11165-012-9314-1

Settlage, J. (2013). On acknowledging PCK’s shortcomings. Journal of Science Teacher Education, 24(1), 1–12. doi:10.1007/s10972-012-9332-x

Shulman, L. S. (1987). Knowledge and teaching: Foundations of the new reform. Harvard Educational Review, 57(1), 1–22. doi:10.17763/haer.57.1.j463w79r56455411

Shulman, L. S. (2004). Teaching as community property: Essays on higher education. San Francisco, CA: Jossey-Bass.

Taylor, J. A., Getty, S. R., Kowalski, S. M., Wilson, C. D., Carlson, J., & Van Scotter, P. (2013). An efficacy trial of research-based curriculum materials with curriculum-based professional development. Colorado Springs, CO: Biological Sciences Curriculum Study. Retrieved from https://bscs.org/efficacy-trial-research-based-curriculum-materials-and-professional-development

Wells, J. G. (2008, November). STEM education: The potential of technology education. Paper presented at the 95th Annual Mississippi Valley Technology Teacher Education Conference, St. Louis, MO. Retrieved from http://www.mississippivalley.org/wp-


-60-

content/uploads/2015/12/Wells_2008_MississippiValleyConference_STEM-ED_TE-Potential.pdf

Wells, J. G. (2016). PIRPOSAL model of integrative STEM education: Conceptual and pedagogical framework for classroom implementation. Technology and Engineering Teacher, 75(6), 12–19.

Williams, J., Eames, C., Hume, A., & Lockley, J. (2012). Promoting pedagogical content knowledge development for early career secondary teachers in science and technology using content representations. Research in Science & Technological Education, 30(3), 327–343. doi:10.1080/02635143.2012.740005

Williams, J., & Lockley, J. (2012). Using CoRes to develop the pedagogical content knowledge (PCK) of early career science and technology teachers. Journal of Technology Education, 24(1), 34–53. doi:10.21061/jte.v24i1.a.3

About the Authors

Tyler S. Love ([email protected]) is Associate Professor and Coordinator of Technology & Engineering Education at the University of Maryland Eastern Shore

John G. Wells ([email protected]) is Associate Professor of Integrative STEM Education at Virginia Tech

Kelly A. Parkes ([email protected]) is Associate Professor in the Department of Arts and Humanities at Teachers College, Columbia University


-61-

Appendix A

Propositional Knowledge: Science Content

6a. The lesson involved detailed explanations and examples about fundamental science content identified by the curriculum.

0 1 2 3 4

7a. The lesson promoted strongly coherent conceptual understanding of science content.

0 1 2 3 4

8a. The teacher had a solid grasp of the science content presented in the lesson.

0 1 2 3 4

9a. Elements of scientific abstraction (e.g., symbolic representations, theory building) were encouraged when it was important to do so.

0 1 2 3 4

10a. Connections with science content to other content disciplines and/or real world phenomena were explored and valued.

0 1 2 3 4

Propositional Knowledge: T&E Content

6b. The lesson involved detailed explanations and examples about fundamental T&E content identified by the curriculum.

0 1 2 3 4

7b. The lesson promoted strongly coherent conceptual understanding of T&E content.

0 1 2 3 4

8b. The teacher had a solid grasp of T&E content presented in the lesson. 0 1 2 3 4

9b. Elements of T&E abstraction (e.g., symbolic representations, theory building) were encouraged when it was important to do so.

0 1 2 3 4

10b. Connections with T&E content to other content disciplines and/or real world phenomena were explored and valued.

0 1 2 3 4


-62-

Procedural Knowledge: Science Practices

11a. Students used a variety of means (simulations, drawings, graphs, concrete materials, manipulatives, etc.) to represent science phenomena.

0 1 2 3 4

12a. Students made predictions, estimations and/or hypotheses about key science concepts, and devised means for testing them.

0 1 2 3 4

13a. Students were actively engaged in thought-provoking activity that often involved the critical assessment of science procedures.

0 1 2 3 4

14a. Students were reflective about their science learning. 0 1 2 3 4

15a. Intellectual rigor, constructive criticism, and the challenging of scientific content were valued.

0 1 2 3 4

Procedural Knowledge: T&E Practices

11b. Students used a variety of means (models, prototypes, drawings, graphs, concrete materials, manipulatives, etc.) to represent T&E phenomena.

0 1 2 3 4

12b. Students made predictions, estimations and/or hypotheses about key T&E concepts, and devised means for testing them.

0 1 2 3 4

13b. Students were actively engaged in thought-provoking activity that often involved the critical assessment of T&E procedures.

0 1 2 3 4

14b. Students were reflective about their T&E learning. 0 1 2 3 4

15b. Intellectual rigor, constructive criticism, and the challenging of T&E content were valued.

0 1 2 3 4


-63-

Appendix B

Question # 0 1 2 3 4

6a & 6b

Detailed explanations and

examples

Targeted fundamental content was never mentioned.

Targeted fundamental content was rarely mentioned and was not a focal point of the lesson.

The lesson had very little focus on targeted fundamental content.

The lesson was focused to some extent on targeted fundamental content.

The lesson was focused entirely around targeted fundamental content.

7a & 7b

Coherent conceptual

understanding

Targeted content was not interrelated with any other concepts.

Targeted content was vaguely interrelated with other concepts to rarely increase its meaning.

Targeted content was vaguely interrelated with other concepts to sometimes increase its meaning.

Targeted content was vaguely interrelated with other concepts to moderately increase its meaning.

Targeted content was strongly interrelated with other concepts to greatly increase its meaning.

8a & 8b

Solid grasp of content

No evidence that the teacher has a solid grasp of the content in the information presented to the class.

Teacher rarely illustrates a solid grasp of the content in the information presented to the class.

Teacher sometimes illustrates a solid grasp of the content in the information presented to the class.

Teacher frequently illustrates a solid grasp of the content in the information presented to the class.

Teacher regularly illustrates a solid grasp of the content in the information presented to the class.

9a & 9b

Elements of abstraction

Relationships were never represented in abstract and/or symbolic ways when it was important to do so.

Relationships were rarely represented in abstract and/or symbolic ways when it was important to do so.

Relationships were sometimes represented in abstract and/or symbolic ways when it was important to do so.

Relationships were often represented in abstract and/or symbolic ways when it was important to do so.

Relationships were regularly represented in abstract and/or symbolic ways when it was important to do so.

10a & 10b

Connections to other

disciplines/real world

Targeted content was never connected with content across disciplines or with a real world application example.

Targeted content was rarely connected with content across disciplines or a real world application example.

Targeted content was sometimes connected with content across disciplines or included an example of a real world application.

Targeted content was often connected with content across disciplines or included 2 examples of real world applications.

Targeted content was regularly connected with content across disciplines and included more than 2 examples of real world applications.


-64-


11a & 11b

Means to represent

phenomena

Students never used a variety of practices (models, drawings, graphs, concrete materials, manipulatives, etc.) to represent targeted science phenomena.

Students incompletely used a variety of practices (models, drawings, graphs, concrete materials, manipulatives, etc.) to represent targeted science phenomena.

Students sometimes (once or twice) used a complete practice (models, drawings, graphs, concrete materials, manipulatives, etc.) to represent targeted science phenomena.

Students often (three) used a variety of complete practices (models, drawings, graphs, concrete materials, manipulatives, etc.) to represent targeted science phenomena.

Students consistently (four or more) used a variety of complete practices (models, drawings, graphs, concrete materials, manipulatives, etc.) to represent targeted science phenomena.

12a & 12b

Made Predictions/ Estimations/ Hypotheses

Students were not led to state predictions, estimations, and/or hypotheses associated with the targeted content, and did not have to devise ways to test it.

Students were vaguely led to state predictions, estimations, and/or hypotheses associated with the targeted content, and did not have to devise ways to test them.

Students were clearly led to state predictions, estimations, and/or hypotheses associated with the targeted content, and did not have to devise ways to test them.

Students were explicitly led to state predictions, estimations, and/or hypotheses associated with the targeted content, and devised ways to test each.

Students were explicitly led to state predictions, estimations, and/or hypotheses associated with the targeted content, and devised several ways to test each.

13a & 13b

Actively engaged in thought-

provoking activity and critical assessment

Students were never involved with the investigation, nor engaged in thought-provoking activity leading to critical assessment of procedures.

Students were involved with the investigation, but rarely engaged in thought-provoking activity leading to critical assessment of procedures.

Students were involved with the investigation, and sometimes engaged in thought-provoking activity leading to critical assessment of procedures.

Students were involved with the investigation, and often engaged in thought-provoking activity leading to critical assessment of procedures.

Students were involved with the investigation, and regularly engaged in thought-provoking activity leading to critical assessment of procedures.


-65-


14a & 14b

Reflective

Students were never reflective about their learning on the targeted content or concepts in tasks.

Students were vaguely reflective about their learning on the targeted content or concepts in tasks with a vague prompt and inappropriate time allowed.

Students were minimally reflective about their learning on the targeted content or concepts in tasks with a minimal prompt and minimal time allowed.

Students were clearly reflective about their learning on the targeted content or concepts in tasks with a clear prompt and adequate time allowed.

Students were reflective on multiple occasions about their learning on the targeted content or concepts in tasks with clear prompts and ample times.

15a & 15b

Intellectual rigor/criticism/cha

llenging

Teacher never allows ideas to be presented, challenged, or negotiated by students on the targeted content.

Teacher rarely allows ideas to be presented, challenged, or negotiated by the students on the targeted content, but without evidence.

Teacher sometimes allows some ideas to be presented, challenged, or negotiated by the students on the targeted content with very little accurate evidence.

Teacher often allows a variety of ideas to be presented, challenged, or negotiated by the students on the targeted content with some accurate evidence.

Teacher always allows a variety of ideas to be presented, challenged, or negotiated by the students on the targeted content with adequate and accurate evidence.

examining the teaching of science, and technology and ... · the rtop (piburn & sawada, 2000)...

Documents