Computational Thinking Skills and Its Impact on TIMSS Achievement: An Instructional Design Approach

Dalia Alyahya
King Saud University

Amal Alotaibi
King Saud University

The instructional design process has a significant impact on students' achievement in public schools in the United States. 'Instructional designers' use this process to improve students' achievement through the systematic design, development, and evaluation of instruction (Dick, 1986). Instructional designers start with the analysis of the learners, determine learning goals, arrange learning activities, and finally develop and implement assessment procedures to achieve this goal. All these activities are driven by learning theories, instructional methods, and strategies (Czerkawski, 2013). Morris, Uppal, and Wells (2017) stated that the important task for the designer is to identify the characteristics that are most critical for initiating a learner analysis. Computational thinking (CT) is a cross-curricular topic that can be infused throughout the curriculum. The main issue associated with CT is its way of conceptualization and implementation by the instructional designers (Czerkawski, 2013).

In Saudi Arabia, improving the achievement level of standardized global tests, such as TIMSS, is one of the strategic objectives of the Ministry of Education (Saudi Vision 2030, 2016). It is linked to the objectives of Vision 2030 in terms of establishing positive values, building an independent personality for the citizen, and providing citizens with the necessary knowledge and skills to fulfil the needs of the future labor market. This objective has three verification indicators that are centered on average student results in international tests. The regional and global standards have been developed for comparing performance in each test (Saudi Vision 2030, 2016).

The International Association for the Evaluation of Educational Achievement (IEA's) and Trends in International Mathematics and Science Study (TIMSS) is a series of international assessments of student knowledge in mathematics and science around the world. The participating students come from a diverse set of educational systems (countries or regional jurisdictions of countries). These students are different in terms of economic development, geographical location, and population size. A minimum of 4,500 to 5,000 students are evaluated in each of the participating educational systems. Contextual data about the conditions are collected from the students, their teachers, their principals, and their parents via questionnaires in which participating students learn mathematics and science (Martin, Mullis, & Foy, 2015).

Students' mathematics achievements are often associated with the future economic power and competitiveness of a country. Therefore, the desire to understand and identify factors with meaningful and consistent relationships with mathematics achievement has been shared among national policy makers and educators around the world. Today, the main purpose of educational systems to teach students how to apply mathematics in a range of contexts has been widely accepted. Higher Order Thinking Skills (HOTS), such as logical thinking, critical thinking, and reasoning are the basic skills for daily life, apart from the academic achievements in the schools (Marshall & Horton, 2011).

In this context, Wing (2006) described CT as "a fundamental skill for everyone, not just for computer scientists. To reading, writing, and arithmetic, an individual should add computational thinking to every child's analytical ability." According to Wing (2008), CT uses an approach similar to mathematical thinking for problem solving. Designing and assessing in engineering thinking, and understanding concepts such as calculability, mind, brain, and human behaviors. Students who learn CT across the curriculum can begin to see a relationship between academic subjects, as well as between life inside and outside of the classroom (Google, 2016). These learners are provided a more realistic view of fields like science, technology, engineering, and mathematics (STEM) by bringing computational tools and practices into mathematics and science classrooms. CT also prepares students for pursuing careers in these disciplines and helps students to be more adept STEM citizens in the future (Augustine, 2005; Gardner, 1983; Weintrop et al., 2016). The UK Department of Education has provided statutory guidance for CT in the national curriculum with the purpose of implementing high-quality computer education that equips pupils to use computational thinking understand and change the world (DFE, 2013).

Internationally, efforts to include CT in K-12 education are being made in Australia, China, Singapore, and South Korea (Wing, 2016). In TIMSS 2015, these three countries outperformed all other participating countries in mathematics for first, third, and fourth grades, respectively (Martin et al.,2015). In Saudi Arabia, the National Transformation Program 2020 started in the beginning of 2016 with participation of the Ministry of Education in the country's sectors. The challenges faced by the education sector were compiled, including the general objectives of education and performance measurement indicators. This resulted in educational reform and the development of new subjects and curricula which would be more relevant to the needs of the country (Ministry of Education, 2016).

Literature Review

CT competence has had a remarkable impact on performing daily activities more effectively (Lee, Martin, & Apone, 2014). Instructional design adds new insight to the learning process; it associates learning with continuous replacement and alterations. Advances in technology have made branched constructivist approaches possible for instructional design models. The instructional designer's toolbox must contain an ever changing and increasing number of theoretical applications and physical possibilities, whether they are designing for training or education. The modern designer will find solutions to the learning requirements of the 21st century with intelligent application of learning theory strategies and technology (McLeod, 2003).

Trends in International Mathematics and Science Study (TIMSS)

TIMSS is a project guided by the International Association for the Evaluation of Educational Achievement (IEA), and it is directed by the TIMSS & PIRLS International Study Center (Olson, Martin, & Mullis, 2008). TIMSS is conducted on a 4-year-cycle and involves collecting data from students, teachers, and school principals. Data collection includes educational contexts such as gender performance, students' backgrounds, home environments, students' attitudes toward learning, school facilities, educational support, resource availability, curriculum and instructional approaches, and teacher preparation (Mullis, Martin, Foy, & Arora, 2012).

In addition to student/teacher/school context, TIMSS 2011 provided four-point scale international benchmarks that can be used to interpret how students' mathematical competences vary with their scores. These benchmarks are: Advanced International Benchmark, High International Benchmark, Intermediate International Benchmark, and Low International Benchmark. TIMSS test items were designed to measure curriculum content in numbers, algebra, geometry, and cognitive processes for knowing, applying, and reasoning domains (Mullis et al., 2012).

The students need to draw on a range of cognitive skills to respond correctly to TIMSS test items. The first domain, knowing, covers the facts, concepts, and procedures that the students need to know. The second domain, applying, focuses on the ability of students to apply knowledge and conceptual understanding to solve problems or answer questions. The third domain, reasoning, goes beyond the solution of routine problems to encompass unfamiliar situations, complex contexts, and multi-step problems. These three cognitive domains are used for both grades, but there is difference in balance of testing time, reflecting the difference in age and experience of students in the two grades. For the fourth and eighth grades, each content domain include items that are developed to address each of the three cognitive domains. For example, the number domain will include knowing, applying, and reasoning items as well as the other content domains (Grønmo, Lindquist, Arora, & Mullis, 2015).

The presence of effective cognitive skills is associated with familiarity of mathematical concepts and fluency in mathematical skills. There is an increase in the potential for engaging in a wide range of problem-solving situations with more relevant knowledge recalled and a wider range of concepts understood by the student. The students find purposeful mathematical thinking impossible without access to a knowledge base that enables easy recall of the language, basic facts, and conventions of number, symbolic representation, and spatial relations. Facts encompass the knowledge that provides the basic language of mathematics, as well as the essential mathematical concepts and properties that form the foundation for mathematical thought. Procedures in each, such as numeric or algebraic expressions, functions, equations, geometric figures, or statistical data sets, form a bridge between basic knowledge and the use of mathematics for solving problems, especially those encountered by individuals in their daily lives (Grønmo et al., 2015).

The applying domain involves the application of mathematics in a range of contexts. In this domain, the facts, concepts, and procedures, as well as the problems, need to be familiar to the student. In some items aligned with this domain, students apply mathematical knowledge of facts, skills, and procedures, or they employ their understanding of mathematical concepts to create representations. Representation of ideas forms the core of mathematical thinking and communication. The ability to create equivalent representations is fundamental to success in the subject. Problem solving is central to the applying domain, with an emphasis on more familiar routine tasks. Problems may be set in real-life situations, or they may be concerned with purely mathematical questions, such as numeric or algebraic expressions, functions, equations, geometric figures, or statistical data sets (Grønmo et al., 2015).

Mathematical reasoning involves logical and systematic thinking. It includes intuitive and inductive reasoning based on patterns and regularities that can be used to arrive at solutions to problems set in novel or unfamiliar situations. Such problems may be purely mathematical or may have real-life settings. Both types of items involve transferring knowledge and skills to new situations and interactions among reasoning skills are usually a feature of such items. Each of the cognitive skills represents a valuable outcome of mathematics education; however, many of these skills may be applied while thinking about and solving a variety of novel or complex problems, with the potential to influence learners' thinking more generally. For example, reasoning involves the ability to observe and make conjectures. It also involves making logical deductions based on specific assumptions and rules and justifying results (Grønmo et al., 2015).

Computational Thinking

Enhancing students' thinking has been the goal of educational studies and programs for decades (Boddy, Watson, & Aubusson, 2003; de Bono, 1976; Ennis, 1989; Kuhn, 1999; Miri, David, & Uri, 2007; Watts, Jofili, & Bezerra, 1997). Each of these studies has its own definition of thinking; some use the phrase 'cognitive skills' (Leou, Abder, Riordan, & Zoller, 2006; Miri et al., 2007; Zoller, 2001), and others refer to 'thinking skills' (Miri et al., 2007; Resnick, 1987; Zohar & Dori, 2003). However, the studies clearly distinguish between higher- and lower-order skills (Miri et al., 2007). Higher Order Thinking Skills (HOTS), such as critical thinking, problem solving, and creativity, are the keys to 21st century lifelong learning (Trilling & Fadel, 2009). Hopson, Simms, and Knezek, (2001) identified higher-order thinking skills as one of the instructional areas that could be improved by using the computer.

Wing (2006) stated that CT represents a universally applicable attitude and skill set for everyone, not just for computer scientists. CSTA and ISTE (2011) define CT as a problem-solving process that includes formulating problems, logical organization, data analysis, representing data through abstractions, automating solutions through algorithmic thinking, identifying, analyzing, and implementing possible solutions, and transferring the problem-solving process to a wide variety of problems. Computational thinking has a long history within computer science. It reflects mental orientation for formulating problems as conversions of some input to an output and looking for algorithms to perform the conversions. Today, the term has been expanded to include thinking with many levels of abstractions, for example use of mathematics to develop algorithms, and examination of how well a solution scales across different sizes of problems (Denning, 2009).

ISTE (2015) defined computational thinking as the common reflection of creativity, algorithmic thinking, critical thinking, problem solving, and the cooperative thinking. These skills are discussed most in the literature. However, these skills explain a brand-new thinking skill that is called computational thinking when they are taken into consideration together. It shall be beneficial to explain other skills used to define this thinking skill to properly understand computational thinking.

Creative thinking is related to divergent and flexible forms of thinking. It increases the capacity of individuals to think 'out of the box' and to welcome ideas, which strongly differs from ordinary forms of thinking (Boden, 1990). Aksoy (2004) describes creativity as a concept that has always existed in the life of human beings and which covers different viewpoints. Creative thinking is a prominent concept that literally means to generate and form. It has found a place for itself in many areas, such as politics, economics, art, technology, and science (as cited in Korkmaz, Cakir & Ozden, 2017).

Algorithms play a major role in problem solving related to the field computer science, especially in repetitive problems. This aspect of computational thinking is perhaps the most closely aligned to computer science itself. Algorithmic thinking can also be viewed as strategic thinking, or a step-by-step processing (Morris et al., 2017). Algorithmic thinking, when applied to general problem-solving activities, can greatly improve efficiency, especially when dealing with multiple problems of a similar nature (Shute, Sun, & Asbell-Clarke, 2017). The expert consensus from the American Philosophical Association defined critical thinking as purposeful, self-regulatory judgment which results in interpretation, analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations on which that judgment is based (Stupple et al., 2017).

Critical thinking can also be defined as the ability to critique data, identify whether conclusions are supported by evidence, and distinguish a significant effect from random noise and variability (Holmes, Wieman, & Bonn, 2015). The term 'problem solving' is used in numerous disciplines, sometimes with different perspectives, visuals, and terminologies. For instance, it is considered a mental process in psychology and a computerized process in computer science (Schacter, Cillbert, & Wegner, 2009). Brandell (2010) defined problem solving as a cognitive–affective–behavioral process through which an individual (or group) attempts to identify, discover, or invent effective means of coping with problems encountered in everyday living.

Cooperative learning involves the instructional use of small groups so that students work together to maximize their own and each other's learning. It has been demonstrated that cooperative learning produces higher achievement, positive relationships among students, and healthier psychological adjustment than competitive or individualistic experiences. These effects, however, do not automatically appear when students are placed in groups (Johnson, Johnson, & Smith, 1991). In a similar context, the present study examines the relationship between computational thinking skills and mathematics achievements in TIMSS tests. The study also uncovers the relationships between each CT skill and mathematics achievements in TIMSS tests. This study is based on a set of assumptions:

H01: Computational thinking skill does not affect students' achievements in the TIMSS mathematics test.

H02: Creativity skill does not affect students' achievements in the TIMSS mathematics test.

H03: Algorithmic thinking skill does not affect students' achievements in the TIMSS mathematics test.

H04: Cooperative skill does not affect students' achievements in the TIMSS mathematics test.

H05: Critical Thinking skill does not affect students' achievements in the TIMSS mathematics test.

H06: Problem solving skill does not affect students' achievements in the TIMSS mathematics test.

Methodology

The present study employed quantitative research following correlational research design. It aims to explore the relationship between students' computational thinking skills and their results in the mathematics test in TIMSS. The study also tested the impact of each computational thinking skill in the mathematics test in TIMSS.

Study Sample

The participants for this study were recruited using a random sampling technique. The participants included 46 female students from 8th grade, studying in the 2017-2018 academic year, from six different schools of the Shaqra province in the central region of Saudi Arabia. On average, there were 16 students in each class, with an average age of 14 years old. The tests that were not filled out properly or with a large amount of missing information were not included in the data analysis. None of the schools in the current study had participated in a TIMSS assessment before. All participating schools were following the national mathematics curriculum and were using the same 8th grade mathematics textbook at the time of the study.

Study Instruments

Two instruments were used to achieve the study's objective. The first was the mathematics achievement test, which was compiled from released TIMSS items (2003-2007-2011). There were 20 questions in the test that included multiple choice questions (70%) and questions that required constructed response (30%). The highest possible score was 20. The test was composed of three domains: knowing, applying, and reasoning. Seven of these questions were from the knowing domain, 8 were from the application domain, and 5 were from the reasoning domain.

The second data collection tool used in the research was the Computational Thinking Scale. The original form of this scale was developed for university students. Its adaptation to secondary schools was conducted by Korkmaz, Çakır, and Özden (2016). There were 22 items included in this questionnaire. This scale consists of five factors;

• "Creativity" consists of 4 items.
• "Algorithmic Thinking" consists of 4 items.
• "Cooperation" consists of 4 items.
• "Critical Thinking" consists of 4 items.
• "Problem Solving" consists of 6 items.

Cronbach's alpha consistency coefficient was calculated for the adapted scale; this was 0.809.

Data Collection

The CT scale was performed first; it was applied to the study group by means of an online questionnaire. This was followed by the mathematics achievement test. Students were offered instruction on the objective of the research before application of the data collection tools. Participants in the study were not asked to provide any personal information such as name or student number so that they answered honestly. The students were allocated 15 minutes for the CT scale and 45 minutes for the mathematics achievement test.

Validity and Reliability

The questionnaire was applied after translation and the Pearson correlation coefficient was calculated to determine the internal truth of the questionnaire. The coefficient of correlation between the degrees of each of the axis expressions within the model was calculated by the total degree of the axis to which the term belongs. The results of the Pearson correlation coefficient of the expressions showed the internal consistency of the model, indicating its internal honesty. The Cronbach's Alpha coefficient, which was α= .653, was used to confirm its stability and measure the stability of the questionnaire. This indicated high degree of stability. The test was arbitrated by four arbitrators (two of them were majoring in mathematics), and all the provided notes were analyzed.

Data Analysis

The data gathered through the questionnaire was entered into the Statistical Package of Social Sciences (SPSS) version 20.0. The frequencies and percentages were calculated to identify the primary data of the study members and determine their responses included in the study tool. Mean values were calculated to observe the high or low responses of the study participants. Standard Deviation was calculated to identify the extent of the responses of the study participants concerning the study variables and each of the main axes. The relationship between test and questionnaire elements was tested using the Pearson correlation coefficient.

Results

The level of significance between the test and CT skills was calculated to be less than 0.05, except for the critical thinking skill, which was 0.085 (see Table 1). This indicated that there was no statistically significant relationship between critical thinking skills and mathematics achievement in TIMSS. However, there was a statistically significant relationship between other skills and mathematics achievement in TIMSS.

A bivariate correlation analysis (Spearman) was conducted to examine the relationship between students' achievement in the TIMSS mathematics test and CT skills scale (see Table 2). The results depicted a positive correlation between students' CT skills and their total mathematics achievement (r =.669, p >.01).

* Correlation is significant at the p<0.05 level, ** Correlation is significant at the p<0.01 level.

There was a positive correlation between students' creativity skills (r =.337, p > .05) and their total mathematics achievement, and this was the lowest correlation between CT skills. Moreover, there was a positive correlation between students' algorithmic thinking skills (r =.408, p>.01) and their total mathematics achievement. The results depicted positive correlations between students' cooperative skills (r =.519, p >.01) and their total mathematics achievement. However, there was no correlation between critical thinking skills (r =.254) and students' total mathematics achievement. The results have depicted a positive correlation between students' problem-solving skills (r=.696, p>.01) and their total mathematics achievement with highest correlation between CT skills.

Discussion

The findings of the present study have shown that high computational thinking levels depict higher performance on mathematics achievement in the TIMSS. The problem solving skill has the highest impact on achievement in the TIMSS; in contrast, creativity has the lowest impact and critical thinking has no impact on mathematics achievement in TIMSS.

TIMSS's items assess students' abilities to demonstrate their knowledge, apply what they have learned, and solve problems through analysis and logical thinking (Martin et al., 2015). Computational thinking enables those who implement it to model problems and situations (69% impact in TIMSS) that may yield a computational solution. Computational thinking promotes problem decomposition and use of logical algorithms instead of separating problems and their solutions (40% impact in TIMSS). It is a combination of logical, arithmetic, efficient, scientific, and innovative thinking, along with qualities such as creativity (37% impact in TIMSS) (Curzon, Black, Meagher, & McOwan, 2009).

Considering TIMSS for applying and reasoning items (65% from TIMSS tests), the students are expected to use different problem-solving strategies and abstract skills such as hypothesizing, analyzing, synthesizing, drawing conclusions, generalizing, and evaluating (Martin et al., 2015). It is reasonable that problem solving skills have the highest impact on achievement on the TIMSS (69% impact). According to Wing (2006), computational thinking can be defined as a method or approach of solving problems, designing systems, and understanding human behavior by drawing on the concepts fundamental to computer science.

Cooperative learning has a preferable place among the learning methods because of its contributions such as contribution to academic success, sharing information, and establishing social relations (Korkmaz et al., 2017; Nam, 2014). Cooperative learning has a significant effect on students' performance in mathematics and their attitudes toward mathematics (Hossain & Tarmizi, 2013). This is in line with the results of the present study.

Algorithmic thinking is the skill of understanding, applying, assessing, and producing the algorithms (Brown, 2015). In TIMSS, the second domain, applying (40%), focuses on using this knowledge to model and implement strategies to solve problems. The third domain, reasoning (25%), includes analyzing, synthesizing, generalizing, and justifying through mathematical arguments or proofs (Martin et al., 2015). The key skill in CT is the ability to logically dissect a problem and systematically devise an algorithm suitable for solving it (Grover & Pea, 2013). Students who have mastered this practice will be able to analyze a given set of data, make claims, and draw conclusions based on the findings from their analysis (Weintrop et al., 2016).

Definitions of critical thinking range from the well-reasoned evaluative judgment to the thinking that involves more than the mere acquisition and recall of factual information. A more inclusive definition of critical thinking embraces all thought processes that are "deeper" than memorization and recall of factual information. When students think critically, they think deeply; they do not only know the facts, but they take the additional step of going beyond the facts to do something with them. Since the TIMSS assessments primarily use two item formats: multiple-choice (70% - 15 items) and constructed-response (30% - 5 items), this skill could not be measured with this number and type of questions (Martin et al., 2015).

The evolution of resources in CT depicted the changing roles of computer science teachers within the classrooms. CT not only assists teachers from the field of computer science; rather, it provides various strategies for undertaking issues and problems of the digital age (Czerkawski & Xu, 2012). In CT, the association between computers and human brain is important as it has evolved as a multidisciplinary thinking skill. This is the reason CT is basically a thinking that is cultivated in all subject areas and fields, rather than exclusively for a computer science or technology techniques. It is important to understand the utilization of pattern identification, abstraction of concepts and algorithmic instructions and translation by the computing professionals in today's complex world (Czerkawski & Xu, 2012).

A similar study conducted by Quinn (2016) provided a framework that aimed at supporting the faculty members working with the challenges of transitioning into a higher education level. The results depicted that challenges related to addition of large number of online learning opportunities was associated with shift of the traditional classroom to online pedagogy daunting. CT offers a framework that supports complex social systems that are likely to undergo change, and which would benefit from a clear change implementation strategy. Also, García-Peñalvo (2018) reviewed CT across a range of academic fields. He concluded that "teachers have a rich range of possibilities with which to create scenarios and learning activities that are effective by combining the tools and methodologies available to them: computational thinking, programming, robotics, teamwork, critical thinking, etc." (p. 18).

CT is likely to provide data driven support for designing the curriculum. This is the reason scholars are approaching this field enthusiastically and understanding its potential to succeed in their fields. Computational thinking skills can be classified into nine different levels, which include parallelization, data representation, data collection, data analysis, abstraction, algorithms, simulation, automation, and problem decomposition. A step by step approach is required to address these nine levels (Czerkawski and Xu, 2012). On the contrary, educational technology is described as the ethical practice and study to facilitate learning and to improve performance by using, managing, and creating adequate technological procedures and resources (AECT, 2008). In addition, educational technology is considered as a cross-curricular field, as its main features facilitate the learning and teaching of any skill at any level (Czerkawski and Xu, 2012).

Conclusions and Recommendations

The results of the present study have indicated a relationship between CT skills and mathematics achievement in TIMSS in eighth grade. Problem solving skills had the highest impact on the test results while the creativity skill was the least influential. Critical thinking depicted no impact on TIMSS achievement. Future studies could determine the relationship between TIMSSs' cognitive domains and CT skills and, if this exists, the direction of the relationship. Discovering which learning skills contribute to high mathematics achievement should motivate instructional designers to identify the educational environments that improve students' learning by integrating CT skills across the instructional materials. The instructional models, after identification, can be designed to optimize the development of these skills and help student enhance their scores.

Acknowledgement

The author is very thankful to all the associated personnel in any reference that contributed in/for the purpose of this research. Further, this research holds no conflict of interest and is not funded through any source.