Untitled Document

Synopsis of May 2009 Pilot & Subsequent Work

In order to ascertain, and subsequently refine IAB framework, ERC ran the first IAB pilot in May 2009. The pilot consisted of a comprehensive exam that covered materials taken in grades 10, 11 and 12 in physics, mathematics and Arabic language.

Method

Themes covered in each field are given in Figure 1. These themes provided the context for two common benchmarks, systems, and state and change of state (or constancy and change), targeted in the pilot. Corresponding outcomes measured in the pilot were set according to a novel profiling schema. The schema consists of a 3x5 matrix for laying out conceptions, processes and dispositions pertaining to the scope, composition, structure, quantification and expression of each benchmark (Figure 2).
The three tests (physics, mathematics and Arabic) were given during the month of May at participating schools. Physics and mathematics were offered in English, French and Arabic. All tests were provided in paper booklets. Students were asked to write their answers in the same booklet, and finish their work within 100 minutes, on each test.
A total of 490 grade 12 students, 35% of whom were females, participated in the pilot. These students were enrolled in 16 private schools in KSA, Egypt, Jordan, and Lebanon. In countries where different streams are available in grade 12, only students from scientific streams participated in the pilot.

Scope: What does the benchmark represent, or correspond to, and under what conditions?
Composition: What are the primary objects associated with the benchmark, and what primary properties of each object should be accounted for?
Structure: How are the different objects and properties related to each other (internal structure), and to other entities within and outside the defined scope (external structure)?
Quantification: How to measure various aspects of the benchmark, or ascertain linguistic / social quantifiers?
Expression: How to express all the above in meaningful ways, with various kinds of representation?

Physics
1. Newtonian Mechanics
1.1    Reference Frames
1.2    Kinematics of linear motion
1.3    Newton’s laws of dynamics: 1st, 2nd, and 3rd laws
1.4    Work-energy and conservation principles
2. Electricity & Magnetism (E&M)
2.1    Motion of a charged particle in an electric or magnetic field
2.2    E&M dynamics: Laws of Laplace, Lorentz, Faraday and Lenz
2.3    DC resistive circuits & daily life applications

Mathematics
1. Complex numbers
1.1 Modulus and argument of a complex number
1.2 Algebraic, geometric and exponential forms of a complex number
1.3 Solution of a quadratic equation in the set of complex numbers
2. Functions
2.1 Function properties
2.2 Derivative of a function
2.3 Interpretation of functions and their graphical representations
2.4 Application of functions in science

اللغة العربيّة
التواصل اللغوي في تناوله موضوع الاخلال بالتوازن البيئي
- حواشي النص
- حقول معجميّة
- تلخيص نص
- وظائف الكلام
- أنواع النصوص
- أنماط النصوص
- أساليب الكلام (التعيين والتضمين، الصور البيانيّة والمحسّنات البديعيّة، أنواع الجمل...)

المعجم البيئي المتخصّص
التنوّع البيولوجي، التوازن البيئي، المنظومة البيئيّة، التصحّر، الموارد الطبيعيّة، الأنواع النباتيّة، الغلاف الحيوي، النفايات الإلكترونيّة، الغطاء الجليدي، التلوّث، الثورة الصناعيّة

Figure 1: Themes covered in the pilot tests

Conceptions: What concepts, laws and/or other conceptual entities and theoretical statements are needed to define each of the above dimensions?
Processes: What mental operations (eventual skills) need to be carried out in order to construct and deploy various dimensions?
Dispositions: What meta-cognitive factors (values, attitudes, beliefs, etc.) are needed to control all the above in meaningful and productive ways?

		Conceptions	Processes	Dispositions
Scope
Composition
Structure	Int
	Ext
Quantification
Expression
Figure 2: Dimensions of the profiling schema used to set the taxonomy of the two benchmarks, system and state & change of state, in the context of the themes of Figure 1.

Results

Overall performance of all participating students was below average on all three tests, as measured by raw scores, and the profile of all students was, in general, not satisfactory on any of the dimensions ascertained by the profiling schema.
Such performance was somewhat expected because of at least three reasons. First, the IAB framework and, especially profile, are novel and unique in many ways which most secondary schools around the world may not be used to. IAB drives for experiential development of a profile consisting of a balanced mix of conceptions, processes and dispositions that are critical for the development of the big picture within and across disciplines. Contrary to what educational reform has been calling for in the last two decades, including in the Arab World, instruction and assessment around the world continue to focus more on conceptual than any other aspect of covered materials, and to pay little attention, if any, to the big, especially cross-disciplinary picture. Second, the three tests are comprehensive (they cover grades 10, 11 and 12), and focus more on meaningful learning rather than rote learning of course materials. Secondary school students are not used to being tested on materials covered in previous grades. Third, the pilot had a dual objective: to get necessary baseline data and to ascertain IAB framework and profile. In order to get a comprehensive picture of participating students’ cognitive state, the level of complexity and difficulty was somewhat stretched on the three tests towards the higher end of the cognitive spectrum. Items of various degrees of complexity and difficulty were included, with a focus more on higher-order thinking that is characteristic of meaningful, experiential learning than on routine operations and recall of materials that could be learned by rote.
Data analysis revealed the following:

Most participating students do not retain in their memory what they have learned in previous grades. Except in rare instances, and especially in physics and mathematics, participating students could not answer items pertaining to topics covered in grades 10 and 11. This is typical of students who learn things by rote under conventional instruction of lecture and demonstration rather than in meaningful ways under an experiential approach similar to the one promoted in IAB.
Predominance of rote learning was also reflected in items ascertaining specific conceptions. For instance, the physics test included items that ascertain student conceptual understanding of basic concepts and laws in mechanics and electricity. The overall mean of participating students on those items was 36%, which is at about the same international level for students who learn physics by rote under conventional instruction. Students who learn physics meaningfully following an experiential approach reach, in many places around the world, an average above 80% on such items.
Most participating students are unable to carry out complex tasks, and especially decide on their own what strategies are appropriate for such tasks. Items with problematic situations were given in all three tests where students were supposed to analyze the situations at hand, and decide on their own what conceptions and/or what operations are most appropriate to deal with those items. The overwhelming majority of students were unable to handle the situations in question.
Most students showed lack of critical dispositions associated with various disciplines, especially consistency and coherence, objectivity and precision, self-evaluation and regulation.
Overall student performance was best on conceptions items (though in reproductive rather than productive ways), followed by processes and dispositions items, on the one hand. On the other hand, performance was best on expression aspects, followed, and in this order by structure, quantification, composition and scope.
Student overall performance was best in Arabic, followed by mathematics and then physics.
ANOVA revealed significant differences among participating countries, and among schools within the same country, on all three tests.
ANOVA also revealed that, overall, participating female students performed significantly better than their male peers on all three tests. However, results were mixed in some schools with mixed gender participants.
Pearson correlation coefficient among the three pilot tests ranged from .51 (p=10-20), between Arabic and mathematics, to .59 (p=10-31), between mathematics and physics. This provides significant support to the cross-disciplinary framework underlying the three tests, and subsequently the overall target profile. Common benchmarks and the profiling schema are responsible to a large extent for such high correlation.

It is though worth noting here that items were given in each of the three tests that relate to fields that are the object of the other two tests (e.g., physics-related items in the mathematics test or mathematics-related items in the Arabic test). Most students had serious trouble with such items.