One of the aimed for outcomes of the project was the evaluation of the INSET strategy through an identification of the changes, if any, in teachers' perceived and actual teaching approaches which might have arisen as a result of the INSET and their use of the new curriculum materials. A second was an analysis of the utility of qualitative long-term evaluation methods for monitoring the effectiveness of INSET activities in a developing country context. This section of the report relates to these aspects of the project.
The 17 teachers that participated in the INSET were monitored closely with a view to determining firstly, if they understood and accepted the purposes and approaches of the new curriculum and secondly, the extend to which the desired outcomes of the INSET were subsequently reflected in their teaching and professional behaviour. In approaching these aspects attempts have been made to fit the field data to two theoretical models. Firstly, a model of professional development related to stages of concern as discussed by Constable and Long (1989) and secondly a practical typology of INSET outcomes model developed by Kinder et al. (1991). The approach taken used data collected at various stages in the processes of introducing and implementing the innovation.
Data was collected both formally and informally, from the group as a whole and from individual teachers. Formally, data was collected by individually completed, end-of-workshop questionnaires and by three rounds of semi-structured personal interviews in school, the first carried out before the INSET, the second after the INSET event but before any teaching with the new materials had been undertaken, and the third carried out in schools after teachers had taught with the new materials. Most interviews were taped and transcribed others were recorded in note form. During the final session of the INSET workshop, group data was gained through a formal evaluative discussion. Informal data collection was through casual conversations and discussion with teachers before, during and after the INSET event and via observations of workshop activities and classroom teaching. Due to limits on time and resources and various practical considerations not all teachers were interviewed in each round of data collection nor were all those interviewed asked the full set of interview questions. However, although the data set may be seen as incomplete it is still substantial. In addition to observation records, field notes and pre-INSET task records, it consists of 17 teacher questionnaires and 37 teacher interviews, 30 of which were conducted after the INSET. Here it needs to be noted that some teachers were difficult to contact before the INSET and so this data set is the most incomplete (it is also seen as of less importance than the post-INSET data). It also needs to be noted that once it was judged that a teacher had established teaching inconsistent with the intent of the innovation they were not interviewed again. Incomplete interviews arose when it became clear to the interviewer that the teacher had little more to contribute or when the teacher had no more time to spend with the interviewer, often as a result of giving extensive answers to early questions. The maintenance of goodwill and cultural sensitivity were important elements in judging when to terminate an interview.
The report here deals with the following aspects:
(a) teacher expectations of INSET
(b) teacher opinion of the three characteristics of the new curriculum
(c) teachers' evaluation of the INSET workshop
(d) teachers' understanding and implementation of the new teaching approach
(e) teachers' advocacy of the new curriculum
(f) how the data fits the theoretical models
(g) the utility of qualitative long-term evaluation methods
Before turning to the data it is of value to note the backgrounds of the 17 teachers who attended the INSET and who then went on to work with the new materials in school. Four categories of teacher were recognised as being represented at the INSET. These are seen as beginning teachers (4), novice teachers (4), experienced diploma teachers (5) and experienced degree teachers (4). Table 1 describes their characteristics and lists their reference code numbers which were used throughout the study. This characterisation and the teachers reference codes are used at several stages in the presentation of the analysis which follows.
Table 1 Categories of teachers who participated in the INSET.
|
Beginning Teachers |
Novice Teachers |
Experienced Diploma Teachers |
Experienced Degree Teachers |
|
untrained, inexperienced degree holders |
trained but inexperienced degree or diploma holders |
trained diploma holders with long experience |
trained degree holders with long experience |
|
4 |
4 |
5 |
4 |
|
01 10 12 17 |
03 05 09 14 |
02 04 11 13 16 |
06 07 08 15 |
figures in bold are the numbers of teachers in each group other figures are the reference code numbers of the teachers
(a) teacher expectations of INSET
Teachers were informed about the INSET workshop by word of mouth from their head teacher or by one of the project team who visited the school. They were informed that the workshop would introduce them to new curriculum materials and prepare them to use these in school. They were asked to carry out a pre-workshop exercise to compare a SWISP lesson with a Matsapha lesson. In fact, only half the participants completed the pre-workshop task. This and the feedback collected at the end of the workshop suggested that most teachers had a clear expectation that they would be passive participants in the INSET and expected just to be told about the curriculum rather than have a more active role in the INSET. Teacher 02 was typical in expecting to "listen to lectures". Teacher 09 represented a minority who expected to "do a few practicals on Electricity and Air and Life". Most teachers thus described the workshop as "not exactly what I expected, especially the peer teaching" (teacher 15). While the more active role was not expected, it was appreciated by many to be "very worthwhile" (teacher 13).
|
The induction strategy included pre-workshop tasks comparing existing practice with the new approach. These tasks were executed by only few teachers, not because of the difficulty of such tasks, but because of the majority's expectation of INSET as 'being told what to do'. |
Noting that most teachers expected a passive INSET experience it is of interest that only 3 teachers suggested that notification of the expectations of the workshop might have been clearer. A further 3 wished for the materials to be issued prior to the workshop rather than in the course of the workshop. The main demand in terms of how to improve the INSET was for more time for peer teaching: a novel INSET experience for all the teachers and which, as noted later, was seen as the most helpful workshop activity.
(b) teacher opinion of the three characteristics of the new curriculum
Table 2 summarises the opinions of the teachers at the end of the workshop on the three characteristics adopted as the technological approach of the Matsapha materials. It is clear that all the features are welcomed by the majority of members of all the groups of teachers and that while a few teachers see difficulties and challenges, mainly with investigations, most see opportunities to improve motivation and bring relevance to learning.
Table 2 Teachers' opinions of the characteristics on the Matsapha materials
|
Group |
Contextualisation |
Application |
Investigation |
|
Beginning Teachers (4) |
2 good |
1 good |
1 good |
|
Novice Teachers (4) |
1 good |
1 good |
2 good |
|
Experienced Diploma Teachers (5) |
2 good |
3 good |
1 good |
|
Experienced Degree Teachers (4) |
1 very good |
2 good |
2 good |
Immediately after the INSET the views of the four groups of teachers were collected on their confidence to teach the two Matsapha units to their Form 2 classes. The evidence indicates that all but one of the teachers felt confident to teach both units and that some expressed the view that they were very confident or even highly confident. The data suggests that, perhaps unsurprisingly, the experienced degree teachers may have a higher level of self confidence in their teaching ability than the other groups.
(c) teachers' evaluation of the INSET workshop
The INSET activity incorporated a number of different professional development strategies. Of particular interest was the perceived value and effectiveness of these, and in particularly simulated teaching (=peer teaching), teacher-led sessions and in-school support.
From the post workshop evaluation questionnaires more than half the group spontaneously suggested that peer teaching was the most helpful activity. When selecting from amongst eight different workshop activities, all but one teacher chose peer-teaching amongst their two most helpful activities. Reasons why peer teaching was seen as helpful were because it:
(1) provided experience of teaching;
(2) enabled learning from others;
(3) covered the curriculum;
(4) provided time to consolidate learning;
(5) built confidence.
Although peer teaching was welcomed, over 50% also saw this as the most difficult activity, because it:
(1) was unfamiliar;
(2) required a change from a traditional teaching approach;
(3) opened up discussion on teaching performance;
(4) was done with little preparation time.
The main critique of the peer teaching was that it required more time to be devoted to it.
|
Peer teaching is considered to be a very worthwhile INSET activity to provide an overview of teaching units and of learning activities. Appropriate time needs to be given to it. |
Some of the INSET workshop sessions were led by a teacher who had been involved in the development of the curriculum materials and had taught the trial units. There were many positive comments made about his contribution during interviews with teachers after the workshop and after they had taught the new units. His contribution was seen by the majority of teachers as uniquely helpful as he:
· knew the school situation and how pupils actually responded;· gave the feeling that if it was possible for him then it was also possible for other teachers to teach with the new materials.
However, not all teachers felt the need for his sessions. Most considered that the INSET leader (an experienced and respected teacher educator) could have led his sessions equally well. Others were confused by the input as they had not yet come to terms with the curriculum (a timing issue and not a staffing issue).
|
The main contribution to the INSET of the practising teacher leading INSET sessions was one of building the professional confidence of the participants. However, most participants suggested that any INSET provider with general credibility as a classroom practitioner could have offered the demonstration teaching sessions equally well. |
In-school support was provided to teachers once they started to use the new materials in their schools. This was provided by the INSET workshop leader. There was some apprehension among some teachers about such a visit. The concern was that the visitor might evaluate or even take over the teaching. These concerns of a small minority were more than balanced by those who welcomed the opportunity to get feedback, to have another adult in the class and to share teaching. As it turned out, visits were difficult to arrange. Several wasted journeys were made to distant schools only to find teachers absent or sudden timetable changes resulting in no opportunities to work with teachers as planned. Consequently, not all teachers were visited and not all those that were visited were teaching with the Matsapha materials on the arranged visiting day. All but one of those supported in class (who did not understand the purpose of the visit and thought she was being checked up on) spoke highly of the class visit. Some reported the benefits of help in class and the positive influence on pupils while others spoke of the opportunity to gain skills and to learn from demonstration teaching. Clearly, some teachers saw the purpose of the visit to be for the immediate benefit of the pupils while others saw it as an aid to their own professional development and of longer term benefit to pupils.
|
The evidence is that in-school support is valued and important. It is likely to be of greatest benefit when its purpose is clearly understood to be different from a 'teaching-practice situation'. Great difficulties must be overcome to make the best use of the time of INSET staff. |
As discussed below (section d) a key factor to the success of the curriculum was the extent to which teachers understood the features of the materials. Data collected after the workshop and after teachers had taught the units throws light on those INSET workshop activities which teachers considered important to improve their understanding of contextualisation, application and investigation.
With regard to contextualisation, peer teaching and INSET leader presentation are perceived as the only contributing activities, with teachers' opinions about equally divided between the merits of these two. Opinions about the development of understanding of the applications approach were more varied. One or more teachers perceived teacher-led presentations, peer-teaching and laboratory practical work as the most useful aid to understanding. The bulk of opinion, however, supported INSET leader presentations. Here it needs to be noted that several teachers considered that they had a full understanding of teaching about applications before attending the workshop (see below). All the responses about investigations record that laboratory practical work is the most useful activity aiding understanding.
|
The above underlines the importance of peer teaching activities in an INSET strategy to familiarise participants with contextualisation. However the value of expert input and practitioner-led sessions are most effective ways of conveying teaching approaches to applying science to everyday problems. The opportunity for practical experimentation aids the acceptance of the investigative aspects. We argue that it is the blend of activities that can give strength to INSET. The right blend may have a greater effect than the sum of the parts. |
(d) teachers' understanding and implementation of the new teaching approach
A key element to the success of the curriculum was seen to be the extent to which the teachers understood the features of the Matsapha materials and interpreted the learning approach in the way intended by the development team and promoted in the INSET workshop. To gain insights into this, an analysis of the INSET feedback questionnaires, post workshop interview transcripts and post teaching interview transcripts was carried out. In analysing the pre-teaching documents what was being looked for was evidence of an understanding of the concepts on contextualisation, application and investigation and how teaching might be transacted in the classroom. In analysing the post-teaching data what was being looked for was evidence that what had gone on in the classroom could be identified as contextualisation, application or investigation.
While there is evidence that four out of five teachers understood contextualisation after the workshop only two out of three teachers understood application and investigation. These proportions for contextualisation and application did not change after teaching but those for investigations did. Here less than half the teachers showed an understanding.
|
High levels of understanding of the Matsapha approach after the INSET workshop translated into high levels of implementation except for the investigative work. Here most teachers were unable to give relevant examples from their teaching and frequently equated investigations with any type of practical work. Those unable to convey a practice or understanding of application saw this as merely an illustration of a concept unrelated to any problem solving situation. Misunderstandings of contextualisation tended to result from confusion with application. |
Here it might also be worth noting that no teacher claimed to have come to the INSET event with an understanding of contextualisation but 6 claimed an understanding of application and 2 an understanding of investigation.
(e) teachers' advocacy of the new curriculum
In relation to the wider adoption of the new approach, we were interested in the extent to which teachers tried to alter the practice of their teaching of other topics to reflect the technological aspects of the Matsapha materials. In order to promote wider practice, some time during the INSET workshop was devoted to brainstorming appropriate contexts for lessons on other topics.
More than a third of the teachers were able to cite examples of contextualised lessons in other science topics they taught. A quarter stated that they had introduced applications to other lessons and a third claimed to have carried out investigations with pupils not following the Matsapha curriculum. What needs to be kept in mind here is that, in the light of data presented above, their notions of contextualisation, application and investigation may not be uniform and at one with those of the curriculum developers.
The teachers were also asked about the level of interest shown by their colleagues in the Matsapha materials and the extent to which they had tried to interest them. While few teachers reported self-initiated interest by their colleagues more than half reported that they had tried to interest colleagues.
There was no expectation that teachers who had not attended the workshop would use the new materials. One teacher commented that "Other teachers did read but since they were not at the workshop they did not use this method" and indeed the need for workshop support was emphasised by another teacher who commented "I think that there will be some problems using it without a workshop". Less encouraging comments reported no interest or feedback from colleagues when shown the new materials, one teacher explaining this because "Teachers are lazy. It is not easy to teach a dog new tricks". More encouraging were comments such as "The HOD has seen them. He liked the approach" and "Yes, they are asking me why only these topics".
|
All this points to a low level of curriculum diffusion to other than pupils taught by teachers working with the new materials. Clearly, a brainstorming session as a practice of creative thinking and resulting in a list of ideas as a teaching resource has a limited effect on the transfer of a new teaching approach to other topics. Also, it seems accepted by participating teachers and their colleagues that specific induction is needed before adoption of a new teaching strategy may be expected. |
(f) how the data fits the theoretical models
The actions and statements of teachers recorded in interviews undertaken after they had the opportunity to teach the new curriculum have been mapped against a range of stages of concern as recognised by Constable and Long (1989). These stages can be described as follows:
0 awareness: shows little interest in the innovation and is not concerned with its implications1 informational: shows a general awareness and interest in the innovation but only in the gross features.
2 personal: is concerned with personal adequacy to cope with the innovation and the extent to which any required change will conflict with present practice
3 management: is concerned with implementation of the innovation and managing the tasks required
4 consequence: concern is for the students in the teacher' classes and the impact of the innovation on their learning
5 collaboration: concern is with co-ordination and co-operation with others regarding the innovation
6 refocusing: concern is with the widespread benefits of the innovation and its potential to replace current practice
Table 3 is arranged so that the teacher codes appear in category order across the top. The 3 marks in the table indicate where there is evidence to indicate that the teacher exhibits this level of concern. The · indicates that the evidence is such that this is a major area of concern.
|
Teacher |
01 |
10 |
12 |
17 |
03 |
05 |
09 |
14 |
02 |
04 |
11 |
13 |
16 |
06 |
07 |
08 |
15 |
|
Stages of Concern |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
0. awareness |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. informational |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2. personal |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3. management |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
4. consequence |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
5. collaboration |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
6. refocusing |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Understanding of Features |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
contextualisation |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
application |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
investigation |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
01 |
10 |
12 |
17 |
03 |
05 |
09 |
14 |
02 |
04 |
11 |
13 |
16 |
06 |
07 |
08 |
15 |
= evidence for stage of concern/feature of curriculum understood
= main stage of concern
= feature of curriculum not understood
For example teacher 10 (a beginning teacher) never progressed beyond an awareness of the innovation as evidenced in his post-teaching interview response of:
It is almost the same as SWISP, isn't it? I only think that the Matsapha lessons are simpler. But I failed to separate the SWISP
Although he received the materials to use in class he did not use them as intended and effectively dropped out of the project. On the other hand teacher 06 (an experienced degree teacher) presented evidence for him to be seen as one who saw the impact of the innovation on his students and the universal benefit of the new curriculum as indicated by the following interview comment:
One thing I like about these new materials is that it increases motivation for the students. If it could only be broadened to other topics. To take over from SWISP.
The table also includes an indication of whether there is evidence to indicate if a teacher understood or misunderstood any of the three characteristics features of the new curriculum. Thus while teacher 14 (a novice teacher) welcomed the potential positive impact of the innovation on his pupils the evidence from interview indicates that he did not internalise the ideas of contextualisation, application and investigation. Teacher 09 in the same category had the same concern but she indicated an understanding and adoption of the approach being advocated.
From this mapping of concerns, stances to the innovation of the teachers in the four categories (beginner, novice, experienced diploma holder and experienced degree holder) can be recognised. These are illustrated in Table 4.
Table 4 Stance of different teacher categories towards the curriculum innovation
|
|
Beginning teachers |
Novice Teachers |
Experienced Diploma Teachers |
Experienced Degree Teachers |
|
Rejecters |
01 10 17 |
|
11 13 |
|
|
Personal Users |
|
03 05 |
|
|
|
Manager Adopters |
|
|
16 |
|
|
Innovators |
12 |
09 14 |
02 04 |
06 07 08 15 |
· the numbers represent teacher codes· bold code numbers are those teachers who understood at least two of the three of features of the curriculum innovation
· underlined codes are teachers who saw the curriculum innovation being appropriate and necessary for the system as a whole
All but one of the Beginning Teacher group (untrained inexperienced degree holders) rejected the innovation. For most this can be explained by a lack of insight into the advantages and disadvantages of one teaching approach versus another. It is argued that these teachers do not adopt a teaching approach for any special educational reasons. The two experienced diploma holders who also rejected the innovation, one of whom provided evidence that she understood its features, did not have secure backgrounds in science or mathematics and had stronger personal and professional commitments to teaching arts subjects. Both underprepared groups have survival skills which they are reluctant to change regardless of the quality of education they are providing. This is not yet a concern for them. The one exception in the group of Beginning Teachers who has been classified as an innovator was characterised by her consistently expressed concern to provide the best possible science education for her pupils, rather than focusing on the innovations' requirements for the teacher practitioner.
Two Novice Teachers understood the purpose of the curriculum but had concerns dominated by their personal adequacy to cope with its demands. Two others were seen as innovators. One of the five experienced diploma teachers is seen as being concerned with managing the innovation whereas another two exhibit not only an understanding of the characteristics of the innovation but look to its more widespread adoption and influence on the science curriculum as a whole. These two are part of a group of 9 (more than half the original INSET group and nearly three quarters of those that used the materials in school) whom we characterise as innovators in that they all have a concern to improve the learning of students. This is well illustrated by teacher 2 who stated:
The main thing is the pupils. Do they understand? That is what I want.
It is of note here that all the Experienced Degree Teachers fall into this group and that all the group except one of the Novice Teachers interprets the characteristics of the curriculum as intended. Three of the 9 have a general concern for the quality of science education and see the type of curriculum which the project has introduced as reflecting the needs of system as a whole.
|
The mapping of the data onto a professional concerns grid has allowed the identification of various stances to the innovation. An implication of this is that classroom innovation is most likely to arise from this type of INSET provision through support for qualified teachers, experienced or inexperienced, who, it is argued, because they are secure in their school science knowledge and have had the opportunity to reflect on practice are more likely to see the benefits of new approaches to teaching and learning and have the confidence to try these with pupils. Furthermore, it is emphasised that introducing new teaching methods needs to be preceded by content confidence building and a basic awareness of teaching methodology. More effective implementation may be achieved if the INSET activities consistently focus on the improvement of learning rather than teaching. |
(g) the utility of qualitative long-term evaluation methods
This section looks back at the methods used to monitor and evaluate the INSET and considers the possibilities for the more widespread adoption of long term qualitative evaluation methods in developing country contexts. There is no attempt made here to compare the merits of qualitative and qualitative evaluation methods. The first point to note here is that our data collection has been collected from teachers and not pupils or classrooms and relies on what teachers say and claim to do. It is thus an indirect reflection on practice. As such it has weaknesses but these are shared with other evaluation methods which centre on teacher feedback. A strength is that data is collected over time and so there is a greater chance that an evaluator can gain a sense of the reality of understanding and implementation of a curriculum innovation in a natural teaching context rather than capture the after-glow of a positive INSET experience orchestrated by enthusiasts and usually (and in our case) away from the reality of schools and pupils. Also, opinions and judgements are likely to have matured and understandings consolidated with time. Teachers can be given the opportunity to articulate their understandings and to express their opinions in a variety of ways which need not be restricted by closed questionnaires, time or ability of expression in writing. The task of the evaluator is to formulate questions, probe for answers through a variety of means, encourage openness and interpret responses. Having said that, and reflecting on the positive outcomes of the adoption of the long term qualitative evaluation in this project, a number of points must be made
· questionnaire, interview and observation methods were used· considerable time was spent by experienced staff on the design, construction and piloting of the questionnaire and interview schedules
· experienced research staff conducted and transcribed the interviews
· not all teachers who participated in the INSET contributed a full set of data to the evaluation
· data analysis though done by experienced staff was time consuming
· the long term nature of the evaluation revealed aspects not evident in the data collected during or immediately after the INSET and between the INSET and when teachers started teaching with the materials the INSET introduced
· the unrestricting nature of the data collection methods allowed for insightful comment from teachers and free interpretation by the researchers
· dialogue with teachers was facilitated by the non threatening position of the evaluators and the trust built up between teachers and interviewer INSET providers and evaluators were identified to teachers and their roles explained at the outset
Our contention is that qualitative long term evaluation methods can be applied to effect in a developing country context but that a number of conditions will need to be satisfied if a high level of success is to be achieved.
|
The conditions for successful use of longitudinal INSET evaluation methods are that: · the teachers have a high level of personal trust in the evaluators so that they are secure in their knowledge that feedback is confidential and will not be used against them but to help improve systems; · the evaluation should be done by people other than the INSET providers; · the evaluators are trained and can achieve a standard of practice likely to achieve meaningful outcomes; · the evaluators have adequate time and resources to carry out the study; · the teachers know what is being evaluated, the methods being used, the time scale of the exercise and its purposes. |
