Chapter 6 - Discussion

In this chapter we will discuss the following aspects of the research.

1. Interpreting the profiles
2. The reliability and validity of the levels of assessment

Figures 1 and 2 in Chapter 5 show that in Malawi, for English and Chichewa 27% of children and 40.7% of children respectively achieved level 3. In terms of the arbitrary norms set by teachers children could be seen to be under-performing in this learning area. It is encouraging, however, that 38.8% of children are achieving at least a level 2 in English and if the profile takes the attainment of both level 2 and level 3 into account then the overall level of achievement in this domain of literacy is promising.

The verbatim records kept by teachers of children’s oral language shows that children have a reasonable to good vocabulary in English in this age group. It would appear that the difficulty lies in constructing sentences or extended discourse in English. Children at all levels appear to lapse readily into Chichewa to convey meaning. A sustained discourse in English without inserting phrases and sentences in Chichewa to clarify meaning is rarely achieved.

In Sri Lanka the oral competencies of children in English is similarly below the norms set by teachers. Figure 6 in Chapter 6 reveals that only 20.3 % of children achieved level 3. It is however, worth noting that 42.5% of children achieved level 2 illustrating that in this country like in Malawi oral competencies reside more in the knowledge the children have of English words rather than in their ability to sustain a discussion. The oral samples collected by teachers illustrate a reasonable vocabulary but there is a definite weakness in children’s capacity to build up a discussion on the basis of what they see in an illustration and to go beyond the illustration to use more imaginative language.

The Reading Profiles

The assessment of reading in both countries revealed interesting similarities. On the whole children were reasonable to good readers in both Malawi and Sri Lanka if one went by their accuracy scores contained in the running records maintained by teachers. The Profiles of Achievement for reading in English and in Chichewa in Malawi shows that 32.4% of children achieved level 3 in English and 39.2% in Chichewa. It is also worth noting that 37.6% of children achieved level 2 in reading in English and 17% achieved a level 4. The reading records maintained by teachers showed that children understood what they read and were able to comment on the text. Those children who achieved levels 3 and 4 very often demonstrated (much to the surprise of their teachers), capacities to provide interesting endings to passages in reading tasks which stopped short of providing an ending or could give alternative outcomes to stories when asked to do so.

The reading tasks revealed for teachers that students enjoyed reading and had a positive attitude towards the activity. In Sri Lanka the reading tasks were conducted in interesting and varied ways. Most tasks were constructed in a role play fashion in which children had to engage in dialogue. Readers took on the roles of mother, uncle and Rani and showed an acute awareness of how language functions in social and cultural contexts such as talking to fruit sellers at the fair.

The Profiles of Reading in English, Tamil and Sinhala show that teacher norms based on the expectation that at least 50% of children would achieve level 3 in each case were not borne out. In English the majority of children achieved level 2 (44.2%) while only 30% achieved level 3. In Tamil and Sinhala the majority of children achieved level 3 (37.2% and 43% respectively) and a slightly smaller number achieved level 2. It is worth noting here that in Tamil 25.8% of children achieved level 4.

The Writing Profiles

The Profiles of Achievement for writing in English and Chichewa in Malawi are interesting. 40.5% of children achieved level 3 for writing stories in English and 42.8% achieved level 3 for story writing in Chichewa. The quality of written work, both story writing and descriptive writing in Malawi is high.

The samples of writing maintained as evidence by the teachers revealed that children were capable of using language imaginatively and meaningfully. This appears to be in contrast to children’s use of language in the domain of speaking. It is perhaps worth reflecting here that the assessment of oracy is not a standard feature in Malawi and the unfamiliarity of the task is likely to have an effect on the outcome.

In Sri Lanka the Profile for writing in English was unsurprising. Over 300 samples of writing were analysed. Most children, 83.6% of the sample achieved level 1; 13.6% achieved level 2 and only 2.5% achieved level 3.

As noted in above, English is introduced to children in Sri Lanka in year 3. The pattern of teaching English to children in the first instance is to develop their communicative competence in the area of speaking and reading. Writing receives less attention in the form of the production of extended text. What the writing samples reveal is that children are competent in naming things and spelling words correctly and indeed in many cases they are capable of producing simple sentences correctly. However, the use of conjunctions and other devices to link sentences has not been widely established at this level.

It is clear from the results therefore that the expectations of teachers of the levels children could attain were too high.

The Maths Profiles

The assessment of pupils’ work on the mathematics tasks was carried out using an item analysis which revealed correct answers, mistakes and pupil responses to the open questions.

In general, pupils showed a high level of success in the straightforward knowledge and computational problems using the four ‘rules’.

Word problems presented greater difficulty but it was often difficult to distinguish between difficulties caused by the language (English) as such or the need in word problems to use less algorithmic approaches i.e. needing to decide on appropriate strategies and operations to achieve a solution.

Teachers regarded language as the primary cause of difficulty but evidence from countries where English is a first language suggests that the other cause is frequently a problem where mathematical activity has become routinised through a narrowly conceived algorithmic approach.

How reliable and valid were the tasks as a way of collecting data on children’s achievements?

Mathematics Tasks

In Malawi, the infant tasks were prepared in English and it was necessary for teachers to translate the instructions and the word problems into the vernacular although words for numbers have been learned in English. The group task appeared to present organisational problems with such young pupils in large groups.

Teachers, however, indicated that the tasks were appropriate for the pupils at each level.

More open questions which had no unique answer presented problems for the teachers in making an assessment. For example, Senior primary pupils were asked to ‘write down a number which is smaller than 10, but which is as close to 10 as you can make it’. Teachers were asked to ‘mark’ some pupil scripts before the teacher meetings but not to add marks to achieve an overall ‘score’. However most did add up the number of right answers, presumably through force of habit. Knowing whether each answer is right or wrong becomes important in such a process if one is to achieve a total. Experience of what can be learned from the variety of pupil responses and how they might be used as a starting point for further teaching is necessary if the potential of such questions is to be realised.

While most of the time in teacher meetings was spent analysing and discussing pupils’ work on the tasks, teachers’ ways of thinking became apparent. Significant amongst these were

· personal uncertainties about possible answers to some of the more open questions; as was to be expected the range of competence and confidence in mathematics varied considerably within each group.

· ‘bounded’ thinking and setting of expectations in relation to mathematical knowledge e.g. ‘children in Standard x do not know about numbers greater than y because it is not on the syllabus’. The continuity and underlying patterns in much of mathematics are easily lost, as is the desirable opportunity for pupils to extend into areas with which they are unfamiliar through appropriate problems and challenges.

· strongly held beliefs as to why children make certain mistakes and what is needed to rectify them. The latter were usually expressed in the form ‘they need more practice at this type of question’. Beliefs appear frequently to centre on ‘practice making perfect’; an awareness of the significance of conceptual understanding was seldom evident.

Regarding the group work task, it was felt that this did not motivate the less able although sometimes the mixture of strong and weak pupils in the group prompted the latter with the right answer but without understanding, and sometimes the answer was copied. It was felt it would be better if the task were done individually. Some teachers had tried competition between groups with success. In the chain question the instructions in the example are not clear.

The first and second tasks were felt to be easy, but the third, which needed multiplication and addition was found to be difficult. However, overall the children enjoyed the tasks done in a small group.

In Tamil a good deal of time was spent discovering which words children had found difficult or easy to read in terms of the number of letters. It was found that words with 5 or 6 letters caused difficulty and this needed to be addressed by introducing more longer words into the curriculum. Teachers in all the groups were invited to identify individually and collectively examples of good, average, poor and, in some cases, very poor, work as basis for building up descriptions of pupils’ achievements in terms of agreed criteria of competence.

Language Tasks

The language assessment tasks were divided into four areas - reading, writing, listening and speaking, although in Year 4 only listening and speaking are in the curriculum. The tasks used did not include an assessment of listening.

Reading

In Sri Lanka children were given tasks for reading and writing, although these are not in the text. books. The tasks revealed a surprising amount of knowledge which had not been formally taught. In reading, there were three different tasks. In the first one ‘Let’s go to the fair’, students appeared to be motivated by the presentation of a real conversation and the role play approach. Even the weaker ones who habitually do not seek to answer questions or become involved wanted to show what they could do. Many of the children were concerned about the marking which took place in front of them, given their unfamiliarity with this kind of assessment. Some had to be reassured that it was not a test. Having understood the nature of the task, the children both enjoyed it and were able to correct themselves. It was felt that this task is easier to read than conventional reading exercises in the text book and encouraged the children to use appropriate text patterns. The teachers found various ways of implementing the instructions given with the task. One teacher for example gave all the other children in the class a different task to do whilst concentrating on the three involved in the dialogue. Another teacher repeated the task using different children as a drama. In the event of the course, the subsequent participants would have had the benefit of hearing earlier attempts. However, the teachers felt there were no real practical problem to using the tasks. For task 2 which is longer, the children apparently found it more difficult, and more words were mispronounced such as ‘stall’ and ‘too’. Some children did not pause in the appropriate place or were not sure if it was a question.

In Task 3 there were a number of specific mistakes. ‘For’ was read a ‘off, ‘toys’ was pronounced ‘tows’, ‘has’ was read as ‘how’, ‘the’ and ‘them’ were new words to the children and they needed help with them, and ‘guess’ was pronounced ‘goss’ and was also too difficult. It was apparent that schools which have a limited English language use, for example Sinhala-medium only using the text book for English, found much more difficulty than schools where extra reading material and stimulus material generally is supplied with more emphasis on English-medium context. The teachers were very clear that from a curriculum point of view there was strong evidence of the desirability of exposing children to as much English as possible in all different forms, with as much variety and interest built in. They were impressed with how much the children could do, that they would not have expected.

Speaking

The children’s ability to speak fluently was affected by the degree to which they were concerned about doing the task correctly. This caused some to hesitate, although they were potentially competent. Familiarity with the role play built in to the text book helped overcome this problem. The tasks had been correctly set up with the setting the scene methodology used in the text book. The children were able to produce sentences. Setting was important to the child’s subsequent success. The quality of the children’s work varied in terms of its creativity, its accuracy, its complexity, its fluency, variety of structure, the enjoyment shown by the children, how meaningful it was and the vocabulary used. Specific problems were identified such as difficulties in using the word ‘the’. An example of the benefit of this kind of assessment occurred in one classroom where a teacher gave a child a task to do. The teacher said he would not be able to do it, but he was successful, thereby showing the benefit of systematic assessment of this kind.

Writing

It was found that the children tended to write the sentences that they had already spoken. The standard achieved in writing was regarded as less good than that achieved in the other areas and the teachers felt that the curriculum should give more time to it. They identified a number of dimensions of quality including spelling, the use of paragraphs, words or phrases or sentences; the use of new vocabulary not in the text book, grammatical accuracy, a variety of tenses, the use of linguistic pictures not in the text book, punctuation and capital letters.

How reliable and valid were the Levels of Achievement?

The question about the reliability and validity of the Levels of Achievement is an important consideration for profiling systems. There are a number of methods of constructing a profiling system, some which make stronger claims than others to reliability and validity indices. The KEEP profiling system in Australia for example does not claim to provide numerical indices of reliability or validity and argues that the intention of the system is to profile performance-based assessments in a non psychometric fashion. The Victoria English Profiles on the other hand make claim to strong reliability and validity indices. Teachers are required to rate student’s achievements with reference to indicators specific for each of nine bands. A score of 3 is obtained if all behaviours associated with a band are consistently displayed by a student. A 2 is achieved if most of the behaviours are present. A 1 is achieved if some of the behaviours are developing and a 0 if none of the behaviours have yet been observed. This scoring system is then subjected to Guttman reliability estimates (see Rowe and Hill, 1996:328) as well as test/re-test reliability’s and inter-rater reliability (see Rowe and Hill, 1996: 329). The main source of evidence for the reliability and validity of the reading profile is drawn from the Victorian 100 schools study (Griffin and Rowe, 1988) in which the profile was completed for 5,000 pupils.

Like the KEEP system, the profiling framework used in this study does not make strong claims for reliability. The levels were determined by through teachers’ professional knowledge and their analysis of samples of pupils work. High inter rater reliability coefficients were obtained.

Were teachers able to maintain records of achievement for individual children?

The study found that most of the teachers involved in the study were able to maintain a record of achievement for the children they chose to assess. (It is worth noting here that not all children in the class were assessed. We shall return to this point later).

The research established that, on the whole, all teachers involved in the study were able to apply the profiling framework in one form or another, in collecting evidence of children’s achievement in literacy. Teachers agreed that the descriptions of the levels of achievement provided them with a meta-language by which to describe (to children, colleagues and parents) what children were achieving. Teachers also submitted that the profiling system gave them a way of assessing domains of literacy which were previously either not assessed or assessed in a fairly basic fashion. Teachers agreed that the application of the framework sensitised them to children’s learning in a way that was not possible before. This augurs well for the diagnostic purposes of assessment, embodied in profiling systems.

Could teacher’s collect evidence?

The collection of evidence proved to be a new and often ‘daunting’ task for teachers. Most of the teachers in Malawi had no history of developing portfolio’s of children’s work and the collection of evidence was particularly challenging. Resource limitations proved to be an influence upon the amount and range of evidence of achievement which could have been accumulated. The tasks were all accompanied by teacher record sheets and pupil work sheets. This allowed teachers to collect samples of children’s work as evidence of their achievements. Beyond the tasks however, the collection of additional information about children’s performance was limited.

Annotating evidence of children’s work produced interesting differences between teachers. Teacher’s in both countries had little prior experience of commenting in detail on pupils work. In both countries, the deficit model of learning (right or wrong answers) was strongly held by teachers and a model which encouraged them to comment not on the answers but on what children might be doing and the learning processes they were experiencing was quite challenging. Teachers in both countries however rose remarkably to this challenge and provided substantive annotations of children’s work. These annotations and comments is evidence that teachers are gaining more of an understanding of the ‘how’ of learning.

Even those comments and annotations on student’s work, which provided less of an insight into the learning processes on a particular aspect of work, could be interpreted as giving insights into the broader context in which education takes place. Take the following comments by a teacher in Malawi who is looking at the relative performance of two students on a mathematics tasks:

‘The performance of this child in mathematics is average because wherever he has been given work to do he tries as much as possible to get it right. At times he manages to do so but at times with difficulties, for example when an example has been given the teacher must repeat it several times before he understands what is expected to do. A number of skills and methods must be used in order for him to grab a thing. For example, if the teacher is to explain about a triangle using bottle tops he/she has to demonstrate, then do practice together with the pupil, then ask him to do by himself. In so doing is able to do it correctly.

The problem in understanding is not because the pupil is not intelligent but because he lacks continuous practice at home as well as at school, because of the swollen numbers of pupils in one class. Given a chance to learn in a class where the number of pupils is reasonable and the teacher is able to do individual help probably he can do better than at present’.

According to the performance of this pupil, it shows that the girl is able to do a good number of activities with little supervision. For example once she has been given an example in Mathematics, she is able to do the rest on her own. After thorough investigations it was discovered that she is able to grab things fast because before her primary education she had attended nursery school. She has got sisters who are in private schools where English is fluently spoken. Her family is well to do. According to reliable sources her parents once worked outside this country where probably life is a bit advanced...’.

Whether the theory is right or wrong is not the concern. Rather, they show that teachers think about the factors which influence learning as well as the processes of learning. It is also true to say that these are always inter-related.

How manageable was the system?

Another question addressed by the study is to do with the manageability of task administration as a method of gathering evidence of children’s achievements. According to Sheil and Forde (1995) the manageability of a profiling system is vital to its success. They argue that administration of tasks, documentation and reporting procedures are time consuming and when demands on teachers become excessive, the profiling system may be regarded as unmanageable.

The study found that administering the learning tasks were demanding of teachers time. Teachers found that using the tasks as assessment tools per se was not necessarily new to them, but the practice of making notes or recording their findings, constituted a cultural shift.

The physical size of the classrooms in the schools involved in the study was also a factor. None of the teachers involved in the study were able to use the tasks with all the children in their classes. Rather, teachers in all schools selected children in their classes (normally between six to ten pupils in a class) for inclusion in the study.

Sampling within a class does not necessarily invalidate the results of a study. It is perfectly reasonable to argue that a stratified random sample which is based on teachers’ intuitive judgements of ‘weak’, ‘average’ and ‘good’ learners is a useful way to obtain a sample within a class. Monitoring the achievements of these learners over time would provide a legitimate basis for generalising about the relative performances of larger classes as a whole.