5 The 1994 research

5.1 Description
5.2 Results of the English reading test
5.3 Results of the local language reading tests

5.1 Description

5.1.1 Aims
5.1.2 Methods
5.1.3 Schools and testees
5.1.4 Administration

5.1.1 Aims

The 1994 research was undertaken in order to investigate reading proficiency at year 5 in English and the local language. However, it had the further aims of investigating in more depth individual pupils' reading in English and the local language. This individual investigation examined a sample of pupils who had achieved high and low scores on the English reading tests, to see whether there were any factors in their background associated with their scores. It also looked at the comprehension strategies of high and low scorers, the difficulties they report, and the ways in which they try to overcome their difficulties.

5.1.2 Methods

The methods employed to collect data were as follows:

(i) a 30 item reading test in English

(ii) a 30 item reading test in the local language (namely Chichewa in Malawi, and ChiNyanja in Zambia)

(iii) a structured interview on pupil background and reading habits carried out either in English or the local language with a local researcher

(iv) a passage in English to be read and discussed either in English or the local language with a local researcher

(v) a passage in the local language to be read and discussed with a local researcher

The 1994 Reading Tests: The group reading tests were again, as in the 1992 research, modified cloze tests with a total of 30 items each. The language for the English test was taken from the English language course books English in Malawi and New Zambia Primary Course, English Language at or below fourth year level (i.e. a year below that of the testees). The test consists of four different passages, with a fairly steep grading of difficulty. In terms of socio-cultural appropriacy the passages contain topics and episodes with which the pupils were, according to teachers, familiar, and which closely resemble those in their course books (for sample extracts, see Appendix F).

The texts for both local language tests were modified versions of Zambian school textbooks (Werenga Cinyanja). The Malawian version of the texts was appropriately modified in terms of spelling, lexis and morphology by a Chichewa expert from the Malawi Institute of Education. This results in very small differences between the versions (see Appendix G for an extract). The same gaps were inserted in both, and the texts may on the basis of close similarity be regarded as of equivalent difficulty. The number of items was set at 30 (rather than 60 as in 1992) since this test was aimed exclusively at year 5 pupils and in principle did not have to allow for such a range of ability as the 1992 test.

5.1.3 Schools and testees

In each country data were collected from year 5 pupils in 4 rural schools and 2 urban schools. In Malawi, three schools were selected in the Zomba area (two rural, one urban) and three in the Lilongwe area (two rural, one urban). In Zambia, four rural schools and two rural schools were selected in the Lusaka area. All schools were different from those tested in the 1992 research, which broadens the base of the testing research to 11 schools in each country. The reason for choosing pupils in the fifth year is partly that it "closes the gap" in the 1992 research, while from a comparative perspective it is also the year in which Malawi starts the switch to English medium, and where any inter-country differences might be maximised.

The subjects for these tests were all pupils present in the year 5 class on the day of testing, with a sample of 50 being randomly selected in cases where classes were above 60. A total of 290 pupils were tested in Malawi, and 227 in Zambia. The data collected on each child included their date of birth, sex and home language. The mean age of testees in Zambia was 12 years 3 months, and in Malawi 13 years and 7 months. All the children in Malawi and Zambia claimed to be able to speak Chichewa and Nyanja respectively, although 18% of the Malawian testees spoke a language other than Chichewa at home, and 46% of the Zambians a language other than Nyanja (the lack of fit of the 1994 Zambian testees when compared with those of 1992, in terms of mean age, and percentage claiming Nyanja as home language, is due to the 1994 rural schools being in the Lusaka rural area, rather than the Eastern Province rural areas).

Table 17: Malawi: Testees for Reading Tests 1994

School	A	B	C(U)	D	E	F(U)	Total
Girls	0	32	25	23	21	28	129
Boys	50	0	25	32	26	28	161
Total	50	32	50	55	47	56	290

Table 18: Zambia: Testees for Reading Tests 1994

School	A(U)	B	C	D	E(U)	F	Total
Girls	17	11	12	14	27	19	100
Boys	22	28	19	20	28	10	127
Total	39	39	31	34	55	29	227

5.1.4 Administration

The group reading test in English was administered first, followed by a break of 15 to 30 minutes, and then the group reading test in the local language. Pupils were given up to 35 minutes to complete each test, with many finishing in half that time. Practice sessions for both language tests were carried out.

5.2 Results of the English reading test

5.2.1 General
5.2.2 Overall means
5.2.3 Location (urban/rural)
5.2.4 Gender
5.2.5 Location and gender

5.2.1 General

The cloze tests each had a maximum score of 30 points. The following tables provide the mean, standard deviation, maximum, median and minimum scores. Statistical analyses were performed by the Applied Statistics Department at Reading University, using the SAS package. The conventional significance level of 0.05 was chosen. The reliability of the tests (KR-21) varied from 0.75 to 0.95. The overall means were as follows:

Table 19: Overall 1994 Results of English Reading Test for Malawian Primary Schools

N	Mean	SD	Max	Median	Min
290	12.84	6.22	30	12	1

Reliability Index (KR-21): 0.84

Table 20: Overall 1994 Results of English Reading Test for Zambian Primary Schools

N	Mean	SD	Max	Median	Min
227	11.72	9.48	30	8	0

Reliability Index (KR-21) 0.95

5.2.2 Overall means

The means are very slightly in favour of Malawi, but not at a statistically significant level, and support the findings of the 1992 research, namely that there is no difference in reading ability in English between children in Zambia and children in Malawi. The evidence here again suggests that Zambian children who have officially had the first four years of education through the medium of English are not superior to Malawi children who have officially had Chichewa as a medium of instruction for those years.

The mean score of approximately 12 out of 30 for both countries is not high, given that the test was drawn from course books at year 4 and below. However, this mean score conceals considerable differences between schools, sexes, and the rural/urban divide, as well as between individual pupils within schools.

5.2.3 Location (urban/rural)

Table 21: 1994 Results of English Reading Test for Malawian Primary Schools by Location

Loc.	N	Mean	SD	Max	Median	Min
Rural	184	11.43	5.18	28	11	3
Urban	106	15.29	7.08	30	15	1

Table 22: 1994 Results of English Reading Test for Zambian Primary Schools by Location

Loc.	N	Mean	SD	Max	Median	Min
Rural	133	8.46	7.49	30	6	0
Urban	94	16.32	10.10	30	17.5	0

In Malawi there are clear differences (over 3 points) in favour of urban schools but this is not statistically significant. In Zambia there are much larger differences (over 7 points, with the urban score almost double that of the rural) in favour of urban schools but these just fail to. attain statistical significance (p = 0.0526 which is marginally over the significance level of 0.05). In strict statistical terms, the view that urban schools outperform rural schools is therefore not supported. However, the statistician's written comment was "It seems likely that a location effect is present in both countries, but that the small number of schools [in each country] in the study prevents this being detected" (personal communication, Department of Applied Statistics, Reading University, 1995). Moreover, when the data sets from the two countries are combined, the location effect is in fact significant (p < 0.02).

The contrast in Zambia is clearly displayed in the histogram of the distribution of English test scores for all children (see Appendix H). Unusually, this does not display a normal bell-shaped curve, such as one would expect from one population. The curve, in fact, is U-shaped with high scores clustering on the right hand side, and low scores on the left. In fact, this U-shape results from rural scores being skewed towards the bottom, and urban scores being skewed towards the top. Rather than suggesting the outline of a single population with most pupils scoring around the average, these results suggest rather the outline of two populations, one rural, one urban. The implications for education in Zambia give cause for concern.

5.2.4 Gender

Table 23: 1994 Results of English Reading Test for Malawian Primary Schools by Sex

Sex	N	Mean	SD	Max	Median	Min
Boys	161	13.72	6.40	30	13	3
Girls	129	11.75	5.83	30	11	1

Table 24: 1994 Results of English Reading Test for Zambian Primary Schools by Sex

Sex	N	Mean	SD	Max	Median	Min
Boys	127	11.13	9.51	30	7	0
Girls	100	12.46	9.42	30	10	0

In Malawi the difference between the sexes in the English test is statistically significant with boys scoring some 2 points higher than girls. The pattern of scoring is revealed in the histogram (Appendix H) with noticeably fewer high scoring girls than boys. In Zambia there is no statistically significant sex difference although girls have in general a higher mean than boys. This somewhat unexpected result is consistent with those obtained in the 1992 research, where Zambian girls again had higher means, but not to a statistically significant extent.

5.2.5 Location and gender

Table 25: 1994 Results of English Reading Test for Malawian Primary Schools by Location and Sex

Location	Sex	N	Mean	SD	Max	Median	Min
Rural	Boys	108	12.46	5.57	28	12	3
Rural	Girls	76	9.97	4.20	26	10	3
Urban	Boys	53	16.28	7.23	30	16	4
Urban	Girls	53	14.30	6.86	30	13	1

Table 26:1994 Results of English Reading Test for Zambian Primary Schools by Location and Sex

Location	Sex	N	Mean	SD	Max	Median	Min
Rural	Boys	77	8.73	7.92	30	5	0
Rural	Girls	56	8.09	6.92	26	6	0
Urban	Boys	50	14.84	10.60	30	14.5	0
Urban	Girls	44	18.02	9.31	30	20	1

Combining the effects of location and sex reveals, as would be expected, considerable differences. In Malawi urban boys score most highly, followed by urban girls, rural boys and rural girls. The difference between the first group and the last is some 6 points. In Zambia urban girls score most highly, with urban boys next, then rural boys and rural girls. The differences are considerable with the mean score of urban girls being more than double that of rural girls.

5.3 Results of the local language reading tests

5.3.1 Overall results
5.3.2 Malawian pupils results on Zambian test
5.3.3 Location and sex in local language 1994 results
5.3.4 Testees from non-Chichewa or non-Nyanja speaking homes
5.3.5 Comparison of English and Chichewa 1994 results
5.3.6 Gender and location discrimination of English tests

5.3.1 Overall results

The most striking feature of the overall means, as in 1992, is the vastly superior performance of the Malawian children. In fact, the overall mean for Zambia at 4.4 is only just above the chance level of 4. The results suggest that Malawian children read better in Chichewa than Zambian children read in Nyanja.

Table 27: Overall 1994 Results of Chichewa Reading Test for Malawian Primary Schools

N	Mean	SD	Max	Median	Min
290	19.88	5.44	30	20.5	4

Reliability Index (KR-21): 0.80

Table 28: Overall 1994 Results of Nyanja Reading Test for Zambian Primary Schools

N	Mean	SD	Max	Median	Min
227	4.4	3.70	22	3	0

Reliability Index (KR-21): 0.75

5.3.2 Malawian pupils results on Zambian test

However, despite the fact that the Malawian and Zambian versions of the reading tests differed mainly in spelling, one cannot conclusively demonstrate that the two versions were of equivalent difficulty. To counter any suspicion that it might be more difficult, the Zambian Nyanja test was administered to pupils in two Malawian schools (in addition to the Malawian test). The results were:

Table 29: Malawi Pupils' Overall Results on Nyanja (Zambian) Reading Test

N	Mean	SD	Max	Median	Min
74	18.66	5.49	30	19	8

Reliability Index (KR-21): 0.79

The Malawian pupils clearly achieve higher scores in Nyanja than the Zambian pupils, despite the fact that they are reading a version intended for Zambians. These results suggest that the two tests are indeed of comparable difficulty, and confirms that the Malawian pupils are superior to the Zambian pupils in local language reading performance.

5.3.3 Location and sex in local language 1994 results

Table 30:1994 Results of Chichewa Reading Test for Malawian Primary Schools by Location and Sex

Location	Sex	N	Mean	SD	Max	Median	Min
Rural	Boys	108	20.19	5.16	30	21	10
Rural	Girls	76	18.36	5.11	30	19.5	4
Urban	Boys	53	20.68	6.04	30	22	6
Urban	Girls	53	20.62	5.54	30	21	4

Table 31: 1994 Results of Nyanja Reading Test for Zambian Primary Schools by Location and Sex

Location	Sex	N	Mean	SD	Max	Median	Min
Rural	Boys	77	4.00	3.52	21	3	0
Rural	Girls	56	3.23	2.57	13	2.5	0
Urban	Boys	50	5.18	3.40	16	4	0
Urban	Girls	44	5.68	4.91	22	4	1

The results for Malawi show that there is no statistically significant difference between the sexes in the Chichewa test, the boys having only a slightly higher mean score than the girls. This contrasts with the results in English, where boys were superior to a statistically significant degree. Likewise the Malawi results yield very low urban/rural differences of no statistical significance.

In Zambia there is no significant sex effect, and although the Zambian results demonstrated a statistically significant location difference in favour of urban children, the means (3.66 rural; 5.41 urban), are so close to the chance score of 4, that this result does not warrant consideration.

5.3.4 Testees from non-Chichewa or non-Nyanja speaking homes

The test was administered in areas where the majority of pupils claimed Chichewa (in Malawi) or Nyanja (in Zambia) as their home languages. In both countries, however, there was a minority of children who spoke another language at home. When the results were analysed for home language effect, those from Chichewa or Nyanja speaking homes had a very slight (1.6 and 0.9 points respectively) but not statistically significant advantage over those who had a different home language.

5.3.5 Comparison of English and Chichewa 1994 results

Analysis of the results (Spearman Correlation Coefficient) suggests, as in 1992, that there is a weak tendency for the pupils who score well in English to also score well in local language, while those with low scores in English tend to have low scores in the local language. Although the correlations were not high (0.57 for Malawi (p < 0.0001), and for Zambia 0.38 (p < 0.0001)) we may infer from them that reading proficiency in one language is probably not acquired at the cost of reading proficiency in another.

5.3.6 Gender and location discrimination of English tests

One striking finding from Malawi is that in the Chichewa test results show smaller differences with respect to the effects of gender, location (urban/rural) and school, than do the English tests. This suggests that English tests discriminate more against girls, and against rural pupils, than do Chichewa tests.

The reason is possibly that both rural and urban children, and both boys and girls, are equally exposed to Chichewa in the social environment and so acquire it in roughly equal measure; English on the other hand, is more available in urban environments giving urban children an advantage. In addition, learning of English is more dependent on "being taught" and therefore variables such as teacher attention and accessibility of books may favour boys.