4.4 Monitoring progress & problems in the pilot projects

4.4.1 Bangladesh
4.4.2 progress and problems in Uganda
4.4.3 Progress and problems in El Salvador
4.4.4 Cross-case study analysis

4.4.1 Bangladesh

Observations on the progress of REFLECT circles

Rate of progress

By November 1994, just four months into the literacy course the learners had covered about ten Units, representing a much more rapid progress through the manual than in Uganda and El Salvador. Partly this was due to the regularity of classes (six days a week compared to two or three days a week in Uganda and El Salvador). In November various circles were visited by participants from India, Uganda, South Africa, El Salvador, Spain and the UK - all of whom had come for an International Workshop on REFLECT held in Dhaka. Whilst probably disrupting progress a little this also added to the motivation of learners - a feeling that they were part of a much wider experiment with REFLECT.

By February, the time of the first evaluation, most circles had reached Unit 14 and by May (the time of the second evaluation) most had reach Unit 19 or 20. No attempt had been made in the design of the manual to determine what time period was required to cover each Unit or the whole course. This indeed had been actively resisted as it was felt inappropriate to impose timing on a method which had no previous field application. Moreover, it was felt that literacy programmes which define a time period (whether 6 months, 9 months or a year) run the risk that learners will feel they have finished at the end of the period and will leave the Centre, only to lose their skills a few months later. Success in adult literacy depends in part on continuity.

Overall participation

In the evaluation visits during a three to four hour meeting with the REFLECT Centres (involving construction of matrices, structured group discussion and interviews), notes were kept on the level of participation of different women. This may not be a fully accurate reflection of participation in the normal functioning of the circle (which would perhaps be higher) as it involved dealing with male outsiders to the community.

The high level of participation is striking and the table below, though impressionistic, reflects what many people have observed in practice. This is in clear contrast to the Control Group centres where levels of participation are more muted and the confidence of women does not seem so high.

Observations on the methodology in practice

The evaluators reviewed each map and matrix produced by seven of the literacy circles and a mark out of ten was given based on the following criteria:

· accuracy of the graphic construction (in relation to manual);
· quality of the graphic (care over detail);
· level of interpretation and analysis of the graphic;
· value of graphic to daily lives as expressed by learners;

The results of this review are detailed overleaf (each literacy circle is referred to by the local shomiti number):

Shomiti	CU	CU	JI	JI	JI	JI	JI	TOTAL
number	109	115	193	209	195	203	207
	M	H	M	M	M	M	H
Total present	12	10	11	11	9	11	10	74
Very active	8	7	6	9	9	7	8	54= 73%
Some participation	3	3	3	2	0	2	2	15= 20%
Almost no particip.	1	0	2	0	0	2	0	5 = 7%

(M = Moslem group. H = Hindu group.)

Shomiti	CU	CU	JI	JI	JI	JI	JI	Total	Ave.
number	109	115	193	209	195	203	207
	M	H	M	M	M	M	H
1. HOUSEHOLD MAP	10	+ 8	+ 9	+ 9	+ 10	+ 7	+ 10	=63-7	9.0
2. NATURAL RESOURCE MAP	7	+ 9	+ 7	+ 9	+ 9	+ 7	+ 4	= 52-7	7.4
3. AGRICULTURAL MAP	6	+ 8	+ 5	+ 4	+ 5	+ 3	+ 3	= 34-7	4.8
4. RAINFALL CALENDAR	8	+ 4	+ 5	+ 4	+ 6	+ 9	+ 10	=46-7	6.6
5. GENDER WORK CALENDAR	7	+ 3	+ 7	+ 7	+ 8	+ 8	+ 9	= 49-7	7.0
6. FOOD AVAILABILITY CAL.	9	+ 6	+ 7	+ 8	+ 3	+ -	+ 7	= 40-6	6.6
7. INCOME/EXPENDITURE CAL.	7	+ 10	+ 6	+ 8	+ 9	+ 6	+ 9	= 55-7	7.9
8. BASIC COMMODITIES CAL.	8	+ 9	+ 7	+ 9	+ 9	+ 8	+ 10	=60-7	8.6
9. CREDIT MATRIX	8	+ 4	+ 7	+ 2	+ 6	+ 8	+ 10	=45-7	6.4
10. SAVINGS/CREDIT BOOKS	-	-	-	-	+ 8	+ -	+ -	= 8-1	8.0
11. PREFERENCE USE OF LOANS	5	+ 6	+ -	+ 3	+ 10	+ -	+ 5	= 29-5	5.8
12. PROJECTIONS OF LOAN USE	9	+ 6	+ -	+ -	+ 9	+ -	+ -	= 24-3	8.0
13. EDUCATION MATRIX	8	+ 8	+ 7	+ 8	+ -	+ -	+ 8	= 39-5	7.8
14. HEALTH CALENDAR	9	+ 6	+ 6	+ 10	+ 10	+ 9	+ 7	= 57-7	8.1
15. HEALTH SEVERITY MATRIX	5	+ 6	+ -	+ 8	+ 8	+ 6	+ -	= 33-5	6.6
16. HEALTH CURATIVE MATRIX	8	+ 8	+ 7	+ 10	+ 10	+ -	+ 9	= 52-6	8.6
17. HERBAL MEDICINE MATRIX	9	+ 9	+ 8	+ 10	+ 8	+ 9	+ 8	=61-7	8.7
18. HEALTH AND HYGIENE MAP	8	+ 7	+ 9	+ 10	+ 9	+ 9	+ -	= 52-6	8.7
19. VACCINATION CARD	-	-	-	-	+ 9	+ -	+ -	=9-1	9.0
TOTAL:	131	117	97	119	146	89	109	=808
no. of units seen	17	17	14	16	18	12	14	=108
average per unit seen:	7.7	6.8	6.9	7.4	8.1	7.4	7.7	=7.4

Participants from a REFLECT circle display one of their graphics

It is interesting to note that whilst there was some variation between Centres the variations were not huge. Learners and facilitators in all Centres had succeeded in constructing effective maps and matrices.

It can be seen that certain Units were more effective than others. For example the Household Map, the Basic Commodities Calendar, the Income and Expenditure Calendar, the Health Calendar, Curative Matrix, Herb Matrix and Health Map were all of consistently high quality. Some Units proved too complex and require simplification, particularly:

· Preference Ranking on the use of Loans,
· Projections of Loan Use
· Health Severity Matrix.

Generally the learners are very enthusiastic about their maps and matrices and quite rightly proud of their achievements. When challenged about the purpose of the graphics they are quick to respond and justify the work they have put into them. The learners seems to have become the biggest promoters of REFLECT. On one occasion we asked "why waste your time on all of this? What is the point of all these lines on this piece of paper?" The response was immediate from one learner, leaning to point at a part of the calendar on display: "that's not just a line, that means we don't have enough food to eat in that month!"

One remarkable feature of REFLECT in Bangladesh compared with Uganda and El Salvador is the quality of the maps and matrices produced by the learners. These are usually kept in pristine condition and are full of colour and detail. In most cases the learners themselves have drawn them, having initially etched them in the mud of the compound and illustrated them with seeds and other materials. The learners' books are also full of drawings whether of birds, animals, flowers or ornate patterns. There is a feeling of "release"- a sense of wonder at what can be done with just a pencil and a blank page - and there is a real joy in many of the images. The learners' books in the control groups have none of these pictures and seem full of copied words in comparison. The value of allowing, indeed encouraging, the development of drawing skills within a literacy class is not usually recognised - but the evidence here seems to strongly indicate a role for drawing - both for increasing motivation of learners, enabling them to have fun and at the same time providing them with the manual dexterity skills necessary for writing.

"That's not just a line; that means we don't have enough food to eat in that month."

Another striking element about the REFLECT workbooks in comparison to those usually found in literacy classes is that there is a lot of writing. Rather than just having repetition of letters or syllables (of which there is very little) and rather than having words and phrases that are routinely copied (so all books are the same), each book has different phrases and different work which appears to show an emphasis on creative writing - based on the learners' own thoughts. There are also a lot of numbers to be seen, particularly with copies of the calendars or matrices and with calculations based on these.

Enrolment, attendance and drop out

The table below provides basic details about the 10 REFLECT Centres: the number enrolled, level of permanent drop out, average attendance after 6 months and average attendance after 10 months.

It should be noted that "Average attendance" was defined as the average number of learners in the circle over the previous month. It usually represents fewer than the total number who are still attending and indicates irregularity of attendance of different individuals for various reasons. This is analysed in the next section.

In total 78% of those who initially enrolled are still attending the literacy circles (though some are irregular). This compares to just 55% of those in the control groups. Three of the ten control group centres were closed down after four months owing to virtual non attendance.

Shomiti	CU	CU	JI	JI	JI	JI	JI	JI	JI	CU
Number	109	115	193	195	203	209	207	127	201	125	Total
Initial enrolment	15	16	14	15	15	15	12	15	21	16	154
Permanent drop out	3	4	0	6	4	4	1	7	2	3	34
AVE. attendance (Dec)	10	8	7	7	8	7	7	3	6	7	70
AVE. attendance (May)	9	8	7	7	10	8	6	2	5	5	67

The 22% (34 women) in the REFLECT circles who permanently dropped out over the ten month period were followed up to determine their reasons -which are classified below:

opposition of men	10
marriage	5
permanent migration	5
lack of time	8
disinterest	5
other	1
	34

The first three factors are largely beyond the control of the learners and can be considered external. These external factors account for 59% of permanent drop outs. It should be noted that opposition of husbands or fathers was the most serious cause of drop out and did in various cases include beatings. In spite of such opposition some women have continued to attend.

It is also revealing to explore the reasons for irregular attendance. Through interviews the following were identified as the major reasons:

· Seasonal work-loads.

· Ongoing work pressures in the home.

· The rains: during the rainy season circles had no shelter and were forced to meet indoors in the badly lit and cramped conditions of someone's house.

· Men's attitudes: some men opposed the literacy circle from the beginning, seeing it as inappropriate activity for women. This of course varied. In one village it was commented: "the men treat us badly if we are ignorant and they treat us worse if we try to learn". One elderly man specifically complained that the literacy circle was creating too much noise and was disturbing his afternoon sleep. The circle was forced to stop until they could reason with him (which they finally managed to do).

· Religious leaders: some religious leaders questioned whether the women should be learning Bengali and studying anything other than the Koran. They also criticised the drawing of human images -which is contrary to Islam. This could have been a serious block but was generally resolved by avoiding human figures or drawing symbols rather than literal images when human figures were needed. Most disputes with religious leaders were resolved early on though they reappeared during Ramadan when there was pressure for the women to either withdraw or switch to learning Arabic.

Some attendance dropped during Ramadan but it picked up again afterwards.

· Some women temporarily migrated in search of work with their husbands.

· Two circles suffered the loss of a facilitator (one married and one migrated out of the area) which disrupted learning for a period until a replacement could be found and trained.

· One common comment made by women who were irregular attenders was that they could not attend because they would have to leave their house unprotected which would leave them at risk from burglary. This goes to show how little the women usually leave their homesteads and the fear they have of making these first steps.

· Health problems were (as in all literacy programmes) a factor in irregular attendance.

Additional problems

The loss of the Education Coordinator who had translated the manual, coordinated the initial training and was providing follow up to the circles affected the REFLECT programme seriously. His departure threatened the continuity of the circles, particularly as another key staff member (the Monitoring and Evaluation coordinator) was not able to dedicate the amount of time needed to provide permanent support. Two new staff members became involved. Whilst both were very committed and capable staff neither had a background in education (one was previously working on credit, the other on tubewells) and neither had received the initial training in REFLECT. They learnt on the job and in the circumstances have done admirably in practice.

A wider problem that affected the programme indirectly involved staff morale within ACTIONAID Bhola. Following an external review of future options for the whole development programme in July 1994 the prospect of phaseout became associated with fears of widespread redundancy. There was also no Project Director in Bhola at this time which added to the confusion and loss of morale. Only with the appointment of a new Project Director did the mood improve (from January 1995). Once the strategic direction of the Bhola programme was re-defined morale was fully restored as staff became committed to a new vision of creating a People's Organisation which would continue after the departure of ACTIONAID.

4.4.2 progress and problems in Uganda

Observations on the progress of REFLECT

Sixty-five classes started up in January 1994. However, due to loss of facilitators through mortality and dismissal by learners, the total number reduced to sixty-one by mid 1995.

The majority of classes had been running for one year, when graduation to took place. In that time, they had covered ten or eleven units as follows:

Natural Resources Map
Human Resources Map
Household by Household Map
Mobility Map
Ideal Futures Map
Agricultural Calendar
Crop Preference Matrix
Gender Workload Calendar
Rainfall Calendar
Hungry Season Calendar
Health Matrix - only covered by some groups.

The analysis of health issues had been done by some groups as a response to the epidemic of fatal dysentery in the period immediately preceding graduation. Facilitators had discussed whether or not to bring forward the unit in their parish groups, and then taken a decision with their learners.

The rate of progress through the course had not been part of the original REFLECT design, and in general ACTIONAID staff had encouraged facilitators to let the learners take their time, and not to feel that it was a mark of their own success to have pushed their group further through the course. This would have led to superficial discussions and easily-forgotten literacy skills.

It was very fortuitous, however, that learners were of a standard to graduate, at the same time as they were about to start the part of the course devoted to Savings and Credit management. This will clearly give a renewed sense of purpose to the circle meetings, (which will otherwise follow the same format).

The graduation ceremonies for learners were held in March - May in all four parishes. Facilitators had assessed their learners on a continuous basis: testing the reading aloud of words, sentences and paragraphs; writing letters in class, completing calculations on the blackboard, and understanding of the graphics. They discussed grading criteria in parish groups and with ACTIONAID staff, and categorised learners into pre-literate (did not graduate); low literate; mid literate, and high literate.

The majority of learners therefore received a certificate, and all agreed that no-one should be chased away from the class. It was considered important to show appreciation for learners' achievements, and the community felt it was inappropriate to start new literacy classes before current ones had graduated, so learners and facilitators organised a big ceremony in each parish. These ceremonies included drumming, dancing and drama focused on creative interpretations of REFLECT units (Hungry Season Song, women and men singing together to show new unity, Family Planning role-play etc) and testimonies on the value of literacy. All the messages presented showed articulate communities recording changes they perceived to be the result of REFLECT, and were thus considered to be snap-shot evidence for the evaluation process itself.

Observing a REFLECT circle meeting in a circular shelter constructed by the participants.

Observations on the methodology in practice

In all the sample classes visited in Bundibugyo, evaluators looked carefully at the maps and calendars drawn from each unit and asked learners to interpret their work. This was in order to assess the understanding of the learners and their facilitator of the process of graphic construction, and the analytical discussion vital for consolidation onto paper. This understanding was again explored in all interviews.

In almost all cases, the sequence and clarity of the annotated graphics was satisfactory to the evaluators. The consensus of all those questioned, was that the process assisted the learners in looking objectively at the advantages and disadvantages of specific aspects of their environment, and representing these in a visual code. "We know where we are...and where other things are." said one learner to explain this point. The graphics have the power of a collective view or decision - even before any actions are taken. They are perceived as records of discussion, and as frames of reference for the future (literally hanging on the walls of the literacy shelter).

Learning to read and write using annotations to the accumulating graphics, and learner-generated language from the free-ranging but related discussions, seemed to be an established process in all classes. Exercise books showed that with the exception of words and sentences freely formed from syllables covered, writing was based on unit discussions, and action points. The learner's level did not seem to make any difference to the perceived connection with the graphic and discussion. Whether writing single words or mini project proposals, learners presented themselves as being fully participating members in a dynamic group process.

The same could not be said about the role of numeracy in the REFLECT process. The basic writing of numbers was related to calendars (eg. marking months 1-12) and to the household by household map through the counting of different categories of people. This map also allowed the introduction of the multiplication table in a meaningful context. After this stage, however, there was no evidence of numeracy activities relating to agriculture, gender, hungry season etc. Learners were practising the four calculations in the abstract and it was hard to see how they would either remember or apply the skills in a practical context.

In general, especially as regards meaningful reading and writing, there was understanding and satisfaction in the community. Several facilitators made unsolicited comments on the effectiveness of the method, in relation to how it could be used to improve methods currently used in primary schools, showing their own understanding of the principles of the work. The only people who did not have much insight into the REFLECT process were some of the Parish Councillors and RCs. This was balanced, however, by those leaders who were actively involved themselves as learners or facilitators, or through having close family members in these roles.

To determine the learners' interest in different activities in class a table was constructed to see how much time they would like to spend on each in an ideal classroom setting. The aim was to test whether learners saw graphic construction and discussion as either a waste of time compared to acquiring "real skills", or whether they were interested in these activities to the exclusion of literacy and numeracy. The table showed that learners would like to spend more time on graphic construction and discussion, but not as large an increase as desired for literacy and numeracy skills. In general there was a relative level of satisfaction with existing time allocation.

It was noted that no-one in the community had received any exposure to other literacy techniques, and had an enthusiastic tendency to regard REFLECT as the only method, the best method and their own. This certainly reduced their ability to look at the method itself objectively. The evaluation team tried to provide a balance by reading exercise

books looking for relevance, and by asking challenging questions about the maps and calendars. Learners were impressively articulate in explaining the graphics and the contents of their own books.

An examination of similar questions in the Control Group brought very different answers. There seemed to be no dynamic process around discussion of the Picture Charts (large posters with different pictures of everyday life which were used as a supplement to the primer). These were generally described as mere illustrations for the lesson; presenting situations either well known and obvious to the learners or unrealistic pictures, difficult to interpret. As codifications brought to the class for learners to deconstruct, they seemed to fail. There was no strategy for stimulating discussion. It seemed clear that most instructors would be able to spend very little time listening to learners before delivering their own pre-set conclusions on the topic for the day. It seemed possible for only an exceptionally knowledgeable and charismatic instructor to manage the class so that learners both developed analytical skills, and reached the required conclusion for the topic.

One instructor interviewed had used her own creativity by preparing songs and role plays to begin lessons in a different way, avoiding the tedium of always starting in the same way. However, it seemed to place a heavy burden on individual instructors and made the REFLECT method, where the pressure is equally balanced between learners and facilitator look much simpler to practice.

Enrolment, attendance and drop out

The coverage of the pilot, and the success rate in statistical terms, are shown in the table below.

	Total	Male	Female
Illiterates	3, 853	1, 313	2, 540
Enrolled - January, 1994	1, 763 100%	617 35%	1, 146 65%
Not enrolled	2, 090 100%	696 33%	1, 394 67%
Graduated -March, 1995	1, 209 100%	441 36%	768 64%
Retention rate	69%	71%	67%

There were many significant incidents in the area whilst the literacy programme was underway. Almost immediately after the programme started there was a large scale influx of refugees from neighbouring Zaire (pushed over the nearby border following increasingly intense fighting between the Zairean army and guerrillas). Most refugees were Lubwisi speaking and integrated with local communities, sometimes joining the literacy circles. However, a couple of months later the refugees were collected together and moved into refugee camps further inside Uganda. This caused some drop out and loss of momentum.

Meetings with the local Resistance Councils in May helped to revitalise many circles, but then in July came the "hungry season" - the time of year when there is least food or income available and many people have to migrate or work extra hours as day labourers to survive. At this time there was also a serious epidemic of cholera. In some villages the circles were actually closed down for a period -particularly when facilitators were recruited into a government "defence" campaign for 3 weeks.

By October the literacy circles had picked up again and were running well, but heavy rains in late November and December, building up to the Christmas break caused some new reductions. In early January 1995 new meetings were organised with local government all the way from village level through parish and sub-district to district level - in order to re-mobilise the circles. Attendance rose well. At the time of graduation in February/ March each literacy circle had an average of about 25 participants (having started with between 30 and 35).

Whilst all of the above "special" factors influenced attendance the "everyday" factors must also be considered. Finding time for literacy when there are so many other responsibilities is not easy. Other priorities often have to take precedence, whether producing crops, selling and buying goods, producing and caring for children, taking community roles, and coping with emergencies. Non-literates in Bundibugyo (as elsewhere) are usually the least powerful and therefore most heavily burdened members of the community, and an examination of registers revealed that the typical learner was indeed irregular, and attended approximately 100 hours of classes out of a possible 208 hours. The main reasons for irregular attendance as expressed by learners in interviews are classified overleaf:

Barriers to attendance	Scores
Health problems (self and family)	55
Guests, domestic problems, violence	18
Agricultural and domestic work	17
Funerals, ceremonies, community meetings	17
Markets	2
Childbirth	2

It was noted from learners and female facilitators, that a healthy pregnancy and delivery was not considered a barrier to attending literacy classes. Usually pregnancy necessitated only a month's absence. This had previously been considered a "problem" of the woman learner. The unreliability of the facilitator was not considered a barrier by any learner interviewed (in contrast to the control group). Lastly, heavy rain, holiday periods and so on, tended to cause class to be cancelled altogether, rather than individuals not attending.

Baby in one hand - pen in another. A healthy pregnancy and delivery was not an obstacle to attendance.

In the control groups in Mityana the drop out of learners from the centres was a major problem. It proved difficult to maintain learner motivation and only 40% of the learners were still attending the centres after a year (compared to 69% in the REFLECT circles). The causes of drop out were similar in some ways (eg workload) though there was some evidence that learners left the controls groups through frustration with their lack of achievement (which was not a factor in the REFLECT circles).

4.4.3 Progress and problems in El Salvador

Observations on the progress of REFLECT

CIAZO and COMUS had difficulty at first in assimilating the basic principles and methods of Participatory Rural Appraisal. They had no previous experience of PRA and had some fundamental doubts which did not surface within the initial period of developing the draft manual. These uncertainties led to a rather confused and overlong final version of the manual, visual cards that (despite being beautifully drawn and comic) were so detailed as to be un-useable and training for facilitators which failed to communicate some of the basic aspects of the REFLECT approach.

This situation gradually improved as the training progressed and COMUS and CIAZO personnel came to understand the methodology. However, a lot then depended on the training sessions and some of the voluntary facilitators undoubtedly had difficulty grasping the REFLECT methodology. For example, in most classes the graphics were constructed on tables (which COMUS promoters agreed with, arguing that for cultural reasons they could not use the ground) and in some cases they were constructed directly on to paper (rather than using moveable objects). This limited participation and meant whoever held the pen (almost invariably the facilitator) held power. There was even one case where a literacy facilitator said that he prepared maps and matrices in advance of the classes so as to progress more rapidly! This completely undermines the REFLECT approach: the graphics are not an end in themselves - they are a means to an end.

Whoever holds the pen holds the power. Constructing graphics directly onto paper limits participation.

The failure to clearly conceive the role of the graphics meant that in some circles the construction of the graphics was regarded as a separate activity and the reading, writing and numeracy work was not integrated with the graphics. The graphic would be constructed and then put to one side as the other activities were undertaken.

In recognition that there was an uneven take up of the REFLECT approach, in March 1995 it was agreed by all involved to classify the circles according to the extent that they had used the REFLECT method. Those circles where the method had been applied could be used to evaluate the impact of the method. In the circles where the method had not been applied the focus of the evaluation needed to be on why the method had not been used properly.

This classification was done through separate interviews with three COMUS promoters and the CIAZO adviser - each person was asked to score the circles according to the extent of their use of the method, focusing on the process by which graphics were constructed and then used. It was made very clear that this was not be a scoring of the overall success or achievement of the literacy circle (some circles might have been very successful using very traditional methods). The scores given by each person interviewed were very consistent and these were averaged out to give a mark out of 10. The circles with more than 5 (out of 10) can be said to have applied the method more or less. Those with less than 5 cannot really be said to have applied the method effectively.

Enrolment, attendance and drop out

The table below presents basic information about the literacy circles classified in order according to the extent to which they have used the REFLECT approach and showing basic information about the facilitators, enrolment and graduation.

	Level of use of method	Grade of Educ OF FAC.	No. of Train. Days	Date Start		Enrolment			Graduation Apr '95 **	New Circles ***
						Initial	Final
					M	W	T	T
Joya	9	11	11	7/2	20	9	29	17	14	Y2
La Pita	9	6/6	12 (5)	28/3	5	3	8	9	9	Y
Los Rios	8	6/6	11 (9)	1/6	13	6	19	12	11	Y2
El Carmen	7	6	7	14/2	4	8	12	12	12	Y2
Las Conchas	7	6	13	15/2	11	5	16	12	10	Y2
La Pena	6	4/9	5/5	15/2	10	10	20	13	10	Y2
Quesera	6	3	12	14/4	11	1	12	12	9	Y2
SUBTOTAL:							116	87 (75%)	75 (86%)
Loma Pacha	5	6	9	15/2	18	8	26	12	8	N
Galingagua	4	*	*	28/3	21	8	29	15	11	Y2
Horcones	4	6	3	1/10	14	3	17	17	7	N
El Coroz	3	4	6	13/7	11	3	14	8	9	Y
Jobalitos	3	3	10	22/2	15	6	21	6	?	N
Zungano	3	3/9	9/8	11/3	5	6	11	7	4	Y
SUBTOTAL:							118	65 (55%)	30 (46%)
TOTAL:							234	152 (65%)	85 (56%)

NOTES:
? = suspended

* In Galingagua there were 5 different facilitators, all with an education level between 3rd & 6th grade.

** The statistics on graduation will be analysed in more detail in the next section on reading, writing and numeracy.

*** The column on "new circles" means those circles which were reactivated for second level/ post-literacy work in 1995. The number "1" after "Y" means that as well as a circle continuing to second level a new first level circle has started up in the same community. This is considered a useful indicator of the community's own perception of the value of the literacy circle - as it represents "word of mouth" support for the experience within the community.

Analysis

1. There is a clear correlation between the level of education of the facilitator and the effective application of the REFLECT method. Those with less than sixth grade primary education had more difficulties. Those with more than sixth grade performed better. In comparison, the attendance at training sessions made little difference.

2. In literacy circles where the REFLECT method was effectively applied there has been a desire to continue learning in every case - with circles continuing to second grade. Moreover, in 6 of the 7 communities new literacy circles have been opened for other learners. Where the method has not been applied only 50% are continuing to second level and only one in six of the communities has shown a demand for a new first level circle.

3. This table shows an overall rate retention rate of 65% of those initially enrolled. However, it is worth emphasising that in those circles where the method has been effectively applied the retention rate has been significantly better (at 75%).

The reasons for irregular attendance provided by learners are tabulated below (with solutions suggested by the learners themselves).

Problem	Possible solution
52% agricultural work	Do work early
39% household work	Do work early
35% health problems	No comments
24% lack of water	Build a water tank
14% other meetings	No comments
14% distance of the circle	Change location
11 % absence of facilitator	Get a new facilitator
11 % celebrations	No comment
11 % sports events	No comment

Clearly work-load is the most pressing problem and it is thus not surprising that at times of peak agricultural labour some of literacy circles were temporarily suspended. Suspension was unavoidable in those communities were people migrated to work on coffee harvests elsewhere in the country. This was not planned for sufficiently in advance and thus caused some demoralisation. It proved difficult to reactivate circles after they had been suspended in this way (whereas if the circles had each agreed an original schedule/ calendar allowing for peak seasons then the suspension would not have been demoralising).

The learners who permanently dropped out of circles gave the following reasons:

22%	household/ agricultural work
22%	eyesight
17%	suspension of circle
11%	the facilitator
11%	shame
5%	the party political matrix
12%	no response

This is revealing. Workload, whilst an obstacle for many people, was not usually the main reason for drop out. Poor eyesight appears very high which reminds us of the importance of addressing eyesight problems in tandem with literacy programmes.

Problems with the facilitator or the suspension of the circle appear significantly (in El Coroz, Cerro Verde, Galingagua and El Zungano the facilitators left for various reasons).

It is interesting that one learner mentioned the matrix on political parties as a cause for drop out. This matrix was poorly designed and could easily be used by facilitators for party political lectures (in direct contradiction with the intention of the REFLECT method). The use of it so early in the literacy programme (the elections were in March -just a month after the circles started) probably alienated many people.

During the process of the literacy programme another factor which was widely commented on was the failure of COMUS to live up to its promises of giving credit to the literacy circles. Two months into the programme the COMUS leadership agreed to make credit (of 5, 000 colones) available to literacy centres for group productive projects (designed in part to help reinforce literacy skills) at a reduced interest rate. Each literacy circle discussed what they could do and prepared a credit request to COMUS. However, COMUS failed to fulfil its promise for more than six months which led to widespread cynicism, at times undermining the facilitator's credibility and weakening the potential for linking the literacy circles to the wider integrated programme of COMUS.

Additional problems

Other problems can be broadly divided between those relating to the Volunteer Facilitators and those of a more general nature.

The Volunteer facilitators had problems in carrying out their role, perhaps mostly because of a lack of time (mentioned by 67% of them) or a lack of support (mentioned by 27%). The lack of support from COMUS or CIAZO was particularly mentioned by facilitators in remote communities which are difficult to reach. The COMUS promoters and CIAZO adviser had just one motorcycle between them to reach the communities and otherwise depended on walking on hitching lifts with other vehicles.

In relation to workload, the fact that the facilitators come from the same community as the learners means that they have the same workload as the learners outside the literacy circle. With training, preparation for classes and the need to be present everyday (whereas learners could miss a day or two more easily when under pressure of work), the time involved was significant and often meant a loss of income.

Under such economic pressures it was difficult to sustain voluntary motivation. Pressures mounted when local health promoters who were previously volunteers started to receive pay from PROSAMI (a USAID funded programme within the national reconstruction process). COMUS's own "zonal promoters" (their first point of contact with each community) had also been paid in the past (though this was ended just before the literacy programme started). Comparing their position with others led the facilitators to desire a greater recognition of their commitment. Some facilitators left, demoralised - and when new ones were recruited to replace them it was not possible to offer intensive initial training so the new facilitator's struggled to pick up the methodology.

Within a few months each literacy circle progressed at a different rate, which is only natural, but this irregular progression was exaggerated where circles were temporarily suspended. This led to difficulties in the follow up training workshops which could not focus on practice of different Units which would be of immediate relevance to all facilitators.

The facilitators' own educational level proved to be a very important factor in the pilot experience. Those who had only third grade primary education had serious difficulties in reading the manual and problems taking notes in the training workshops. This made it difficult for them to replicate the methodology in practice. Moreover, their own problems in writing caused problems in their circles for the learners, for example when they wrote on the blackboard mixing small and capital letters. Those facilitators who had these problems also failed to maintain their notebooks.

As already mentioned the manual and visual cards proved to be a source of many problems. The manual was unnecessarily long (over 150 pages) and dense (with no illustrations) with too many technical or academic words. The guidelines were not clear in many cases. The sections on "Information" which were added to the manual were also much to long and tended to give the impression that this was the "truth" or the "answer" or the "real knowledge" given by the "experts". The visual cards were very detailed pictures in many cases (almost codifications themselves) and could not be copied by the facilitators. The fact that there are so many of them and they are not numbered or referenced made them effectively un-useable by facilitators who thus had to improvise their own drawings (which many were embarrassed to do). Some facilitators hardly used visual cards and this probably limited participation of learners in the construction of graphics (there was less large scale visualisation).

General/ Organisational Problems

Just a month after the launch of the literacy circles national (and local) elections were held in El Salvador (in March 1994) - the first democratic elections following the peace accords. This was a critical moment in Salvadorean history with the FMLN guerrillas who had been very strong in Usulutan, participating as a political party for the first time. There were political meetings and mobilisations in almost every community and this interrupted the process of the literacy circles almost before they had begun. It was difficult to focus on literacy and local developments when everyone was talking about party politics and national change. COMUS staff were making a big effort to be neutral although most personally felt very strongly pro-FMLN.

With the victory of ARENA in the national elections and of the Christian Democrats locally, there were further problems. Some COMUS staff could not believe the result and were very demoralised. Some even felt that they should have campaigned for the FMLN openly - and started blaming people for not voting for the "frente". Locally there were accusations of vote-rigging, with dead people appearing on the electoral register and many live people not being given polling cards. ARENA and

PDC provided transport for their voters to the polls and reputedly intimidated people or bought their votes. The lack of awareness of the electoral process meant many people did not realise that their vote was truly confidential.

At the community level, COMUS tended to have links with the more active or politicised people -and this was reflected in the literacy circles and in the literacy facilitators. Many thus felt disillusioned and demoralised after the elections - wondering why it was that the FMLN had laid down arms if there was no access to political power through the ballot box. The demoralisation was widespread amongst other COMUS staff and this weakened the capacity to link the literacy work to other sectoral initiatives of COMUS for many months.

The elections, however, were only the most recent in a series of experiences which acted as blocks to "participation" for the campesinos of Usulutan. Contrary to the international image of Latin America being a place where "participation" and active mobilisation is easy to promote, the historical and social context of rural El Salvador presents many obstacles to real participation. The first block is the deeply ingrained traditional relationship between the campesinos and their patrons (the large plantation owners/ landowners) - which was always very hierarchical, leaving only a passive role for the campesino.

The civil war tried to break these relations but the vertical military command structure of the guerrillas did not give a positive alternative experience for most campesinos. Then, most recently with the arrival of large scale international aid for reconstruction the majority of projects have been based on direct assistance, service delivery or donations - reinforcing the traditional passive and "dependent" role of campesinos. This "asistencialismo" builds upon the expectations of campesinos who have had few opportunities for (or experience of) active participation. In this context it is in fact extremely hard to promote community participation and initiative.

Related to the above is the reaction encountered in some communities who have had a lot of exposure to aid/ development projects. There is a weariness with "process" and a desire for "solutions". As commented by one person: "we already know our problems and don't want to discuss them again - what we want is real solutions, which require money!"

There were many obstacles to community mobilisation - not just the difficulties in promoting initial participation. Perhaps most fundamental was the fact that for the first months of the literacy programme very few communities had any legal titles to their land (and only some of them acquired titles during the process). As land reform was delayed (particularly after the elections) learners had little interest in mobilising for community actions relating to agriculture (the dominant themes of the early Units) because most involved making medium or long term investments in land over which they had no security (eg terracing for soil conservation, establishing nurseries, diversifying crops, experimenting with new methods). The "climate" was not appropriate for long term planning, risk taking or new initiatives - though in communities were land was then "verified" there was indeed that incentive.

In cases where there was an incentive to act, the literacy circles often found themselves faced with a problem of lacking the "authority" to act. The learners were only a section of their community or cooperative - and most communities had some nominal (or real) democratic authority (a "junta directiva"/ local council/ cooperative committee etc) which was the body responsible for deciding on local actions. The status of the literacy circle was not always clear (particularly when it was small) and relations between the circle and the community committees were sometimes important for achieving local action.

This background to "participation" and "action" is particularly important given the significant impact which the literacy programme did indeed achieve in these respects (see section 4.6).

4.4.4 Cross-case study analysis

There were clearly differences in the development of the REFLECT methodology in each of the pilot programmes. Since REFLECT was not a "prepackaged method" and since it was applied in such different settings this is unsurprising. It is, in effect, through the creative adaptations of the basic ideas in these pilots that what is now known as REFLECT (and which is consolidated in the "Mother Manual") has evolved.

In Bangladesh and Uganda the methodology was adapted to the local situation with considerable success and a high degree of understanding of the basic principles. However, it is sometimes from the mistakes that one learns the most and aspects of the El Salvador pilot proved particularly revealing. In some of the literacy circles in Usulutan it became clear how the REFLECT methodology could be distorted with the graphics being regarded as ends in themselves rather than as a means to an end -and with the literacy and numeracy work not being closely interwoven with the graphics. It is important to note that a similar problem arose to some extent in Uganda with the numeracy work which was not well designed in the original manual and which was therefore separated off and became more abstract than would be desirable. This was not a problem in Bangladesh where the numeracy work was a central focus of the pilot.

In contrast to the above, some of the adaptations and innovations of the pilot programmes do not undermine the basic principles of the REFLECT approach. For example, it is not inherently contradictory for the maps and matrices to be constructed on tables (as was done in most circles in El Salvador) instead of on the ground (as was the norm in Uganda and Bangladesh), so long as movable objects are used. Likewise, the materials on which the graphics are drawn (whether large sheets of manila paper as in Bangladesh or harder cardboard as in Uganda) and even the visual quality of the resulting maps and matrices (meticulous and colourful detail in Bangladesh compared to simpler graphics in Uganda and a broad range of styles in El Salvador), though significant differences to an outside eye, are not in themselves key to the methodology. It is the process of constructing these materials and how they are then used which are the essential things to consider.

It is important to note the different rate of progress in the three pilots. In Uganda and El Salvador, most circles had covered just ten Units (though some had done 15). In Bangladesh, which started later (though circles met 6 days a week compared to 2/3 days a week in Uganda and El Salvador) most circles covered 20 Units. This is not in itself a sign of greater achievement. Programme managers in all three pilots were keen to emphasise that no pressure was put on circles to "get through" the manual quickly. The key is the process. A single Unit could take two days, two weeks or a month depending on the intensity of discussion that the graphic produced and the amount of literacy and numeracy work that flowed from it.

The observations on the problems encountered within the pilot literacy programmes are a very important introduction to the results of the evaluations. They help the reader to see that these were not "specially preserved" or "uniquely attended" pilots or models. Each programme was operating in a very specific moment in time with very real problems. Every literacy programme is unique and so we cannot generalise too much from any one programme. But when we compare the three programmes some wider conclusions can be drawn.

4.5 Reading, writing and numeracy

4.5.1 Bangladesh
4.5.2 Uganda
4.5.3 El Salvador
4.5.4 Analysis

4.5.1 Bangladesh

The results of the External Evaluation established that the REFLECT approach was much more successful at teaching literacy and numeracy than the methods used in the control group (using FIVDB materials). REFLECT groups scored 43% better in reading, 79% better in writing, 64% better in numeracy and 150% better in visual literacy.

It was further noted that "in the control groups performance was better in the upper age group while in the intervention [REFLECT] groups performance was better in the lower age group". The 15-19 year olds in the REFLECT circles did exceptionally well - with 62% getting a Grade A for reading, 75% getting Grade A for writing, 100% getting Grade A for numeracy.

The numbers who achieved literacy and numeracy in the REFLECT circles are tabulated below:

Analysing this we can observe the following:

78% of those who initially enrolled are still attending the literacy circle (though some are irregular).

77.5% of those who have stayed on have learnt basic literacy.

60.3% of the total of those enrolled have achieved literacy.

70% of those enrolled achieved literacy if we exclude those women who were forced to permanently drop out for reasons beyond their control (permanent migration owing to marriage or search for work and violent opposition of husbands).

The Bengali alphabet is much more complex than the Roman alphabet having many more simple letters and complex compounds. Nevertheless REFLECT participants learnt rapidly

In the control groups only 47% of those who were tested passed, leaving an overall success rate (given 45% drop out) of just 26%. The REFLECT circles, with an overall success rate of 60%, were more than twice as successful.

Literate habits

The women were asked what they have actually read in the past six months. In the first two communities we asked whether they had read a lot or a little of each but this proved too time consuming and so we restricted the questions to yes or no. Out of 59 women interviewed the results were as follows:

42=71% School texts
36=61% Pass book*
33 = 56% Health leaflets
31 = 53% Medicines/prescrips.
30 = 51% Cinema Posters
29 = 49% Directions eg buses
29 = 49% Letters

Shomiti	CU	CU	JI	JI	JI	JI	JI	JI	JI	CU
number	109	115	193	195	203	209	207	127	201	125	Total
Initial enrolment	15	16	14	15	15	15	12	15	21	16	154
Permanent drop out	3	4	0	6	4	4	1	7	2	3	34
Achieved literacy	13	12	10	9	7	7	9	5	9	12	93

24 = 41 % Signs (eg shops)
23 = 39% Fertiliser instr.
19=32% Agricultural/ IG leaflets
17=29% Legal documents
16 = 27% Religious texts
12 = 20% Newspapers
9 = 15% Minutes of meetings

* a higher percentage was hoped for but it should be noted that the existing Pass Book is badly designed, including many compound letters and small text. If it is re-designed and made easier then this percentage will probably rise substantially.

The women were also asked what they have actually written in the past six months. Out of 59 women interviewed the results were as follows:

48=81% Signature
18 = 30% Personal letters
15 = 25% Pass books
12=20% Shopping lists
3 = 5% Other letters
3 = 5% Minutes of meetings

Finally the women were asked what they have actually used written numbers for in the past six months. Out of 59 women interviewed the results were as follows:

15 =25% Pass book
11=18% Household accounts
8 = 14% For buying/selling
3 = 5% Other accounts
1 = 2% Measurements

One of the most positive signs for the medium term is the near universal demand for books which the literacy programme has generated. If this can be responded to successfully with books appropriate for newly literates and if a rickshaw library can be established as planned then the literacy programme will have contributed significantly to the creation of a more literate environment which will benefit the wider community. One additional possibility is the subsidising of a local newspaper set up as a business concern.

4.5.2 Uganda

The same basic literacy and numeracy tests, based on a standard design used elsewhere in Uganda, were conducted with learners in both REFLECT groups and control groups. All learners in the sample of 24 REFLECT circles passed the test except for one learner who was regarded as still being pre-literate. The test papers were independently reviewed to verify this. In the control group only 55% of learners passed. The average score of REFLECT learners was 55%, with half the learners scoring over 50%. This compares with an average score of 36% in the Control Groups.

The average REFLECT learner after one year (with a typical attendance of one hundred hours) can: a) read a paragraph aloud and understand it. She finds difficulty, however, with silent reading. b) write a letter on a familiar topic eg, letter to ACTIONAID, asking for a loan, of about one paragraph. She writes clearly and with confidence. c) copy and calculate using the four signs. Four figure numbers are handled well in addition and subtraction, and two figures in multiplication and division.

Bangladesh: Rickshaw with video, TV and generator -a similar system will be used for books.

Uganda: REFLECT circle with graphics on permanent display on the walls.

The average woman obtained a higher score than the average man, though there was a wider spread of ability (women's scores in the REFLECT circles ranged from 10% - 86%, and men's scores from 47% - 88%).

Test results were broken down into skill components:-

i) Reading

This was the weakest area and women performed relatively badly particularly on reading comprehension requiring silent reading of unknown material. Learners have almost no experience of this activity, and many did not attempt the task. However, it was also the last question in the test and there may have been time and tiredness factors at work.

ii) Writing

This was a strong area, and most learners attempted some independent writing, and were not worried about spelling errors etc. They were confident about setting out their answers on a separate sheet of paper and wrote quickly and clearly. This was impressive as they had never done a test before. Only one learner seemed to be in the pre-literate stage.

iii) Numeracy

Learners scored high in this area, and women did better than men. All the questions were based on conventional maths, and even if not answered accurately were set out correctly as a record. Women were more methodical than men in their setting out of the calculations. Their scores were a surprise considering that they had expressed a desire for more time on numeracy and the numeracy work had been criticised for being delinked from the graphics.

In the learners' self evaluation of progress (PRA activity), 84% were satisfied with their overall progress in reading, writing, numeracy and discussion with a slightly lower figure of 81% satisfied with their progress in numeracy.

Examination of learners' notebooks, the test results and talking to learners and facilitators, led to the conclusion that REFLECT enabled learners to achieve their maximum potential, and there were no constraints such as experienced by learners in the Control Group. The learners are encouraged to experiment with independent writing, and do so.

The main weakness of the methodology is that the course is cumulative, using phonetic languages to build up a repertoire of syllables, but there are no strategies for remedial action when adult learners miss a number of classes. Facilitators have no idea what to do except to give up their time free of charge to irregular (or slow) learners. There is obviously no reason why they should do this as an integral part of their work.

Control groups

The most noticeable characteristic of Control Groups was a lack of self confidence. REFLECT learners took the papers, were happy to work on their own (indeed were anxious that other learners should not copy them!) and were certain that they would do well. In the Control Group, learners had to be persuaded to try, and were always asking for help from the instructor and each other.

"We cannot take the test. We are still in Primary 1" was a typical comment.

This lack of confidence clearly led to the test being done poorly. It showed (with the exception of some individuals with very high scores) that the learners had indeed achieved only the very limited goals set out in the methodology for Basic Literacy. They could have achieved much more in the same period using REFLECT.

Reading of exercise books in Control Groups showed that learners spent most of the time copying, and did not understand what they had written. Reading with comprehension, as opposed to merely reading syllables aloud, was rare in the sample classes. This was compounded by the problem of learners who were not fluent in Luganda which was the only language offered.

Numeracy exercises had in the main no connection with the Picture Chart or the topic, and this seemed to be due to the restriction to single figure calculations. This was recognised by facilitators to be well below learners' capacity for mental arithmetic (used to reading and calculating very large figures in Ugandan shillings), but the only practical numeracy supplied was the explanation of relevant weights and measures eg. weighing coffee on a kilogram scale.

At the end of the course, at best learners could write key words, (often only the key "generative words" which the primer had given them). They were not in a meaningful context, and it appeared

that the adult learners were being taught in the same way as a child for whom language and context are both new worlds to explore. The result was low achievement (with just 55% passing and an average score of just 36% in the tests) and little prospect with the Primer methodology of creating a dynamic process linking literacy and action (see section 4.6).

Literate habits & language empowerment

Having chosen to teach in a language that was not previously written there were (at first) apparently no materials for people to use to develop literate habits. A survey of people's habits showed that they were creatively using their skills to read some things (including Bibles, labels, signs, directions, instructions etc) often in other languages (including English). However, it is clear that their main reading activities were in the literacy class itself. This has left learners with very little conception of their future choices of reading materials and presents a serious challenge.

One learner expressed her lack of experience in reading using the following Lubwisi proverb: "If a child always eats at her own home, she always thinks her mother is the best cook."

If the literacy skills are to be consolidated into literate habits in Bundibugyo, much depends on the creation of a literate environment. There are some positive signs. There are initiatives from the communities themselves: notices of meetings, funerals and other events are being put up in the villages on improvised notice boards; an increase in letter-writing (both personal and for "official purposes"); the preparation of local project proposals and the writing down of oral histories. There is also a "demand". Many learners commented in their interviews that, following the REFLECT circles they wanted to read more, in order to feel "enlightened" about the world around, and also to gain access to practical information -even if they were usually unaware of the range of things which they could read.

From "above" as it were, ACTIONAID is responding to this by printing some of the local stories written by learners (and other local materials) in a newsletter - as well as translating texts on health, agriculture, politics (eg the new constitution) and law into Lubwisi. There are plans for simple low cost printing (eg silk-screen) facilities to be made available to each literacy class.

Maintaining a momentum and the demand for literacy is critical. This has been helped by an apparent change of people's attitude towards literacy. As one put it, the local definition of literacy is now "that people are naturally clever but they also need to learn reading and writing". The pressure on numbers for the new set of literacy classes is ample evidence that those previously uninterested in literacy are changing. Interviews with Parish Councillors and RC1s revealed that over 50% had a real understanding of the REFLECT process happening in the classes. This made their commitment to support the existing classes, as well as the new ones starting, seem more realistic and genuine.

Most significantly however is the shifting status of the main language used in the literacy programme, Lubwisi. Whereas two years ago it was unwritten and many local people commented that they felt "ashamed" to speak it in the markets (or on the few occasions they travelled outside the area), now there is a sense of pride in the language. Not only this, the language itself is gathering a momentum, with the local bank in Bundibugyo town now printing forms and cheque books in the language, government agencies starting to recognise the language and even the Ministry of Education now accepting its use in the first three years of primary school. This is a spectacular turnaround in a short space of time. The process is helped by the present political climate in Uganda which emphasises decentralisation. It would not be so easy in other contexts. Nevertheless the extent of change in how Lubwisi is identified and used can only increase the prospects of local people creating their own literate environment in their own language, consolidating their literacy skills in the process.

The same level of "language empowerment" has not yet taken place with Lukonjo (as there were only a handful of Lukonjo circles) but once the literacy programme extends to more Lukonjo speaking villages it will be interesting to observe the changes. There is certainly a danger that Lubwisi may end up being elevated above other local languages (like Lukonjo) and dialects (like Lwamba) thus disempowering or undermining other local groups. If the original pilot programme had included more Lukonjo speaking villages this danger would have been reduced. The circles which have learnt in Lukonjo have shown that the REFLECT methodology is able to adapt to such a challenge. However the need for follow up resources in printing and publishing in different

languages also needs to be considered. The cost of this would be reduced if the emphasis is placed on creating a literate environment from below through low cost village level printing.

4.5.3 El Salvador

As emphasised in section 4, 4.3 the REFLECT approach had been applied in varying degrees across the different literacy circles in El Salvador and in some cases it had really not been implemented sufficiently to serve as the basis for evaluating the impact of the method. The evaluation therefore focussed on the eight literacy circles where the REFLECT methodology had been applied. The table in section 4.4.3 provides details of these circles. This is a small sample and may raise some concerns about the wider validity of the results. However, the detail of the results is sufficiently substantial to counterbalance some of the concerns about the size of the sample.

Analysing this table we can observe that in the literacy circles where the REFLECT method has been applied there has been considerably more success. 75% of those who enrolled are still in the literacy circles and 86% of them have graduated using a standard literacy and numeracy test. The other literacy circles performed less well (only 55% continued to attend to the end and only 46% passed the test).

Different control groups were used to ensure a good range of comparison. Firstly the municipal programme of CIAZO, regarded as their most successful programme at present, which benefits from considerable logistical support from UNICEF and local government. A sample of five communities was taken from this programme (selected as representative) and these results revealed that 49% of those enrolled completed the course (compared to 75% in the REFLECT circles in COMUS). Of those who completed, 90% graduated (which compares to 86% of those using the similar test in COMUS REFLECT circles). In overall terms this means that 43% of those who enrolled successfully completed the course in CIAZO's municipal programme (compared to 65% in the COMUS REFLECT circles). It is worth noting that the facilitators had an average level of 11 grades education in this Control Group (almost twice the average education level of the COMUS facilitators).

A more representative comparison in relation to the literacy and numeracy tests was taken from the second control group - ClAZO's work with other grassroots organisations. A set of 74 completed tests were selected at random and reviewed to provide comparative results. The results were broken down in detail according to the different tests involved. The results are provided below both for the control group and for the COMUS REFLECT circles:

	Control Total*	Group %	COMUS %	Difference (+ or-)
Dictated words	55	74%	98%	COMUS + 24%
Sentence construction	65	88%	93%	COMUS + 5%
Creative writing	28	38%	84%	COMUS + 44%
Additions	60	81%	86%	COMUS + 5%
Subtraction	62	84%	64%	COMUS - 20%
Problem solving	45	61%	66%	COMUS + 5%
Overall passes	61	82%	86%	COMUS + 4%

* 74 people of whom 39 were women (of these 33 graduated = 85%). There were also no notable gender differences in COMUS circles.

The results here are slightly poorer than for the municipal programme of CIAZO. What is missing here is detail of the percentage of learners who dropped out during the course before coming to graduation, Reports from CIAZO advisers indicate that a drop out of at least 50% is common (which is consistent with the municipal programme). The greatest difference with the COMUS REFLECT programme appears to be a greater ability to retain learners in the circle. Having retained more learners the success rates of those sitting the final test are also (marginally) higher than in CIAZO's normal programme.

It is interesting to observe the details of the above results. In general the COMUS REFLECT circles performed much better in the detail and passed with better results. The COMUS REFLECT circles performed much better in writing - particularly creative writing. There were more difficulties in mathematics. However, the results in numeracy were distorted by a serious failure in one circle, Galingagua. If the results of this circle are separated out then the numeracy scores improve substantially.

Self-evaluation of learning

Objective tests of literacy skills tell only half the story. Learner's own perceptions of their progress, their subjective view, is equally important. Understanding how learners themselves see their progress in relation to their own initial expectations offers some insight into their motivation to continue learning (and whether they would recommend the experience to others). A matrix was constructed in each of 7 REFLECT circles, with 34 learners, where each learner had to indicate their level of satisfaction with different activities in relation to their initial expectations: reading, writing, mathematics and discussion. A similar matrix was constructed with just 7 learners in a control group. The results showed that:

91% of REFLECT learners expressed satisfaction with their reading compared to 42% of Control group learners.

94% of REFLECT learners expressed satisfaction with their writing compared to 71% of Control group learners.

82% of REFLECT learners expressed satisfaction with their numeracy skills compared to 57% of Control group learners.

Literate and numerate habits

In structured interviews with 37 REFLECT learners and 18 control group learners the following emerged:

What things have you actually read (in the past months)?

COMUS	CONTROL
30% newspapers/ newsletters	28%
28% health leaflets/ materials	22%
22% children's school texts	11%
19% posters	22%
17% agriculture leaflets/ materials	16%
16% documents about cooperative	0%
14% The Bibles	22%
11 % Stories/ comics	5%
8% letters	5%

REFLECT participant sharpens his pencil with a machete.

What things have you actually written (in the past months)

COMUS	CONTROL
24% signatures	22%
16% shopping lists	11%
11 % personal letters	11%
11 % messages	0%

On what occasions have you used written numbers?

COMUS	CONTROL
43% personal accounts/ other accounts	22%
27% buying/selling of products	27%
11% measurements	0%
8% credit requests/ records	0%
6% calendar	0%
3% telephone numbers	0%

In general the REFLECT learners have used their skills slightly more than those in the control groups though there is not a very substantial difference. The clearest differences are in relation to school texts, cooperative documents and the keeping of accounts which REFLECT learners mentioned much more.

In general however the results might be considered disappointing and this could be related to the fact that COMUS had made no systematic attempt to promote or create a more literate environment. There has been no attempt to establish village libraries, to promote the distribution of written materials, to create a newsletter or to liaise with other sectoral staff to ensure a coherent strategy. This is a serious missed opportunity. No literacy programme can be fully effective unless clear attempts are made to back it up with strategies to create a more literate environment. This is not therefore a reflection on the REFLECT methodology but on the way in which it has been implemented in this pilot.

4.5.4 Analysis

Abadzi (1994) in her review of adult literacy programmes over the last thirty years estimated an effectiveness rate of just 12.5%. This figure is arrived at in a rather formulaic way but does resonate with experience for many people working in literacy:

· on average 50% of those who enrol in programmes drop out within a few weeks.
· 50% of those who remain in the literacy programme fail to complete the course or fail to graduate.
· 50% of those who graduate lose their skills through disuse within a year or so.
If these figures are taken as an overall benchmark and we take the Control groups in each country as local or national benchmarks then we can observe that the REFLECT programmes in all three pilots have performed well with a higher percentage staying on in the literacy circles and a higher percentage achieving literacy.

	Retained	Achieved Literacy	% overall
Salvador REFLECT	75%	86%	65%
Salvador Contro	49%	90%	43%
Bangladesh REFLECT 78%	77.5%	60%
Bangladesh Control	55%	47%	26%
Uganda REFLECT	69%	99%	68%
Uganda Control	40%	55%	22%
World Bank	50%	50%	25%

Given the time frame of the evaluations it is too early to be measure the retention of skills after the completion of the REFLECT literacy programme. However, there are positive signs:

· REFLECT learners are generally motivated to continue learning
· most REFLECT learners are starting to use the skills in their daily lives

In general terms in all three pilot programmes the REFLECT learners performed very well in tests of writing, particularly creative writing. This is perhaps

unsurprising given the focus on literacy circles producing their own materials (rather than using pre-printed texts or simply copying out words from a primer).

Reading skills were also better than in the control groups (43% better in Bangladesh) though not so dramatically. This may be an area of concern - reflected perhaps by a rather "purist" approach in the pilots. Having rejected primers, personnel in the REFLECT pilots were hesitant to introduce other pre-printed materials and the emphasis was placed very much on learners producing everything for themselves. A balance needs to be sought in which learner generated materials forms the backbone of the course but supplementary reading materials are also supplied (particularly for home reading). Numeracy skills were the weakest area in Uganda and El Salvador though in Bangladesh the REFLECT learners significantly out-performed the control group. This reflects the fact that numeracy was scarcely addressed in the manuals for the first two pilots (Uganda and El Salvador) but became a dominant focus in Bangladesh in the context of the savings and credit programme.

Many of the ideas concerning the use of the REFLECT approach for numeracy were only developed in Bangladesh so hopefully the positive results there are a sign of the future potential of the approach.