(AKU-EB) Evaluation of the Aga Khan University Examination Board
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Evaluation of the Aga Khan
University Examination Board
(AKU-EB)
TASK ORDER NO. REE-I-00-05-00053-00, TASK
ORDER #8 WITH
ACADEMY FOR EDUCATIONAL DEVELOPMENT
April - May 2008
This program evaluation is made possible by the support of the American People through the
United States Agency for International Development (USAID.) The contents of this evaluation are
the sole responsibility of the Academy for Educational Development and its authors, and do not
necessarily reflect the views of USAID or the United States Government.Evaluation of the Aga Khan
University Examination Board
(AKU-EB)
Task Order No. REE-I-00-05-00053-00, Task Order
# 8 with the
Academy for Educational Development
David Carroll
Academy for Educational Development
Washington, DC
April - May 2008
This program evaluation is made possible by the support of the American People through the
United States Agency for International Development (USAID.) The contents of this evaluation
are the sole responsibility of Academy for Educational Development and its authors, and do not
necessarily reflect the views of USAID or the United States Government.TABLE OF CONTENTS ACKNOWLEDGMENTS 1 EXECUTIVE SUMMARY 5 LIST OF ACRONYMS 6 BACKGROUND STATEMENT 8 EVALUATION METHODOLOGY 12 OVERALL PROGRAM EFFECTIVENESS 15 PROGRAM ACHIEVEMENTS/STRENGTHS 19 PROGRAM CHALLENGES AND OPPORTUNITIES 34 COST-EFFECTIVENESS OF THE PROGRAM 49 AKU-EB SUPPORT FOR GOVERNMENT POLICY 51 RECOMMENDATIONS 57 CONCLUSION 63 APPENDIX A – POWERPOINT PRESENTATION 65 APPENDIX B – LIST OF CONTACTS 66 APPENDIX C: LIST OF SITES VISITED AND SCHOOLS INTERVIEWED BY TELEPHONE 71 APPENDIX D – INTERVIEW/FOCUS GROUP PROTOCOLS 73 APPENDIX E – LIST OF DOCUMENTATION REVIEWED 77 APPENDIX F – CASE STUDIES OF SCHOOLS 82 APPENDIX G – INTERNATIONAL TRAVEL AND CONSULTANT INPUTS, 2003-07 90 APPENDIX H: DATA ABOUT TRAINING ACTIVITIES 91 APPENDIX I – BRIEF NOTES OF FEEDBACK FROM RAWALPINDI SCHOOLS 94 APPENDIX J – PHOTO: CONDUCTING EXAMINATIONS 95
ACKNOWLEDGMENTS
The authors would like to thank the following people for their assistance during this
evaluation:
• The Director and staff of the Aga Khan University Examination Board, who gave so
generously of their time, both while we were in Karachi and afterwards, to answer our
innumerable queries, and provide us much valuable information.
• The teachers, head teachers and others in the AKU-EB affiliated schools, who were
so willing to spend time with us, and to talk in depth about matters which went to the
heart of their professional lives. Without their openness, patience and courtesy, this
report would be much less informative.
• Those in all levels of federal and provincial departments, and in particular the officers
of the various Boards of Intermediate and Secondary Education that we visited, who
gave us of their time and insight, and much valuable information.
• USAID, specifically Mr. Fazal Rabbi and Dr. Randy Hatfield, for their guidance,
support and valuable comments. Also their generous quantities of briefing material.
• AED, both in Washington DC and on the ground in Islamabad (in the person of Mr.
Zia Asad Khan), for their ongoing overall support, as well as direct assistance with
editing and logistics.
For a complete list of those the team met with, and whose contribution we should like to
acknowledge, see appendix B.
AKU-EB Evaluation 1 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentKey Evaluation Questions – Summary of Findings
Question Key Relevant Findings
1) Did the program complete a) The AKU-EB has been established with 37 full-time staff, mainly
the activities in the trained in-house
cooperative agreement and b) At least 1,147 educators have participated in writing syllabuses,
in annual workplans? questions, assessment units, etc.
c) A state-of-the-art computer-based examination processing system
has been developed and implemented
d) In 2008, AKU-EB offered HSSC-I (grade XI) in 24 subjects ; SSC
Composite in 26 subjects; part-wise SSC-I in 25 subjects.
e) In 2008, 5219 students from 83 schools took AKU-EB
examinations.
f) 161 schools have affiliated; 172 principals and 1812 teachers were
oriented.
g) 912 teachers (633 female) trained in item writing; a teachers
trained in item writing; 300 teachers trained in delivering science
practicals; 837 teachers (591 female) participate in writing 98
assessment units
h) 68,550 copies of assessment units were distributed (some free,
some at charge)
2) Did the program have an a) AKU-EB Examinations require teachers to plan lessons, and use
impact on the teaching new questioning techniques and diverse learning materials.
methods of teachers trained b) Teachers have been trained and given access to learning materials.
by the AKU-EB? c) Teachers report that the training was effective in establishing the
required changes in approach and outlook.
d) Teachers report that the learning materials are innovative and of
good quality.
e) Teachers who work in schools where such methods are practiced
in lower grades welcome the change in approach.
f) Teachers who know only the public sector BISE approach
struggled to cope.
g) Practical examinations require schools to actually carry out
practical assessments.
h) This has resulted in major changes in teaching of practical skills in
science.
3) Did the program have an a) AKU-EB exams require a much higher level and quantity of
impact on learning understanding and application than public sector BISE exams.
outcomes and teaching b) In this, they are more faithful to the national curriculum
methods? c) But teachers have to teach and students to learn in new ways.
d) Those who are accustomed to the approach from lower grades
find the AKU-EB exams liberating; they consider the support
provided by AKU-EB sufficient.
e) Those who are not find it very demanding, and generally feel that
they need more support, and the two-year SSC program is too
short to make such a big change.
AKU-EB Evaluation 2 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentQuestion Key Relevant Findings
4) Did the program contribute a) AKU-EB is an innovative example of PPP, intended to stimulate
to the Government of change in public schools and BISEs; but public schools were
Pakistan’s (GoP) education denied affiliation by government; and public BISEs resisted the
reform measures? model. Benefits therefore went mainly to the private sector.
b) The AKU-EB program directly supported GoP reform strategy in:
i. curriculum, by setting out specific objectives, giving guidance
on how to teach and assess them, and disseminating the
information;
ii. textbook reform, by making available lists of recommended
materials and developing assessment units;
iii. assessment reform, by defining a full range of assessment
objectives and assessing them validly.
5) Did the AKU-EB model a) AKU-EB is a model of good practice consistent with the ESR
influence other local Action Plan 2001-05 in:
examination boards? i. research and development;
ii. paper setting;
iii. computerization of exam processes; and
iv. promoting fairness and transparency.
b) However, its influence on the public BISEs has so far been limited
by resistance from public BISEs.
c) AKU-EB has therefore been more a stimulus to change than a
model the BISEs followed.
Summary of Key Recommendations
AKU-EB to:
develop as a matter of urgency a new marketing strategy in the light of experience to
date, to focus on growing its core business as rapidly and sustainably as possible;
support this by developing a more extensive development and support program for
middle-class schools, perhaps in collaboration with the IED;
strengthen in-house and overseas professional development activity with a view to
succession planning;
consider a change to a name which would be perceived more neutrally;
review the Subject Officers' workload, to ensure that they have enough time for their
examination-related duties;
hold workshops to disseminate good practice in the use of feedback reports to
schools;
set up at least one branch office, in Lahore;
establish a clearinghouse for learning materials developed by AKU-EB schools;
identify schools which training programs and Professional Development Centers, and
AKU-EB Evaluation 3 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational Developmentlocal resource persons, to implement AKU-EB training locally;
upload a much wider range of material to the AKU-EB web site, including model
question papers, assessment units and training packages;
establish one or more internet chatrooms for teachers in AKU-EB affiliated schools.
IBCC to:
establish a working group to evaluate the AKU-EB innovations, and propose ways to
achieve similar quality improvements within the public sector BISEs;
promote the development of national or provincial examination syllabuses on the
AKU-EB model;
promote improved question paper quality and pos-examination analysis, on the AKU-
EB model;
promote a more rigorous approach to practical testing, similar to that of the AKU-EB;
develop approaches to standard setting and reporting which would be more
curriculum- or objectives-referenced, and stable across BISEs, years and subjects.
USAID to:
use its leverage to push for the computer-based examination management system
to be made available as widely as possible at a price which will allow developing
countries to purchase the system;
affiliating with AKU-EB, and candidates in low-fee schools presenting for AKU-
EB examinations.
AKU-EB Evaluation 4 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentEXECUTIVE SUMMARY
The program requested by USAID to be evaluated, the Aga Khan University
Examination Board (AKU-EB), is the first private-sector examination board to be
established in Pakistan. The purpose of the evaluation was to determine whether the
program completed the activities in the cooperative agreement, and what impact it had on
teaching methods, learning outcomes and the public sector examination boards.
The evaluation methodology included: document study; visiting the AKU-EB to observe
at first hand its systems and processes, public-sector examination boards for purposes of
comparison, and schools affiliated to the AKU-EB; focus group interviews with AKU-EB
staff, school principals and directors, teachers and students; and telephone and in-depth
interviews with school directors, BISE and IBCC officials, and distinguished experts
working in education in Pakistan.
The achievements of the AKU-EB program include the computer-based examination
processing system which has been established at very low cost in the AKU-EB's
headquarters; AKU-EB's success in developing examinations and question papers of
uniquely high quality; the innovative approach AKU-EB has taken to practical testing in
science; AKU-EB's technically sophisticated approach to setting and maintaining
standards; the support AKU-EB provides to teaching via its in-service professional
development programs; and its exceptionally user-friendly customer service. In all of
these, AKU-EB is quite unique amongst Pakistani BISEs.
However, AKU-EB has also faced a number of challenges. Beginning in 2004, before
the Board was even offering examinations, there was politico-religious agitation against
it. It also faced resistance from the public-sector BISEs, which manifested itself in
difficulties in areas such as the affiliation of public-sector schools, and dual affiliation
(i.e., to a public-sector BISE and AKU-EB). The general public has also shown caution
about the AKU-EB examinations, mainly through fear of the results. This fear may be
largely unfounded; but it has reduced the numbers of candidates presenting for AKU-EB
examinations well below what was predicted. As a result, AKU-EB faces a significant
financial and marketing challenge if it is to achieve sustainability. As far as the team was
able to assess AKU-EB's use of funds, the cost-benefit analysis showed that the AKU-EB
made effective use of the funds provided.
The AKU-EB is an innovative example of public-private partnership in education; but its
obvious potential to stimulate and support change in the public sector has not so far been
fully realized, because public schools are unable to affiliate to the AKU-EB, and the
public-sector BISEs seem somewhat resistant to the influence of the AKU-EB. The
AKU-EB has been in the forefront of changes being introduced in education, and
especially public examinations, by the Government of Pakistan. It introduced a number
of innovations which were subsequently introduced on a wide scale in the public sector.
However, because of the resistance of the public-sector BISEs, the impact of the AKU-
EB has been less than its potential.
The AKU-EB is a brilliant technical example, which is a potential model for the future;
but its impact depends in large part on its achieving financial self-sufficiency, and on
reducing the resistance of the public sector BISEs.
AKU-EB Evaluation 5 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentLIST OF ACRONYMS
AED Academy for Educational Development
AJK Azad Jammu and Kashmir
AKES(P) Aga Khan Education Services (Pakistan)
AKF Aga Khan Foundation
AKU Aga Khan University
AKU-EB Aga Khan University – Examination Board
ARI Annual Rate of Increase
BISE Board of Intermediate and Secondary Education
CRQ Constructed Response Question
EO Examination Officer
ESR Education Sector Reforms
FANA Federally Administered Northern Areas
FATA Federally Administered Tribal Areas
FBISE Federal Board of Intermediate and Secondary Education
FY Fiscal Year
GoP Government of Pakistan
HIV Human Immunodeficiency Virus
HSSC Higher Secondary School Certificate
IBCC Inter-Board Committee of Chairmen
IPCME Inter-Provincial Committee of Ministers of Education
IR Intermediate Result
IRT Item Response Theory
JI Jama’at-e Islami
KBIE Karachi Board of Intermediate Education
KBSE Karachi Board of Secondary Education
LMO Learning Management Officer (ED-LINKS)
MCQ Multiple Choice Question
MMA Muttahida-Majlis-e-Amal
MNA Member of the National Assembly
MoE Ministry of Education
MoU Memorandum of Understanding
NEC National Education Census
NEP National Education Policy
NETS National Educational Testing Service
NOC No Objection Certificate
NWFP North-West Frontier Province
OMR Optical Mark Reading
PITE Provincial Institute of Teacher Education
PPP Public-Private Partnership
SBA School-Based Assessment
SDPI Sustainable Development Policy Institute
SLO Specific Learning Objective
SO Strategic Objective
AKU-EB Evaluation 6 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentSSC Secondary School Certificate
TRC Teachers’ Resource Center
USAID United States Agency for International Development
WHO World Health Organization
AKU-EB Evaluation 7 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentBACKGROUND STATEMENT
Purpose
The purpose of this evaluation was to evaluate the impact and effectiveness of
USAID support to The Aga Khan University Examination Board (AKUEB) program.
The evaluated program was implemented in support of the Interim Strategic Plan for
fiscal years (FY) 2004 - 2006, which provides specific goals, strategic objectives
(SOs), and intermediate results (IRs) for all funded programs.
The overall goal of the USAID Pakistan strategy is to “promote equality, stability,
economic growth and improved well-being of Pakistani families”. This overall goal is
divided into four sector strategic objectives, with the USAID/Pakistan SO 3 encompassing
the education sector.
USAID's Strategic Objective 3 (S03) aims 'to increase knowledge, training and
infrastructure to develop high quality education programs for girls and boys
throughout Pakistan'. Five Intermediate Results (IRs) were defined, as follows:
IR 3.1: Strengthened Education Sector Policy and Planning
IR 3.2: Improved Capacity of Teachers and Education Administrators
IR 3.3: Improved Youth and Adult Literacy
IR 3.4: Expanded Public-Private Partnerships to Improve Access and Delivery
of Education Services.
IR 3.5: Increased access to higher education
The AKU-EB program directly supported IR 3.2 and IR 3.4. The chart below describes
the SO and IRs directly related to the AKU-EB program:
AKU-EB Evaluation 8 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentSO3
Increased Knowledge, Training, and Infrastructure Provided to Develop High Quality
Education Program for Girls and Boys Throughout Pakistan
Indicators
Number of USAID sponsored policies developed at the national, provincial levels or district
level (ARI)
Annual percentage increase in student enrollment in target schools in target districts (ARI)
Number of students benefiting from Higher Education Assistance
IR 3.2 – Improved capacity of teachers IR 3.4 – Improved access to and delivery of
and education administrators education services
Indicators Indicators
• Number of teachers and education • Amount of private sector (profit and non-
administrators trained (ARI) profit) investment in schooling
• % of teachers meeting improved • Number of USAID-sponsored agreements
performance standards formalized between private sector entities and
• increase in students demonstrating public education
improved performance • Number of schools regularly developing and
implementing School Improvement Plans in
target districts
• Number of assisted infrastructure facilities
brought into use
Aga Khan University Examination Board Program (AKU-EB)
The AKU-Examination Board was established in Pakistan under a presidential
ordinance dated November 8, 2002. The ordinance stipulates that the certificates
awarded by the AKU-EB for educational achievement up to the higher secondary level or
its equivalent will be accorded full credit and recognition in Pakistan.
The Aga Khan University (AKU) implemented the program, “Establishment of the
AKU-Examination Board” in Pakistan. The program was awarded in August 2003 and
ended on December 31st 2007. The program aims were:
To promote improvement of the quality of education in schools through
examinations for the secondary school certificate (SSC) and the higher secondary
school certificate (HSSC), training of teachers, and development of supplementary
materials.
To design and offer high quality public examinations based on the national
curriculum for secondary and higher secondary education, with the
consequential objective of improving the quality of education in schools.
To promote the development of expertise in educational assessment and tests
in Pakistan.
The program planned to achieve these aims by establishing an examination system using
technical expertise and equipment from abroad, rather than by inviting an existing
examining body to establish the system, to facilitate the growth of expertise in
educational assessment and tests in the University and the country.
AKU-EB Evaluation 9 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentDevelopment was phased, to allow for the registration of schools, preparation of teachers
and students as well as examiners, and development of examination papers and systems.
Work was scheduled to begin in the first quarter of 2003, and the first SSC
examination as planned to be given in April-May of 2006.
The examination system was envisaged as being open to both government and private
schools from the beginning, with examinations based on the national curriculum for
secondary and higher secondary education, offered at an affordable cost, in both English
and Urdu. In principle, any school would be able to affiliate, regardless of location,
because the examinations and tests offered were based on attainment targets derived from
the objectives of the national curriculum, rather than the content of specific textbooks, as
is the case with the existing BISE examinations.
The system aimed to offer examination questions which assess the higher
intellectual abilities of comprehension, application, logical thinking and problem
solving, rather than being restricted to recall of the content of the textbooks. The learning
milieu in affiliated schools would be improved through accreditation and periodic review
leading to development of human resources, review of teaching-learning strategies, and
provision of learning resources. When a school presented candidates for examination,
their performance would be analyzed in detail and feedback provided to teachers,
schools, curriculum planners and educational policy makers in order to improve school
effectiveness, curriculum design, examinations, educational policies and support.
In this way the program aimed to demonstrate a credible and efficient model that might
lead the local examining boards towards improvement of the quality of their
examinations, and at the same time create capacity in Pakistan for education and training
in educational assessment and tests.
Rationale for implementing the AKU-EB Program
The Pakistani education system is often characterized as “examination driven”, in that
access to higher education and future earning capacity is closely linked to examination
success. As a result, schools and parents tend to judge academic performance through
examination scores achieved by students.
There are two main public examinations at this time. The SSC, taken in grades 9 and
10, is both a terminal certificate for school graduates who then seek employment and a
basis for admission to the next stage of education. HSSC is taken at grades 11 and
12, and forms the basis for selection of students for higher education and
employment. Only high HSSC scores can secure admission to professional colleges of
medicine, engineering, business administration and science and technology, the highest
competitive options for higher education. Previously, grades 11 and 12 were known as
“intermediate”, and formed the first stage of higher education. Students graduated from
secondary school at the end of grade 10, and sought admission to a college. Current
government policy is that grades 11 and 12 should become part of secondary education,
as is the norm internationally. However, this reform has been slow to take root,
especially in the private sector.
AKU-EB Evaluation 10 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentThe Boards of Intermediate and Secondary Education (BISE) are responsible for
organizing and conducting SSC and HSSC examinations. At present, there are 26
examination boards in the country, each established by an Act of the respective
Provincial or National legislatures. The locale of these examination boards is as
follows: 20 BISEs with jurisdiction for major cities, regions or provinces/areas, a Board
of Secondary Education (BSE) and another of Intermediate Education (BIE) for
Karachi, the Federal Board of Intermediate and Secondary Education (FBISE) with
national and international jurisdiction,; and three Boards of Technical Education, one
each in Sindh, Punjab and NWFP.
At present the public examination system is dysfunctional in that, although the
objectives of the national curriculum for the different levels of education are
appropriate, the SSC and HSSC examinations evaluate mainly recall of prescribed
texts, not attainment of the curricular objectives. This leads teachers to emphasize rote
learning, and largely to neglect comprehension of concepts and application of knowledge.
Over several decades, this unremitting preoccupation with rote learning has been a
major cause of declining quality of education. Propagation of this pattern of rote learning
prevents the classroom application of teaching methodologies that encourage
understanding of subject content and the ability to analyze and apply this knowledge.
When students taught in this way become teachers themselves, they tend to lack both the
necessary cognitive skills and a basic awareness of the weaknesses of the existing system,
and the possible alternatives. The high stakes involved also tend to encourage corruption
at all levels, from cheating in examination centers, through selling of questions or question
papers up to exploitation of senior management roles for personal gain.
The need for reform has long been realized; but the public examination system has shown
remarkable inertia. There is no apex body at federal or provincial level with a mandate to
regulate the system as a whole. BISEs are responsible for the efficient conduct of the
examination process within their geographical area, and are indeed quite efficient; but
they have no significant role in developing the system; nor do they have the specialist
resources in-house to develop new models, train question setters and teachers, and so on.
Parents and teachers know the present system, and are reluctant to take risks with the
future of the current generation of students.
This has led to strong demand, particularly from the private sector, for an alternative to
the present public sector examination system. The small minority who can afford top-
quality private education tend towards international qualifications such as the '0' and 'A'
level examinations conducted in Pakistan by the UK Boards. The disadvantages of
these are well understood, however, and the policy of GoP has been to encourage
students to take national examinations.
In 1995, sixteen private schools requested the Aga Khan University (AKU) to establish
an independent examination board. In 1998, the AKU appointed a Task Force to carry out
a feasibility study for establishing an alternative examination system. The Task Force
recommended the creation of an examination service as a function of the AKU, and the
Board approved the recommendation.
AKU-EB Evaluation 11 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentA proposal for establishing an alternative examination service was made to the
Ministry of Education in 1999 and again in 2001, and was approved in November 2002 by
the issuing of an Ordinance establishing the Examination Board of the Aga Khan
University. The Ordinance was consistent with the Government's policy to improve the
public examinations by introducing an independent examining body as a competitor,
through public-private partnership, given in the publication of the Ministry of Education
titled “Education Sector Reform 2001-2004”.
EVALUATION METHODOLOGY
Process
In 2008, USAID/Pakistan contracted Academy for Education Development (AED), to
evaluate the Aga Khan University Examination Board program, and to determine its
effectiveness in regards to cost as well as to stated program goals and objectives. The key
strategic and priority questions tasked to the evaluation team to consider included:
1) Did the program complete the activities in the cooperative agreement and in
annual work plans?
2) Did the program have an impact on the teaching methods of the teachers trained
by the AKU-EB?
3) Did the program have an impact on learning outcomes and teaching methods?
4) Did this program contribute to the Government of Pakistan's (GOP) education
reform measures?
5) Did the AKU-EB model influence other local examination boards?
The process used for gathering data for this evaluation included:
Meeting with SO3 Team: the evaluation team had an initial briefing from the
SO3 team.
Document review: the team carried out an extensive review of briefing materials
and documents provided by USAID/Pakistan and AKU-EB as well as the
Government of Pakistan’s Ministry of Education resources, documents
downloaded from the BISE internet sites and received directly from BISEs during
visits, and a survey of the archives of the English-language Pakistani press
covering the period of the AKU-EB’s interaction with the general public.
Telephone interviews with school principals and directors: the team contacted
as wide a range as possible of principals and directors of schools affiliated with
AKU-EB, in areas to which it was not feasible for team members to travel.
Site visits: the team paid an extended visit to the AKU-EB in Karachi, to meet
senior management and key staff members, including training managers, ICT staff
AKU-EB Evaluation 12 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational Developmentand subject officers, and to review the AKU-EB processes and products. One
member of the team also attended the AKU-EB Board meeting on May 12th. For
purposes of comparison, team members also visited the two Karachi Boards, of
Secondary and Intermediate Education, where they met with the senior
management to review processes and products. The team also visited Karachi
University and the IBCC.
Interviews and focus group discussions with school principals, teachers and
students: the team visited selected schools and held focus groups and/or in-depth
interviews with school principals, teachers and students. The team also made
follow-up interviews for the purpose of developing case studies.
Interviews with other key persons: the team also met selected individuals
having a particular interest in assessment and examinations in Pakistan. These
individuals were chosen essentially on the basis of recommendations.
Appendix B contains a list of people met. Appendix C contains a list of sites visited, and
Appendix D the basic Interview Protocols followed for both interviews and focus groups.
Appendix E contains a list of documents reviewed.
Selection of Schools to be Investigated
In identifying schools to be investigated, the team worked from the list of affiliated
schools provided to USAID by AKU-EB. This list provided information about all
schools that had sought affiliation, whether it was granted or not, and did not distinguish
the schools currently presenting candidates from those not presenting.
In order to capture as much as possible of the diversity in the private sector within a
limited time and budget, the team investigated geographical clusters of schools in detail,
rather than sampling individual schools at random. Schools within a cluster were
contacted by telephone, to gather details of their affiliation history and plans for their
future relationship with the AKU-EB. Where possible, principals and teachers from the
schools in a cluster were then invited to join a focus group meeting. Some schools were
then investigated in greater depth, as a basis for developing case studies:
Rawalpindi-Islamabad: in the case of Rawalpindi/Islamabad, very few schools
in the cluster had presented candidates, and planned to continue to do so. Three
schools in the cluster were selected for follow-up, one each from among the elite
private schools which also present candidates for O- and A-level, the lower-status
for-profit private schools, and a charitable foundation running two schools for
high-ability children of poorer people, one in Azad Kashmir, one near
Rawalpindi.
Karachi: in Karachi, appointments were mostly made through the AKU-EB. The
team visited a diverse group of schools, including a charitable foundation running
a school in a less affluent area, a large private school cooperates with the
AKES(P), and an AKES(P) school. In addition, two focus groups were held, one
with head teachers and one with teachers. Participants came from a range of
AKU-EB Evaluation 13 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational Developmentschools presenting candidates for AKU-EB examinations.
Lahore: in Lahore, all schools affiliated with AKU-EB were contacted by
telephone. Of the twenty-seven in the area, twelve agreed to attend a meeting and
focus group discussion held in the Lahore Grammar School.
In addition, a number of other schools outside these main clusters were contacted,
to confirm the provisional findings. Table 1 below summarizes the data gathered
from schools.
Table 1 – Schools contacted, visited, interviewed
Rep. Attended Representative Used as a Case
Location Contacted Visited
Focus Group Interviewed Study
Rawalpindi 9 1 0 9 3
Karachi 19 3 19 3 0
Lahore 27 1 6 4 3
Other 9 0 0 9 0
Total 64 5 19 25 6
Strengths of the Evaluation Process
As indicated above, the selection of schools was designed to illuminate the diversity of
the private sector, rather than to be statistically representative, and in this the team
believes it was successful. The three clusters of schools sampled provided examples of
very elite schools and schools run by charitable foundations for poor pupils, as well as the
range of for-profit schools which are the backbone of private sector education in
Pakistan. They yielded very different perspectives on the AKU-EB enterprise, and helped
to illuminate the market in which AKU-EB has to find its niche.
All informants were very generous with their time. AKU-EB staff met with the team for
extended periods, and provided generous follow-up information. School principals and
teachers in particular were willing to meet us outside school hours, and discuss the AKU-
EB at length, and very frankly and openly, with the team.
Challenges to the Evaluation Process
The time frame of the evaluation did not allow the team to gather data from all the 245
schools that have sought affiliation, even by means of a phone or mail questionnaire. The
approach taken was therefore designed more to identify issues than to determine the
statistical incidence of the different groups of schools identified. For example, a major
issue has proved to be schools that having affiliated with AKU-EB, do not send up
candidates. The team has therefore identified a number of such schools to determine
what their reasons are.
That said, the team has been in some sort of contact with nine schools in Rawalpindi,
twenty-seven schools in Lahore, and twenty-one in Karachi. These represent more than
one in five of all the schools that have sought affiliation. In most, the team talked with
both teachers and principals/directors. In a few, it was also possible to talk with students.
AKU-EB Evaluation 14 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentThe team was able to analyze documents, both AKU-EB documents and the equivalents
from different BISEs. Thus, statements about the differences between AKU-EB
instruments and procedures and their BISE counterparts are based on, and supported by,
evidence. The team was not, however, able to gather classroom observation data, nor
(except in a very limited way) to analyze individual students’ performance. The team
therefore had to rely mainly on the reports of informants, backed up, where possible, by
figures they provide.
OVERALL PROGRAM EFFECTIVENESS
In discussing overall program effectiveness, the evaluation team first looked at the
indicators defined in the cooperative agreement. The AKU-EB’s final report on the
program to USAID uses a different matrix; but this was not available to the team until the
very end of the evaluation period. The following tables were therefore adapted from the
results matrix in the final quarterly report prepared by the AKU-EB1.
Objective 1: To promote improvement of the quality of education in schools through
examination for the SSC and HSSC, training of teachers, and development of
supplementary learning materials
Method of
Performance Target and
Definition Data Responsibility Baseline
Indicator Achievements
Collection
% of A1 and In 2006 practice
A grades in there were 1%
Positive change
affiliated A+ grades and
in SSC and Routine
schools in the 8% A grades.
1.1 Improved HSSC AKU-EB
year prior to The same
quality of performance post-exam.
AKU-EB first AKU-EB candidates in
education in school by school, analysis by
examination, 2007 composite
schools with standards subject and
and in first scored 10% A+
maintained at a school
AKU-EB and 24% A
constant level
examination grades
Met for 2004
1.2 At least 50 Number of and 2005; not
schools per schools attracted AKU-EB met for 2006 (31
year affiliate to the AKU-EB records of 2004 April schools added);
AKU-EB
with AKU-EB and able to meet affiliation data set 2007 (11) or
beginning in affiliation fees 2008 (2 in
April 2004 requirements January-April).
Not met overall.
1
18th USAID Quarterly Report Oct to Dec 2007.doc
AKU-EB Evaluation 15 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentMethod of
Performance Target and
Definition Data Responsibility Baseline
Indicator Achievements
Collection
AKU-EB
T’s have copies
inspectors Textbook is not
of examination
1.3 Teachers check primary focus in
syllabus and use
move from teachers Not at least 1 period
more than one
relying on have established but a week in 70%
reference to AKU-EB
single text to syllabus; currently of subjects in
support teaching. inspectors
attainment review reputed to be 80% of affiliated
P’s interact more
targets in exam. lesson close to zero. schools.
with teacher and
syllabus plans; No data
each other, less
interview gathered
with textbook
students
Pedagogic skills
and subject
understanding
Data from all Significant year
enhanced by
BISE in 2004 on year increase
exposure to Annual
examination in difference
1.4 Training of assessment school
AKU-EB and following between school
teachers material; inspection
years for both and private
participation in visits
SSC and candidates due
marking;
HSSC. to training.
detailed
feedback to
schools.
SLMs provide
subject matter
enrichment for
teacher; learning
Materials
1.5 activities for
prepared for
Development pupil; criterion- Inventory
AKU-EB and 100% of SSC
of referenced of material
affiliated 0% and HSSC
supplementary assessment tasks; available to
schools syllabus topics
learning and form to schools.
deemed to need
material (SLM) report class
them
results to AKU-
EB for
improving
materials
The implementation of program activities was excellent. The wide range of
supplementary learning materials, providing subject matter enrichment for teachers,
learning activities for pupils, and criterion-referenced assessment tasks were of good
quality (indicator 1.5). The AKU-EB have not prepared materials for 100% of SSC and
HSSC syllabus topics deemed to need them; but have provided a stimulus to schools and
teachers to identify and develop further materials.
The AKU-EB provided a one-day orientation to 1812 teachers and 172 school principals.
The Board also trained 912 teachers in item writing, 300 teachers in conducting practical
examinations, and 837 teachers in developing assessment units. Overall participants
rated the quality of the training as very high, and asserted that it had an impact on their
teaching practices.
AKU-EB Evaluation 16 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentThe team was not able to confirm directly that interaction patterns had changed, or that
teachers relied on attainment targets rather than textbooks (indicator 1.3); but teachers
interviewed did report having the examination syllabus, engaging in lesson planning and
using materials from different sources. Librarians met reported strong demand for
resource materials, although they also reported some difficulty in getting some materials
from out of province.
After a promising start, AKU-EB is currently not achieving its affiliation targets
(indicator 1.2). Applications for affiliation peaked at 120 in 2005, and have fallen in each
succeeding year (although the 2008 figure is for four months only). Therefore, the target
for school affiliations has not been met to date; and the slowing rate of affiliations
suggests it is not likely to be met in the immediate future.
Table 2 – School Affiliations, 2003-08
Schools 2003 2004 2005 2006 2007 20082 Total
Applied 1 66 120 38 18 4 247
Affiliated 1 44 72 31 11 2 161
Active - - - 60 85 83
As table 2 above shows, the number of schools currently affiliated with the AKU-EB,
according to the database supplied by USAID, is 161. Not all of these affiliated schools
are defined as “active” (i.e., presenting candidates for SSC or HSSC). Eighty-three, or
52%, presented candidates for one or other examination in 2008; but this number is likely
to fall in 2009, because seventeen of the schools with candidates in class X in 2008 have
none in class IX (i.e., are not presenting candidates for the first year of the new two-part
examination).
The baseline proposed for indicator 1.1 (percentage of A+ and A grades in affiliated
schools in the year before AKU-EB’s first examination compared with the percentage
achieved in the first AKU-EB examination) was probably inappropriate, because the
public sector BISE examinations are acknowledged to primarily test recall rather than
thinking, and vary widely in difficulty. The revised baseline, performance in AKU-EB’s
practice examination, may be more appropriate; but it is vulnerable to distortion by low
student motivation (because it is low-stakes), and “practice effect” (because it is the first
such test paper students have attempted). The first actual AKU-EB SSC examinations
might therefore be a better baseline; but these took place only in 2007. It is therefore too
soon to report on trends in achievement.
2
January to April only.
AKU-EB Evaluation 17 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentObjective 2: To design and offer high quality public examinations based on the National
Curriculum for Secondary and Higher Education
Performance Method of Data Target and
Definition Responsibility Baseline
Indicator Collection Achievements
Specification grid
2.1 16 SSC showing marks
subject exams allocated to content
designed and topics and objectives,
published by April and illustrated with AKU-EB Examination Target met for SSC.
AKU-EB WWW
2004 specimen materials. Subject papers of the HSSC syllabuses
pages
16 HSSC subject Goals for knowledge, Committees other boards not yet published.
exams designed understanding,
and published by application; no more
April 2006 than 50% knowledge
marks in any subject.
2.2 Schools
servicing the
middle income AKU-EB
Schools pay affiliation
group, dissatisfied records of Not At least 10% annual
fees; register candidates AKU-EB
with quality of affiliation and established increase on baseline
for the examinations
BISE exam candidate fees
affiliate to AKU-
EB
The target of developing examination syllabuses and specification grids for SSC has been
exceeded (target 2.1). The AKU-EB web site offers twenty SSC syllabuses for
download. These syllabuses are well structured and highly innovative in Pakistani terms.
HSSC syllabuses are not yet available on the web site. However, the team was able to
verify that AKU-EB unquestionably has delivered its full quota of examinations
syllabuses and specification grids, and more. In 2008, it will conduct: HSSC-I (grade XI)
examination in 24 subjects across five subject-groups (pre-engineering, pre-medical,
science general, humanities and commerce); SSC Composite examination in 26 subjects;
and part-wise SSC-I in 25 subjects.
Schools serving the middle income group are undoubtedly dissatisfied with the quality of
both the public sector BISE examinations and their customer service (target 2.2); by
contrast, the AKU-EB examinations and customer service were reported on very
favorably. As a result, there was an initial rush to affiliate to the AKU-EB. However,
many schools found preparing students for the AKU-EB examinations much more
challenging than they had anticipated. As a result, many affiliating schools have never
presented candidates; and a significant minority have given up after presenting candidates
for one or two years. This now also appears to be reflected in a tapering-off in
affiliations. The AKU-EB therefore needs to give attention to strengthening its business
model.
AKU-EB Evaluation 18 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentObjective 3: To promote the development of expertise in educational assessment and tests in
Pakistan
Performance Method of Data Target and
Definition Responsibility Baseline
Indicator Collection Achievements
At least two staff
members exposed to
training per year, 2005-
2008
The board in collaboration
with IED is mounting a
Overseas training in At least six Certificate in Educational
Contractual Assessment Course which
assessment training months
requirement of AKU-EB Nil is now in its second cohort
successfully per annum from 1
training institution
completed Jan, 2005
One attended five months
course, 2 conference
participants of 1 week
each, and 2 attended the
security printings of 2
weeks each.
According to information provided by AKU-EB, overseas travel for training purposes has
been limited to attendance at international conferences plus short orientation visits to
examining bodies, as outlined in table G-1 (annex G) below.
The team is of the opinion that given the major differences in functions and
organizational structure between AKU-EB and the public-sector BISEs, there is a lack in
the national market of both desirable skills in assessment and examination, and relevant
experience in managing a board on the AKU-EB model. More therefore might have been
done during the program period to develop the professional/technical capacity of AKU-
EB’s middle management to face the challenges of the post-program period; and given
the relative similarities between AKU-EB and examining bodies in various
Commonwealth countries, a program of longer attachments (1-3 months) with
Commonwealth examining bodies, plus tailored training in examination practice, might
have been considered.
PROGRAM ACHIEVEMENTS/STRENGTHS
The program demonstrated significant technical strengths comparative to the public
sector BISEs, which can be directly linked to the funding provided by USAID. These
include: the examination processing system; the system for examination development; the
support provided for teaching and learning; and the customer service procedures. The
professional and technical capacity of the AKU-EB far exceeds that of all the public
sector BISEs combined; and although it is not possible to assess at this stage the long-
term impact on learning of participating in AKU-EB examinations, it is potentially very
significant.
The AKU-EB Examination Processing System
Supported by program funding, the AKU-EB has established an integrated system for the
development and management of examinations that makes extensive use of information
and communication technology, and stands in marked contrast to the examination
AKU-EB Evaluation 19 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational Developmentprocessing systems of the public-sector BISEs, which continue to depend largely on
manual labor, with limited technological support. This system has five main elements.
1) Registration of Candidate Details: Schools pre-register potential candidates in
December, either by completing a scannable sheet or (where schools have the
capacity) via a web-based form. The web-based process, called by AKU-EB e-
enrollment, allows schools to update their candidate information on demand.
Beginning in January, potential candidates have to register for the actual
examination. AKU-EB sends out individual scannable candidate registration
forms based on the pre-registration lists. In February/March, candidates confirm
their registration, and if necessary, correct the information on the form and/or
change their subject choices.
E-enrollment is clearly superior to paper-based enrollment in both speed and
accuracy. It has been very favorably received by users; in due course, it is
expected that registration will be done mainly via the web-based system; although
a paper-based option will be retained for schools lacking web access.
2) Printing of Question Papers: Once the registration data have been finalized, the
AKU-EB is able to establish the database for examination processing, plan for the
required examination centers, and print the required examination papers. A
secure printer (the printer currently used is based in Dubai) is sent camera-ready
copy for the test booklets, and, at the same, an electronic candidate list. State-of-
the-art digital printing technology then allows the printer to print customized
examination papers for all candidates (including candidate details and bar-codes
on each page as required) and bundle the papers for eventual distribution.
This approach is potentially highly secure. Question papers are only available in-
country prior to the examination in electronic form. There is no need for human
intervention in counting and bundling papers; and although the physical sealing of
the packets still requires some human intervention, the packets can be sealed
outside Pakistan, and sent directly via Habib Bank to the examination centers.
Digital printing of question papers with bar codes is an integral element in the e-
marking of question papers, because it allows data to be linked electronically,
even when pages are separated physically.
3) Marking of Question Papers: Data for responses to multiple-choice questions
are gathered on separate answer sheets and captured using a conventional OMR
scanner; but the most striking innovation in the AKU-EB examination processing
system may be the facility for e-marking of constructed-response questions.
Candidates write their answers to constructed-response questions directly into the
question-answer booklets. For processing, the booklet is cut into separate sheets.
The cover page is discarded, and the separate pages are then scanned using an
image scanner, and stored as a separate graphic image in the computer system. At
this point, the pages are linked only by the bar code printed on each. Thus, it is
not possible for anyone to know who wrote the answer by looking at the graphic
AKU-EB Evaluation 20 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational Developmentimage itself; and linking up the separate pages of a candidate's answer book can
only be done by reading the bar codes.
At this time, each scanner is capable of scanning and storing about 1000 page
images per hour – i.e., 10,000 page images during a ten-hour working day.
Marking is carried out on-line, using the AKU-EB’s proprietary e-marking
software. Under the overall leadership of the relevant subject officer, markers
work in teams of five markers plus a team leader. Each marker works at a
networked workstation. Questions are distributed amongst markers (each marker
marks a limited number of questions, rather than marking whole papers). The
software presents the marker with the graphic image of a candidate’s answer, plus
the marking key or marking suggestions, and records the mark awarded. The
team leader is able to monitor the standards of the team members in real time,
with the option of sending some or all of any member’s papers for re-marking.
During the 2007 examination cycle, e-marking was carried out not only at the
AKU-EB’s main building in Karachi, but at a satellite site in Lahore, allowing
teachers in a wider range of locations to participate.
This system has many benefits. It promotes marker reliability, by: limiting the
number of questions each marker marks; providing them with marking hints; and
allowing real-time statistical monitoring of individual markers’ standards against
the standard set by the chief examiners. It is virtually immune to corruption at the
level of the marker, because: no marker sees more than a very few questions; it is
all but impossible to guarantee that any individual’s paper will go to any given
marker; and team leaders are able to monitor a much larger proportion of
markers’ work than would be possible with paper-based marking.
As a result, all the problems associated with managing large quantities of paper
are eliminated. Booklets do not get lost. It is theoretically possible for team
members to work in any physical location, and still be coordinated, because the
scripts can be accessed on-line. Teachers of the relevant grade can participate in
marking, because their access is limited to the specific questions for the specific
individuals they have to mark; and they cannot identify individual candidates'
work. This is beneficial both for the quality of the marking and for the skills of
the teachers. It is very efficient in its use of markers’ time, and much less prone
to errors in transfer of results than any paper-based system.
4) Results Processing: Results are made available on-line to registered candidates,
as well as in the form of printed certificates. In terms of its public interface, the
AKU-EB system is similar to those of the public sector BISEs; but because it
makes much more extensive use of information technology, it is quicker, more
accurate and less prone malpractice.
A further significant advantage of the system of scanned images of answer books
is that it makes archiving all the documents connected with an examination a
relatively straightforward process. There is no longer any need to store large
AKU-EB Evaluation 21 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational Developmentquantities of paper (a factor which normally severely limits not only the number
of years for which examination-related documents can be kept, but also their
accessibility). The graphic images can easily be stored in a compact and
permanent form, for an indefinite period.
5) Feedback to Schools: Another big advantage of e-marking is that it provides test-
and item-level candidate data which can be aggregated to inform schools and
teachers on their relative performance. AKU-EB has developed a format for a
school report, including graphical representations of data. A report in this format
is sent to each school which sends up candidates, which includes:
a. average achievement and grade distribution of candidates from school,
overall and by subject paper, compared with previous year(s);
b. average achievement and grade distribution of candidates from school,
overall and by subject paper, compared with all schools;
c. performance of candidates from the school in questions measuring
knowledge, understanding and application, compared with all schools;
d. item-wise performance of candidates from school, compared with all
schools.
All respondents expressed enthusiasm for the format of the feedback report, and
said they used it. One school, St. Mary’s Academy in Rawalpindi, shared with
the team a PowerPoint presentation based on the feedback report which formed
the basis of a day spent by the staff team in analyzing the implications of the
results, in terms of changes in the teaching program, motivation of students, and
contact with parents.
The AKU-EB System for Examination Development
Box 1 – excerpt from a Punjab BISE syllabus document
In comparison with the
public sector BISEs, the
AKU-EB exercises much
greater control over the test
development process. This
is done initially by means of examination syllabuses, derived from the national
curriculum. The existence of these examination syllabuses, which were developed by the
AKU-EB for the first time in Pakistan, and their distribution to all teachers, is a crucial
factor in making the curriculum accessible to teachers as a set of goals, rather than simply
as defined textbook content.
AKU-EB Evaluation 22 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentThe content for the whole is
Paper carrying total marks:- 75 (except English & Urdu)
divided into topics, and the
topics are divided into Subjective 30 Marks Time: 1:30 Hour
specific learning objectives Objective 45 Marks Time: 1:30 Hour
(SLOs). SLOs are Detail of Marks for Objective Part will be as Follows
categorized according the
MCQ 10 Item 10 Marks
cognitive level (knowledge,
understanding, application). Completion 10 Item 10 Marks
On this basis, a scheme of Matching 09 Item 09 Marks
assessment is developed.
SLOs are summarized Short answers 08 Item 16 Marks
according to topics and Box 2 – Typical BISE Scheme for Setting Question Papers
levels. Marks are then
allocated to each topic
roughly corresponding to the number of sub-topics, and divided into MCQ and CRQ. It
is AKU-EB policy that every topic will be examined. The public sector BISEs have no
such specific policy, and also currently lack the tools to implement it, because of the
nature of their syllabuses (see Box 1). The examination syllabus then gives brief
guidance on the teaching-learning approaches to be used, and a short list of recommended
texts and reference materials. The syllabus also identifies areas where development of
learning support materials would be beneficial. The syllabus includes a list of definitions
of terms. Finally, the scheme of studies and the list of practicals (where appropriate) are
given as annexes.
This comprehensive syllabus document, available from the AKU-EB’s website, contrasts
with the approach of the public sector BISEs, where the syllabus is in effect the textbook
(see box 2), and only the number and type of items are defined3, with the specific content
left to the paper setter (see box 2). The AKU-EB approach is greatly superior as a means
of controlling the content (and by implication quality) of the test paper, because it defines
both the content topics to be tested and the cognitive levels at which questions should be
set. It also assures that teachers and students need to cover the whole content of the
curriculum, and provides them with a structure for planning that coverage.
Question Setting and Question Paper Assembly
As a national Board which offers SSC and HSSC qualifications for both English- and
Urdu-medium schools, the AKU-EB follows the Schemes of Studies issued by the
Federal Curriculum Wing.
Unlike the public-sector BISEs, where question paper setting is largely delegated to
outside examiners, with limited in-house moderation, the AKU-EB actively involves
teachers from participating schools in question setting, and has a relatively protracted
process of question paper development, involving Examiners, Chief Examiners and
Subject Officers. Having been recruited and trained, question setters send questions to
3
The fact that some of the item types used by the public-sector BISEs, such as true-false, fill in the
blanks and matching, are not technically very sound, should also be noted.
AKU-EB Evaluation 23 David Carroll, Shaista Shahid,
April-May 2008 Mulazim Hussain Mujahid
Academy for Educational DevelopmentYou can also read
