Using Experts' Opinions Through Delphi Technique
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
A peer-reviewed electronic journal.
Copyright is retained by the first or sole author, who grants right of first publication to the Practical
Assessment, Research & Evaluation. Permission is granted to distribute this article for nonprofit, educational
purposes if it is copied in its entirety and the journal is credited.
Volume 12, Number 4, May 2007 ISSN 1531-7714
Using Experts’ Opinions Through Delphi Technique
Muhammad Imran Yousuf
University of Arid Agriculture, Rawalpindi, Pakistan
The Delphi technique provides different opportunities to researchers than survey research. Essential
components of the Delphi technique include the communication process, a group of experts, and essential
feedback. This paper provides the foundations of the Delphi Technique, discusses its strengths and
weaknesses, explains the use and stages followed, discusses panel selection, and explains how consensus
among participants is reached.
The Delphi technique is a group process used to survey This paper presents how to apply the Delphi
and collect the opinions of experts on a particular Technique and then discusses its strengths and
subject. Linestone and Turoff (1975) provides a basic weaknesses, panel selection, and consensus among
definition of the Delphi technique: “Delphi may be participants.
characterized as a method for structuring a group
communication process so that the process is effective in BACKGROUND
allowing a group of individuals, as a whole, to deal with a The technique was named after the ancient Greek oracle
complex problem” (p.3). It has application whenever at Delphi from which prophecies were given (Koontz &
policies, plans, or ideas have to be based on informed O'Donnell, 1976). An oracle refers to a statement from
judgment. This technique is useful where the opinions someone of unquestioned wisdom and knowledge or of
and judgments of experts and practitioners are needed infallible authority (Funk & Wagnells, 1966). The Delphi
but time, distance, and other factors make it unlikely or technique was developed by Olaf Helmer and his
impossible for the panel to work together in the same associates at the Rand Corporation in the early 1950s
physical location. when they were working on defense research. Rieger
(1986) described Delphi's development in five stages: (1)
The Delphi technique, by definition, is a group secrecy and obscurity, (2) novelty, (3) popularity, (4)
process involving an interaction between the researcher scrutiny, and (5) continuity.
and a group of identified experts on a specified topic,
usually through a series of questionnaires. Delphi has The first stage was secrecy, during which the Delphi
been used to gain a consensus regarding future trends technique was classified by the military. Delphi
and projections using a systematic process of techniques were developed to gain consensus within a
information gathering. The technique is useful where the group of military experts on a very sensitive problem.
opinions and judgments of experts and practitioners are This stage lasted from the early 1950s to the early 1960s,
necessary. It is especially appropriate when it is not when it was declassified. The second stage, novelty,
possible to convene experts in one meeting. Skutsch and lasted from the mid-1960s to the late 1960s. During this
Hall (1973) identified the Delphi technique as a method stage the technique was used primarily by corporate
for gaining judgments on complex matters where precise planners as a forecasting tool for industry and human
information is unavailable. services. The third stage, popularity, lasted from the late
1960s to the mid-1970s. During this time various articles,Practical Assessment, Research & Evaluation, Vol 12, No 4 2
Yousuf, Delphi Technique
papers, reports and dissertations appeared on the topic. heads of a group of knowledgeable people. He further
The fourth stage, scrutiny, began in 1975 with Sackman’s stated that a well-designed and properly managed Delphi
unexpected attack on the Delphi technique itself. The could be a highly motivating environment for
attack was not unchallenged. The fifth and final stage, respondents.
continuity, is the Delphi's present stage of development.
A policy Delphi is one which seeks to generate the
Areas of application strongest possible opposing viewpoints on a policy issue
from an expert panel. Rather than consensus, the
Linstone and Turoff (1975) argued that Delphi has emphasis is on identifying differing opinions and
application in the following areas: divergent responses through a process of debate carried
out though the rounds of Delphi (Needham, 1990).
• Gathering current and historical data not The policy Delphi is given other names also, such as
accurately known or available. focus Delphi and decision Delphi. A normative Delphi
• Evaluating possible budget allocations. (also called a consensus Delphi), focuses on establishing
what is desirable in the form of goals and priorities. It
• Exploring urban and regional planning options. does not focus on speculating about what is probable
• Planning university campus and curriculum within a given time frame in the future; instead it is an
development. attempt to "... structure a set of properties which could
be integrated into a normative future--properties based
• Putting together an educational model. on the criterion of desirability rather than likelihood ..."
(Sutherland, 1975, p.466).
• Delineating the pros and cons associated with
potential policy options.
Most Delphi studies in educational settings are
• Distinguishing and clarifying real and perceived normative and are perceived as particularly useful. Rieger
human motivations. (1986) reported 83 percent of the dissertations
completed during the 1981-1984 period which used the
• Exploring priorities of personal values, social Delphi technique were of the normative type. He went
goals, etc. on to state, "... it seems reasonable to claim that Delphi
This research technique allows educators, amongst is continuing to be a much used tool in the search for
others, to communicate and effectively develop trends, answers to normative questions, especially in education
needs, or other factors relative to a particular area of areas, but also in other fields".(p.198)
education. In selecting the most appropriate research
tool, however, Linstone and Turoff (1975) caution the PROCESS
researcher to consider the circumstances surrounding the The process for each type of Delphi is essentially the
"necessarily associated group communication process" same; however, the purpose of a study determines the
(p. 6). They suggest these guiding questions: "Who is it type of Delphi used. The Delphi's process is similar to
that should communicate about the problem, what the nominal group technique (NGT), except Delphi does
alternative mechanisms are available for that not require the physical presence of group members
communication, and what can we expect to obtain with (Mitchell & Larson, 1987). An interaction process still
these alternatives?" (p. 6) Depending on the answers to takes place between the members of the group (Delphi
these questions, one may then choose the Delphi as the panel) and the researcher, with the researcher acting as a
most effective research tool for the study at hand. facilitator.
Forms of Delphi The basic steps of the Delphi process were outlined
by Pfeiffer (1968):
The original intent of Delphi was as a forecasting
technique, designed to predict the likelihood of future 1. The first questionnaire which is sent to the
events. Additional names have been given to this panel of experts may ask for a list of opinions
process. Dailey (1988) described it as an exploratory involving experiences and judgments, a list of
Delphi. Van Dijk (1990) called it a conventional Delphi. predictions, and a list of recommended
According to Dalkey (1972) the Delphi is a procedure activities.
that is a rapid and efficient way to cream the tops of thePractical Assessment, Research & Evaluation, Vol 12, No 4 3
Yousuf, Delphi Technique
2. On the second round, a copy of the collective 6. The results of Questionnaire Four are tabulated
list is sent to each expert and the expert is asked and presented as the final statement of group
to rate or evaluate each item by some criterion consensus.
of importance.
Worthen and Sanders (1987) stated that this
3. The third questionnaire includes the list, the "interactive procedure can continue for several more
ratings indicated, and the consensus, if any. The rounds, but the payoff usually begins to diminish quickly
experts are asked to either revise their opinions after the third round" (p.312). Brooks (1979) included an
or discuss their reasons for not coming to additional step prior to beginning the procedure: assess
consensus with the group. the willingness of potential panel members to participate
in the study. Several steps, as identified by Brooks
Scheele (1975) illustrated a process where the
(1979), are involved in using the Delphi Technique:
opinions and judgments of people familiar with or
associated with a subject and they listed a typical 1. Identifying the panel of experts.
sequence of events in the Delphi process in six steps.
2. Determining the willingness of individuals to
According to Issac and Michael (1981, p.115) the Delphi
serve on the panel.
process has six steps:
3. Gathering individual input on the specific issue
1. Identify the group members whose consensus
and then compiling it into basic statements.
opinions are sought. If the study goes beyond an
intact group such that representatives must be 4. Analyzing data from the panel.
selected, care must be taken to insure that all the
5. Compiling information on a new questionnaire
various publics or positions are proportionately
and sending to each panel member for review.
sampled.
6. Analyzing the new input and returning to the
2. Questionnaire One. Have each member
panel members the distribution of the
generate a list of goals, concerns, or issues
responses.
toward which consensus opinions are desired.
Edit the results to a manageable summary of 7. Asking each panel member to study the data and
items presented in random order. Prepare the evaluate their own position based on the
second questionnaire in an appropriate format responses from the group. When individual
for rating or ranking (Note: If an established or responses vary significantly from that of the
acceptable listing of such items already exists, group norm, the individual is asked to provide a
this first step can be bypassed.). rationale for their differing viewpoint while
limitations are placed on the length of the
3. Questionnaire Two. Have each member rate or
remarks in order to keep responses brief.
rank the resulting items.
8. Analyzing the input, and sharing the minority
4. Questionnaire Three. Present the results of
supporting statements with the panel. Panel
Questionnaire Two in the form of
members are again asked to review their
Questionnaire Three, showing the preliminary
position and if not within a specified range, to
level of group consensus to each item. Where
justify the position with a brief statement.
the individual differs substantially from the
group, and chooses to remain so on
Questionnaire Three, the respondent should
CHARACTERISTICS OF THE
provide a brief reason or explanation. DELPHI TECHNIQUE
5. Questionnaire Four. The results of Dalkey (1967) has identified the following basic
Questionnaire Three are presented in the form characteristics of the Delphi technique:
of Questionnaire Four, showing the new level of
group consensus for each item and repeating the 1. Anonymity - - the use of questionnaires or other
member's latest rating or ranking, along with a communication where expressed responses are
listing by item of the major reasons members not identified as being from specific members of
had for dissent from the prevailing group the panel allows for anonymity.
position. Each member rates or ranks each item 2. Controlled feedback from the interaction - -
for the third and final time, in light of the Controlled feedback allows interaction with a
emerging pattern of group consensus and the large reduction in discord among panel
reasons for dissent. members. Interaction consists of allowingPractical Assessment, Research & Evaluation, Vol 12, No 4 4
Yousuf, Delphi Technique
interaction among group members in several Helmer (1983) agreed that Delphi is a technique
stages, with the results of the previous stage frequently used for eliciting consensus from within a
summarized and group members asked to group of experts that has application in reliability and has
reevaluate their answers as compared to the many advantages over other methods of using panel
thinking of the group. decision making. Helmer (1983) agrees with Linstone
and Turoff (1975) in regards to the application of Dephi.
3. Statistical group response - - the group opinion
Helmer (1983), Linstone and Turoff (1975), and Dalkey
is defined as a statistical average of the final
(1972) all found that one of the major advantages of
opinions of the individual members, with the
using Delphi as a group response is that consensus will
opinion of every group member reflected in the
emerge with one representative opinion from the
final group response.
experts.
At the same time as Dalkey (1967) was identifying
There are many additional advantages. The
the basic characteristics of the Delphi technique, Helmer
technique is simple to use. Advanced mathematical skills
(1967) supported the validity and reliability of the
are not necessary for design, implementation, and
technique as an acceptable method of data collection
analysis of a Delphi project. Because the Delphi provides
from an identified group. Further he said that Delphi
confidentiality, many barriers to communication are
Technique is efficient in both group decision making
overcome. Some of these barriers are reluctance to state
situations and in other areas where order of magnitude
unpopular views, to disagree with one's associates, or to
estimates are required (Helmer, 1983). A Delphi study
modify previously stated positions (Barnes, 1987).
carried to the extreme degree could be an expensive
undertaking in both time and money on the part of the It helps prevent a groupthink, as earlier mentioned,
researcher and the respondents. particularly with one or two dominant people. A major
strength of the technique is the flexible, but limited, time
Strengths parameters with which individuals have to respond to
The Delphi technique is beneficial when other methods the questionnaires (Brooks, 1979). This flexibility allows
are not adequate or appropriate for data collection. It is individuals, who may be restricted by daily schedules and
particularly useful when geographic location, the opportunity to respond at times
available to them.
1. The problem does not lend itself to precise
analytical techniques but can benefit from Limitations
subjective judgments on a collective basis.
Delphi is not without limitations. The consensus reached
2. The individuals needed to contribute to the in a Delphi may not be a true consensus; it may be a
examination of a broad or complex problem product of specious or manipulated consensus. A
have no history of adequate communication and specious consensus does not contain the best judgment.
may represent diverse backgrounds with respect Instead, it is a compromise position (Mitroff & Turoff,
to experience and expertise. 1975).
3. More individuals are needed than can effectively Delphi appears to be a straightforward approach to
interact in a face-to-face exchange. doing research in the area of forecasting and for building
4. Time and cost make frequent group meetings consensus. Researchers, at first glance, think of Delphi as
infeasible. a simple technique that can be done easily. However, one
must carefully consider the problems associated with
5. The efficiency of face-to-face meetings can be Delphi before designing a Delphi study.
increased by a supplemental group
communication process. Linstone and Turoff (1976, p.6) suggested that there
are five common reasons for Delphi to fail:
6. Disagreements among individuals are so severe
or politically unpalatable that the 1. Imposing monitor views and preconceptions of
communication process must be refereed a problem upon the respondent group by over
and/or anonymity assured. specifying the structure of the Delphi and not
allowing for contribution of other perspectives
7. The heterogeneity of the participants must be related to the problem.
preserved to assure validity of the results, i.e.,
avoidance of domination by quantity or by 2. Assuming that Delphi can be a surrogate for all
strength of personality ("bandwagon effect"). other human communications in a given
(Linstone & Turoff, 1975, p.4) situation.Practical Assessment, Research & Evaluation, Vol 12, No 4 5
Yousuf, Delphi Technique
3. Poor techniques of summarizing and presenting (1) "the problem does not lend itself to precise
the group response and ensuring common analytical techniques but can benefit from subjective
interpretations of the evaluation scales utilized judgments on a collective basis.
in the exercise.
(2) "individuals who need to interact cannot be
4. Ignoring and not exploring disagreement so that brought together in a face-to-face exchange because of
discouraged dissenters drop out and an artificial time or cost constraints" (p.275).
consensus is generated
5. Understanding the demanding nature of a DISCUSSION
Delphi and the fact that the respondents should
be recognized as consultants and properly The outcome of a Delphi sequence is nothing but
compensated for their time if the Delphi is not opinion; the results of the sequence are only as valid as
an integral part of their job function. the opinions of the experts who made up the panel. The
panel viewpoint is summarized statistically rather than in
Delkey (1972) pointed out that Delphi is not terms of a majority vote. It is very important to ensure
sufficient to be a defining property for an uncertain understanding of the aim of the Delphi exercise by all
question because the expert’s cultural bias can lead to the participants. Otherwise the panelists may answer
similar answers to some questions which in fact are inappropriately or become frustrated and lose interest.
poorly known; or there could be an instance where the The respondents to the questionnaire should be well
experts legitimately do not know the answer. According informed in the appropriate area yet some literature
to Linstone and Turoff (1975), the virtual problems do suggests that a high degree of expertise is not necessary.
not affect the utility of Delphi but rather how to select The minimum number of participant to ensure a good
the respondent group. performance is somewhat dependant on the study
Barnes (1987) has listed additional disadvantages of design.
the technique: The Delphi method has got criticism as well as
1. Judgments are those of a select group of people support. The extensive critique of the Delphi method
and may not be representative; were (a) being unscientific; (b) having a low level
reliability of judgments among experts and therefore
2. Tendency to eliminate extreme positions and dependency of forecasts on the particular judges
force a middle-of-the-road consensus; selected; (c) the sensitivity of results to ambiguity in the
3. More time consuming than the nominal group questionnaire that is used for data collection in each
process; round; and (d) the difficulty in assessing the degree of
expertise incorporated into the forecast. The support
4. Should not be viewed as a total solution; underlines the fact that Delphi is a method of last resort
5. Requires skill in written communication; in dealing with extremely complex problems for which
there are no adequate models. Sometimes reliance on
6. Requires adequate time and participant intuitive judgment is not just a temporary expedient but
commitment (about 30 to 45 days to complete in fact a mandatory requirement.
the entire process) (p.63).
The essence of the technique is fairly
Fortune (1992) indicated that an additional reason straightforward. The main point behind the Delphi
for Delphi failure is that the panel members may not be method is to overcome the disadvantages of
able to see the vision or the big picture in which they are conventional committee action. The group interaction in
involved. This problem arises when the panel members Delphi is anonymous, in the sense that comments,
chosen are so close to the problem that they cannot see forecasts, and the like are not identified as to their
the future. originator but are presented to the group in such a way
Appropriateness as to suppress any identification. The convenience of
electronic communication has steered the evolution of
An overriding factor in the selection of the Delphi the Delphi toward computer-mediated studies. This
technique is the appropriateness of the technique for a could foster further developments, including support
particular study. Linstone (1978) identified two from multi-media, simulation and modeling tools and
circumstances where Delphi techniques are most altogether boost new research opportunities for the
appropriate: method. To maximize the quality of the outcome and
address concerns for methodological rigor, the DelphiPractical Assessment, Research & Evaluation, Vol 12, No 4 6
Yousuf, Delphi Technique
study may be triangulated with a parallel electronic on a given subject, selection of the panelists amongst
survey and follow-up evaluation techniques. experts, development of the rounds, transmission to the
panelists and preparation of a report by the analysis. It is
Caveats most important to understand the aim of the Delphi
Overall the track record of the Delphi method is mixed. exercise by all participants, otherwise the panelists may
There have been many cases when the method produced answer inappropriately or become frustrated and lose
poor results. Still, some authors attribute this to poor interest. The respondents to the questionnaire should be
application of the method and not to the weaknesses of well informed in the appropriate area but the some
the method itself. It must also be realized that in areas literature suggests that a high degree of expertise is not
such as science and technology forecasting the degree of necessary. The minimum number of participants to
uncertainty is so great that exact and always correct ensure a good group performance is somewhat
predictions are impossible, so a high degree of error is to dependent on the study design. The outcome of a
be expected. Another particular weakness of the Delphi Delphi sequence is nothing but opinion. The results of
method is that future developments are not always the sequence are only as valid as the opinions of the
predicted correctly by iterative consensus of experts, but experts who made up the panel, whereas the panel
instead by unconventional thinking of amateur outsiders. viewpoints are summarized statistically rather than in
Experts tend to judge the future of events in isolation terms of a majority vote. The information obtained by
from other developments. A holistic view of future the Delphi study is only as good as the experts who
events where change has had a pervasive influence participate on the panel.
cannot be visualized easily. At this point cross-impact
analysis is of some help. While addressing the
manipulation of Delphi, the responses can be altered by References
the monitors in the hope of moving the next round Anderson, F. T. T. (1975). A modified Delphi study of the
responses in a desired direction. political feasibility of critical issues affecting educational reform
After each round, an administrator provides an in Maryland. Unpublished doctoral dissertation,
anonymous summary of the experts’ forecasts and their Virginia Polytechnic Institute and State University,
reasons for them. When experts’ forecasts have changed Blacksburg, Virginia.
little between rounds, the process is stopped and the Barnes, J. L. (1987). An international study of curricular
final round forecasts are combined by averaging. Delphi organizers for the study of technology. Unpublished
is based on well-researched principles and provides doctoral dissertation, Virginia Polytechnic Institute
forecasts that are more accurate than those from and State University, Blacksburg, Virginia.
unstructured groups. However, Turoff and Hiltz (1996) Brooks, K. W. (1979). Delphi technique: Expanding
warn that because of its emphasis upon communication, applications. North Central Association Quarterly, 53,
Delphi can be in danger of dismissal as merely a form of 377-385.
data collection, when it is much more than this. Its
iterative feedback method develops an insight, which in Cetron, M. J. (1969). Technological forecasting. New York:
its totality, is more than the sum of the parts. This Gordon and Breach.
technique is a very unethical method of achieving Dailey, A. L. (1988). Faculty consensus at a multi-
consensus on a controversial topic in group settings. It campus college through Delphi. Community/Junior
requires well-trained professionals who deliberately College Quarterly, 12, 21-26.
escalate tension among group members, pitting one Dalkey, N. C. (1967). Delphi. Santa Monica, CA: The
faction against the other, so as to make one viewpoint RAND Corporation.
appear ridiculous so the other becomes "sensible" Dalkey, N., & Helmer O. (1963). An experimental
whether such is warranted or not. application of the Delphi method to the use of
CONCLUSION experts. Management Science, 9(3), 458-467.
Flanders, F. B. (1988). Determining curriculum content for
Delphi is a method pertaining to the utilization of expert nursery/landscape course work in vocational agriculture for
opinions. Essential components of the Delphi technique the 21st Century: A future’s study utilizing the Delphi
include the communication process, a group of experts, technique. Doctoral Dissertation, University of
and essential feedback. The Delphi method has been Georgia.
used in a variety of ways in government, business, and
education. The steps for the Delphi method include Forsyth, D. R. (1990). Group dynamics (2nd ed.). Pacific
formation of a team to undertake and monitor a Delphi Grove, CA: Brooks/Cole Publishing Company.Practical Assessment, Research & Evaluation, Vol 12, No 4 7
Yousuf, Delphi Technique
Fortune, J. C. (1992, May). [Interview with Dr. Jimmie C. Reeves, G., & Jauch, L. R. (1978). Curriculum
Fortune, Professor at Virginia Tech]. Blacksburg, development through Delphi. Research in Higher
VA: Virginia Tech. Education, (8), 157-68.
Goldschmidt, P. G. (1975). Scientific inquiry or political Rieger, W. G. (1986). Directions in Delphi
critique? Remarks on Delphi assessment, expert developments: Dissertations and their quality.
opinion, forecasting, and group process by H. Technological Forecasting and Social Change, 29, 195-204.
Sackman. Technological Forecasting and Social Change, 7, Scheele, D. S. (1975). Reality construction as a product
195-213. of Delphi interaction. In H. A. Linstone & M.
Helmer, O. (1967). Systematic use of expert opinions (Report Turoff (Eds.), The Delphi method: Techniques and
No. P-3721). Santa Monica, CA: The RAND applications (pp.37-71). Reading, MA: Addison-
Corporation. Wesley Publishing Company.
Helmer, O. (1983). Looking forward: A guide to future Skutsch, M., & Hall, D. (1973). Delphi: Potential uses in
research. Beverly Hills, CA: Sage. education planning. In J. L. Barnes, (1987). An
Isaac, S., & Michael, W. B. (1981). Handbook in research international study of curricular organizers for the study of
and evaluation. San Diego, CA: EdITS Publishers. technology. Unpublished doctoral dissertation, Virginia
Polytechnic Institute and State University,
Janis, I. L. (1972). Victims of groupthink. Boston:
Blacksburg, Virginia.
Houghton-Mifflin.
Spencer-Cooke, B. (1989). Conditions of participation in
Koontz, H., & O'Donnell, C. (1976). Management: A
rural, non-formal education programmes: A Delphi
systems and contingency analysis of managerial functions (6th
study. Educational Media International, 26(2), 115-124.
ed.). New York: McGraw-Hill.
Sutherland, J. W. (1975). Architecting the future: A
Linstone, H. A. (1978). The Delphi technique. In J.
Delphi-based paradigm for normative system-
Fowlers (Ed.), Handbook of futures research (pp. 273-
building. In H. A. Linstone & M. Turoff (Eds.), The
300). Westport, CT: Greenwood Press.
Delphi method: Techniques and applications. Reading, MA:
Linstone, H. A., & Turoff, M. (1975). The Delphi method: Addison-Wesley.
Techniques and applications. Reading, Massachusetts:
Uhl, N. P. (Ed). (1983). Using research for strategic planning.
Addison-Wesley Publishing Company.
San Francisco: Jossey-Bass.
Mitchell, T. R., & Larson, J. R., Jr. (Eds.). (1987). People in
Van Dijk, J. A. G. M. (1990). Delphi questionnaires
organizations: An introduction to organizational behavior.
versus individual and group interviews: A
New York: McGraw-Hill Book Company.
comparison case. Technological Forecasting and Social
Mitroff, I. I., & Turoff, M. (1975). Philosophical and Change, 37, 293-304.
methodological foundations of Delphi. In H. A.
Weatherman, R., & Swenson, K. (1974). Delphi
Linstone & M. Turoff (Eds.), The Delphi method:
technique: Futurism in education. In J. L. Barnes,
Techniques and applications (pp. 17-35). Reading MA:
(1987). An international study of curricular organizers for
Addison-Wesley Publishing Co.
the study of technology (pp. 56-69). Unpublished
Needham, R. D. (1990). Geographic: The policy Delphi: doctoral dissertation, Virginia Polytechnic Institute
Purpose, structure, and application. The Canadian and State University, Blacksburg, Virginia.
Geographer, 34(2), 133-142.
Worthen, B. R., & Sanders, J. R. (1987). Educational
Pfeiffer, J. (1968). New look at education. Poughkeepsie, evaluation: Alternative approaches and practical guidelines.
NY: Odyssey Press. New York: Longman.Practical Assessment, Research & Evaluation, Vol 12, No 4 8
Yousuf, Delphi Technique
Citation
Yousuf, Muhammad Imran (2007). Using Experts’ Opinions through Delphi Technique. Practical
Assessment Research & Evaluation, 12(4). Available online: http://pareonline.net/getvn.asp?v=12&n=4
Author
Correspondence concerning this paper should be addressed to
Muhammad Imran Yousuf
Division of Continuing Education
University of Arid Agriculture
Rawalpindi, Pakistan.
E-mail miyphd@yahoo.com
tel/fax 54 223 4752266You can also read
