Expert judgement tools

What is expert judgement?

   On the PMBOK Guide 4th Edition, this term was mentioned several times, showing a proof that Expert Judgment has a vast importance on overall processes in project management. Example is in the process of Project Integration, expert judgment is often used to access the inputs needed to develop the project charter. Such judgment and expertise is applied into any technical and management details during this process. Therefore, expert judgment helps identify all the work including project management deliverables and those other deliverables required.
     Since in this post we are to identify the best tool use in expert judgment, let me first give you the definition of the term “expert judgment”. There’s a lot of definition on the internet, it’s up to you what you will consider first, but the thought is all the same with mine. Expert judgment is an expression on one’s or group’s opinions for finding solutions and their response are either based on their experience or knowledge or both. Anyone who has worked on a large company appreciates its importance on making good decisions. Project managers must not hesitate to ask or consult experts on different topics such as what methodology to follow or programming language to use and so on.

Expert Judgment is use for

four situations which require recourse to expert judgment by [Lannoy & Procaccia, 2001]:
- completing, validating, interpreting and integrating existing data; assessing the impact of a change,
- predicting the occurrence of future events and the consequences of a decision,
- determining the present state of knowledge in one field,
- providing the elements needed for decision-making in the presence of several options.
The uncertainty of data in Expert Judgment
Expert judgment depends on experts (knowledge, experience, motivation,…), the state of knowledge on the topic and the dialogue between experts and analyst. So, according to Cooke, the most important tool in using expert judgment is the representation of uncertainty [Cooke, 1991].

Actors in expert judgement methods

The are two kinds of actors in an expert judgement method:
- the experts, Ballay defines the expert as being the "person who has the knowledge" [Ballay, 1997],
- and the analyst who carry on the expert judgement exercise.
Advantages and disadvantages of using Expert Judgment
Expert judgment uses the experience and knowledge of experts to estimate the cost of a software project. An advantage of this method is the experience from past projects that the expert brings to the proposed project. The expert also can factor in project impacts caused by new technologies, applications, and languages. Examples of popular expert judgment techniques include the Delphi and Wideband Delphi methods. Expert judgment techniques are suitable for assessing the differences between past and future programs; and are especially useful for new or unique programs for which no historical precedent exists. However, the expert's biases and sometimes insufficient knowledge may create difficulties. It can be hard to document the factors used by the expert who contributes to the estimate. Although Delphi techniques can help alleviate bias problems, experts are usually hard-pressed to accurately estimate the cost of a new software program. Therefore, while expert judgment models are useful in determining inputs to other types of models, they are not frequently used alone in software cost estimating.

Expert Judgment Tools

The first two methods using expert judgement were developed by the RAND Corporation in the United-States after Word War II [Cooke, 1991] they are Scenario Analysis and the Delphi method.

Scenario Analysis

Herman Kahn is regarded as the father of scenario analysis [Cooke, 1991]. In The Year 2000 [Kahn & Wiener, 1967], Kahn defines scenarios as hypothetical sequences of events constructed for the purpose of focusing attention on causal processes and decision-points. They answer two kinds of questions:
- Precisely how might some hypothetical situation come about, step by step ?
- What alternatives exists, for each actor, and each step, for preventing diverting, or facilitating the process. 
          The method as applied in projecting the year 2000 works basically as follows. The analyst first identifies what he takes to be the set of basic long-terms trends. These trends are then extrapolated into the future, taking account of any theoretical or empirical knowledge that might impinge on such extrapolations. The result is termed the surprise-free scenario. The surprise-free scenario serves as a foil for defending alternative futures or canonical variations. Roughly speaking, these are generated by varying key parameters in the surprise-free scenario.
          Scenario analysis can also be used to illuminate "wild cards." For example, analysis of the possibility of the earth being struck by a large celestial object (a meteor) suggests that whilst the probability is low, the damage inflicted is so high that the event is much more important (threatening) than the low probability (in any one year) alone would suggest. However, this possibility is usually disregarded by organizations using scenario analysis to develop a strategic plan since it has such overarching repercussions.
Scenario planning is a useful way of challenging the assumptions you naturally tend to make about the situation in which your plans will come to fruition. By building a few alternative scenarios, you can foresee more unknowns that may come to pass, and therefore you will be able to plan measures to counteract or mitigate their impact.
The Delphi method
             The Delphi method was developed at the RAND corporation in the early 1950s as a spin-off of an Air Force-sponsored research project, “Project Delphi”. The original project was designed to anticipate an optimal targeting of U.S. industries by a hypothetical Soviet strategic planner. In the middle 1960s and early 1970s the Delphi method found a wide variety of applications, and by 1974 the number of Delphi studies has exceeded 10,000 [Linstone & Turoff, 1975].
            The Delphi method has undergone substantial evolution and diversification. The method was developed by mathematicians and engineers, and enjoyed considerable popularity among research managers, policy analysts, and corporate planners in the late 1960s early 1970s. By the middle of 1970s psychometricians, people trained in conducting controlled experiments with humans, began taking serious look at the Delphi methods and results. According to Cooke [Cooke, 1991], the most significant study in this regard is Sackman’s Delphi Critique (1975). As a result the whole question of evaluating expert opinion and developing methodological guidelines for its use has moved into the foreground. The Delphi exercises seem to have disappeared, and play almost no role in contemporary discussions of expert opinion.
The basic idea of the Delphi method is as follows:
• create a list of statements/questions
• have the experts give their ratings/answers/etc.
• make a report - send it out to everyone
• have the experts revise their answers
• make the second report
These make Delphi method as the best tool in securing Expert Judgment.

References:
State of the art on expert judgement.pdf
http://fast.faa.gov/pricing/c1919-5.htm#19.5.4
http://ec.europa.eu/regional_policy/sources/docgener/evaluation/evalsed/sourcebooks/method_techniques/collecting_information/delphi_survey/index_en.htm