Policy Analysis 2.0 – Structured Collaboration Using Projected Outcome Matrices from the 
                       Eightfold Path 
                           
                      By Zev Winkelman 
       
                A dissertation submitted in partial satisfaction of the 
                    requirements for the degree of 
                      Doctor of Philosophy 
                          in 
                       Public Policy 
                         in the 
                      Graduate Division 
                         of the 
                   University of California, Berkeley 
                           
                      Committee in charge: 
                    Professor Michael Nacht, Chair 
                    Professor Eugene Smolensky 
                     Professor Steve Weber 
                           
                        Spring 2012 
       
                   
                        Abstract
       
        Policy Analysis 2.0 – Structured Collaboration Using Projected Outcome Matrices from the 
                       Eightfold Path 
                         by 
                       Zev Winkelman 
                  Doctor of Philosophy in Public Policy 
                   University of California, Berkeley 
                    Professor Michael Nacht, Chair 
                           
                           
      The Eightfold Path is a methodology for policy analysis taught at the Goldman School of Public 
      Policy, at the University of California, Berkeley. At the center of this methodology is a matrix in 
      which each cell represents the projected outcome of the policy alternative corresponding to the 
      cell’s row, as measured by the criterion corresponding to the cell’s column. Policy analysts use 
      this methodology to explore complex policy problems, consider the tradeoffs between different 
      alternatives, and orient decision makers as to the choices they face.  A similar methodology has 
      been developed for the analysis of competing hypotheses (ACH) regarding intelligence 
      assessments of streams of evidence. The ACH method, also based on a matrix, represents 
      evidence, hypotheses, and analyst’s assessments of the consistency of a particular piece of 
      evidence with a given hypothesis. The ACH methodology has been implemented as open source 
      software that enables collaboration between intelligence analysts and contributes to more robust 
      analysis. This dissertation starts with the open source implementation of ACH, and adapts it to fit 
      the Eightfold Path methodology. Debates from a 2009 Congressional hearing on sunsetting 
      provisions from the USA PATRIOT Act are then used as a case study to demonstrate that 
      benefits from the ACH system can also be captured by policy analysts using the Eightfold Path. 
                   
                          1 
       
       
       
       
       
       
       
       
       
       
       
       
                For both of my grandfathers, whom I never met, 
              both of my grandmothers, who spoiled me with their affection, 
            and my parents, without whom none of this would have been possible. 
                        Thank you.       
                          i 
       
      Table of Contents 
      Abstract ........................................................................................................................................... 1 
      I. Introduction ................................................................................................................................. 1 
       Motivation ................................................................................................................................... 2 
        Case Study ............................................................................................................................... 2 
        Methodology ............................................................................................................................ 3 
       Findings ....................................................................................................................................... 5 
       Outline ......................................................................................................................................... 5 
      II. Language Models ....................................................................................................................... 6 
       Manual Coding ............................................................................................................................ 6 
       Inter coder Reliability.................................................................................................................. 7 
       Automation .................................................................................................................................. 7 
      III. Visualizations ............................................................................................................................ 8 
       Reference Design ........................................................................................................................ 9 
       Coverage Analysis....................................................................................................................... 9 
       Revised Design .......................................................................................................................... 11 
      IV. Application ............................................................................................................................. 19 
       Process ....................................................................................................................................... 19 
        The Eightfold Path Ontology ................................................................................................. 19 
       Individual Matrices ................................................................................................................... 21 
        Policy Matrix: Kris Testimony .............................................................................................. 21 
        Policy Matrix: Spaulding Testimony ..................................................................................... 30 
        Policy Matrix: Wainstein Testimony ..................................................................................... 34 
        Policy Matrix: Graves Testimony .......................................................................................... 38 
        Policy Matrix: Fine Testimony .............................................................................................. 42 
       Group Matrix ............................................................................................................................. 46 
        Linking Alternatives and Criteria ....................................................................................... 46 
       The Group View ........................................................................................................................ 50 
      V. Conclusions .............................................................................................................................. 51 
       Meeting Design Objectives ....................................................................................................... 51 
       Content ...................................................................................................................................... 58 
        Enumerating Issues of Disagreement .................................................................................... 58 
                          ii