QUALITATIVE RESEARCH METHODS IN PROGRAM EVALUATION: 
                CONSIDERATIONS FOR FEDERAL STAFF 
                            
                            
                            
                Office of Data, Analysis, Research & Evaluation 
                Administration on Children, Youth & Families 
                            
                            
                            
                        May 2016 
         
                    
                    
       Table of Contents 
       EXECUTIVE SUMMARY ............................................................................................................. i 
       INTRODUCTION .......................................................................................................................... 1 
       PART ONE: DECIDING WHEN TO USE QUALITATIVE METHODS.................................... 3 
        What are Qualitative Methods? .................................................................................................. 3 
        The Role of Qualitative Methods in Program Evaluation .......................................................... 4 
         TEXT BOX 1: The Role of Qualitative Methods in Program Evaluation .............................. 6 
        Summary ..................................................................................................................................... 7 
       PART TWO:  GETTING STARTED ............................................................................................. 8 
        Researcher Experience ................................................................................................................ 8 
         Questions to ask about researcher credibility ......................................................................... 9 
        Budgeting and Time Planning .................................................................................................... 9 
         Questions to ask about time and cost estimates .................................................................... 11 
        Summary ................................................................................................................................... 11 
       PART THREE:  A BRIEF PRIMER ON QUALITATIVE METHODS ..................................... 12 
        Research Design:  The Structure of the Study .......................................................................... 12 
         Questions to ask about research design ................................................................................ 13 
        Sampling: What Gets Measured ............................................................................................... 14 
         Sampling Strategies in Qualitative Research ........................................................................ 14 
         Sample Size ........................................................................................................................... 15 
         Questions to ask about sampling........................................................................................... 16 
        Qualitative Data Collection: Methods and Other Considerations............................................. 16 
         Interviews and Focus Groups................................................................................................ 17 
         Asking Open-Ended Questions ............................................................................................. 18 
         Capturing Interview Data ...................................................................................................... 19 
         Qualitative Observation ........................................................................................................ 19 
         Document Review ................................................................................................................. 20 
         On-Site Data Collection and Researcher Bias ...................................................................... 20 
         A Few Words About Data Collection By Program Staff  ..................................................... 22 
         Questions to ask about data collection .................................................................................. 23 
        Qualitative Data Analysis ......................................................................................................... 24 
         TEXT BOX  4 Overview of the Qualitative Data Analysis Procedures ............................... 25 
         Inter-Rater Reliability ........................................................................................................... 26 
         Questions to ask about qualitative data analysis ................................................................... 26 
        Summary ................................................................................................................................... 27 
       PART FOUR: THE CREDIBILITY OF QUALITATIVE FINDINGS ....................................... 28 
        Internal Validity:  Ruling Out Alternative Explanations .......................................................... 28 
         Rival Conclusions ................................................................................................................. 29 
         Contrasting and Disconfirming/Negative Cases ................................................................... 30 
         Triangulation ......................................................................................................................... 31 
         Getting Feedback from Participants/Informants ................................................................... 32 
        Questions to ask about internal validity .................................................................................... 32 
        External Validity:  Generalizing the Findings .......................................................................... 33 
         Sample-to-population extrapolation ...................................................................................... 34 
         Case-to-case transfer ............................................................................................................. 35 
         Analytic generalization ......................................................................................................... 35 
        Summary ................................................................................................................................... 36 
         Questions to ask about generalizability ................................................................................ 36 
       CONCLUSION ............................................................................................................................. 37 
       REFERENCES ............................................................................................................................. 38