DCC / ICEx / UFMG
                                                                                    Types of Research
                      Empirical Strategies                                          
 We can identify two main types of 
                                                                                      research for empirical studies
                                                                                       Exploratory Research
                                                                                       Explanatory Research
                                  Eduardo Figueiredo                                
 The two types are complementary 
                               http://www.dcc.ufmg.br/~figueiredo                     rather than competitive
                       Exploratory Research                                         Explanatory Research
                      
 It is concerned with studying objects                       
 It is mainly concerned with quantifying 
                         in their natural settings                                    a relationship or comparing groups 
                          Letting the findings emerge from                            The aim often is to identify a cause-effect 
                            observations                                                 relationship
                      
 It has a flexible research design                           
 It is usually conducted through a 
                                                                                      controlled experiment (fixed design)
                      
 It is usually supported by qualitative                         Supported by quantitative data
                         data                                                          Promotes comparison and statistical 
                                                                                         analysis
                       Quantitative vs. Qualitative                                 Empirical Strategies
                     
 A quantitative investigation can answer                      
 The main empirical strategies are
                         How much a new inspection method                             Survey
                           decreases the number of faults found in                     Case Study
                           tests?                                                      Experiment / Quasi-Experiment
                     
 A qualitative investigation can answer                       
 Empirical strategies are not orthogonal
                         What are the sources of faults between                       Some studies may be viewed as a 
                           different inspection groups?                                  combination of strategies
                                                                                                                                          1
                                                                                   Survey
                      Survey                                                      
 A survey is an empirical strategy for 
                                                                                    collecting information from people
                                                                                     Used to describe, compare, or explain 
                                                                                       their knowledge, attitudes, and behavior
                                                                                  
 Data is collected by interviews or 
                                                                                    questionnaires
                                                                                  
 Survey is often performed in retrospect
                                                                                     Data is analyzed to derive conclusions
                       A Survey Questionnaire                                      Generalization
                     
 In a survey, a questionnaire is answered                  
 The purpose of a survey is to 
                        by a sample of developers                                  understand the population
                                                                                    Example: by interviewing 25 developers, 
                     
 Collected information can support both                         we aim to know the opinion of a population
                        quantitative and qualitative analyses
                                                                                 
 Conclusions in a survey can often be 
                     
 Be careful: surveys with many questions                     generalized
                        are tedious for respondents
                       Examples of Survey
                      
 Example 1:                                                Case Studies
                          A tool has been used for a while
                          A survey is conducted to assess its 
                           advantages and drawbacks
                      
 Example 2
                          A pool is used to determine how a 
                           population will vote in the next election
                                                                                                                                         2
                      Case Study                                                  Case Study Arrangements
                      
 Case Study is an empirical strategy                     
 A case study can be applied as a 
                         that draws on multiple sources of                         comparative research strategy
                         evidences                                                  A comparison of results using a new 
                          It relies on one instance (or small set                   method against historic data
                           of instances) within its real-life context               A sister project: one using a new method 
                                                                                     and another using the typical method
                      
 It normally aims at tracking a                              Apply to some components of a larger 
                         specific attribute or at establishing                       project and compare results with other 
                         relationships between attributes                            components
                      Confounding Factor                                          Advantages and Drawbacks
                     
 It is hard to distinguish the effects of two              
 Advantages
                       factors from each other in case studies                       Case studies are easier to plan
                        Example: A better result may be either due                  Results are more realistic
                          to a new tool or due to the user experience
                     
 Confounding factors are common in                         
 Drawbacks
                       case studies due to low level of control                      Data are hard to interpret
                                                                                     Results are difficult to generalize
                      Survey vs. Case Study
                      
 Survey is usually done in retrospect                     Experiments
                         Case study is done while a project is 
                           executed
                      
 The purpose of surveys is to 
                        understand the population
                         Case Study targets a particular project
                                                                                                                                       3
                       (Controlled) Experiment                                     Quantitative (+ Qualitative)
                        
 Experiment is an empirical strategy                      
 Experiments are almost pure 
                          that manipulates one factor (or                            quantitative
                          variable) of the studied setting                            Statistical methods are usually applied
                        
 Different treatments are applied to 
                          the variable (or variables)                              
 Qualitative data may be used to help 
                           Other variables are kept constant                        in the interpretation and conclusions
                        
 Experiments are mostly done in a 
                          laboratory
                           They require a high level of control
                       Human vs. Technology                                        Baseline
                    
 Experiments can be human-oriented or                        
 It is common to consider the current 
                      Technology-oriented                                           (or typical) situation as baseline
                    
 Human-oriented experiments                                     Baseline (control group) is one level of 
                       Humans apply different treatments to objects                   the independent variable
                       Two groups use different inspection methods               
 The new situation (evaluated group) is 
                    
 Technology-oriented experiments                               the one we want to evaluate
                       Two different tools (e.g., two testing tools) are            Another level of the independent variable
                         applied to different objects                             
 Values of other variables should stay 
                                                                                    the same, i.e., controlled
                       Quasi-Experiment                                            Case Study vs. Experiment
                      
 Quasi-Experiment is similar to                           
 Different environments
                         experiment                                                  Case studies run in real environment
                      
 However, treatments cannot be                                Experiments run in controlled environment
                         based on randomization                                  
 Experiments are more controlled
                          They emerge from characteristics                          Control is lower in a case study
                           of the subjects or objects                            
 Experiments rely on measurements and 
                          Example: it is hart to randomize                         manipulation of variables
                           programming experience in a class                         Case studies are most observational
                                                                                                                                         4