Sheet 1: HEADER

	
	
	
	
	
	
	
	

		


		


		


		


		


		


		


		


		


		


		


		


		


	
	

		


		


		


		


		


		


		


		


		


		


		


		


		


	
	

		
Data Quality Assessment (DQA) Tool
		


	
	

		


		


		


		


		


		


		


		


		


		


		


		


		


	
	

		


		
Protocol 1:

Assessment of Data Management and Reporting Systems
		


		


		


	
	

		


		
Number of intermediate aggregation levels
		


		


		


		


		


		


		


		

 
		
		


		


		


	
	

		


		
Number of higher level intermediate aggregation sites (e.g. regional)
		


		


		


		


		


		


		


		

 
		
		


		


		


	
	

		


		
Number of intermediate aggregation sites (per region)
		


		


		


		


		


		


		


		

 
		
		


		


		


	
	

		


		
Number of service delivery points
		


		


		


		


		


		


		


		


		


		


	
	

		


		
Reset survey
		


		


		


		


		


		


		


		

 
		
		


		


		


	
	

		


		


	
	

		
Draft:  July 2008
		


	
	

		
Important notes for the use of this spreadsheet:
		


	
	

		
1.  In order to use the Data Quality Audit tools you will need to ensure that your 'macro security' is set to something less than 'high'.  With the spreadsheet open go to the 'Tools' pull-down menu and select 'Macro', then 'Security'.  Select 'medium'.  Close Excel and re-open the file.  When you open file the next time you will have to select 'Enable Macros' for the application to work as designed.

		


	
	

		
2.  On the HEADER Page (this page), please select number of intermediate aggregations levels (from the dropdown list) the data are reported through from service delivery point up to national level.  Then select the number of sampled intermediate aggregation sites (IAS) from the next dropdown list.   IAS are typically the district level health unit of the Ministry of Health.  Next, enter the number of sampled service delivery points (SDP), e.g. health units or facilities,  reporting to each intermediate site.  If you have an unequal number of SDP per district select the number corresponding to largest number of sampled SDP in a sampled district.  If SDPs report directly to national level select '0' from the IDS dropdown list.  Then select the number of SDPs in the sample.  Please see the INSTRUCTIONS page for more detail.
		


	
	

		
3.  Please enter the health facility, or site, names into the sheet labeled 'Information Page'.  These names will then auto-populate the appropriate forms in the spreadsheet and help ensure a well organized assessment and good quality data.
		


	

 
 
 


Sheet 2: Information_Page

	
	
	

		
BACKGROUND INFORMATION
		
	

		


		


		


		


		


		


		


		


		


		


		


		


		


		


		


	
	

		
Name of Orgnization Implementing the DQA
		


		
	

		


		


		


		


		


		


		


		


		


		


		


		


		


		


		


	
	

		
Country
		


		
	

		


		


		


		


		


		


		


		


		


		


		


		


		


		


		


	
	

		
Program Area(s)
		


		
	

		


		


		


		


		


		


		


		


		


		


		


		


		


		


		


	
	

		
Indicator(s)
		


		
	

		


		


		


		


		


		


		


		


		


		


		


		


		


		


		


	
	

		
Grant Number(s)
		


		
	

		


		


		


		


		


		


		


		


		


		


		


		


		


		


		


	
	

		
M&E Management Unit at Central Level
		
	

		
Name of Site
		
Date of Audit
		
	

		
1-
		


		


		
	

		
Service Delivery Points (SDPs)
		
	

		
Name of Site
		
Date of Audit
		



Sheet 3: INSTRUCTIONS

	
	

		
INSTRUCTIONS (Protocol 1 - System's Assessment)
		
	

		
Note:  The Audit Team should read the document "Data Quality Audit:  Guidelines for Implementation" in addition to these instructions prior to administering this protocol.
		
	

		


		
	

		
OBJECTIVE
		
	

		
The objective of this protocol is to review and assess the design and functioning of the Program/Project’s data management and reporting system to determine if the system is able to produce reports with good data quality. 
		
	

		


		
	

		
CONTENT
		
	

		
The review and assessment includes several steps, including a preliminary desk review of information supplied by the Program/Project, and follow-up reviews at the M&E Unit, at selected Service Delivery Points (SDPs) and Intermediate Aggregation Levels (IALs).  Each level (i.e. M&E Unit, SDPs and IALs) has a separate worksheet in the Excel file. 

The protocol is arranged into five sections: 1 - Organisation of M&E Structure and Functions; 2 - Definitions, Policy and Guidelines; 3 - Data-collection and Reporting Forms / Tools; 4 - Data Management Processes; 5 - Links with National Reporting System. These assessment areas are critical to evaluating whether the Data Management and Reporting System is able to produce quality data.  Questions related to the five areas are asked at different levels of the reporting system (see the "All Questions" worksheet for the list of all questions).  
		
	

		
Modifying the Excel template

Selecting Levels and Sites
The Systems Assessment Excel template is designed to follow the DQA sampling methodology with regard to the selection of Intermediate Aggregation Levels and Service Delivery Sites.  On the "HEADER" tab, the Auditing Team can select the number of Intermediate Aggregation Sites, and the number of Service Delivery Sites reporting to each Intermediate Aggregation Site, according to the understanding reached with the Organization Commissioning the DQA.  In the event there more than, or less than, one intermediary level to the flow of data from service delivery level to the national level, the required numbers of intermediate levels and sites per level can be selected.  First select the number of Intermediate Aggregation Levels (e.g. region or district).  If the data passes through both the region and the district, select two levels.  If the data only passes through the district before being sent to national level, select one level.  In some countries the data is reported from the service delivery point directly to national level.  In this case, select zero intermediate levels.  

Next select the number of sites within each intermediate level.  The most likely sampling scheme calls for the random selection of three
clusters (intermediate sites), and three Service Delivery Sites within each cluster.  In the event of two intermediate levels, the sampling
scheme will most likely be two regional clusters, with two district clusters within each region.  Then two service points will be selected
within each district cluster for a total of eight Service Delivery Sites.  Modifications to this sampling scheme may become necessary due
to constraints of time, distance and resources.  All changes to the methodology should be agreed upon with the Organization
Commissioning the DQA before the field visits. 

		
	

		
Editing the content of the Excel Template
The Excel Templates should be modified with care.  To enable the selective addition or removal of tabs for the different sites and 
levels the templates have been programmed in VBA (Visual Basic for Applications).  Modifications to the worksheets can disrupt the
programming and render the template inoperable.  However, it is possible to make certain changes that will make the template more 
useful during the DQA, and more descriptive of the information system.  For example, additional cross-checks and spot-checks can be
added to individual worksheets provided they are added at the bottom of the worksheet. Please do not 'insert' rows or columns in the
middle of the sheet as this could alter the calculation of certain performance indicators.  Other additions to the sheets can also be 
made at the bottom of the worksheet.

Up to six Service Delivery Sites within four Intermediate Aggregation Sites can be selected (i.e. 24 sites) so it is unlikely that service
points or intermediate sites will need to be added to "Information Page".   Simply select the desired number of each from the dropdown
lists on the “HEADER” page.  These selections can be modified later if need be.

It is acceptable to rename worksheet tabs as needed.

The worksheets are protected to guard against inadvertent changes to the contents .  To edit the contents of cells, unprotect the
worksheets by selecting 'Protection' from the 'Tools' pull-down menu.  Then select 'Unprotect'.  Remember to Re-protect the worksheet
after making the modifications.

		
	

		
AUDITORS INSTRUCTIONS FOR USING THE PROTOCOL
		
	

		
Because the quality of reporting of the M&E system may vary among indicators and may be stronger for some indicators than others, the Audit Team will need to fill out separate "Protocol 1:  Assessment of Data Management and Reporting Systems" for each indicator audited through the DQA.  However, if indicators selected for auditing are reported through the same data reporting forms and systems (e.g. ART and OI numbers for HIV/AIDS, or TB Detection and Successfully Treated numbers), only one protocol may be completed for these indicators.
		
	

		
Desk Review

Based on preliminary documentation received from the M&E Unit (refer to the guidelines for the list of documents requested in advance from the country), the Audit Team should begin completing the M&E Unit worksheet .  
		
	

		
Site visits

Once in the country, the Audit Team will need to administer the protocol at:

-  the M&E Management Unit (at central level);
- selected Intermediate Aggregation Levels - e.g. Regional or District offices (if any);
- sample Service Delivery Points.

The Audit Team should complete the relevant worksheets for each site visited.  The worksheets include space for comments and for following up on the supplied documentation.  The Audit Team will need to address unanswered questions and obtain documentary support for those questions at all levels. 
		
	

		
Instructions for filling in the columns of the checklist:

A list of questions appears on the left on each worksheet (column B).  The Audit Team should read each question (statement) and  complete columns C, E and F.  Column C has a pull down menu (in the right bottom corner) for each question with response categories: "Yes - completely", "Partly", "No - not at all" and "N/A".  Column E is for auditor notes and column F should be filled in with "Yes" if the Audit Team deems that issues raised by the response to a question(statement) are sufficient to require a "Recommendation Note" to the Program/project.     
		
	

		
Instructions for filling out Audit Summary Question Sheet:

To answer these questions, the Audit Team will have the completed worksheets for each site visited and the summary table and graph of findings from the protocol (generated on the "Summary_Table" and "Spider_Graph" worksheets) .  Based on these information, the Audit Team should use its judgment to develop an overall response to the Audit Summary Questions.    
		
	

		


		
	

		
OUTPUTS
		
	

		
Based on all responses to the questions, a Summary Table will be automatically generated, as will a Summary Graph of the strengths of the data management and reporting system.  The results generated will be based on the number of “Yes, completely”, “Partly” and “No, not at all” responses to the questions in the worksheets for the M&E Unit, the IALs and the SDPs.
		
	

		
Interpretation of the Output:

The scores generated for each functional area on the Service Delivery Level, Intermediate Aggregation and M&E Unit pages are an average of the responses which are coded 3 for "“Yes, completely”, 2 for “Partly”, and 1 for “No, not at all”.  Responses coded "N/A," 
or "Not Applicable," are not factored into the score.  The numerical value of the score is not important; the scores are intended to be compared across functional areas as a means to prioritizing system strengthening activities.  That is, the scores are relative to each 
other and are most meaningful when comparing the performance of one functional area to another.  For example, if the system scores 
an average of 2.5 for 'M&E Structure, Functions and Capabilities' and 1.5 for 'Data-collection and Reporting Forms / Tools,' one would reasonably conclude that resources would be more efficiently spent strengthening 'Data-collection and Reporting Forms / Tools' rather than 'M&E Structure, Functions and Capabilities'.  The scores should therefore not be used exclusively to evaluate the information system.  Rather, they should be interpreted within the context of the interviews, documentation reviews, 
data verifications and observations made during the DQA exercise.