Time Calculation In Excel Sheet 46293 | Collaborator Roi 04 01 18

Partial file snippet.

Sheet 1: ROI Calculation Worksheet

	
	
	
	
	
	
	
	
	
	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		 Collaborator ROI Worksheet
		

		

		

		

		

		

	
	
		

		

		

		

		

		

	
	
		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		How can you quantify the benefits of a code review tool? This spreadsheet provides a method to calculate how much time and money 
		
	
		Collaborator is likely to save your team. Fill in your own values in the orange cells to get a personalized calculation.
		
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		We find the following two metrics clear and easy to track. They comprise the focal point of this ROI worksheet:
  ▪  Developer time saved by using  Collaborator to do reviews - Step 1 below
  ▪  Number of bugs found - we use this number to compare the cost to find and fix bugs discovered in QA or after product release with the cost to find and fix bugs found by Dev 
      during during code review) - Step 2
		
	
		 
		

		

		

		

		

		

		

		

		

		

		

	
	
		Background
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  The default values here assume 25 person team with 2 people doing each review for a total of 40 reviews per month (a fairly low review rate).  
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  Cost to fix defects comes from industry standard information and SmartBear customer benchmarks.
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  This ROI is designed to be customized to match your team's review needs. We've provided default values as guidelines in case your own numbers are not available.
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Step 1:    Enter data from your existing review process
		

		

		

		

		

		

		

	
	
		Number of people per review:
		2
		

		

		

		

		

		

		

		

		

		

	
	
		Total number of developers:
		25
		

		

		

		

		

		

		

		

		

		

	
	
		Defects found per review:
		8
		

		

		

		

		

		

		

		

		

		

	
	
		Hours spent per review (per person):
		1
		

		

		

		

		

		

		

		

		

		

	
	
		Labor cost per hour:
		50
		

		

		

		

		

		

		

		

		

		

	
	
		# Reviews Per Month:
		40
		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Step 1 Calculation: SPEND LESS TIME ON REVIEWS
		

		Current Process
		

		With  Collaborator
		

		Additional Information
		
	
		With Collaborator, developers do reviews faster and spend less time on reviews.
		

		

		

		

		

		

		
	
		

		

		

		

		

		

		

		
	
		Number of people per review
		

		2
		

		2
		

		

		
	
		Defects found per review
		

		8
		

		8
		

		

		
	
		Hours spent per review (per person)
		

		1
		

		0.83
		

		Studies show developers find a bug every 10-15 min with  Collaborator - avg 12.5 min per bug or find 4.8 bugs/hr. Assuming you're finding the same number of bugs, this value shows how long it would take to find them with Collaborator. Equals Defects Found / 4.8 bugs per hour / Number of Developers
		
	
		Total hours spent per review (all participants)
		

		2
		

		1.67
		

		Equals Number of reviewers * Hours spent per review
		
	
		

		

		

		

		

		

		

		
	
		Labor cost per hour
		

		 $50.00 
		

		 $50.00 
		

		Hourly cost of a developer
		
	
		# Reviews Per Month
		

		40
		

		40
		

		

		
	
		

		

		

		

		

		

		

		
	
		Avg review time to find one bug
		

		0.25
		

		0.21
		

		Average review hours spent for a developer to find one bug. 
12.5 min per bug inspection rate/60 min
		
	
		Cost to Find Bugs
		

		$4,000
		

		$3,333
		

		Equals Total hours spent per review (all participants) * Human cost per hour * Number of reviews per month
		
	
		Cost per Defect
		

		 $12.50 
		

		 $10.42 
		

		Equals Total cost to find bugs / # Defects found per review / # reviews per month
		
	
		MONTHLY COST SAVINGS
		

		

		

		$667
		

		Equals Cost to find bugs w/ current process - Cost to find bugs w/ CC
		
	
		ANNUAL COST SAVINGS
		

		

		

		$8,000
		

		

		
	
		Percent Cost Savings
		

		

		

		17%
		

		Equals Cost to Find Bugs with Collab / Cost to find bugs without
		
	
		Percent Time Savings
		

		

		

		17%
		

		

		
	
		

		

		

		

		

		

		

		
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Step 2:    Enter data on your cost to fix defects 
		

		

		

		

		

		

		

		

	
	
		Cost to fix defects in Dev:
		 $25 
		

		

		

		

		

		

		

		

		

		

	
	
		Cost to fix defects in QA:
		 $200 
		

		

		

		

		

		

		

		

		

		

	
	
		Cost to fix defects at Customer:
		 $1,000 
		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Step 2: FIND MORE DEFECTS IN DEV
		

		Current Process
		

		With Collaborator
		

		Additional Information
		
	
		It's less expensive to find bugs in Development than it is in QA or once the product reaches a customer. With Collaborator, you'll find more bugs. This calculation looks at the cost of finding & fixing defects if they are not found in code review, but instead progress to QA and customers. 
		
	
		

		

		

		

		

		

		

		
	
		% more defects found and fixed using Collaborator
		

		N/A
		

		30%
		

		One customer surveyed (Customer B below) found 18% more defects, while another (Customer A) found 100% more. We'll choose 30% as a default but feel free to change it. If your current code review process is not finding many bugs, then you should probably increase this value. 
		
	
		# additional bugs that slip through a review to QA and customers without Collaborator
		

		2.4
		

		0
		

		Equals Defects found per review (a Step 1 input) * % more defects found and fixed using CC
		
	
		Cost to fix defects in Dev
		

		$25
		

		$25
		

		See the Data from Customer Use below for more detail about these default numbers. We assume the cost to fix defects is the same with and without Collaborator.
		
	
		Cost to fix defects in QA
		

		$200
		

		$200
		

		Cost to fix defects in QA is defined in Step 2.
		
	
		Cost to fix defects at Customer
		

		$1,000
		

		$1,000
		

		Cost to fix defects that have reached customers is defined in Step 2.
		
	
		Percentage of bugs found in QA vs Customer
		

		80%
		

		80%
		

		A typical distribution is that 80% of bugs that are not caught in development are generally caught in QA, while the remaining 20% pass through to the customer. See Best Kept Secrets of Peer Code Review for more details.
		
	
		Weighted Avg Cost to find and fix an add'l defect
		

		$360
		

		

		

		Determines the Avg cost to find and fix a defect in QA & in customers' hands using dollar values entered in step 2 and the percentage of bugs you expect to find in QA vs at customers (entered above). Feel free to insert your own number here.
		
	
		

		

		

		

		

		

		

		
	
		Cost to find and fix additional defects Per Review
		

		$864
		

		$0
		

		Cost per review to find and fix additional defects that made it past dev. Equals
# of addl' bugs that slip through a review * Weighted Avg. Cost
		
	
		

		

		

		

		

		

		

		
	
		Monthly Cost to find and fix bugs that progressed beyond Dev
		

		$34,560
		

		$0
		

		Cost to find and fix bugs that slipped through to QA and customers as a result of not finding them during review. Equals Cost to find and fix add'l defects per review * number of reviews per month
		
	
		

		

		

		

		

		

		

		
	
		MONTHLY COST SAVINGS
		

		

		

		$34,560
		

		

		
	
		ANNUAL COST SAVINGS
		

		

		

		$414,720
		

		

		
	
		

		

		

		

		

		

		

		
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		COST OF COLLABORATOR
		

		Current Process
		

		With Collaborator
		

		Additional Information
		
	
		

		

		

		

		

		

		
	
		Base License Fee
		

		

		$8,495
		

		Floating license cost = $1699 * (1/5) Total Number of Developers. Includes first yr maintenance.
		
	
		Cost per month
		

		

		$708
		

		Cost per month during first year of usage.
		
	
		

		

		

		

		

		

		
	
		Annual Maintenance Fee
		

		

		$1,954
		

		Maintenance renews at ~23% of the initial license cost
		
	
		Total License Cost over 3 years
		

		

		$12,403
		

		Base License fee plus 2 subsequent years of maintenance fees
		
	
		

		

		

		

		

		

		
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		SUMMARY: Save Review Time and Find Addititional Defects using  Collaborator (Results of Steps 1-3 combined with  Collaborator cost)
		

		

		

		

		

		

		

		

		

		

		

	
	
		Derived ROI Values
		

		Current Process
		

		With  Collaborator
		

		Additional Information
		
	
		Number of Hours Saved in Review Time Monthly
		

		

		

		13
		

		Equals (Total hours spent currently - Total hours spent using Collaborator) * Number of Reviews per month
		
	
		Money Saved Finding Defects Sooner - Per Month
		

		

		

		$34,560
		

		Money saved per month, based on the fact the developers find more bugs when using Collaborator.
		
	
		Number of Additional Bugs Found Monthly
		

		

		

		96
		

		Equals # Add'l bugs that slip through to QA & customers when not using  Collaborator * # of reviews per month
		
	
		Total Cost Savings - Monthly
		

		

		

		$34,519
		

		Equals Monthly Cost Savings Associated w/ Review Time + Monthly Cost Savings Associated w/ Addl Defects found - Cost of Collaborator. Note that we've converted time into dollars here for consistency and to determine the ROI.
		
	
		Total Cost Savings - Annually
		

		

		

		$414,225
		

		Annual Cost Savings Associated w/ Review Time + Annual Cost Savings Associated w/ Addl Defects found - Cost of Collaborator
		
	
		Total Cost Savings - 3 Years
		

		

		

		$1,255,757
		

		Cost Savings associated with use of Collaborator over a 3 year period. (Taking into consideration an annual maintenance fee of 22% for years 2 and 3)
		
	
		

		

		

		

		

		

		

		
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Data from Customer Use
		
	
		
	
		
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Customer A
		
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Customer A is a national division of a multi-national corporation, with sales in the $100-300M/year range. Their division has hundreds of programmers working on a variety of projects, 
		

		

		

		

		

		

		

		

		

		

		

	
	
		located across the country.
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		This SmartBear customer previously conducted review meetings on some of its code, when time permitted or contracts dictated. These meetings tied up a team for 1.5-2 hours at a time. 
		

		

		

		

		

		

		

		

		

		

		

	
	
		Then they deployed a Collaborator pilot project that was so successful that now reviewing ALL code using Collaborator is mandatory.
		

		

		

		

		

		

		

		

		

		

		

	
	
		Using Collaborator, Customer A's programmers now spend 20-30 min reviewing code, for a time savings of about 75%.
		

		

		

		

		

		

		

		

		

		

		

	
	
		They estimate they find about twice as many bugs using Collaborator.
		

		

		

		

		

		

		

		

		

		

		

	
	
		Overall results: Find twice as many bugs in 25% of the time. 
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Other useful metrics:
		

		

		

		

		

		

		

		

		

		

		

	
	
		This customer calculates the cost of a programmer's time at $60/hr (which we consider lower than average).
		

		

		

		

		

		

		

		

		

		

		

	
	
		If each programmer now spends 10 hours per project reviewing code rather than 40, that saves $1800 per programmer per project.
		

		

		

		

		

		

		

		

		

		

		

	
	
		This customer estimates that if a bug costs $X to fix when found in Development, that same bug costs 10-12X to fix when found during QA and 30X to fix when found after the product has shipped.
		

		

		

		

		

		

		

		

		

		

		

	
	
		Let's give a value to X: 
		$25 
		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Since Collaborator finds ~4.8 bugs/hour, assume in those 10 hours each programmer finds 48 bugs. By their admission, that's twice as many as they would otherwise have found.
		

		

		

		

		

		

		

		

		

		

		

	
	
		So if they review for 10 hours, they're finding an additional 24 bugs per programmer per project.
		

		

		

		

		

		

		

		

		

		

		

	
	
		Additional cost that would have been incurred when finding/fixing those 24 bugs in QA and in customers' hands (assume 80/20 split):
		

		

		

		

		

		

		

		$8,880 
		

		

		

	
	
		That's a SAVINGS PER PROGRAMMER PER 10 HRs of Review
		

		

		$8,880 
		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Customer B
		
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Customer B is a $2-3B/year multi-national software company. They tried Collaborator in a four-month pilot project with about 50 developers and compared the results to their existing code 
		

		

		

		

		

		

		

		

		

		

		

	
	
		review process. This pilot program was so successful that the company deployed Collaborator to more than 1000 developers.
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Results
		

		

		

		

		

		

		

		

		

		

		

	
	
		The company calculated that Collaborator would save them more than $800,000 per year vs. their existing review process. When subtracting out the licensing and approximate 
		

		

		

		

		

		

		

		

		

		

		

	
	
		implementation costs, the total ROI (for the first year) is about $720,000 – but that includes the entire purchase price of the software!  And, they were able to review 100%, up from 60%.
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  As a benchmark, their cost of a developer's time is $63.
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  This customer found 18% more bugs during code review than with current process. 
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  That 18% broke down as follows: 16% were bugs previously found during QA while 2% were bugs previously found at customer.
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  With Collaborator, they review code 33% faster than with their old process  (400 Lines of Code per hour vs 300).
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Costs to Fix Bugs from Various Sources
		
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		According to one customer, costs to find and fix bugs were   
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  During Development: $275 total cost to find and fix bugs during the Development phase. They found 11 bugs in Development. Cost per bug found in Dev = $25
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  During QA: $33,000 total cost to find and fix bugs in QA. They found 163 bugs. Cost per bug found in QA = $202 / bug  (which is about 8X the cost per bug in Dev)
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  Once released to customers: $141,000 total cost to fix bugs that reach customers. Customers found 141 bugs. Cost per bug found after release = $1000/bug (about 40X the cost per bug in Dev)
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Let's assume 100 bugs were found and fixed in Code Review. Of those, 85 would otherwise have been found and fixed in QA, and 15 would otherwise have been found and fixed
		

		

		

		

		

		

		

		

		

		

		

	
	
		after release to customers. Using the customer data above:
		

		

		

		

		

		

		

		

		

		

		

	
	
		Cost to find & fix bugs during Dev
		 $25 
		

		

		

		

		

		

		

		

		

		

	
	
		Cost to find & fix bugs in QA
		 $202 
		

		

		

		

		

		

		

		

		

		

	
	
		Cost to find & fix bugs released to customers
		 $1,000 
		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Cost to find & fix 85 bugs in QA and 15 at customer
		

		 $32,000 
		

		

		

		

		

		

		

		

		

	
	
		Cost to find & fix 100 bugs found during Dev w/code review
		

		$2,500
		

		

		

		

		

		

		

		

		

	
	
		Cost savings for every 100 bugs that exist in the code
		

		$29,500
		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Another Set of Industry Data
		

		

		

		

		

		

		

		

		

		

		

	
	
		Note: This information is outlined in detail in the free book, Best Kept Secrets for Peer Code Review, available at www.CodeReviewBook.com.
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Costs to Find and Fix Bugs
		One SmartBear customer
		HP Study*
		Costs for Customer detailed in book
		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

	
	
		In Dev
		X
		X
		X
		

		

		

		

		

		

	
	
		In QA
		10-12X
		

		8X
		

		

		

		

		

		

	
	
		Once Product is Released
		30X
		100X (HP)
		40X
		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		*See Best Kept Secrets of Peer Code Review for more info on the HP study. As a manufacturing company for B2C products, HP has a very high cost to fix bugs once they're in the field.
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Background, Benefits and Facts
		
	
		
	
		
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		SmartBear Software has done an extensive study on the effectiveness of tool-assisted code review, and we’ve collected specific ROI data from our customers. We’re sharing this information 
		

		

		

		

		

		

		

		

		

		

		

	
	
		to help you decide whether our Collaborator tool for code review makes sense – and saves dollars – for your team.
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Code Review with Collaborator helps development teams…
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  Find more defects in less time, which equates to better product quality and cost savings
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  Collaborate and communicate to write better code
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  Improve overall team knowledge level, resulting in better code quality
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  Automatically captures review metrics for easy reporting, so you can see the benefits
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Most of all, code reviews aren’t tedious – they can even be fun! With Collaborator, developers actually do them – quickly, efficiently, and at a time that’s convenient for them.
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		Facts
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  While this number can vary widely based on many factors, the industry standard rate for finding bugs during code review is about 45% (without a tool). In other words, if 10 bugs exist 
		

		

		

		

		

		

		

		

		

		

		

	
	
		       in a piece of code, developers are likely to find 4.5 of them during code review, rather than later during the QA phase or after the product is released to customers.
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  In our case study at Cisco, which spanned 50 developers, 2500 reviews, and 3.2 million lines of code, we found that with Collaborator, developers are more likely to find 70-90% 
		

		

		

		

		

		

		

		

		

		

		

	
	
		      of the bugs (almost twice as many), in half the time.
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  We found lightweight code review to be equally effective at finding bugs as heavyweight processes, but substantially more efficient, with reviews taking 1/5th - 1/7th the time of their
		

		

		

		

		

		

		

		

		

		

		

	
	
		       heavyweight counterparts.
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  Across four studies of heavyweight inspections, the average defect detection rate was 2.6 defects per hour; our reviews were seven times faster. This time savings isn’t surprising, 
		

		

		

		

		

		

		

		

		

		

		

	
	
		      since our reviews didn’t include two hour inspection meetings with 3-5 participants.
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		How is Collaborator so much easier and more efficient than other code review methods?
		

		

		

		

		

		

		

		

		

		

		

	
	
		The product is designed specifically to facilitate code review, simplify the process and save time:
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  Differences between versions ("diffs") are highlighted so you can see them – programmers don’t have to shuffle through a bunch of printouts
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  You can comment directly on the code – in real-time (chat-style) or when it’s convenient
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  You can review on your own schedule, at your convenience. Days are not disrupted with meetings, and meetings don’t get sidetracked onto other topics
		

		

		

		

		

		

		

		

		

		

		

	
	
		  ▪  Since you go straight to the designated area of change, reviews are more accurate.
		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

		

		

		

	



Sheet 2: Summary

	
	
	
	
	
	
	
	
		

		

		

		

		

		

		

		
 
		
		
	
		Collaborator ROI Summary
		

		

		

		

		

		

		
	
		Collaborator vs Review Using Other Methods
		

		

		

		

		

		

		
	
		

		

		

		

		

		

		

		
	
		

		

		

		

		

		

		

		
	
		

		

		

		

		

		

		

		
	
		

		

		

		

		

		

		

		

		

	
	
		This summary is based on your review data and presents an accurate overview of the ROI you can expect with Collaborator
		

		

		

		

		

		

		

		

	
	
		
 
		
		

		
 
		
		
	
		

		

		

		

		

		Total Savings
		

	
	
		

		

		

		

		

		Time
		

		Defect
		

	
	
		

		

		

		

		

		 $667 
		 Monthly 
		 $34,560 
		

	
	
		

		

		

		

		

		 $8,000 
		 Annually 
		 $414,720 
		

	
	
		

		

		

		

		

		

		*Details below
		

		

	
	
		

		

		

		

		

		

		

		

		

	
	
		SUMMARY: Save Review Time and Find Addititional Defects using Code Collaborator (Results of Steps 1, 2, and 3 on ROI Calculation Worksheet)
		

		

		

		

		

		

		

		

	
	
		Savings Using Collaborator
		

		Monthly
		Annually
		Over 3 Years
		

		

		

		Additional Information
	
	
		

		

		

		

		

		

		

		

		

	
	
		Number of Hours Saved in Review Time
		

		13
		160
		480
		Collaborator reviews are faster and more efficient than other review types, so it saves developers many hours per release cycle. This number is calculated in the ROI Calculation Worksheet Summary section using (Total hours spent currently - Total hours spent with Collaborator) * Number of Reviews per month.
		
	
		Number of Additional Bugs Found
		

		96
		1152
		3456
		Collaborator helps teams find this many additional bugs per month that would otherwise have slipped through to QA and to customers. This value comes from the Number of additional bugs Code Collaborator helps find per review times the Number of reviews conducted (see Step 2 of the Calculation Worksheet to identify the percent of additional bugs found using Collaborator.)
		
	
		Cost Savings realized from spending less time on code reviews
		

		 $667 
		 $8,000 
		 $24,000 
		Collaborator allows robust reviews to be completed in less time, freeing your developers to focus on other projects or higher-value activities. 
		
	
		Money Saved finding defects in review vs. in QA or after release to customers
		

		 $34,560 
		 $414,720 
		 $1,244,160 
		By helping teams find bugs during Development rather than QA or after release (when they are much more expensive to address), Collaborator saves this amount of money, based on the fact the developers find more bugs when using Collaborator. This value is calculated in Step 2 of the Calculation Worksheet using Cost to find and fix additional defects per month times Number of reviews conducted per month.
		
	
		Total Cost Savings
		

		 $34,519 
		 $414,225 
		 $1,242,675 
		This Total Cost Savings represents the total amount Collaborator is likely to save your team based on the inputs given. It includes two cost-saving components: Cost (time) saved doing reviews and money saved finding defects during code review rather than in QA or after release to customers. The calculation comes from the Worksheet Summary and is comprised of Cost Savings Associated w/ Review Time plus Cost Savings Associated w/ Addl Defects found minus Cost of Code Collaborator. We took into consideration an annual maintenance fee of ~23% for years 2 and 3. Note that we've converted time into dollars here for consistency and to determine the ROI. 
		
	
		

		
	
		

		

		

		

		

		

		

		

		

	
	
		In 13 fewer hours over one month, Collaborator helped developers find 96 additional bugs at a cost savings of $667 in review
		

		

		

		

		

		

		

		

	
	
		time alone (from ROI Calculation Worksheet Step 1).  When you include the higher cost to fix bugs after Development, Collaborator saved $34519.
		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

	
	
		Assumptions & Background
		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

	
	
		  ▪  This ROI calculates data for two separate Code Collaborator value propositions: 1. Spend less time doing reviews  2. Find more defects in the time spent (while defects are still in Dev).
		

		

		

		

		

		

		

		

	
	
		  ▪  The default values used here assume 25 person team with 4 people doing each reviews for a total of 100 reviews per month (a fairly low review rate).  
		

		

		

		

		

		

		

		

	
	
		  ▪  As a benchmark, Smart Bear's team of 5 developers does 75 reviews/month. For a team of 25, that review frequency equal 375 reviews per month. 
		

		

		

		

		

		

		

		

	
	
		  ▪  Cost to fix defects comes from industry standard information and Smart Bear customer benchmarks (see ROI Calculation Worksheet for more details)
		

		

		

		

		

		

		

		

	
	
		  ▪  The spreadsheet expects input for Number of people per review, Defects found per review, Hours spent per review (per person), Labor cost per hour, # Reviews conducted
		

		

		

		

		

		

		

		

	
	
		    per month, costs to fix defects in Dev, QA, and at Customers, % of bugs found in QA vs at customers, and % more defects found using Code Collaborator.
		

		

		

		

		

		

		

		

	
	
		 
		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

	
	
		Results
		

		

		

		

		

		

		

		

	
	
		Collaborator automatically captures metrics than can produce reports, so you can tell exactly how many bugs your team is capturing and how much time they used to find those bugs.
		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

	
	
		¨   Collaborator typically pays for itself in no more than two weeks.
		

		

		

		

		

		

		

		

	
	
		¨  Even if it saved each developer one hour per month (very unlikely), Collaborator would pay for itself in 4 months (when amortizing the cost per seat over a year). 
		

		

		

		

		

		

		

		

	
	
		       And if it's not going to do that, don't buy it!
		

		

		

		

		

		

		

		

	
	
		¨  Collaborator is fast and easy enough to let your team review ALL of your code, not just 25% or 50%, which is all many teams have time to do now. That's a lot 
		

		

		

		

		

		

		

		

	
	
		       more bugs that don't go out to customers.
		

		

		

		

		

		

		

		

	
	
		¨  Developers actually DO code review when it's so simple.
		

		

		

		

		

		

		

		

	
	
		

		

		

		

		

		

		

		

		

	
	
		Note: Indirect impact costs (cost the bugs have on your business in Support, Patches, Reputation, losing market position, Lost Business, etc) are not addressed here.  
		

		

		

		

		

		

		

		

	
	
		But they only make the ROI stronger. We also did not attempt to quantify the “softer” benefits: overall improved code quality, support of best practices, improved communication,
		

		

		

		

		

		

		

		

	
	
		better collaboration, stronger team spirit, more maintainable code, knowledge spread throughout the team, and a safe way for junior programmers to contribute.

The words contained in this file might help you see if this file matches what you are looking for:

...Sheet roi calculation worksheet collaborator how can you quantify the benefits of a code review tool this spreadsheet provides method to calculate much time and money is likely save your team fill in own values orange cells get personalized we find following two metrics clear easy track they comprise focal point developer saved by using do reviews step below number bugs found use compare cost fix discovered qa or after product release with dev during background default here assume person people doing each for total per month fairly low rate defects comes from industry standard information smartbear customer benchmarks designed be customized match s needs ve provided as guidelines case numbers are not available enter data existing process developers hours spent labor hour spend less on current additional faster studies show bug every min avg bugshr assuming re finding same value shows long it would take them equals all participants reviewers hourly one average inspection human defect mo...


Collaborator ROI Summary
Collaborator vs Review Using Other Methods




*This summary is based on your review data and presents an accurate overview of the ROI you can expect with Collaborator*

				Total Savings
				Time		Defect
				$667	Monthly	$34,560
				$8,000	Annually	$414,720
					*Details below

SUMMARY: Save Review Time and Find Addititional Defects using Code Collaborator (Results of Steps 1, 2, and 3 on ROI Calculation Worksheet)
Savings Using Collaborator	Monthly	Annually	Over 3 Years				Additional Information

Number of Hours Saved in Review Time	13	160	480	Collaborator reviews are faster and more efficient than other review types, so it saves developers many hours per release cycle. This number is calculated in the ROI Calculation Worksheet Summary section using (Total hours spent currently - Total hours spent with Collaborator) * Number of Reviews per month.
Number of Additional Bugs Found	96	1152	3456	Collaborator helps teams find this many additional bugs per month that would otherwise have slipped through to QA and to customers. This value comes from the Number of additional bugs Code Collaborator helps find per review times the Number of reviews conducted (see Step 2 of the Calculation Worksheet to identify the percent of additional bugs found using Collaborator.)
Cost Savings realized from spending less time on code reviews	$667	$8,000	$24,000	Collaborator allows robust reviews to be completed in less time, freeing your developers to focus on other projects or higher-value activities.
Money Saved finding defects in review vs. in QA or after release to customers	$34,560	$414,720	$1,244,160	By helping teams find bugs during Development rather than QA or after release (when they are much more expensive to address), Collaborator saves this amount of money, based on the fact the developers find more bugs when using Collaborator. This value is calculated in Step 2 of the Calculation Worksheet using Cost to find and fix additional defects per month times Number of reviews conducted per month.
Total Cost Savings	$34,519	$414,225	$1,242,675	This Total Cost Savings represents the total amount Collaborator is likely to save your team based on the inputs given. It includes two cost-saving components: Cost (time) saved doing reviews and money saved finding defects during code review rather than in QA or after release to customers. The calculation comes from the Worksheet Summary and is comprised of Cost Savings Associated w/ Review Time plus Cost Savings Associated w/ Addl Defects found minus Cost of Code Collaborator. We took into consideration an annual maintenance fee of ~23% for years 2 and 3. Note that we've converted time into dollars here for consistency and to determine the ROI.


*In 13 fewer hours over one month, Collaborator helped developers find 96 additional bugs at a cost savings of $667 in review*
*time alone (from ROI Calculation Worksheet Step 1). When you include the higher cost to fix bugs after Development, Collaborator saved $34519.*


Assumptions & Background

▪ This ROI calculates data for two separate Code Collaborator value propositions: 1. Spend less time doing reviews 2. Find more defects in the time spent (while defects are still in Dev).
▪ The default values used here assume 25 person team with 4 people doing each reviews for a total of 100 reviews per month (a fairly low review rate).
▪ As a benchmark, Smart Bear's team of 5 developers does 75 reviews/month. For a team of 25, that review frequency equal 375 reviews per month.
▪ Cost to fix defects comes from industry standard information and Smart Bear customer benchmarks (see ROI Calculation Worksheet for more details)
▪ The spreadsheet expects input for Number of people per review, Defects found per review, Hours spent per review (per person), Labor cost per hour, # Reviews conducted
per month, costs to fix defects in Dev, QA, and at Customers, % of bugs found in QA vs at customers, and % more defects found using Code Collaborator.




Results
Collaborator automatically captures metrics than can produce reports, so you can tell exactly how many bugs your team is capturing and how much time they used to find those bugs.

¨ Collaborator typically pays for itself in no more than two weeks.
¨ Even if it saved each developer one hour per month (very unlikely), Collaborator would pay for itself in 4 months (when amortizing the cost per seat over a year).
And if it's not going to do that, don't buy it!
¨ Collaborator is fast and easy enough to let your team review ALL of your code, not just 25% or 50%, which is all many teams have time to do now. That's a lot
more bugs that don't go out to customers.
¨ Developers actually DO code review when it's so simple.

Note: Indirect impact costs (cost the bugs have on your business in Support, Patches, Reputation, losing market position, Lost Business, etc) are not addressed here.
But they only make the ROI stronger. We also did not attempt to quantify the “softer” benefits: overall improved code quality, support of best practices, improved communication,
better collaboration, stronger team spirit, more maintainable code, knowledge spread throughout the team, and a safe way for junior programmers to contribute.

Related files

Share

Help


Collaborator ROI Worksheet



How can you quantify the benefits of a code review tool? This spreadsheet provides a method to calculate how much time and money
Collaborator is likely to save your team. Fill in your own values in the orange cells to get a personalized calculation.

We find the following two metrics clear and easy to track. They comprise the focal point of this ROI worksheet: ▪ Developer time saved by using Collaborator to do reviews - Step 1 below ▪ Number of bugs found - we use this number to compare the cost to find and fix bugs discovered in QA or after product release with the cost to find and fix bugs found by Dev during during code review) - Step 2

Background
▪ The default values here assume 25 person team with 2 people doing each review for a total of 40 reviews per month (a fairly low review rate).
▪ Cost to fix defects comes from industry standard information and SmartBear customer benchmarks.
▪ This ROI is designed to be customized to match your team's review needs. We've provided default values as guidelines in case your own numbers are not available.


Step 1: Enter data from your existing review process
Number of people per review:	2
Total number of developers:	25
Defects found per review:	8
Hours spent per review (per person):	1
Labor cost per hour:	50
# Reviews Per Month:	40


Step 1 Calculation: SPEND LESS TIME ON REVIEWS		Current Process		With Collaborator	Additional Information
With Collaborator, developers do reviews faster and spend less time on reviews.

Number of people per review		2		2
Defects found per review		8		8
Hours spent per review (per person)		1		0.83	Studies show developers find a bug every 10-15 min with Collaborator - avg 12.5 min per bug or find 4.8 bugs/hr. Assuming you're finding the same number of bugs, this value shows how long it would take to find them with Collaborator. Equals Defects Found / 4.8 bugs per hour / Number of Developers
Total hours spent per review (all participants)		2		1.67	Equals Number of reviewers * Hours spent per review

Labor cost per hour		$50.00		$50.00	Hourly cost of a developer
# Reviews Per Month		40		40

Avg review time to find one bug		0.25		0.21	Average review hours spent for a developer to find one bug. 12.5 min per bug inspection rate/60 min
Cost to Find Bugs		$4,000		$3,333	Equals Total hours spent per review (all participants) * Human cost per hour * Number of reviews per month
Cost per Defect		$12.50		$10.42	Equals Total cost to find bugs / # Defects found per review / # reviews per month
MONTHLY COST SAVINGS				$667	Equals Cost to find bugs w/ current process - Cost to find bugs w/ CC
ANNUAL COST SAVINGS				$8,000
Percent Cost Savings				17%	Equals Cost to Find Bugs with Collab / Cost to find bugs without
Percent Time Savings				17%



Step 2: Enter data on your cost to fix defects
Cost to fix defects in Dev:	$25
Cost to fix defects in QA:	$200
Cost to fix defects at Customer:	$1,000



Step 2: FIND MORE DEFECTS IN DEV		Current Process		With Collaborator	Additional Information
It's less expensive to find bugs in Development than it is in QA or once the product reaches a customer. With Collaborator, you'll find more bugs. This calculation looks at the cost of finding & fixing defects if they are not found in code review, but instead progress to QA and customers.

% more defects found and fixed using Collaborator		N/A		30%	One customer surveyed (Customer B below) found 18% more defects, while another (Customer A) found 100% more. We'll choose 30% as a default but feel free to change it. If your current code review process is not finding many bugs, then you should probably increase this value.
# additional bugs that slip through a review to QA and customers without Collaborator		2.4		0	Equals Defects found per review (a Step 1 input) * % more defects found and fixed using CC
Cost to fix defects in Dev		$25		$25	See the Data from Customer Use below for more detail about these default numbers. We assume the cost to fix defects is the same with and without Collaborator.
Cost to fix defects in QA		$200		$200	Cost to fix defects in QA is defined in Step 2.
Cost to fix defects at Customer		$1,000		$1,000	Cost to fix defects that have reached customers is defined in Step 2.
Percentage of bugs found in QA vs Customer		80%		80%	A typical distribution is that 80% of bugs that are not caught in development are generally caught in QA, while the remaining 20% pass through to the customer. See Best Kept Secrets of Peer Code Review for more details.
Weighted Avg Cost to find and fix an add'l defect		$360			Determines the Avg cost to find and fix a defect in QA & in customers' hands using dollar values entered in step 2 and the percentage of bugs you expect to find in QA vs at customers (entered above). Feel free to insert your own number here.

Cost to find and fix additional defects Per Review		$864		$0	Cost per review to find and fix additional defects that made it past dev. Equals # of addl' bugs that slip through a review * Weighted Avg. Cost

Monthly Cost to find and fix bugs that progressed beyond Dev		$34,560		$0	Cost to find and fix bugs that slipped through to QA and customers as a result of not finding them during review. Equals Cost to find and fix add'l defects per review * number of reviews per month

MONTHLY COST SAVINGS				$34,560
ANNUAL COST SAVINGS				$414,720



COST OF COLLABORATOR		Current Process		With Collaborator	Additional Information

Base License Fee				$8,495	Floating license cost = $1699 * (1/5) Total Number of Developers. Includes first yr maintenance.
Cost per month				$708	Cost per month during first year of usage.

Annual Maintenance Fee				$1,954	Maintenance renews at ~23% of the initial license cost
Total License Cost over 3 years				$12,403	Base License fee plus 2 subsequent years of maintenance fees



SUMMARY: Save Review Time and Find Addititional Defects using Collaborator (Results of Steps 1-3 combined with Collaborator cost)
Derived ROI Values		Current Process		With Collaborator	Additional Information
Number of Hours Saved in Review Time Monthly				13	Equals (Total hours spent currently - Total hours spent using Collaborator) * Number of Reviews per month
Money Saved Finding Defects Sooner - Per Month				$34,560	Money saved per month, based on the fact the developers find more bugs when using Collaborator.
Number of Additional Bugs Found Monthly				96	Equals # Add'l bugs that slip through to QA & customers when not using Collaborator * # of reviews per month
Total Cost Savings - Monthly				$34,519	Equals Monthly Cost Savings Associated w/ Review Time + Monthly Cost Savings Associated w/ Addl Defects found - Cost of Collaborator. Note that we've converted time into dollars here for consistency and to determine the ROI.
Total Cost Savings - Annually				$414,225	Annual Cost Savings Associated w/ Review Time + Annual Cost Savings Associated w/ Addl Defects found - Cost of Collaborator
Total Cost Savings - 3 Years				$1,255,757	Cost Savings associated with use of Collaborator over a 3 year period. (Taking into consideration an annual maintenance fee of 22% for years 2 and 3)




Data from Customer Use



Customer A

Customer A is a national division of a multi-national corporation, with sales in the $100-300M/year range. Their division has hundreds of programmers working on a variety of projects,
located across the country.

This SmartBear customer previously conducted review meetings on some of its code, when time permitted or contracts dictated. These meetings tied up a team for 1.5-2 hours at a time.
Then they deployed a Collaborator pilot project that was so successful that now reviewing ALL code using Collaborator is mandatory.
Using Collaborator, Customer A's programmers now spend 20-30 min reviewing code, for a time savings of about 75%.
They estimate they find about twice as many bugs using Collaborator.
Overall results: Find twice as many bugs in 25% of the time.

Other useful metrics:
This customer calculates the cost of a programmer's time at $60/hr (which we consider lower than average).
If each programmer now spends 10 hours per project reviewing code rather than 40, that saves $1800 per programmer per project.
This customer estimates that if a bug costs $X to fix when found in Development, that same bug costs 10-12X to fix when found during QA and 30X to fix when found after the product has shipped.
Let's give a value to X:	$25

Since Collaborator finds ~4.8 bugs/hour, assume in those 10 hours each programmer finds 48 bugs. By their admission, that's twice as many as they would otherwise have found.
So if they review for 10 hours, they're finding an additional 24 bugs per programmer per project.
Additional cost that would have been incurred when finding/fixing those 24 bugs in QA and in customers' hands (assume 80/20 split):						$8,880
That's a SAVINGS PER PROGRAMMER PER 10 HRs of Review			$8,880



Customer B

Customer B is a $2-3B/year multi-national software company. They tried Collaborator in a four-month pilot project with about 50 developers and compared the results to their existing code
review process. This pilot program was so successful that the company deployed Collaborator to more than 1000 developers.

Results
The company calculated that Collaborator would save them more than $800,000 per year vs. their existing review process. When subtracting out the licensing and approximate
implementation costs, the total ROI (for the first year) is about $720,000 – but that includes the entire purchase price of the software! And, they were able to review 100%, up from 60%.
▪ As a benchmark, their cost of a developer's time is $63.
▪ This customer found 18% more bugs during code review than with current process.
▪ That 18% broke down as follows: 16% were bugs previously found during QA while 2% were bugs previously found at customer.
▪ With Collaborator, they review code 33% faster than with their old process (400 Lines of Code per hour vs 300).



Costs to Fix Bugs from Various Sources

According to one customer, costs to find and fix bugs were
▪ During Development: $275 total cost to find and fix bugs during the Development phase. They found 11 bugs in Development. Cost per bug found in Dev = $25
▪ During QA: $33,000 total cost to find and fix bugs in QA. They found 163 bugs. Cost per bug found in QA = $202 / bug (which is about 8X the cost per bug in Dev)
▪ Once released to customers: $141,000 total cost to fix bugs that reach customers. Customers found 141 bugs. Cost per bug found after release = $1000/bug (about 40X the cost per bug in Dev)

Let's assume 100 bugs were found and fixed in Code Review. Of those, 85 would otherwise have been found and fixed in QA, and 15 would otherwise have been found and fixed
after release to customers. Using the customer data above:
Cost to find & fix bugs during Dev	$25
Cost to find & fix bugs in QA	$202
Cost to find & fix bugs released to customers	$1,000

Cost to find & fix 85 bugs in QA and 15 at customer		$32,000
Cost to find & fix 100 bugs found during Dev w/code review		$2,500
Cost savings for every 100 bugs that exist in the code		$29,500


Another Set of Industry Data
Note: This information is outlined in detail in the free book, Best Kept Secrets for Peer Code Review, available at www.CodeReviewBook.com.

Costs to Find and Fix Bugs	One SmartBear customer		HP Study*	Costs for Customer detailed in book

In Dev	X		X	X
In QA	10-12X			8X
Once Product is Released	30X		100X (HP)	40X

*See Best Kept Secrets of Peer Code Review for more info on the HP study. As a manufacturing company for B2C products, HP has a very high cost to fix bugs once they're in the field.



Background, Benefits and Facts



SmartBear Software has done an extensive study on the effectiveness of tool-assisted code review, and we’ve collected specific ROI data from our customers. We’re sharing this information
to help you decide whether our Collaborator tool for code review makes sense – and saves dollars – for your team.

Code Review with Collaborator helps development teams…
▪ Find more defects in less time, which equates to better product quality and cost savings
▪ Collaborate and communicate to write better code
▪ Improve overall team knowledge level, resulting in better code quality
▪ Automatically captures review metrics for easy reporting, so you can see the benefits

Most of all, code reviews aren’t tedious – they can even be fun! With Collaborator, developers actually do them – quickly, efficiently, and at a time that’s convenient for them.

Facts
▪ While this number can vary widely based on many factors, the industry standard rate for finding bugs during code review is about 45% (without a tool). In other words, if 10 bugs exist
in a piece of code, developers are likely to find 4.5 of them during code review, rather than later during the QA phase or after the product is released to customers.
▪ In our case study at Cisco, which spanned 50 developers, 2500 reviews, and 3.2 million lines of code, we found that with Collaborator, developers are more likely to find 70-90%
of the bugs (almost twice as many), in half the time.
▪ We found lightweight code review to be equally effective at finding bugs as heavyweight processes, but substantially more efficient, with reviews taking 1/5th - 1/7th the time of their
heavyweight counterparts.
▪ Across four studies of heavyweight inspections, the average defect detection rate was 2.6 defects per hour; our reviews were seven times faster. This time savings isn’t surprising,
since our reviews didn’t include two hour inspection meetings with 3-5 participants.

How is Collaborator so much easier and more efficient than other code review methods?
The product is designed specifically to facilitate code review, simplify the process and save time:
▪ Differences between versions ("diffs") are highlighted so you can see them – programmers don’t have to shuffle through a bunch of printouts
▪ You can comment directly on the code – in real-time (chat-style) or when it’s convenient
▪ You can review on your own schedule, at your convenience. Days are not disrupted with meetings, and meetings don’t get sidetracked onto other topics
▪ Since you go straight to the designated area of change, reviews are more accurate.

Related files

Share

Share to social media

Help

Login Area