Software benchmarking typically refers to a process by which the development cycle of a software manufacturer is compared to other businesses to determine the efficiency of the company. This is often done to determine where improvements can be made to the development process, and to ensure that a company is running at peak efficiency. These comparisons are usually quantitative in nature, as they deal with numerical values and figures to gauge how quickly and effectively a company is performing. This type of software benchmarking should not be confused with computer hardware benchmarking, which evaluates the effectiveness of different devices and computers.
The primary purpose of software benchmarking is to determine how well one software developer is performing when compared to another. Initially, information about a developer needs to be gathered and considered in a quantifiable way. The length of time required for that company to create its latest program, the time required for bugs to be fixed and for patches to be released, and the number of programmers involved in these processes can all be considered. Once this data is gathered, then it can be used in software benchmarking against other companies using comparable metrics.
Third-party companies are often involved in software benchmarking, by gathering information about numerous developers, which is frequently provided by those businesses for comparison. A benchmarking company might have data from several dozen, or even a hundred, software developers detailing the quantifiable values for time spent in development, costs for this process, and achievements based on numbers of programmers. Comparisons are then made by this organization between these different developers, and used to establish a number of values for each one. Once this process is complete, then each company can be notified about how well it performed within this larger pool and can use this software benchmarking data to determine improvements that should be made for future developments.
This type of software benchmarking is somewhat similar in nature to hardware benchmarking, but they should not be confused. Hardware benchmarks are often determined by running different computer setups and pieces of hardware through various tests to generate data about those devices. This data is then used to compare different pieces of hardware, in order to determine what processors or graphics cards are theoretically superior to others within a given context.
There is a form of software benchmarking that can be performed in a way similar to this, though it is more often considered a part of support testing. The same program can be run on multiple computers with different devices and hardware setups, to see what systems it is compatible with. This type of benchmarking is often used to determine the minimum requirements for a program and to ensure that as wide a user base as possible can run the software.