CS302 --- Analysis of Algorithms and Selection of Algorithms

Analysis of Algorithms

Measures of Times

Average Case: The amount of time the algorithm takes on an "average" set of inputs.

Worst Case: The amount of time the algorithm takes on the worst possible set of inputs.

Quantifying Time

Size of input: We like to be able to express a program's running time as a function of the size of the input. The size of the input is typically denoted as N and the running time of the algorithm as T(N). By plugging in the size of the input, we can get a rough estimate of the cost of executing the algorithm.

Typical Running Times of Algorithms

1: The running time of the program is constant (i.e., it is independent of the size of the input).

Example: Find the 5th element in an array.

log N: Typically achieved by dividing the problem into smaller segments and only looking at one input element in each segment.

Example: Binary search

N: Typically achieved by examining each element in the input once

Example: Find the minimum element

N log N: Typically achieved by dividing the problem into subproblems, solving the subproblems independently, and then combining the results. Unlike the log N algorithms, each element in the subproblems must be examined.

Example: Merge sort

N^2: Typically achieved by examining all pairs of data elements.

Example: Selection Sort

N^3: Often achieved by combining algorithms with a mixture of the previous running times.

Example: Matrix Multiplication--Each item in the result matrix is obtained by multiplying together two vectors of size N. This requires N time. Since there are N^2 elements in the result matrix, it takes N^3 time to compute the entire result matrix.

Common Recurrences

Recurrence	Solution
T(N) = T(N/2) + d	T(N) = O(log N)
T(N) = T(N/2) + N	T(N) = O(N)
T(N) = 2T(N/2) + d	T(N) = O(N)
T(N) = 2T(N/2) + N	T(N) = O(N log N)
T(N) = T(N-1) + d	T(N) = O(N)

Rough Estimates of Running Time

Machine time: Ultimately the best measure but subject to variation due to different instruction costs on different machines.

Abstract time: Abstract away from machines by using a model that assigns a cost of 1 to simple instructions, such as assignment or reading one input element. Using this simple cost, count the number of instructions and try to express this count as a function of the input size.

Big O Notation: Abstracts away all but the most significant cost term. In general, the most significant cost term will dominate the running time of the algorithm once input sizes get very large.

Formal definition: A function g(N) is said to be O(f(N)) if there exist constants c and X such that g(N) < cf(X) for all N > X.

Example: 5N^2 + N + 20 is O(N^2).

Proof: Choose c = 6 and X = 5. Then 5N^2 + N + 20 < 6n^2 for N > 5

Big O Notation represents worst case complexity. However, most often, worst case and average case complexity will be comparable

Considerations in Choosing among Competing Algorithms

1. Which is fastest?

Big-O notation and average case analysis only provide rough guides. One often has to implement the algorithms and test them with typical inputs to actually determine which algorithm is faster.

Example: Quicksort is typically faster than the other sorting methods. However, if the input sequences are typically almost sorted (e.g., some card games), insertion sort is faster because it is "almost" O(n)

2. Which is simplest to implement?

Often if two algorithms have almost comparable running times but one is far easier to implement and maintain, the simpler algorithm will be chosen, even if it is the slower of the two.

3. How often will the algorithm be executed?

If the algorithm will only be executed a few times, programmer time may be more important than computer time.

4. What is the size of the input?

If the input size is small, then an algorithm with a higher time complexity but smaller constants may be preferable.

Example: As a rough rule of thumb, selection sort is faster than quicksort for inputs of size less than 50

Strategy for Solving a Problem

Choose the simplest algorithm first and implement it. Consider a faster, but potentially more complicated, algorithm only if speed becomes a problem. Even if it is clear from the outset that a faster, more complicated algorithm is required, consider implementing a simpler algorithm as a way of checking the correctness of the more complicated algorithm.