Algorithms

An algorithm is a set of rules for carrying out some calculation. A problem (say sorting a list of numbers) will have many instances. An instance, in this case, can be the list {23,15, 67}. An algorithm must work correctly on every instance of the problem it claims to solve. There may be several algorithms to solve the same problem. How do we choose the best one?

The empirical approach is to program the competing algorithms and try them on different instances with the help of a computer. The theoretical approach is to determine mathematically the quantity of resources needed by each algorithm as a function of the size of the instances considered. The resources are computing time and storage space. Computing time being the more critical of the two.

The size of an instance is any integer that in some way measures the number of components in an instance.

number of items in a list to be sorted
number of nodes and/or edges in a graph

Time taken by an algorithm can vary considerably between two different instances of the same size. For example, sorting an already sorted list or list that is sorted in reverse order

We usually consider the worst case, i.e. for a given instance size we consider those instances which requires the most time. The average behavior of an algorithm is much harder to analyze.

We analyze an algorithm in terms of the number of elementary operations that are involved. An elementary operation is one whose execution time is bounded by a constant for a particular machine and programming language. Thus within a multiplicative constant it is the number of elementary operations executed that matters in the analysis and not the exact time.

Since the exact time for an elementary operation is unimportant, we say, that an elementary operation can be executed at unit cost. We use the "Big O" notation for the execution of algorithms. The "Big O" notation gives the asymptotic execution time of an algorithm.

Algorithms can be classified using the "Big O" notation.

Constant - O ( 1 )
Logarithmic - O ( log N )
Root N - O ( √N )
Linear - O ( N )
N log N - O ( N log N )
Quadratic - O ( N² )
Cubic - O ( N³ )
Exponential - O ( 2^N )

Constant time operations

Assigning the value to a variable: x = y
Mathematical operations: x = y + 2 * z
Comparisons: if (x > max) max = x
Accessing a value in an array: x = y[3]

Linear Operations

Sum the values in an array

  sum = 0;
  for item in a;
    sum = sum + item

The number of additions depends on the length of the array. Hence the run time is O(N).

Quadratic Operations

Selection Sort - in this algorithm we start at the first element of the array and go through the array and find the minimum element. We swap the minimum element with the element at the first place. We start at the second position in the array and go through the entire array and find the minimum element in the remaining portion of the array. We swap that minimum element with the element at the second position. We start at the third position and repeat the procedure until we have reached the end of the array.

def selectionSort (a):
  for i in range (len(a) - 1):
    // Find the minimum
    min = a[i]
    minIdx = i

    for j in range (i + 1, len(a)):
      if (a[j] < min):
        min = a[j]
	minIdx = j
   
    // Swap the minimum element with the element at the ith place
    a[minIdx] = a[i]
    a[i] = min

Summation Formulae

Σ 1 = N
Σ i = N ( N + 1 ) / 2
Σ i² = N ( N + 1 ) ( 2 N + 1 ) / 6
Σ i³ = N² ( N + 1 )² / 4