Recurrences

Recurrences#

Factorial of n (formula)#

$n!$

For each positive integer $n$, the quantity $n$ factorial denoted $n!$, is defined to be the product of all the integers from $1$ to $n$:

\[n! = n · (n − 1) \dots 3·2·1\]

Zero factorial, denoted 0!, is defined to be 1:

\[0! = 1\]

– Discrete Mathematics with Applications, 4th

Examples

Simplify the following expressions:

code

int fact(int num) {

  if (num == 0) return 1;

  else return num * fact(num - 1);   
}

https://www.geeksforgeeks.org/factorial-formula/

Use Cases

Permutation

gives the number of ways to select $r$ elements from $n$ elements when order matters

\[^nP_r = \frac{n!}{(n – r)!}\]

Example

Three different fruits are to be distributed among a group of 10 people. Find the total number of ways this can be possible.

Combination

gives the number of ways to select $r$ elements from $n$ elements where order does not matter

\[^nC_r = \frac{n!}{r! (n – r)!}\]

Example

Find the number of ways 3 students can be selected from a class of 50 students.

Recurrence Relations #

By itself, a recurrence does not describe the running time of an algorithm

need a closed-form solution (non-recursive description)
exact closed-form solution may not exist, or may be too difficult to find

For most recurrences, an asymptotic solution of the form $\Theta()$ is acceptable

…in the context of analysis of algorithms

Methods of Solving Recurrences #

Unrolling

In the unrolling method, we repeatedly substitute the recurrence relation into itself until we reach a base case. Let’s unroll the recurrence relation for a specific value of $n$:

\[\begin{split}\begin{align} T(n) &= 2T \bigg(\frac{n}{2} \bigg) + n \\ &= 2 \bigg[2T \bigg(\frac{n}{4} \bigg) + \frac{n}{2} \bigg] + n \\ &= 4T \bigg(\frac{n}{4} \bigg) + 2n + n \\ &= 4 \bigg[2T \bigg(\frac{n}{8} \bigg) + \frac{n}{4} \bigg] + 2n + n \\ &= 8T \bigg(\frac{n}{8} \bigg) + 4n + 2n + n \\ &= 2^kT \bigg(\frac{n}{2^k} \bigg) + kn \\ \end{align}\end{split}\]

We continue this process until we reach the base case, which occurs when $\frac{n}{2^k} = 1$. Solving for $k$, we find that $k = log_2\ (n)$. Therefore, the final unrolled expression becomes:

\[ \ \ T(n) = 2^{log_2 (n)} \ T(1) + n \ log_2 \ (n) \ \ \]

Since $T(1)$ is a constant, we can simplify the expression further:

\[\begin{split}\begin{align} T(n) &= nT(1) + n\ log_2 \ (n) \\ &= O(n\ log_2\ (n)) \end{align}\end{split}\]

Thus, the solution obtained through unrolling suggests that the time complexity of the original recurrence relation is $O(n\ log_2 (n))$.

Guessing

In the guessing method, we make an educated guess or hypothesis about the form of the solution based on the recurrence relation. We then use mathematical induction or substitution to prove our guess.

Let’s guess that the solution to the recurrence relation $T(n) = 2T(\frac{n}{2}) + n$ is $T(n) = O(n\ log\ (n))$.

We assume that $T(k) ≤ ck\ log\ (k)$ for some constant $c$, where $k < n$. Now we substitute this assumption into the recurrence relation:

\[\begin{split}\begin{align} T(n) &= 2T \bigg(\frac{n}{2} \bigg) + n \\ &≤ 2 \Bigg(\ c \bigg( \frac{n}{2} \bigg)\ log\ \bigg(\frac{n}{2} \bigg) \Bigg) + n \\ &= cn\ log (n) - cn\ log (2) + n \\ &= cn\ log (n) - cn + n \\ &= cn\ log (n) + (n - cn) \\ \end{align}\end{split}\]

To ensure that $T(n) ≤ cn\ log\ (n)$ holds, we need to find a value of $c$ such that $(n - cn) ≤ 0$. By choosing $c ≥ 1$, we can guarantee that $T(n) ≤ cn\ log\ (n)$.

Hence, the solution $T(n) = O(n\ log\ (n))$ satisfies the recurrence relation.

Recursion Tree

In the recursion tree method, we draw a tree to represent the recursive calls made in the recurrence relation. Each level of the tree corresponds to a recursive call, and we analyze the work done at each level.

For our example, the recursion tree for $T(n) = 2T(\frac{n}{2}) + n$ would have a root node representing $T(n)$, two child nodes representing $T(\frac{n}{2})$, and so on. Each node represents a recursive call, and the work done at each node is represented by the term n.

The recursion tree will have $log_2\ (n)$ levels since we divide the problem size by $2$ at each level. At each level, the work done is $n$, and there are $2^i$ nodes at level $i$. Therefore, the total work at each level is $2^i * n$.

Summing up the work done at each level, we have:

\[ \ \ Total\ work\ = n + 2n + 4n + ... + (n * 2^{log_2\ (n)} - 1) \ \ \]

This is a geometric series with a common ratio of $2$ and the first term being $n$. The sum of this series is given by:

\[\begin{split}\begin{align} Total work &= n * \frac{2^{log_2\ (n)} - 1}{2 - 1} \\ &= \frac{n * (n - 1)}{1} \\ &= n^2 - n \\ \end{align}\end{split}\]

Therefore, the time complexity of the recurrence relation is $O(n^2)$.

Note that in this example, the unrolling and guessing methods led to a time complexity of $O(n\ log\ (n))$, while the recursion tree method resulted in $O(n^2)$. The discrepancy arises because the recursion tree method accounts for the exact work done at each level, while the other methods provide upper bounds or estimations.

Master Theorem

The master theorem is a mathematical formula used to analyze the time complexity of divide-and-conquer algorithms. It provides a solution for recurrence relations of the form:

\[T(n) = aT \bigg(\frac{n}{b} \bigg) + f(n)\]

Where:

$T(n)$ represents the time complexity of the algorithm, $a$ is the number of recursive subproblems, $\frac{n}{b}$ is the size of each subproblem, $f(n)$ is the time complexity of the remaining work done outside the recursive calls.

The master theorem provides three cases based on the relationship between $f(n)$ and the subproblem part $aT(\frac{n}{b})$.

Cases

Case 1: If $f(n) = O(n^c)$ for some constant $c < log_b(a)$, then the time complexity is dominated by the subproblem part, and the overall time complexity is given by

\[T(n) = \Theta(n^{log_b(a)})\]

Case 2: If $f(n) = \Theta(n^c \ log^{k(n)})$ for some constants $c ≥ 0$ and $k ≥ 0$, and if $aT(\frac{n}{b})$ is the dominant term, then the overall time complexity is given by

\[T(n) = \Theta(n^{log_b(a)} \ log^{(k+1)}(n))\]

Case 3: If $f(n) = \Omega(n^c)$ for some constant $c > log_b(a)$, and if $f(n)$ satisfies the regularity condition ($af(\frac{n}{b}) ≤ kf(n)$ for some constant $k < 1$ and sufficiently large $n$), then the non-recursive part dominates the time complexity, and the overall time complexity is given by

\[T(n) = \Theta(f(n))\]

Case 1 Ex

\[T(n) = 4T \bigg(\frac{n}{2} \bigg) + n\]

Case 2 Ex

\[T(n) = 9T \bigg(\frac{n}{3} \bigg) + n^2\]

Case 3 Ex

\[T(n) = 2T \bigg(\frac{n}{2} \bigg) + n^3\]

In our example, the recurrence relation is $T(n) = 2T(\frac{n}{2}) + n$, where $a = 4$, $b = 2$, and $f(n) = n$.

Now, let’s compare the function $f(n)$ with $n^{log_b\ (a)}$:

$f(n) = n$, which is asymptotically smaller than $n^{log_2\ (4)} = n$.

According to the Master Theorem:

If $f(n) = O(n^c)$ where $c < log_b\ (a)$, then $T(n) = \Theta(n^{log_b\ (a)})$. In our case, $f(n) = O(n^1)$, and $c = 1 < log_2\ (4) = 1$. Since the condition is met, the time complexity of the recurrence relation is $Θ(n^{log_2\ (4)}) = \Theta(n)$.

Therefore, according to the Master Theorem, the time complexity of the binary search recurrence relation $T(n) = 2T(\frac{n}{2}) + n$ is $\Theta(n)$, which aligns with the result obtained from the unrolling and guessing methods.

It’s important to note that the Master Theorem is applicable in specific cases, and not all recurrence relations can be solved using this theorem. However, for recurrence relations that follow the prescribed form, it provides a convenient way to determine the time complexity.

Examples#

Dividing Function

Recurrence Relation: Dividing Function

 1, 3, 4
void Test(int n) {       // = T(n)
  if (n > 0) {
    printf("/d", n);     // = 1
    Test(n/2);           // = T(n/2) 
  }
}

\[\begin{split} T(n) = \begin{cases} 1, & \text{$n = 1$} \\[2ex] T(\frac{n}{2}) + 1, & \text{$n \gt 1$} \end{cases} \end{split}\]

Linear

Recurrence Relation: Linear

 1, 3, 4
void Test(int n) {       // = T(n)
  if (n > 0) {
    printf("/d", n);     // = 1
    Test(n - 1);         // = T(n - 1) 
  }
}

\[\begin{split} T(n) = \begin{cases} 1, & \text{$n\ = 0$} \\[2ex] T(n - 1) + 1, & \text{$n\ \gt 0$} \end{cases} \end{split}\]

Divide & Conquer

Recurrence Relation: Divide & Conquer

 1, 3, 6, 7
void Test(int n) {              // = T(n)
  if (n > 1) {
    for(i = 0; i < n; i++>) {   // = n
      // some statement
    }
    Test(n/2);                  // = T(n/2)
    Test(n/2);                  // = T(n/2)
  }
}

\[\begin{split} T(n) = \begin{cases} 1, & \text{$n\ = 1$} \\[2ex] 2T(\frac{n}{2}) + n, & \text{$n\ \gt 1$} \end{cases} \end{split}\]

Tower of Hanoi

Recurrence Relation: Tower of Hanoi

 1 - 4
void Test(int n) {    // = T(n)
  printf("/d", n);    // = 1
  Test(n - 1);        // = T(n - 1)
  Test(n - 1);        // = T(n - 1)
}

\[\begin{split} T(n) = \begin{cases} 1, & \text{$n\ = 0$} \\[2ex] 2T(n - 1) + 1, & \text{$n\ \gt 0$} \end{cases} \end{split}\]