Tail Recursion continued...
Look again at the stack usage of that new version of factorial
we introduced last week:
| | | | |
|(fact 4)|(fact-it 1 1 4)|(fact-it 1 2 4)|(fact-it 2 3 4)| (continued...)
----------------------------------------------------------
| | | |
|(fact-it 6 4 4)|(fact-it 24 5 4)| 24 |
---------------------------------------
The shape is just a horizontal line, which is characteristic of
what's called a "linear iterative process" (note that the process
can be iterative, referring to how the shape is produced, even though
the procedure itself is defined recursively).
To accomplish this, we used a type of recursion called "tail
recursion". In LISP, we obtain tail recursion by making sure
that the step in our procedure which is the recursive step
does not have any "work" being done "outside" the recursive
function call. Compare the corresponding steps from the two
different function definitions. Here's the first version:
(T (* n (factorial (- n 1))))))
That "(* n " part is what was killing us in terms of memory
usage. This kind of recursion is called "augmenting
recursion", meaning that there's some additional computation
added on to the recursive call, and that makes the program
stack grow. In the tail recursive version, however, there's
no augmenting recursion:
(T (factorial-iterative (* counter product)
(+ counter 1)
max-count))))
All the computation is done within the arguments to the
function call, and so the substitutions are one-for-one and
don't cause any growth of the program stack.
Here's a slightly different perspective which may help you
understand the difference between these two types of
recursion. The first version of the factorial function
postponed arithmetic operations as it progressed, and to
postpone those operations, LISP had to remember what those
operations were. To remember all that stuff, LISP had to
use a chunk of memory on the program stack for each postponed
operation.
But when the tail-recursive version of factorial is running,
no arithmetic operations are being postponed. All
computations are being done as they're needed, so no
operations must be remembered, and consequently there's
no corresponding growth in usage of the program stack.
In order to make this work, though, we had to introduce the
equivalent of variables as place holders for these embedded
computations, and we did this by introducing some additional
arguments. Those arguments in this case were established
through the creation of a helping function, which initializes
the arguments being used as variables:
(defun factorial (n)
(factorial-iterative 1 1 n))
The arguments themselves take on the roles of the sorts of
things you'd expect to see in a traditional iterative
solution:
(defun factorial-iterative (product counter max-count)
...
Here, "product" is used as the place where you store your
cumulative product, and it is initialized to the
multiplicative identity (i.e., 1). "counter" is just your
index variable, and "max-count" is the limit on the number of
"iterations" you want this thing to perform.
So, it may look like iteration, but tail recursion is most
definitely a clever and successful attempt to reap some
of the resource savings of iterative solutions while maintaining
design qualities of the functional programming paradigm.
We gain the advantages of iterative solutions without the
corresponding increase in complexity. What kind of complexity?
Well, just think about this for a minute. If I'm debugging
a recursive function that's constructed within the constraints
of the functional programming paradigm (like "factorial-iterative"),
I know that those "variables" won't be changing value while I
examine the execution of that function. That's gonna make my life
easier, in the short run as well as the long run. On the other
hand, if I'm debugging the traditional, loop-based, side-effects-laden
equivalent of "factorial-iterative", I've got three variables
which *are* changing values as that procedure is running, and
that makes it harder for me to follow what's going on, and therefore
harder for me to make sure that the procedure actually does what I
want it to do.
Sometimes the tail recursive solutions are not immediately
obvious, and sometimes they just seem natural. Just take
a look at the "my-member" function to see just how obvious the tail
recursive solution can be. Like many things in this course, it's not
especially hard, but it does require a different slant on thinking
about the problem, and again like many things in this course, it
gets better with practice. In any case, going after
the tail recursive solution is worthwhile, for reasons
we've shown above. Also, some LISP systems have the ability to
recognize and further optimize tail recursive functions. The
object code that is produced is actually a simple loop (which
you never see) with variables corresponding to the arguments being
passed in the recursive function call. This gives you the elegant
expressiveness of recursion with the speed of loops or
whatever your flavor of iterative control structure might be.
You get all the good stuff, and none of the bad stuff.
You might argue, as students occasionally do, that tail recursive
solutions are harder to read, and I believe that's true for
new LISP programmers. (Oddly enough, someone in this class
actually said that tail recursion was naturally easier...I have to
say that's a first for us in 2360!) But as you gain more experience,
the tail recursive solution becomes just another programming
cliche and no longer causes any difficulty in understanding
what's going on. Remember that lots of tasks look difficult
the first time you encounter them, but over time they become
second nature. When you were real little, your primary mode
of locomotion was crawling. At some point you tried standing
up and walking, and this was undoubtedly much more difficult
than crawling at first. But you kept at it, and now you get
around by walking instead of crawling (except perhaps the
morning after one of those particularly intense frat parties).
The moral is simply that if you practice, you get better.
Students will sometimes overgeneralize the "factorial-iterative"
example and assume that all tail recursive functions require some
helping or auxiliary function. Our definition of "my-member"
didn't require any helping function though, and it's certainly
tail recursive. The helping function is only necessary if you
need to introduce "variables" as additional arguments, so don't
get carried away with the helping functions.
The myth of efficiency
One of the big myths that is constantly perpetuated in the
religious wars about programming paradigms and programming
languages is that functional programming in general (and LISP
in particular) is to be avoided because of all this recursion
stuff. It's hard to learn, they say, and it's obviously
inefficient.
It may be true that the concept of recursion itself is
difficult to grasp, but for most folks it seems that the real
problem is that they've been trained to think about
repetitive computations in a certain way, using index
variables and loops and so on. That training interferes with
thinking about the same problems in different ways; you'll
feel that urge to fall back on old familiar habits. That
urge should diminish as you become more comfortable with the
paradigm.
But the knock on efficiency is in fact nothing more than
mythology spread by folks who never learned the whole story.
As we've just seen:
1) If I'm thinking about it instead of acting like I'm brain
dead, I can use tail recursion and get very efficient
results. Remember that brain-dead programmers make
inefficient programs, regardless of their choice of
language or paradigm.
2) Again, efficiency of the programmer, especially on the
back end of the software life cycle, is greatly improved
by the use of recursion (and other functional programming
ideas), assuming of course that the other folks who read
your code also aren't brain dead.
Recursion examples
After exploring the factorial function in great detail, we
went over a bunch of examples, comparing augmenting recursion
examples to tail recursion examples. Our first example was "length":
(defun my-length (input-list)
(cond ((null input-list) 0)
(T (+ 1 (my-length (rest input-list))))))
Note how the augmenting recursion solution maps onto an inductive proof.
The ((null input-list) 0) corresponds to the base case. The
(my-length (rest input-list)) maps onto assuming the proof holds
for the Nth case. And the (+ 1 (...)) wrapper on the call to
"my-length" maps onto proving the N+1th case. Now here's the
tail recursion version of the same thing:
(defun my-length (input-list)
(my-length-helper input-list 0))
(defun my-length-helper (input-list counter)
(cond ((null input-list) counter)
(T (my-length-helper (rest input-list) (+ 1 counter)))))
That one was pretty simple. Here's a more complicated problem.
The "substitute" function works like this:
? (substitute 'x 'a '(a b (a) c a))
(X B (A) C X)
Here's the augmenting recursion approach:
(defun my-substitute (new old input-list)
(cond ((null input-list) nil)
((eql old (first input-list))
(cons new (my-substitute new old (rest input-list))))
(T (cons (first input-list)
(my-substitute new old (rest input-list))))))
And here's the tail-recursive counterpart:
(defun my-substitute (new old input-list)
(reverse (my-substitute-helper new old input-list nil)))
(defun my-substitute-helper (new old input-list result)
(cond ((null input-list) result)
((eql old (first input-list))
(my-substitute-helper new old
(rest input-list) (cons new result)))
(T (my-substitute-helper new old (rest input-list)
(cons (first input-list) result)))))
But "my-substitute" is truly tail-recursive only if "reverse" is
tail-recursive. Here's a tail-recursive "reverse":
(defun my-reverse (input-list)
(my-reverse-helper input-list nil))
(defun my-reverse-helper (input-list result)
(cond ((null input-list) result)
(T (my-reverse-helper (rest input-list)
(cons (first input-list) result)))))
You could also make an augmenting recursion version of "reverse":
(defun my-reverse (input-list)
(cond ((null input-list) nil)
(T (append (my-reverse (rest input-list))
(list (first input-list))))))
But you didn't have "append" in your list of legal functions, so
you'd have to make your own. Here's the augmenting recursion version
of "append":
(defun my-append (list1 list2)
(cond ((null list1) list2)
(T (cons (first list1) (my-append (rest list1) list2)))))
Looking at this version of "append", you can see that appending something
onto the end of a long (or growing) list is computationally expensive.
You have to "cons" together a copy of the first list before you can
run a pointer from the end of that copy to the second list. So if
you have a choice of building a list by consing elements to the front
and reversing the list once versus appending elements to the end,
make sure you use the cons-and-reverse approach. It's not just a little
savings in both time and cons cells, it's a big big big (potentially
exponential) savings in time and cons cells. And it has nothing to do
with whether "append" is implement with augmenting recursion or tail
recursion (that's about stack space, not time or cons cells). And how
would you make a tail-recursive version of "append"? That's part of
your homework assignment.
Here's one more example, or set of examples, that might help illustrate
the the big differences in efficiency. Here's an augmenting recursion
version of "remove":
(defun my-remove (item input-list)
(cond ((null input-list) nil)
((eql item (first input-list))
(my-remove item (rest input-list)))
(T (cons (first input-list)
(my-remove item (rest input-list))))))
Here's a tail-recursive version:
(defun my-remove (item input-list)
(my-reverse (my-remove-helper item input-list nil)))
(defun my-remove-helper (item input-list result)
(cond ((null input-list) result)
((eql item (first input-list))
(my-remove-helper item (rest input-list) result))
(T (my-remove-helper item (rest input-list)
(cons (first input-list) result)))))
That version builds the new list, without the elements being removed,
by consing the list together and reversing it. Here's another approach
that builds the same list by appending things to the end of the list,
which may seem more intuitively obvious, but it turns out to be much
more expensive:
(defun my-remove-ugly (item input-list)
(my-reverse (my-remove-ugly-helper item input-list nil)))
(defun my-remove-ugly-helper (item input-list result)
(cond ((null input-list) result)
((eql item (first input-list))
(my-remove-ugly-helper item (rest input-list) result))
(T (my-remove-ugly-helper item (rest input-list)
(append result
(list (first input-list)))))))
So let's compare the behavior of these two tail-recursive solutions
to "remove" which differ only in whether they use cons or append to
construct the result. I build a 1,000 element list that looks like
this: (a x a x ... a x) and then asked "my-remove" (the first tail-
recursive version above) to remove every other element:
? (time (my-remove 'a x))
(MY-REMOVE 'A X) took 0 milliseconds (0.000 seconds) to run.
8,000 bytes of memory allocated.
Macintosh Common LISP didn't even note how much time was involved...
it was so small that it didn't register. And it only took 8,000 bytes
of memory (in terms of cons cells) to build the list and reverse it.
What did the "ugly" version of the same function do:
? (time (my-remove-ugly 'a x))
(MY-REMOVE-UGLY 'A X) took 125 milliseconds (0.125 seconds) to run.
1,006,000 bytes of memory allocated.
Hmmm. Noticeably worse performance. But look what happens when we
increase the length of the input list by a factor of 10 (i.e., a
10,000 element list):
(MY-REMOVE 'A X) took 13 milliseconds (0.013 seconds) to run.
80,000 bytes of memory allocated.
The time finally registered on the more efficient version, and the memory
needs increased by a factor of 10, proportional to the increased size
of the input list. But when we run the less efficient version, watch
out:
(MY-REMOVE-UGLY 'A X) took 22,457 milliseconds (22.457 seconds) to run.
Of that, 742 milliseconds (0.742 seconds) were spent in The Cooperative
Multitasking Experience.
13,681 milliseconds (13.681 seconds) was spent in GC.
100,060,432 bytes of memory allocated.
Time requirements went up a lot, and LISP spent a lot of time in memory
management, trying to find memory it could use. And the memory requirements
increased not by a factor of 10, but by a factor of 100!
The moral to this story is that simple little decisions like using
append instead of cons can cost you lots of time and lots of memory.
In this case, the difference appears to be linear growth in memory usage
versus exponential growth in memory usage. And it looks like the
differences in growth in time needed to complete the computation is
linear versus exponential too. So while we don't want you to sacrifice
readablility of your code to save a few bytes here or a few cycles there,
on the other hand we DO want you to worry about linear versus exponential
differences...these are the kinds of differences that determine whether
your program will finish in a few seconds or whether it will still be
running when the Sun burns out. Don't be a brain-dead programmer,
regardless of the language you're using or the problem you're tyring to solve.
Copyright 1998 by Kurt Eiselt. All rights reserved.
Last revised: April 17, 1998