Your code ends up with a stack overflow because you recurse into test regardless of the result of the nil-check.
(defun test (a)
(if (equal a nil) 0) ; <-- problem is partly here...
(if (listp (car a))
(print "a")
(print "b"))
(test (cdr a))) ; <-- ...and partly down here
Even if (equal a nil) evaluates to T and the surrounding if therefore evaluates to 0, you basically ignore this result, because the next step you do is checking if a is a list regardless of the outcome of the previous nil-check. Eventually you call test again without taking the result of your comparisons into consideration.
Keep in mind that the result of a function in Lisp is the result of the last evaluated expression, meaning that if you want your function to return 0 you have to make 0 the last evaluated expression.
The following code behaves as you specified:
(defun test (a)
(if (null a)
0
(progn
(if (listp (car a))
(print "a")
(print "b"))
(test (cdr a)))))
- Note that you can use
null to do a nil-check.
- If
a is not nil, then and only then, a is examined further. For this purpose progn lets you evaluate a sequence of expressions and eventually evaluates to the result of its last expression.