Haskell max from parameterized binary search tree

The fundamental problem here is that your type signature (like that of maximum) is a lie. You cannot guarantee to produce the maximal element from a data structure that may contain no elements. A special sentinel value can paper over this issue in some cases, but even in those cases it breaks down if you look carefully. What is the minimum element of Node Nil -2000000 Nil? This is a nonempty tree, so you should be able to get its maximum, but your implementation returns -1000000 instead, as though the tree were empty!

One thing you could try that would do a better job of sweeping the problem under the rug would be to add a Bounded constraint, so that you could use minBound as the "neutral" element. Then at least you don't get erroneous results as in my example, but you still can't tell an empty tree from a tree containing only the minimum.

Better is to adjust your type signature to tell the truth. You can return Maybe t instead of just t, using Nothing to indicate "sorry, this tree was empty". As for implementation, you can just the the obvious brute-force thing of pattern-matching on the recursive calls - it's clunky, but it works.

Better still, though, would be to implement Foldable for your tree type (a good idea in any case), so that you can take advantage of its toList. Presuming that you have done so, this maximum function becomes easy:

maxBST t = case toList t of
  [] -> Nothing
  xs -> Just $ maximum xs

I agree with @amalloy's answer, that returning a Maybe type is the best approach here.

However, there is another approach if you are determined to always return an "actual value" and have it be a sensible "minimum" for the type. And that is to alter the type signature so that the elements of the tree must be of a type that is an instance of Bounded as well as Ord. This would be:

maxBST       :: (Ord t, Bounded t) => BST t -> t
maxBST Nil   =  minBound
maxBST (Node left value right) = max (maxBST left) (max value (maxBST right))

Note that, although this will work on Int, it won't be identical to your original function, because the minBound for Int is much less than -1000000.

Don't goto nil, just destructure left and right and see if both of them are nil and handle the cases

maxBST Nil = error "your function shouldn't reach here"
maxBST (Node Nil value Nil) = value
maxBST (Node left@(Node _ _ _) value Nil) = max (maxBST left) value
maxBST (Node Nil value right@(Node _ _ _)) = max value (maxBST right)