I posted the complete solution and explanation on my blog:
http://www.leetcode.com/2010/11/largest-binary-search-tree-bst-in.html
The idea is to go around the depth and pass the minimum and maximum values from bottom to top instead of top to bottom. In other words, we test the deeper nodes before we verify that the above nodes satisfy the BST requirements.
The main reason for this is when one of the nodes does not satisfy the BST properties, all subtrees above (including this node) must also not satisfy the BST requirements.
Compared to the top-down approach, the bottom-up approach is such an amazing choice, because the results for the total number of nodes can be transmitted along the tree. This saves us from recounting again and again. The total number of nodes for a subtree is simply the total number of nodes on its left and right subtrees plus one.
This algorithm works in O (N) time complexity and O (1) space, which is as efficient as it can be. (where N is the total number of nodes in the binary tree).
Below is the C ++ code. The hard part of the implementation is taking care of how the min and max values are passed from bottom to top. Please note that I did not initialize the minimum and maximum values, as they are initialized at the bottom of the tree.
// Find the largest BST subtree in a binary tree. // If the subtree is a BST, return total number of nodes. // If the subtree is not a BST, -1 is returned. int findLargestBSTSubtree(BinaryTree *p, int &min, int &max, int &maxNodes, BinaryTree *& largestBST) { if (!p) return 0; bool isBST = true; int leftNodes = findLargestBSTSubtree(p->left, min, max, maxNodes, largestBST); int currMin = (leftNodes == 0) ? p->data : min; if (leftNodes == -1 || (leftNodes != 0 && p->data <= max)) isBST = false; int rightNodes = findLargestBSTSubtree(p->right, min, max, maxNodes, largestBST); int currMax = (rightNodes == 0) ? p->data : max; if (rightNodes == -1 || (rightNodes != 0 && p->data >= min)) isBST = false; if (isBST) { min = currMin; max = currMax; int totalNodes = leftNodes + rightNodes + 1; if (totalNodes > maxNodes) { maxNodes = totalNodes; largestBST = p; } return totalNodes; } else { return -1; // This subtree is not a BST } } BinaryTree* findLargestBSTSubtree(BinaryTree *root) { BinaryTree *largestBST = NULL; int min, max; int maxNodes = INT_MIN; // INT_MIN is defined in <climits> findLargestBSTSubtree(root, min, max, maxNodes, largestBST); return largestBST; }