10

What does RandomForestClassifier() do if we choose bootstrap = False?

According to the definition in this link

http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier

bootstrap : boolean, optional (default=True) Whether bootstrap samples are used when building trees.

Asking this because I want to use a Random Forest approach to a time series, so train with a rolling window of size (t-n) and predict date (t+k) and wanted to know if this is what would happen if we choose True or False:

1) If Bootstrap = True, so when training samples can be of any day and of any number of features. So for example can have samples from day (t-15), day (t-19) and day (t-35) each one with randomly chosen features and then predict the output of date (t+1).

2) If Bootstrap = False, its going to use all the samples and all the features from date (t-n) to t, to train, so its actually going to respect the dates order (meaning its going to use t-35, t-34, t-33... etc until t-1). And then will predict output of date (t+1).

If this is how Bootstrap works I would be inclined to use Boostrap = False, as if not it would be a bit strange (think of financial series) to just ignore the consecutive days returns and jump from day t-39 to t-19 and then to day t-15 to predict day t+1. We would be missing all the info between those days.

So... is this how Bootstrap works?

Improve this question

3 Answers 3

Reset to default
14

I don't have the reputation to comment. So I will just post my opinion here. The scikit-learn documentation says the sub-sample size is always the same as the original input sample size but the samples are drawn with replacement if bootstrap=True (default). So if bootstrap = FALSE, I think every sub-sample is just as same as the original input sample.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

12

It seems like you're conflating the bootstrap of your observations with the sampling of your features. An Introduction to Statistical Learning provides a really good introduction to Random Forests.

The benefit of random forests comes from its creating a large variety of trees by sampling both observations and features. Bootstrap = False is telling it to sample observations with or without replacement - it should still sample when it's False, just without replacement.

You tell it what share of features you want to sample by setting max_features, either to a share of the features or just an integer number (and this is something that you would typically tune to find the best parameter for).

It will be fine that you're not going to have every day when you're building each tree - that's where the value of RF comes from. Each individual tree will be a pretty bad predictor, but when you average together the predictions from hundreds or thousands of trees you'll (probably) end up with a good model.

Improve this answer

5 Comments

Crystal clear, thanks @Tchotchke. and good link. Was worried that by doing this would ignore the autocorrelation info from using consecutive days. But I guess RF would take care of that then.
I love ISLR - it's the best intro to machine learning that I've found. Applied Predictive Modeling is also quite good (though not freely available) and the Elements of Statistical Learning (freely available here) goes a lot more in-depth.
When bootstrap is True, the sample size is same as size of original data. What is the sample size when bootstrap is False?
Setting bootstrap = False causes no subsampling of the data. The newer documentation makes this clearer.
The documentation in RandomForestRegressor says "Whether bootstrap samples are used when building trees. If False, the whole dataset is used to build each tree." So, you are saying that the algorithm is still doing sampling from the observations as well as features. False just makes it do without replacement. How about features? How does sampling of features work?
1

According to this definition [1]

A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is always the same as the original input sample size but the samples are drawn with replacement if bootstrap=True (default).

Note: sub-sample size is always the same

But the samples are drawn with replacement if bootstrap=True (default).

So Bootstrap=True (default): samples are drawn with replacement Bootstrap=False : samples are drawn without replacement

[2] In sampling without replacement, each sample unit of the population has only one chance to be selected in the sample. For example, if one draws a simple random sample such that no unit occurs more than one time in the sample, the sample is drawn without replacement.

Visually you could imagine that from a bag of balls (samples), you pick M.

bag of balls

That constitutes your subset number 1, with M balls.

Now, if you trow the balls inside the bag before you pick up another M for your subset 2 then you do "draw with replacement" (bootstrap=True)

But, if you put the subset 1 aside and pick up another M balls from the bag for your subset 2, then none of the balls in subset 1 can be in subset 2 (or any other subset) because you "draw without replacement" (bootstrap=False)

[1] https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

[2] http://methods.sagepub.com/Reference//encyclopedia-of-survey-research-methods/n516.xml

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.