purpose is to produce pure numeric
data.frames that are ready for supervised
predictive modeling (predicting a value from other values). By ready
we mean: a purely numeric data frame with no missing values and a
reasonable number of columns (missing-values re-encoded with indicators,
and high-degree categorical re-encode by effects codes or impact
Part of the
vtreat philosophy is to assume after the
vtreat variable processing the next step is a sophisticated
machine learning method. Under this assumption we assume the machine
learning methodology (be it regression, tree methods, random forests,
boosting, or neural nets) will handle issues of redundant variables,
joint distributions of variables, overall regularization, and joint
However, an important exception is: variable screening. In practice we have seen wide data-warehouses with hundreds of columns overwhelm and defeat state of the art machine learning algorithms due to over-fitting. We have some synthetic examples of this (here and here).
The upshot is: even in 2018 you can not treat every column you find in a data warehouse as a variable. You must at least perform some basic screening.
To help with this
vtreat incorporates a per-variable
linear significance report. This report shows how useful each variable
is taken alone in a linear or generalized linear model (some details can
be found here). However,
this sort of calculation was optimized for speed, not discovery
vtreat now includes a direct variable valuation system
that works very well with complex numeric relationships. It is a
for numeric or regression problems and
for binomial classification problems. It works by fitting two
transformed copies of each numeric variable to the outcome. One
transform is a low frequency transform realized as an optimal
k-segment linear model for a moderate choice of
k. The other fit is a high-frequency trasnform realized as
k-nearest neighbor average for moderate choice of
k. Some of the methodology is shown here.
We recommend using
vtreat::value_variables_*() as an
initial variable screen.
Let’s demonstrate this using the data from the segment fitter
example. In our case the value to be predicted (“
y”) is a
noisy copy of
sin(x). Let’s set up our example data:
##  61 4
Now a simple linear valuation of the the variables can be produced as follows.
##  "vtreat 1.6.4 start initial treatment design Sat Aug 19 12:10:18 2023" ##  " start cross frame work Sat Aug 19 12:10:18 2023" ##  " vtreat::mkCrossFrameNExperiment done Sat Aug 19 12:10:18 2023"
Notice the signal carrying variable did not score better (having a
r-squared and a smaller (better) significance value)
than the noise variable (that is unrelated to the outcome). This is
because the relation between
y is not
Now let’s try
Now the difference is night and day. The important variable
x is singled out (scores very well), and the unimportant
x_noise doesn’t often score well. Though, as with
all significance tests, useless variables can get lucky from time to
time- (an issue that can be addressed by using a Cohen’s-
Our modeling advice is:
sig <= 1/number_of_variables_being_considered.
The idea is: each “pure noise” (or purely useless) variable has a
significance that is distributed uniformly between zero and one. So the
expected number of useless variables that make it through the above
number_of_useless_varaibles * P[useless_sig <= 1/number_of_variables_being_considered].
number_of_useless_varaibles * 1/number_of_variables_being_considered.
number_of_useless_varaibles <= number_of_variables_being_considered
we get this quantity is no more than one. So we expect a constant number
of useless variables to sneak through this filter. The hope is: this
should not be enough useless variables to overwhelm the next stage
supervised machine learning step.
Obviously there are situations where variable importance can not be discovered without considering joint distributions. The most famous one being “xor” where the concept to be learned is if an odd or even number of indicator variables are zero or one (each such variable is individual completely uninformative about the outcome until you have all of the variables simultaneously). However, for practical problems you often have that most variables have a higher marginal predictive power taken alone than they have in the final joint model (as other, better, variables consume some of common variables’ predictive power in the joint model). With this in mind single variable screening often at least gives an indication where to look.
In conclusion the
vtreat package and
vtreat::value_variables_*() can be a valuable addition to
your supervised learning practice.