library(hardhat)

Introduction

For most modeling functions, data must be accepted from the user in some format where the outcomes and predictors are both specified. The next step is often to validate and preprocess that input in some way to prepare it for the actual modeling implementation function. For example, when a formula method is used, R provides some infrastructure for preprocessing the user input through the model.frame() and model.matrix() functions.

But the formula method is not the only way to specify modeling terms. There is also an XY method, where x and y are supplied directly, and, recently, a recipe implementation can be used to preprocess data using a set of sequential steps.

As a developer, you likely won’t want to care about the details of how each of these methods work, but (hopefully) still want to provide all three of these interfaces for your shiny new model. mold() makes this easy on you, and takes care of the details of preprocessing user input from any of these methods.

The intended use of mold() is to be called from your user facing modeling function. To see that in action, have a look at the vignette found here: vignette("package", "hardhat"). The rest of this vignette will be focused on the various different ways to use mold(), but keep in mind that generally it is not used as an interactive function like this.

A First Example

The most familiar interface for R users is likely the formula interface. In this case, terms are specified using the formula notation: outcomes ~ predictors. Generally, as a developer, you have to then call model.frame() and model.matrix() on this result to coerce it into the right format for ingestion into your model. mold() handles all of that for you.

iris_form <- mold(Sepal.Width ~ log(Sepal.Length), iris)

names(iris_form)
#> [1] "predictors" "outcomes"   "blueprint"  "extras"

mold() returns four things. Two of them are immediately useful, and are almost always applicable to the modeling implementation you have created. The first is the predictors, returned as a tibble. All of the required processing has been done for you, so you just have to focus on the modeling implementation.

iris_form$predictors
#> # A tibble: 150 x 1
#>    `log(Sepal.Length)`
#>                  <dbl>
#>  1                1.63
#>  2                1.59
#>  3                1.55
#>  4                1.53
#>  5                1.61
#>  6                1.69
#>  7                1.53
#>  8                1.61
#>  9                1.48
#> 10                1.59
#> # … with 140 more rows

Second is the outcomes, also returned as a tibble. While not used here, any processing on the outcome that was specified in the formula would also be done here.

iris_form$outcomes
#> # A tibble: 150 x 1
#>    Sepal.Width
#>          <dbl>
#>  1         3.5
#>  2         3  
#>  3         3.2
#>  4         3.1
#>  5         3.6
#>  6         3.9
#>  7         3.4
#>  8         3.4
#>  9         2.9
#> 10         3.1
#> # … with 140 more rows

Beyond these two elements, mold() also returns a slot for any extras that might have been generated during preprocessing, but aren’t specifically predictors or outcomes. For example, an offset() can be specified directly in the formula, but isn’t technically a predictor.

mold(Sepal.Width ~ log(Sepal.Length) + offset(Petal.Width), iris)$extras
#> $offset
#> # A tibble: 150 x 1
#>    .offset
#>      <dbl>
#>  1     0.2
#>  2     0.2
#>  3     0.2
#>  4     0.2
#>  5     0.2
#>  6     0.4
#>  7     0.3
#>  8     0.2
#>  9     0.2
#> 10     0.1
#> # … with 140 more rows

Lastly, mold() returns a very important object, the blueprint. This is responsible for knowing how to preprocess both the training data, and any new data at prediction time. As a developer, you should attach the blueprint to your model object before returning it to the user. For more information about this, see the package development vignette, vignette("package", "hardhat").

blueprints

As mentioned above, one of the objects that mold() returns is an blueprint responsible for controlling the preprocessing. There are multiple blueprints available in hardhat, but when you call mold() one is selected automatically for you. The following two calls generate the same result, using the default formula blueprint.

identical(
  mold(~ Sepal.Width, iris),
  mold(~ Sepal.Width, iris, blueprint = default_formula_blueprint())
)
#> [1] TRUE

Each blueprint can be tweaked to change how the processing for that interface occurs, and the options vary per blueprint. To understand why you’d ever want to do this, read on!

Formulas

Now that you have a basic idea of how mold() works, we can talk about some of the more interesting functionality.

Intercepts

One challenge with the standard formula interface is that, by default, intercepts are always implicitly present and are added to your data set automatically. This works great for the simple regression case. However, other models might either always require or never allow an intercept, but still use the formula interface because of its convenience (for example, earth). This has led to many ad hoc solutions that prevent the user from removing or adding an intercept.

To get around this, mold() will never add an intercept by default. Instead, the addition of an intercept is completely controlled by the formula blueprint argument, intercept.

no_intercept <- mold(~ Sepal.Width, iris)

no_intercept$predictors
#> # A tibble: 150 x 1
#>    Sepal.Width
#>          <dbl>
#>  1         3.5
#>  2         3  
#>  3         3.2
#>  4         3.1
#>  5         3.6
#>  6         3.9
#>  7         3.4
#>  8         3.4
#>  9         2.9
#> 10         3.1
#> # … with 140 more rows
with_intercept <- mold(
  ~ Sepal.Width, iris,
  blueprint = default_formula_blueprint(intercept = TRUE)
)

with_intercept$predictors
#> # A tibble: 150 x 2
#>    `(Intercept)` Sepal.Width
#>            <dbl>       <dbl>
#>  1             1         3.5
#>  2             1         3  
#>  3             1         3.2
#>  4             1         3.1
#>  5             1         3.6
#>  6             1         3.9
#>  7             1         3.4
#>  8             1         3.4
#>  9             1         2.9
#> 10             1         3.1
#> # … with 140 more rows

An error is thrown if an intercept removal term is specified:

mold(~ Sepal.Width - 1, iris)
#> Error: `formula` must not contain the intercept removal term: `- 1`.

mold(~ Sepal.Width + 0, iris)
#> Error: `formula` must not contain the intercept removal term: `+ 0` or `0 +`.

Dummy variables

One of the nice things about the formula interface is that it expands factors into dummy variable columns for you. Like intercepts, this is great…until it isn’t. For example, ranger fits a random forest, which can take factors directly, but still uses the formula notation. In this case, it would be great if the factor columns specified as predictors weren’t expanded. This is the job of the blueprint argument, indicators.

expanded_dummies <- mold(~ Sepal.Width + Species, iris)

expanded_dummies$predictors
#> # A tibble: 150 x 4
#>    Sepal.Width Speciessetosa Speciesversicolor Speciesvirginica
#>          <dbl>         <dbl>             <dbl>            <dbl>
#>  1         3.5             1                 0                0
#>  2         3               1                 0                0
#>  3         3.2             1                 0                0
#>  4         3.1             1                 0                0
#>  5         3.6             1                 0                0
#>  6         3.9             1                 0                0
#>  7         3.4             1                 0                0
#>  8         3.4             1                 0                0
#>  9         2.9             1                 0                0
#> 10         3.1             1                 0                0
#> # … with 140 more rows
non_expanded_dummies <- mold(
  ~ Sepal.Width + Species, iris,
  blueprint = default_formula_blueprint(indicators = FALSE)
)

non_expanded_dummies$predictors
#> # A tibble: 150 x 2
#>    Sepal.Width Species
#>          <dbl> <fct>  
#>  1         3.5 setosa 
#>  2         3   setosa 
#>  3         3.2 setosa 
#>  4         3.1 setosa 
#>  5         3.6 setosa 
#>  6         3.9 setosa 
#>  7         3.4 setosa 
#>  8         3.4 setosa 
#>  9         2.9 setosa 
#> 10         3.1 setosa 
#> # … with 140 more rows

Note: It’s worth mentioning that when an intercept is not present, base R expands factors columns completely (also known as one hot encoding) into K columns corresponding to the K levels present in the factor. When an intercept is present, only K-1 columns are generated.

k_cols <- mold(~ Species, iris)

k_minus_one_cols <- mold(
  ~ Species, iris,
  blueprint = default_formula_blueprint(intercept = TRUE)
)

colnames(k_cols$predictors)
#> [1] "Speciessetosa"     "Speciesversicolor" "Speciesvirginica"

colnames(k_minus_one_cols$predictors)
#> [1] "(Intercept)"       "Speciesversicolor" "Speciesvirginica"

Multivariate outcomes

One of the other frustrating things about working with the formula method is that multivariate outcomes are a bit clunky to specify.

.f <- cbind(Sepal.Width, Sepal.Length) ~ Petal.Width

frame <- model.frame(.f, iris)

head(frame)
#>   cbind(Sepal.Width, Sepal.Length).Sepal.Width
#> 1                                          3.5
#> 2                                          3.0
#> 3                                          3.2
#> 4                                          3.1
#> 5                                          3.6
#> 6                                          3.9
#>   cbind(Sepal.Width, Sepal.Length).Sepal.Length Petal.Width
#> 1                                           5.1         0.2
#> 2                                           4.9         0.2
#> 3                                           4.7         0.2
#> 4                                           4.6         0.2
#> 5                                           5.0         0.2
#> 6                                           5.4         0.4

This might look like 3 columns, but it is actually 2, where the first column is named cbind(Sepal.Width, Sepal.Length), and it is actually a matrix with 2 columns, Sepal.Width and Sepal.Length inside it.

ncol(frame)
#> [1] 2

class(frame$`cbind(Sepal.Width, Sepal.Length)`)
#> [1] "matrix" "array"

head(frame$`cbind(Sepal.Width, Sepal.Length)`)
#>      Sepal.Width Sepal.Length
#> [1,]         3.5          5.1
#> [2,]         3.0          4.9
#> [3,]         3.2          4.7
#> [4,]         3.1          4.6
#> [5,]         3.6          5.0
#> [6,]         3.9          5.4

The default formula blueprint used with mold() allows you to specify multiple outcomes like you specify multiple predictors. You can even do inline transformations of each outcome, although if you are doing very much of that, I’d advise using a recipe instead. outcomes then holds the two outcomes columns.

multivariate <- mold(Sepal.Width + log(Sepal.Length) ~ Petal.Width, iris)

multivariate$outcomes
#> # A tibble: 150 x 2
#>    Sepal.Width `log(Sepal.Length)`
#>          <dbl>               <dbl>
#>  1         3.5                1.63
#>  2         3                  1.59
#>  3         3.2                1.55
#>  4         3.1                1.53
#>  5         3.6                1.61
#>  6         3.9                1.69
#>  7         3.4                1.53
#>  8         3.4                1.61
#>  9         2.9                1.48
#> 10         3.1                1.59
#> # … with 140 more rows

XY

The second interface is the XY interface, useful when the predictors and outcomes are specified separately.

x <- subset(iris, select = -Sepal.Width)
y <- subset(iris, select = Sepal.Width)

iris_xy <- mold(x, y)

iris_xy$predictors
#> # A tibble: 150 x 4
#>    Sepal.Length Petal.Length Petal.Width Species
#>           <dbl>        <dbl>       <dbl> <fct>  
#>  1          5.1          1.4         0.2 setosa 
#>  2          4.9          1.4         0.2 setosa 
#>  3          4.7          1.3         0.2 setosa 
#>  4          4.6          1.5         0.2 setosa 
#>  5          5            1.4         0.2 setosa 
#>  6          5.4          1.7         0.4 setosa 
#>  7          4.6          1.4         0.3 setosa 
#>  8          5            1.5         0.2 setosa 
#>  9          4.4          1.4         0.2 setosa 
#> 10          4.9          1.5         0.1 setosa 
#> # … with 140 more rows

iris_xy$outcomes
#> # A tibble: 150 x 1
#>    Sepal.Width
#>          <dbl>
#>  1         3.5
#>  2         3  
#>  3         3.2
#>  4         3.1
#>  5         3.6
#>  6         3.9
#>  7         3.4
#>  8         3.4
#>  9         2.9
#> 10         3.1
#> # … with 140 more rows

This interface doesn’t do too much in the way of preprocessing, but it does let you specify an intercept in the blueprint specific arguments. Rather than default_formula_blueprint(), this uses the default_xy_blueprint().

xy_with_intercept <- mold(x, y, blueprint = default_xy_blueprint(intercept = TRUE))

xy_with_intercept$predictors
#> # A tibble: 150 x 5
#>    `(Intercept)` Sepal.Length Petal.Length Petal.Width Species
#>            <int>        <dbl>        <dbl>       <dbl> <fct>  
#>  1             1          5.1          1.4         0.2 setosa 
#>  2             1          4.9          1.4         0.2 setosa 
#>  3             1          4.7          1.3         0.2 setosa 
#>  4             1          4.6          1.5         0.2 setosa 
#>  5             1          5            1.4         0.2 setosa 
#>  6             1          5.4          1.7         0.4 setosa 
#>  7             1          4.6          1.4         0.3 setosa 
#>  8             1          5            1.5         0.2 setosa 
#>  9             1          4.4          1.4         0.2 setosa 
#> 10             1          4.9          1.5         0.1 setosa 
#> # … with 140 more rows

Vector outcomes

y is a bit special in the XY interface, because in the univariate case users might expect to be able to pass a vector, a 1 column data frame, or a matrix. mold() is prepared for all of those cases, but the vector case requires special attention. To be consistent with all of the other mold() interfaces, the outcomes slot of the return value should be a tibble. To achieve this when y is supplied as a vector, a default column name is created, ".outcome".

mold(x, y$Sepal.Width)$outcomes
#> # A tibble: 150 x 1
#>    .outcome
#>       <dbl>
#>  1      3.5
#>  2      3  
#>  3      3.2
#>  4      3.1
#>  5      3.6
#>  6      3.9
#>  7      3.4
#>  8      3.4
#>  9      2.9
#> 10      3.1
#> # … with 140 more rows

Recipe

The last of the three interfaces is the relatively new recipes interface. The default_recipe_blueprint() knows how to prep() your recipe, and juice() it to extract the predictors and the outcomes. This is by far the most flexible way to preprocess your data.

library(recipes)

rec <- recipe(Sepal.Length ~ Species + Petal.Width, iris) %>%
  step_log(Sepal.Length) %>%
  step_dummy(Species)

iris_recipe <- mold(rec, iris)

iris_recipe$predictors
#> # A tibble: 150 x 3
#>    Petal.Width Species_versicolor Species_virginica
#>          <dbl>              <dbl>             <dbl>
#>  1         0.2                  0                 0
#>  2         0.2                  0                 0
#>  3         0.2                  0                 0
#>  4         0.2                  0                 0
#>  5         0.2                  0                 0
#>  6         0.4                  0                 0
#>  7         0.3                  0                 0
#>  8         0.2                  0                 0
#>  9         0.2                  0                 0
#> 10         0.1                  0                 0
#> # … with 140 more rows

iris_recipe$outcomes
#> # A tibble: 150 x 1
#>    Sepal.Length
#>           <dbl>
#>  1         1.63
#>  2         1.59
#>  3         1.55
#>  4         1.53
#>  5         1.61
#>  6         1.69
#>  7         1.53
#>  8         1.61
#>  9         1.48
#> 10         1.59
#> # … with 140 more rows

The only special thing you can tweak with the recipe blueprint is whether or not an intercept is added.

recipe_with_intercept <- mold(
  rec, iris,
  blueprint = default_recipe_blueprint(intercept = TRUE)
)

recipe_with_intercept$predictors
#> # A tibble: 150 x 4
#>    `(Intercept)` Petal.Width Species_versicolor Species_virginica
#>            <int>       <dbl>              <dbl>             <dbl>
#>  1             1         0.2                  0                 0
#>  2             1         0.2                  0                 0
#>  3             1         0.2                  0                 0
#>  4             1         0.2                  0                 0
#>  5             1         0.2                  0                 0
#>  6             1         0.4                  0                 0
#>  7             1         0.3                  0                 0
#>  8             1         0.2                  0                 0
#>  9             1         0.2                  0                 0
#> 10             1         0.1                  0                 0
#> # … with 140 more rows