data.table, from its very first releases, enabled the
usage of subset and with (or
within) functions by defining the [.data.table
method. subset and with are base R functions
that are useful for reducing repetition in code, enhancing readability,
and reducing number the total characters the user has to type. This
functionality is possible in R because of a quite unique feature called
lazy evaluation. This feature allows a function to catch its
arguments, before they are evaluated, and to evaluate them in a
different scope than the one in which they were called. Let’s recap
usage of the subset function.
subset(iris, Species == "setosa")
# Sepal.Length Sepal.Width Petal.Length Petal.Width Species
# 1 5.1 3.5 1.4 0.2 setosa
# 2 4.9 3.0 1.4 0.2 setosa
# ...Here, subset takes the second argument and evaluates it
within the scope of the data.frame given as its first
argument. This removes the need for variable repetition, making it less
prone to errors, and makes the code more readable.
The problem with this kind of interface is that we cannot easily parameterize the code that uses it. This is because the expressions passed to those functions are substituted before being evaluated.
There are multiple ways to work around this problem.
The easiest workaround is to avoid lazy evaluation in the
first place, and fall back to less intuitive, more error-prone
approaches like df[["variable"]], etc.
my_subset = function(data, col, val) {
data[data[[col]] == val & !is.na(data[[col]]), ]
}
my_subset(iris, col = "Species", val = "setosa")
# Sepal.Length Sepal.Width Petal.Length Petal.Width Species
# 1 5.1 3.5 1.4 0.2 setosa
# 2 4.9 3.0 1.4 0.2 setosa
# ...Here, we compute a logical vector of length nrow(iris),
then this vector is supplied to the i argument of
[.data.frame to perform ordinary “logical vector”-based
subsetting. To align with subset(), which also drops NAs,
we need to include an additional use of data[[col]] to
catch that. It works well enough for this simple example, but it lacks
flexibility, introduces variable repetition, and requires user to change
the function interface to pass the column name as a character rather
than unquoted symbol. The more complex the expression we need to
parameterize, the less practical this approach becomes.
parse / evalThis method is usually preferred by newcomers to R as it is, perhaps, the most straightforward conceptually. This way requires producing the required expression using string concatenation, parsing it, and then evaluating it.
my_subset = function(data, col, val) {
data = deparse(substitute(data))
col = deparse(substitute(col))
val = paste0("'", val, "'")
text = paste0("subset(", data, ", ", col, " == ", val, ")")
eval(parse(text = text)[[1L]])
}
my_subset(iris, Species, "setosa")
# Sepal.Length Sepal.Width Petal.Length Petal.Width Species
# 1 5.1 3.5 1.4 0.2 setosa
# 2 4.9 3.0 1.4 0.2 setosa
# ...We have to use deparse(substitute(...)) to catch the
actual names of objects passed to function, so we can construct the
subset function call using those original names. Although
this provides unlimited flexibility with relatively low complexity,
use of eval(parse(...)) should be avoided.
The main reasons are:
Martin Machler, R Project Core Developer, once said:
Sorry but I don’t understand why too many people even think a string was something that could be evaluated. You must change your mindset, really. Forget all connections between strings on one side and expressions, calls, evaluation on the other side. The (possibly) only connection is via
parse(text = ....)and all good R programmers should know that this is rarely an efficient or safe means to construct expressions (or calls). Rather learn more aboutsubstitute(),quote(), and possibly the power of usingdo.call(substitute, ......).
The aforementioned functions, along with some others (including
as.call, as.name/as.symbol,
bquote, and eval), can be categorized as
functions to compute on the language, as they operate on
language objects (e.g. call,
name/symbol).
my_subset = function(data, col, val) {
eval(substitute(subset(data, col == val)))
}
my_subset(iris, Species, "setosa")
# Sepal.Length Sepal.Width Petal.Length Petal.Width Species
# 1 5.1 3.5 1.4 0.2 setosa
# 2 4.9 3.0 1.4 0.2 setosa
# ...Here, we used the base R substitute function to
transform the call subset(data, col == val) into
subset(iris, Species == "setosa") by substituting
data, col, and val with their
original names (or values) from their parent environment. The benefits
of this approach to the previous ones should be clear. Note that because
we operate at the level of language objects, and don’t have to resort to
string manipulation, we refer to this as computing on the
language. There is a dedicated chapter on Computing on the
language in R
language manual. Although it is not necessary for programming on
data.table, we encourage readers to read this chapter for the sake
of better understanding this powerful and unique feature of R
language.
There are third party packages that can achieve what base R computing
on the language routines do (pryr, lazyeval
and rlang, to name a few).
Though these can be helpful, we will be discussing a
data.table-unique approach here.
Now that we’ve established the proper way to parameterize code that uses lazy evaluation, we can move on to the main subject of this vignette, programming on data.table.
Starting from version 1.15.0, data.table provides a robust mechanism
for parameterizing expressions passed to the i,
j, and by (or keyby) arguments of
[.data.table. It is built upon the base R
substitute function, and mimics its interface. Here, we
introduce substitute2 as a more robust and more
user-friendly version of base R’s substitute. For a
complete list of differences between base::substitute and
data.table::substitute2 please read the substitute2
manual.
Let’s say we want to have a general function that applies a function to sum of two arguments that has been applied another function. As a concrete example, below we have a function to compute the length of the hypotenuse in a right triangle, knowing length of its legs.
${\displaystyle c = \sqrt{a^2 + b^2}}$
The goal is the make every name in the above call able to be passed as a parameter.
substitute2(
outer(inner(var1) + inner(var2)),
env = list(
outer = "sqrt",
inner = "square",
var1 = "a",
var2 = "b"
)
)
# sqrt(square(a) + square(b))We can see in the output that both the functions names, as well as
the names of the variables passed to those functions, have been
replaced. We used substitute2 for convenience. In this
simple case, base R’s substitute could have been used as
well, though it would’ve required usage of
lapply(env, as.name).
Now, to use substitution inside [.data.table, we don’t
need to call the substitute2 function. As it is now being
used internally, all we have to do is to provide env
argument, the same way as we’ve provided it to the
substitute2 function in the example above. Substitution can
be applied to the i, j and by (or
keyby) arguments of the [.data.table method.
Note that setting the verbose argument to TRUE
can be used to print expressions after substitution is applied. This is
very useful for debugging.
Let’s use the iris data set as a demonstration. Just as
an example, let’s pretend we want to compute the
Sepal.Hypotenuse, treating the sepal width and length as if
they were legs of a right triangle.
DT = as.data.table(iris)
str(
DT[, outer(inner(var1) + inner(var2)),
env = list(
outer = "sqrt",
inner = "square",
var1 = "Sepal.Length",
var2 = "Sepal.Width"
)]
)
# num [1:150] 6.19 5.75 5.69 5.55 6.16 ...
# return as a data.table
DT[, .(Species, var1, var2, out = outer(inner(var1) + inner(var2))),
env = list(
outer = "sqrt",
inner = "square",
var1 = "Sepal.Length",
var2 = "Sepal.Width",
out = "Sepal.Hypotenuse"
)]
# Species Sepal.Length Sepal.Width Sepal.Hypotenuse
# <fctr> <num> <num> <num>
# 1: setosa 5.1 3.5 6.185467
# 2: setosa 4.9 3.0 5.745433
# ---
# 149: virginica 6.2 3.4 7.071068
# 150: virginica 5.9 3.0 6.618912In the last call, we added another parameter,
out = "Sepal.Hypotenuse", that conveys the intended name of
output column. Unlike base R’s substitute,
substitute2 will handle the substitution of the names of
call arguments, as well.
Substitution works on i and by (or
keyby), as well.
DT[filter_col %in% filter_val,
.(var1, var2, out = outer(inner(var1) + inner(var2))),
by = by_col,
env = list(
outer = "sqrt",
inner = "square",
var1 = "Sepal.Length",
var2 = "Sepal.Width",
out = "Sepal.Hypotenuse",
filter_col = "Species",
filter_val = I(c("versicolor", "virginica")),
by_col = "Species"
)]
# Species Sepal.Length Sepal.Width Sepal.Hypotenuse
# <fctr> <num> <num> <num>
# 1: versicolor 7.0 3.2 7.696753
# 2: versicolor 6.4 3.2 7.155418
# ---
# 99: virginica 6.2 3.4 7.071068
# 100: virginica 5.9 3.0 6.618912In the above example, we have seen a convenient feature of
substitute2: automatic conversion from strings into
names/symbols. An obvious question arises: what if we actually want to
substitute a parameter with a character value, so as to have
base R substitute behaviour. We provide a mechanism to
escape automatic conversion by wrapping the elements into base R
I() call. The I function marks an object as
AsIs, preventing its arguments from character-to-symbol
automatic conversion. (Read the ?AsIs documentation for
more details.) If base R behaviour is desired for the whole
env argument, then it’s best to wrap the whole argument in
I(). Alternatively, each list element can be wrapped in
I() individually. Let’s explore both cases below.
substitute( # base R behaviour
rank(input, ties.method = ties),
env = list(input = as.name("Sepal.Width"), ties = "first")
)
# rank(Sepal.Width, ties.method = "first")
substitute2( # mimicking base R's "substitute" using "I"
rank(input, ties.method = ties),
env = I(list(input = as.name("Sepal.Width"), ties = "first"))
)
# rank(Sepal.Width, ties.method = "first")
substitute2( # only particular elements of env are used "AsIs"
rank(input, ties.method = ties),
env = list(input = "Sepal.Width", ties = I("first"))
)
# rank(Sepal.Width, ties.method = "first")Note that conversion works recursively on each list element, including the escape mechanism of course.
The example presented above illustrates a neat and powerful way to make your code more dynamic. However, there are many other much more complex cases that a developer might have to deal with. One common problem handling a list of arguments of arbitrary length.
An obvious use case could be to mimic .SD functionality
by injecting a list call into the j
argument.
cols = c("Sepal.Length", "Sepal.Width")
DT[, .SD, .SDcols = cols]
# Sepal.Length Sepal.Width
# <num> <num>
# 1: 5.1 3.5
# 2: 4.9 3.0
# ---
# 149: 6.2 3.4
# 150: 5.9 3.0Having cols parameter, we’d want to splice it into a
list call, making j argument look like in the
code below.
DT[, list(Sepal.Length, Sepal.Width)]
# Sepal.Length Sepal.Width
# <num> <num>
# 1: 5.1 3.5
# 2: 4.9 3.0
# ---
# 149: 6.2 3.4
# 150: 5.9 3.0Splicing is an operation where a list of objects have to be
inlined into an expression as a sequence of arguments to call. In base
R, splicing cols into a list can be achieved
using as.call(c(quote(list), lapply(cols, as.name))).
Additionally, starting from R 4.0.0, there is new interface for such an
operation in the bquote function.
In data.table, we make it easier by automatically enlist-ing
a list of objects into a list call with those objects. This means that
any list object inside the env list argument
will be turned into list call, making the API for that use
case as simple as presented below.
# this works
DT[, j,
env = list(j = as.list(cols)),
verbose = TRUE]
# Argument 'j' after substitute: list(Sepal.Length, Sepal.Width)
# Detected that j uses these columns: [Sepal.Length, Sepal.Width]
# Sepal.Length Sepal.Width
# <num> <num>
# 1: 5.1 3.5
# 2: 4.9 3.0
# ---
# 149: 6.2 3.4
# 150: 5.9 3.0
# this will not work
#DT[, list(cols),
# env = list(cols = cols)]It is important to provide a call to as.list, rather
than simply a list, inside the env list argument, as is
shown in the above example.
Let’s explore enlist-ing in more detail.
DT[, j, # data.table automatically enlists nested lists into list calls
env = list(j = as.list(cols)),
verbose = TRUE]
# Argument 'j' after substitute: list(Sepal.Length, Sepal.Width)
# Detected that j uses these columns: [Sepal.Length, Sepal.Width]
# Sepal.Length Sepal.Width
# <num> <num>
# 1: 5.1 3.5
# 2: 4.9 3.0
# ---
# 149: 6.2 3.4
# 150: 5.9 3.0
DT[, j, # turning the above 'j' list into a list call
env = list(j = quote(list(Sepal.Length, Sepal.Width))),
verbose = TRUE]
# Argument 'j' after substitute: list(Sepal.Length, Sepal.Width)
# Detected that j uses these columns: [Sepal.Length, Sepal.Width]
# Sepal.Length Sepal.Width
# <num> <num>
# 1: 5.1 3.5
# 2: 4.9 3.0
# ---
# 149: 6.2 3.4
# 150: 5.9 3.0
DT[, j, # the same as above but accepts character vector
env = list(j = as.call(c(quote(list), lapply(cols, as.name)))),
verbose = TRUE]
# Argument 'j' after substitute: list(Sepal.Length, Sepal.Width)
# Detected that j uses these columns: [Sepal.Length, Sepal.Width]
# Sepal.Length Sepal.Width
# <num> <num>
# 1: 5.1 3.5
# 2: 4.9 3.0
# ---
# 149: 6.2 3.4
# 150: 5.9 3.0Now let’s try to pass a list of symbols, rather than list call to
those symbols. We’ll use I() to escape automatic
enlist-ing but, as this will also turn off character to symbol
conversion, we also have to use as.name.
DT[, j, # list of symbols
env = I(list(j = lapply(cols, as.name))),
verbose = TRUE]
# Argument 'j' after substitute: list(Sepal.Length, Sepal.Width)
# Error in `[.data.table`(DT, , j, env = I(list(j = lapply(cols, as.name))), : When with=FALSE, j-argument should be of type logical/character/integer indicating the columns to select.
DT[, j, # again the proper way, enlist list to list call automatically
env = list(j = as.list(cols)),
verbose = TRUE]
# Argument 'j' after substitute: list(Sepal.Length, Sepal.Width)
# Detected that j uses these columns: [Sepal.Length, Sepal.Width]
# Sepal.Length Sepal.Width
# <num> <num>
# 1: 5.1 3.5
# 2: 4.9 3.0
# ---
# 149: 6.2 3.4
# 150: 5.9 3.0Note that both expressions, although visually appearing to be the same, are not identical.
str(substitute2(j, env = I(list(j = lapply(cols, as.name)))))
# List of 2
# $ : symbol Sepal.Length
# $ : symbol Sepal.Width
str(substitute2(j, env = list(j = as.list(cols))))
# language list(Sepal.Length, Sepal.Width)For more detailed explanation on that matter, please see the examples
in the substitute2
documentation.
Let’s take, as an example of a more complex function, calculating root mean square.
${\displaystyle x_{\text{RMS}}={\sqrt{{\frac{1}{n}}\left(x_{1}^{2}+x_{2}^{2}+\cdots +x_{n}^{2}\right)}}}$
It takes arbitrary number of variables on input, but now we cannot
just splice a list of arguments into a list call because each
of those arguments have to be wrapped in a square call. In
this case, we have to splice by hand rather than relying on
data.table’s automatic enlist.
First, we have to construct calls to the square function
for each of the variables (see inner_calls). Then, we have
to reduce the list of calls into a single call, having a nested sequence
of + calls (see add_calls). Lastly, we have to
substitute the constructed call into the surrounding expression (see
rms).
outer = "sqrt"
inner = "square"
vars = c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width")
syms = lapply(vars, as.name)
to_inner_call = function(var, fun) call(fun, var)
inner_calls = lapply(syms, to_inner_call, inner)
print(inner_calls)
# [[1]]
# square(Sepal.Length)
#
# [[2]]
# square(Sepal.Width)
#
# [[3]]
# square(Petal.Length)
#
# [[4]]
# square(Petal.Width)
to_add_call = function(x, y) call("+", x, y)
add_calls = Reduce(to_add_call, inner_calls)
print(add_calls)
# square(Sepal.Length) + square(Sepal.Width) + square(Petal.Length) +
# square(Petal.Width)
rms = substitute2(
expr = outer((add_calls) / len),
env = list(
outer = outer,
add_calls = add_calls,
len = length(vars)
)
)
print(rms)
# sqrt((square(Sepal.Length) + square(Sepal.Width) + square(Petal.Length) +
# square(Petal.Width))/4L)
str(
DT[, j, env = list(j = rms)]
)
# num [1:150] 3.17 2.96 2.92 2.87 3.16 ...
# same, but skipping last substitute2 call and using add_calls directly
str(
DT[, outer((add_calls) / len),
env = list(
outer = outer,
add_calls = add_calls,
len = length(vars)
)]
)
# num [1:150] 3.17 2.96 2.92 2.87 3.16 ...
# return as data.table
j = substitute2(j, list(j = as.list(setNames(nm = c(vars, "Species", "rms")))))
j[["rms"]] = rms
print(j)
# list(Sepal.Length = Sepal.Length, Sepal.Width = Sepal.Width,
# Petal.Length = Petal.Length, Petal.Width = Petal.Width, Species = Species,
# rms = sqrt((square(Sepal.Length) + square(Sepal.Width) +
# square(Petal.Length) + square(Petal.Width))/4L))
DT[, j, env = list(j = j)]
# Sepal.Length Sepal.Width Petal.Length Petal.Width Species rms
# <num> <num> <num> <num> <fctr> <num>
# 1: 5.1 3.5 1.4 0.2 setosa 3.172538
# 2: 4.9 3.0 1.4 0.2 setosa 2.958462
# ---
# 149: 6.2 3.4 5.4 2.3 virginica 4.594834
# 150: 5.9 3.0 5.1 1.8 virginica 4.273757
# alternatively
j = as.call(c(
quote(list),
lapply(setNames(nm = vars), as.name),
list(Species = as.name("Species")),
list(rms = rms)
))
print(j)
# list(Sepal.Length = Sepal.Length, Sepal.Width = Sepal.Width,
# Petal.Length = Petal.Length, Petal.Width = Petal.Width, Species = Species,
# rms = sqrt((square(Sepal.Length) + square(Sepal.Width) +
# square(Petal.Length) + square(Petal.Width))/4L))
DT[, j, env = list(j = j)]
# Sepal.Length Sepal.Width Petal.Length Petal.Width Species rms
# <num> <num> <num> <num> <fctr> <num>
# 1: 5.1 3.5 1.4 0.2 setosa 3.172538
# 2: 4.9 3.0 1.4 0.2 setosa 2.958462
# ---
# 149: 6.2 3.4 5.4 2.3 virginica 4.594834
# 150: 5.9 3.0 5.1 1.8 virginica 4.273757In [.data.table, it is also possible to use other
mechanisms for variable substitution or for passing quoted expressions.
These include get and mget for inline
injection of variables by providing their names as strings, and
eval that tells [.data.table that the
expression we passed into an argument is a quoted expression and that it
should be handled differently. Those interfaces should now be considered
retired and we recommend using the new env argument,
instead.
getmgetv = c("Petal.Width", "Sepal.Width")
DT[, lapply(mget(v), mean)]
# Petal.Width Sepal.Width
# <num> <num>
# 1: 1.199333 3.057333
DT[, lapply(v, mean),
env = list(v = as.list(v))]
# V1 V2
# <num> <num>
# 1: 1.199333 3.057333
DT[, lapply(v, mean),
env = list(v = as.list(setNames(nm = v)))]
# Petal.Width Sepal.Width
# <num> <num>
# 1: 1.199333 3.057333evalInstead of using eval function we can provide quoted
expression into the element of env argument, no extra
eval call is needed then.