Example: fuzz testing oneclust

In R, there is a nice R package fuzzr written by Matthew Lincoln. The package offers an example framework for fuzz testing R code. The off-the-shelf functions primarily emphasize the unexpected or non-standard input types, while custom tests can be easily created and evaluated.

Let’s use it to test my R package oneclust released in September 2020. The package is built for maximum homogeneity clustering of one-dimensional data. Although the core is implemented in C++ (statically typed), we will see that the R interface still allows some flexibility on input types.

The core function oneclust::oneclust() has four arguments:

oneclust(x, k, w = NULL, sort = TRUE)

where x is a numeric vector representing the samples to be clustered, k is the number of clusters, w is the optional sample weights vector, and sort indicates if x (and w) should be sorted. Example:

library("oneclust")

set.seed(42)
x <- sample(c(
  rnorm(50, sd = 0.2),
  rnorm(50, mean = 1, sd = 0.3),
  rnorm(100, mean = -1, sd = 0.25)
))

oneclust(x, 3)
#> $cluster
#>   [1] 3 1 3 2 1 1 1 3 2 3 2 2 3 1 1 1 1 1 2 1 1 1 1 1 2 3 2 2 1 1 1 2
#>  [33] 1 1 1 3 1 1 3 1 3 2 1 1 3 2 3 2 1 1 3 3 1 2 3 3 1 1 1 1 3 3 1 1
#>  [65] 1 1 1 3 2 2 2 2 2 1 1 2 3 2 1 2 1 3 2 3 1 2 3 1 3 1 1 2 1 1 2 3
#>  [97] 3 1 2 3 2 3 1 1 2 1 3 1 1 1 1 3 1 1 1 1 1 3 1 2 2 1 1 2 1 1 2 2
#> [129] 2 1 2 1 2 1 3 2 2 1 3 3 2 2 2 1 1 3 1 1 3 1 2 3 2 3 1 3 1 2 1 1
#> [161] 2 3 1 2 2 3 2 1 1 3 3 1 1 1 1 3 1 3 1 3 1 2 3 2 1 3 1 1 1 1 1 1
#> [193] 1 1 1 2 3 3 1 1
#> 
#> $cut
#> [1]   1 101 152

Use fuzzr to test how argument x handles all sorts of input types:

library("fuzzr")
library("kableExtra")

f <- fuzz_function(oneclust, "x", k = 3, tests = test_all())
f |>
  as.data.frame() |>
  kbl() |>
  kable_classic(
    lightable_options = c("striped", "hover"),
    html_font = "inherit",
    font_size = 12
  )

As expected, we see that the character and data frame inputs did not go through (Not compatible with requested type: [type=…; target=double]). Inputs with empty or NA values also returned index error. The tests that returned meaningful results include int_multiple, dbl_mutliple, fctr_multiple, fctr_missing_levels, lgl_mutliple, date_multiple, raw_na, df_one_col. It was a bit surprising for me that a factor with missing levels can run here:

fuzz_call(f, x = "fctr_missing_levels")
#> $fun
#> [1] "oneclust"
#> 
#> $args
#> $args$x
#> [1] a b c
#> Levels: a b c d
#> 
#> $args$k
#> [1] 3

It means that a “character factor” like the one below can actually be clustered:

oneclust(factor(letters[1:3], levels = letters[1:4]), k = 3)
#> $cluster
#> [1] 1 2 3
#> 
#> $cut
#> [1] 1 2 3

After checking the textbooks, we know that factors in R are built on top of integer vectors, so they were probably treated like one. A deeper understanding of R’s vector types helps interpret the other results, too.

Note that we only focused on the input types here. For statistical computing, the exceptions can be caused by subtle numerical issues and strange artifacts in the data, such as the distribution shapes and outliers. In those domain-specific cases, creating your own tests or even frameworks would be helpful.