# Introduction

This package contains a set of tools to classify the pixels of digital images into colour categories arbitrarily defined by the user. It contains functions to

• visualize the distribution of the pixel colours in the images,
• define classification rules
• classify the pixels and to store this information in R objects,
• save these as image files.

It is a simple version of the multivariate technique known as Support Vector Machine (Cortes and Vapnik, 1995; Bennet and Campbell, 2000), adapted to this particular use.

### The procedure

The basic steps of the procedure are the following:

• One or more digital images in JPEG or TIFF format is imported into R. The categories to identify are represented in this set (the test set).
• The values of the three three colour variables (or bands) that compose each image (R, G, and B) are transformed into proportions (r, g and b).
• The pixels of the image are plotted in the plane defined by two of the transformed variables (the user can select them arbitrarily) and, hopefully, they would form separate clusters (pixel categories).
• The user then traces straight lines that separate the pixel clusters. Using the mathematical expression for these rules and the rgb values, each pixel can be tested for membership in each category (see below).
• Recording the results of the tests as 1 or 0 (pass/fail), an incidence matrix is build for that rule. This is the result of the procedure, which can be submitted to posterior analysis or used to create a new version of the original image showing the category of each pixel.

The second step simplifies the problem because it makes one of the variables dependent on the other two (as r + g + b = 1). Moreover, the transformation eliminates colour variations due to differences in illumination.

The expressions for classification rules are the same as the expression for a straight line but using one of the comparison operators $$<$$, $$\leq$$, $$>$$ or $$\geq$$. For example: $$r \geq a g +c$$, being $$a$$ and $$c$$ the slope and intercept of the line, and $$r$$ and $$g$$ the colour variables selected for the classification. A single line can produce two classification rules.

### Using several rules per category

When there are more than two categories, or when the cluster of points has a complex shape, a single rule is not enough. In these cases the procedure has additional steps:

• several rules are defined for each category,
• incidence matrices are created for each rule,
• the incidence matrices are combined with the & operator to obtain the category incidence matrix.

The last step is equivalent to estimate the union of the incidence matrices, i e $$\mathbf{M} = \mathbf{M}_{1} \cap \mathbf{M}_{2} \cap \ldots \cap \mathbf{M}_{p}$$, being p the number of rules.

### Concave category shapes

A caveat of the method is that the rules must delimit a convex polygon to combine the individual rule results successfully (in a convex polygon, a line joining any two internal points is contained in the polygon). Not all clusters have convex shape. In these cases, the cluster must be divided in convex sub-polygons (subcategories) for which rules are defined as before. The incidence matrices of the subcategories are combined using the | operator, i.e. $$\mathbf{M} = \mathbf{M}_{1} \cup \mathbf{M}_{2} \cup \ldots \cup \mathbf{M}_{s}$$, being s the number of subcategories. Note that any polygon, convex or not, can be subdivided in triangles and, as triangles are convex polygons, it is always possible to solve this problem. Note that the goal is to obtain a minimal set of convex polygons, not a complete triangulation. The example presented below is one of such cases.

# The session

What follows is a sample session illustrating both the method and the use of the package functions. It uses an example image and a test set created by cutting small areas out of the example image. It is not a good test set, see below, but it is enough to show how the method works, and its problems.

The package is loaded in the usual way:

library(pixelclasser)


Figure 1 shows the example images included in the package. The goal of this example session is to classify the pixels of the example image into dead, oak and ivy categories. The small images are fragments of the main image and are the test set, i.e. representatives of each class. In a real case, a more extensive test set should be used to represent the whole variation of the categories.

As the images are included in the package as external (non R) data, they are loaded with the following code:

ivy_oak_rgb <- read_image(system.file("extdata", "IvyOak400x300.JPG", package = "pixelclasser"))
test_ivy_rgb <- read_image(system.file("extdata", "TestIvy.JPG", package = "pixelclasser"))
test_oak_rgb <- read_image(system.file("extdata", "TestOak.JPG", package = "pixelclasser"))


The function read_image() performs the first step of the procedure. It stores the image as an array of rgb values, which are the proportion of each colour variable (i.e. R /(R+G+B), and so on). This uses functions from packages jpeg or tiff, and uses the extension in the file name to identify which one to use.

### Pixel distributions in rgb space

Before plotting pixels and lines, it is convenient to define a set of colours to use throughout the session:

transparent_black <- "#00000008"
brown <- "#c86432ff"
yellow <- "#ffcd0eff"
blue <- "#5536ffff"
green <- "#559800ff"


The next step is to visualize the distribution of the pixels in rgb space, but only two variables are needed. Any pair of variables would do, but a particular combination might produce a better display of the clusters. It is a matter of try the three possible combinations to select the most convenient.

Plotting the pixels is a two-step procedure (Fig. 2): a void plot is drawn (using plot_rgb_plane()) and then the pixels are added to the plot (the use of a transparent black colour, #00000008, creates a “density plot” effect):

plot_rgb_plane("r", "b", main = "Image: ivy and oak")
plot_pixels(ivy_oak_rgb, "r", "b", col = transparent_black)


The coloured lines are an aid to interpret the graph: no pixels could be found outside the blue lines, and the red lines converging in the barycentre of the triangle (r, g, b) = (1/3, 1/3, 1/3), define the areas where a colour is dominant. Note that graphical parameters (main to set the title of Fig. 2) can be passed to the function to change the final appearance of the graph. All the auxiliary lines are optional. Figure 3 omits them and uses different colour variables to plot the pixels.

plot_rgb_plane("r", "g", plot_limits = FALSE, plot_guides = FALSE, plot_grid = FALSE)
plot_pixels(ivy_oak_rgb, "r", "g", col = transparent_black)


There are two clear pixel clusters and a small, but noticeable, quantity of pixels in between. Also visible are linear patterns that are artefacts created because the RGB data are discrete variables (eight bit in the most common cases). These are more appreciable in the following graphs, which are restricted to the area occupied by the pixels. In the following examples g and b will be used as variables x and y for plotting and pixel classification.

### Plotting the pixels of the test images

The following code plots the pixels of the example image on the gb plane and then adds the pixels of the test images, using arbitrary colours (Fig. 4). Usually, only the pixels of the test images are plotted before placing the rules, but there both the “problem” and test images were plotted (note that real studies involve many problem images, not one). This was done to assess the quality of the test images (see below). To create the figure, the graphic parameters xlim and ylim were used to limit the extent of the plot to the area occupied by the pixels:

plot_rgb_plane("g", "b", xlim = c(0.2, 0.6), ylim = c(0.1, 0.33))
plot_pixels(ivy_oak_rgb, "g", "b", col = transparent_black)
plot_pixels(test_oak_rgb, "g", "b", col = green)
plot_pixels(test_ivy_rgb, "g", "b", col = blue)
plot_pixels(test_dead_rgb, "g", "b", col = brown)


Figure 4 shows that the clusters of pixels in the ivy_oak_rgb image correspond to dead leaves (on the left), and oak and ivy (on the right).

The small areas taken as test images were not representative of the whole pixel set in the image, as they do not cover the same area as the black pixels. This is not a surprise given that a single sample was collected for each type of pixel. In a real study, the set of test images must be selected to be representative of the classes. This is, obviously, a need for any method that uses a test set, not only for pixelclasser.

Warning: plotting several million points in an R graph is an slow process. Be patient or use images as small as possible. Using a nice smartphone with a petapixel camera sensor to capture images is good for artistic purposes, but not always for efficient scientific work.

### Defining the rules

Defining the rules that classify the pixels is a matter of tracing straight lines to separate the clusters. In this example, a single line more or less equidistant to both clusters should suffice to separate them. The intermediate points will be arbitrarily ascribed to one category.

The rules are defined by setting the name of the rule, the colour variables to use, the coordinates of two points in the plane and a comparison operator. The exact placement of the line is an arbitrary decision, as the method does not include any mechanism to place it automatically.

There are two methods to create the rule. The first uses the function create_rule(), which receives a list with the coordinates of two points defining a line in the selected subspace, and returns a pixel_rule object. In the following example, the points with coordinates (g, b) = (0.345, 1/3), and (g,b) = (0.40, 0.10) defined the position of the first line, and were selected by trial and error. The adequate operator must be included in the rule definition:

rule_01 <- pixel_rule("rule_01", "g", "b", list(c(0.345, 1/3), c(0.40, 0.10)), "<")
rule_02 <- pixel_rule("rule_02", "g", "b", list(c(0.345, 1/3), c(0.40, 0.10)), ">=")


Both rules are described by the same line but use different comparison operator. rule_01 includes the pixels at the left (under) of the line and rule_02 those at the right (over) and on the line, i.e. the dead leaves and the fresh leaves, respectively. Each line can generate two rules, but beware: if > and < define the rules, then the points on the line will not belong to any category, and if >= and <= are used, the points on the line will belong to the two categories simultaneously. The function that classifies the pixels can identify the second type of error, but if there are legitimate unclassified points, the errors of the first type can pass unnoticed.

The second method uses place_rule(), which is a wrapper for graphics::locator() that allows the user to select the two points by clicking in the rgb plot with the mouse:

rp01 <- place_rule("g", "b")
rp02 <- place_rule("g", "b", "v")
rule_07 <- pixel_rule("rule_07", "g", "b", rp02, ">=")


The function returns an object of class pixel_rule_points that can then be passed to pixel_rule() in the parameter line_points. The second example produces a vertical line (use “h” for horizontal lines) When using these options the function adjusts the coordinates, as it would be difficult to create true vertical or horizontal lines with the mouse. To make the code run automatically, create_rule() is used in this vignette, but using place_rule() is the easiest way to define the rules. It is even easier to place the call to place_rule() in the call to create_rule() to avoid creating the intermediate pixel_rule_points object:

rule_07 <- pixel_rule("rule_07", "g", "b", place_rule("g", "b"), "<")


Note that both pixel_rule() and the pixel_rule_points object must use the same colour variables as axis. pixel_rule() throws an error if this condition does not hold.

The rule objects store the values passed as parameters, the parameters of the equation of the line (a and c), and a textual representation of the equation which will be evaluated by the classification function. To check the correctness of the rules, the lines can be added to the plot, as in Fig. 5:

plot_rgb_plane("g", "b", xlim = c(0.2, 0.6), ylim = c(0.1, 0.33))
plot_pixels(ivy_oak_rgb, "g", "b", col = transparent_black)
plot_pixels(test_oak_rgb, "g", "b", col = green)
plot_pixels(test_ivy_rgb, "g", "b", col = blue)
plot_pixels(test_dead_rgb, "g", "b", col = brown)
plot_rule(rule_01, lty = 2, col = brown)


In order to classify the fresh leaves into ivy and oak categories, more rules are needed. The pixels of the oak test image were plotted and used to define additional rules (again by trial and error):

rule_03 <- pixel_rule("rule_03","g", "b", list(c(0.35, 0.30), c(0.565, 0.10)), "<")
rule_04 <- pixel_rule("rule_04","g", "b", list(c(0.35, 0.25), c(0.5, 0.25)), "<")


Figure 6 shows the pixels of the oak test image and the lines that limit this class. Line type and colour were set using the graphical parameters lty and col (see graphics::par)) ... argument of plot_rule():

plot_rgb_plane("g", "b", xlim = c(0.2, 0.6), ylim = c(0.1, 0.33), plot_limits = F, plot_guides = F)
plot_pixels(test_oak_rgb, "g", "b", col = green)
plot_rule(rule_01, lty = 2, col = green)
plot_rule(rule_03, lty = 2, col = green)
plot_rule(rule_04, lty = 2, col = green)