Generic filters
Exact matches only

# Non-Linear Classification in R with Decision Trees

Last Updated on August 22, 2019

In this post you will discover 7 recipes for non-linear classification with decision trees in R.

All recipes in this post use the iris flowers dataset provided with R in the datasets package. The dataset describes the measurements if iris flowers and requires classification of each observation to one of three flower species.

Discover how to prepare data, fit machine learning models and evaluate their predictions in R with my new book, including 14 step-by-step tutorials, 3 projects, and full source code.

Let’s get started. Classification with Decision Trees
Photo by stwn, some rights reserved

## Classification and Regression Trees

Classification and Regression Trees (CART) split attributes based on values that minimize a loss function, such as sum of squared errors.

The following recipe demonstrates the recursive partitioning decision tree method on the iris dataset.

library(rpart)
data(iris)
# fit model
fit <- rpart(Species~., data=iris)
# summarize the fit
summary(fit)
# make predictions
predictions <- predict(fit, iris[,1:4], type=”class”)
# summarize accuracy
table(predictions, iris\$Species)

library(rpart)

data(iris)

# fit model

fit <- rpart(Species~., data=iris)

# summarize the fit

summary(fit)

# make predictions

predictions <- predict(fit, iris[,1:4], type=”class”)

# summarize accuracy

table(predictions, iris\$Species)

## C4.5

The C4.5 algorithm is an extension of the ID3 algorithm and constructs a decision tree to maximize information gain (difference in entropy).

The following recipe demonstrates the C4.5 (called J48 in Weka) decision tree method on the iris dataset.

library(RWeka)
data(iris)
# fit model
fit <- J48(Species~., data=iris)
# summarize the fit
summary(fit)
# make predictions
predictions <- predict(fit, iris[,1:4])
# summarize accuracy
table(predictions, iris\$Species)

library(RWeka)

data(iris)

# fit model

fit <- J48(Species~., data=iris)

# summarize the fit

summary(fit)

# make predictions

predictions <- predict(fit, iris[,1:4])

# summarize accuracy

table(predictions, iris\$Species)

### Need more Help with R for Machine Learning?

Take my free 14-day email course and discover how to use R on your project (with sample code).

Click to sign-up and also get a free PDF Ebook version of the course.

## PART

PART is a rule system that creates pruned C4.5 decision trees for the data set and extracts rules and those instances that are covered by the rules are removed from the training data. The process is repeated until all instances are covered by extracted rules.

The following recipe demonstrates the PART rule system method on the iris dataset.

library(RWeka)
data(iris)
# fit model
fit <- PART(Species~., data=iris)
# summarize the fit
summary(fit)
# make predictions
predictions <- predict(fit, iris[,1:4])
# summarize accuracy
table(predictions, iris\$Species)

library(RWeka)

data(iris)

# fit model

fit <- PART(Species~., data=iris)

# summarize the fit

summary(fit)

# make predictions

predictions <- predict(fit, iris[,1:4])

# summarize accuracy

table(predictions, iris\$Species)

## Bagging CART

Bootstrapped Aggregation (Bagging) is an ensemble method that creates multiple models of the same type from different sub-samples of the same dataset. The predictions from each separate model are combined together to provide a superior result. This approach has shown participially effective for high-variance methods such as decision trees.

The following recipe demonstrates bagging applied to the recursive partitioning decision tree for the iris dataset.

library(ipred)
data(iris)
# fit model
fit <- bagging(Species~., data=iris)
# summarize the fit
summary(fit)
# make predictions
predictions <- predict(fit, iris[,1:4], type=”class”)
# summarize accuracy
table(predictions, iris\$Species)

library(ipred)

data(iris)

# fit model

fit <- bagging(Species~., data=iris)

# summarize the fit

summary(fit)

# make predictions

predictions <- predict(fit, iris[,1:4], type=”class”)

# summarize accuracy

table(predictions, iris\$Species)

## Random Forest

Random Forest is variation on Bagging of decision trees by reducing the attributes available to making a tree at each decision point to a random sub-sample. This further increases the variance of the trees and more trees are required.

The following recipe demonstrate the random forest method applied to the iris dataset.

library(randomForest)
data(iris)
# fit model
fit <- randomForest(Species~., data=iris)
# summarize the fit
summary(fit)
# make predictions
predictions <- predict(fit, iris[,1:4])
# summarize accuracy
table(predictions, iris\$Species)

library(randomForest)

data(iris)

# fit model

fit <- randomForest(Species~., data=iris)

# summarize the fit

summary(fit)

# make predictions

predictions <- predict(fit, iris[,1:4])

# summarize accuracy

table(predictions, iris\$Species)

Boosting is an ensemble method developed for classification for reducing bias where models are added to learn the misclassification errors in existing models. It has been generalized and adapted in the form of Gradient Boosted Machines (GBM) for use with CART decision trees for classification and regression.

The following recipe demonstrate the Gradient Boosted Machines (GBM) method in the iris dataset.

library(gbm)
data(iris)
# fit model
fit <- gbm(Species~., data=iris, distribution=”multinomial”)
# summarize the fit
print(fit)
# make predictions
predictions <- predict(fit, iris)
# summarize accuracy
table(predictions, iris\$Species)

library(gbm)

data(iris)

# fit model

fit <- gbm(Species~., data=iris, distribution=”multinomial”)

# summarize the fit

print(fit)

# make predictions

predictions <- predict(fit, iris)

# summarize accuracy

table(predictions, iris\$Species)

## Boosted C5.0

The C5.0 method is a further extension of C4.5 and pinnacle of that line of methods. It was proprietary for a long time, although the code was released recently and is available in the C50 package.

The following recipe demonstrates the C5.0 with boosting method applied to the iris dataset.

library(C50)
data(iris)
# fit model
fit <- C5.0(Species~., data=iris, trials=10)
# summarize the fit
print(fit)
# make predictions
predictions <- predict(fit, iris)
# summarize accuracy
table(predictions, iris\$Species)

library(C50)

data(iris)

# fit model

fit <- C5.0(Species~., data=iris, trials=10)

# summarize the fit

print(fit)

# make predictions

predictions <- predict(fit, iris)

# summarize accuracy

table(predictions, iris\$Species)

## Summary

In this post you discovered 7 recipes for non-linear classification using decision trees in R using the iris flowers dataset.

Each recipe is generic and ready for you to copy and paste and modify for your own problem.

## Discover Faster Machine Learning in R! #### Develop Your Own Models in Minutes

…with just a few lines of R code

Discover how in my new Ebook:
Machine Learning Mastery With R

Covers self-study tutorials and end-to-end projects like: