# Undersampling + bagging = better generalized classification for imbalanced dataset

This post is reproduced from a post of my Japanese blog.

A friend of mine, an academic researcher in machine learning field tweeted as below.

I've studied how to handle imbalanced data, but I found Wallace et al. ICDM'11 concluded that you should do "undersampling + bagging".

In the other post of my Japanese blog, I argued about how to handle imbalanced data with "class weight" in which cost of negative samples is reduced by a ratio of negative to positive samples in loss function.

However, I thought "undersampling + bagging" would work better, so I would try it here. Please notice that here I only used randomForest {randomForest} in R for this trial just for simplicity and computational cost. If you're interested in any other classifier including deep NN, please try by yourself :P)

Note

If you use Python, already there is a good package for "undersampling + bagging".

### Dataset

I prepared a dataset with 250 positive and 3750 negative samples. Please get it from my GitHub repository below.

First, import it as a data frame "d". In addition, let's create a grid to draw decision boundaries.

```> px <- seq(-4,4,0.05)
> py <- seq(-4,4,0.05)
> pgrid <- expand.grid(px, py)
> names(pgrid) <- names(d)[-3]
```

OK, let's proceed.

### Class weight

If you use randomForest, what you have to do is just to give a ratio of positive to negative samples to "classwt" argument.

```> d.rf <- randomForest(as.factor(label)~., d, classwt=c(1, 3750/250))
> out.rf <- predict(d.rf, newdata=pgrid)
> plot(d[,-3], col=d[,3]+1, xlim=c(-4,4), ylim=c(-4,4), cex=0.5, pch=19)
> par(new=T)
> contour(px, py, array(out.rf, c(length(px), length(py))), levels=0.5, col='purple', lwd=5, drawlabels=F)
``` The result was as I expected; a little more expanded decision boundary than its original one.

### Undersampling + bagging

First, I tried a bagging with 10 sub-classifiers. Please accept my dirty codes :P)

```> outbag.rf <- c()
> for (i in 1:10){
+     set.seed(i)
+     train0 <- d[sample(3750, 250, replace=F),]
+     train1 <- d[3751:4000,]
+     train <- rbind(train0, train1)
+     model <- randomForest(as.factor(label)~., train)
+     tmp <- predict(model, newdata=pgrid)
+     outbag.rf <- cbind(outbag.rf, tmp)
+ }
> outbag.rf.grid <- apply(outbag.rf, 1, mean)-1
> plot(d[,-3], col=d[,3]+1, xlim=c(-4,4), ylim=c(-4,4), cex=0.5, pch=19)
> par(new=T)
> contour(px, py, array(out10.rf.grid, c(length(px), length(py))), levels=0.5, col='purple', lwd=5, drawlabels=F)
``` It looks that the classification region of the positive samples got expanded a little bit. How about 50 sub-classifiers? It appears that jaggy parts got reduced. OK, let's try 100 sub-classifiers. Got it, once it finished.

### Advanced: positive samples as embedded into negative samples

The dataset above was generated by a script below.

```> set.seed(1001)
> x1 <- cbind(rnorm(1000, 1, 1), rnorm(1000, 1, 1))
> set.seed(1002)
> x2 <- cbind(rnorm(1000, -1, 1), rnorm(1000, 1, 1))
> set.seed(1003)
> x3 <- cbind(rnorm(1000, -1, 1), rnorm(1000, -1, 1))
> set.seed(4001)
> x41 <- cbind(rnorm(250, 0.5, 0.5), rnorm(250, -0.5, 0.5))
> set.seed(4002)
> x42 <- cbind(rnorm(250, 1, 0.5), rnorm(250, -0.5, 0.5))
> set.seed(4003)
> x43 <- cbind(rnorm(250, 0.5, 0.5), rnorm(250, -1, 0.5))
> set.seed(4004)
> x44 <- cbind(rnorm(250, 1, 0.5), rnorm(250, -1, 0.5))
> d <- rbind(x1,x2,x3,x41,x42,x43,x44)
> d <- data.frame(x = d[,1], y = d[,2], label=c(rep(0, 3750), rep(1, 250)))
```

In short, positive samples in this dataset are concentrated in the lowest and rightmost part of the 4th quadrant, so I think it's natural that we got a result above. Thus, I tried to move the positive samples more inside, as embedded into the negative samples.

```> d1 <- d
> d1\$label <- c(rep(0,3000), rep(1,250), rep(0,750))
```

First, let's try "classwt".

```> d1.rf <- randomForest(as.factor(label)~., d1, classwt=c(1, 3750/250))
> out.rf <- predict(d1.rf, newdata=pgrid)
> plot(d1[,-3], col=d1[,3]+1, xlim=c(-4,4), ylim=c(-4,4), cex=0.5, pch=19)
> par(new=T)
> contour(px, py, array(out.rf, c(length(px), length(py))), levels=0.5, col='purple', lwd=5, drawlabels=F)
``` OMG... what a tiny decision boundary, even like overfitting. OK, let's try "undersampling + bagging". For simplicity, I tried 100 sub-classifiers here.

```> outbag.rf <- c()
> for (i in 1:100){
+ set.seed(i)
+ train.tmp <- d1[d1\$label==0,]
+ train0 <- train.tmp[sample(3750, 250, replace=F),]
+ train1 <- d1[3001:3250,]
+ train <- rbind(train0, train1)
+ model <- randomForest(as.factor(label)~., train)
+ tmp <- predict(model, newdata=pgrid)
+ outbag.rf <- cbind(outbag.rf, tmp)
+ }
> outbag.rf.grid <- apply(outbag.rf, 1, mean)-1
> plot(d1[,-3], col=d1[,3]+1, xlim=c(-4,4), ylim=c(-4,4), cex=0.5, pch=19)
> par(new=T)
> contour(px, py, array(outbag.rf.grid, c(length(px), length(py))), levels=0.5, col='purple', lwd=5, drawlabels=F)
``` Yes, I didi it. You see a broad decision boundary in the 4th quadrant just near to (0,0), which looks to get a little bit more generalized than usual.

### My comment

Two cases clearly showed that "undersampling + bagging" does better than "class weight" in order to get well-generalized decision boundary. To tell the truth, I rather got shocked because "classwt" causes overfitting shown in these two cases...