Learning Methods of Deep Learning

create by Deepfinder

Agenda

1. Learning from a Teacher: Supervised Learning
2. Seeing the Pattern: Unsupervised Learning
3. Learning Without a Teacher: Self-supervised Learning
4. Learning from a Few Labels: Semi-supervised Learning
5. Learning by Comparison: Contrastive Learning
6. Learning by Transfer: Transfer Learning
7. Learning Through Competition: Adversarial Learning
8. Strength in Numbers: Ensemble Learning
9. Different Paths, Shared Goal: Federated Learning
10. Learning Through Trial and Error: Reinforcement Learning
11. Learning by Asking: Active Learning
12. Learning to Learn: Meta-Learning

Tutorial 08 – Strength in Numbers: Ensemble Learning

In machine learning and deep learning, the performance of a single model is often limited by model complexity, data quality, training methods, and many other factors.

Ensemble learning provides a powerful way to handle these limits. It combines the predictions of multiple models to improve overall performance, robustness, and generalization.

Ensemble learning is widely used in traditional machine learning, such as Random Forest and Gradient Boosting Trees. In recent years, as deep learning has developed quickly, the idea of ensemble learning has also been used successfully in deep learning. It often gives better results on complex tasks such as image classification and natural language processing.

Core Ideas of Ensemble Learning

The basic idea of ensemble learning is to combine the predictions of several base models. This can be understood as “many hands make light work.” The core ideas include the following:

1. Base Model

A base model is a single model that takes part in the ensemble. In deep learning, a base model can be a convolutional neural network (CNN), a recurrent neural network (RNN), and so on.

2. Ensemble Strategies

• Bagging (Bootstrap Aggregating): resample the dataset and train multiple independent base models, such as in Random Forest.
• Boosting: use weighted learning to optimize base models step by step. Each new model focuses more on the data that earlier models handled poorly, such as in AdaBoost and gradient boosting.
• Stacking (stacked generalization): use a meta-learner to combine the outputs of several base models.

3. Voting Mechanisms

• Hard voting: use majority voting based on the class prediction from each model.
• Soft voting: combine the predicted probabilities from each model, usually by weighted averaging or by taking the class with the highest probability.
• With these strategies, ensemble learning can reduce the risk of overfitting in a single model. It can also use the complementary strengths of different models to improve overall performance.

Ensemble Learning in Deep Learning

When combined with deep learning, ensemble learning is useful in several ways:

• Model averaging: combine predictions from multiple models by simple averaging or weighted averaging. This can reduce the prediction bias of a single model.
• Model stacking: use the outputs of multiple models as input features, then train a lightweight learner, such as logistic regression or a shallow neural network, to learn better prediction rules.
• Model diversity: use different network architectures, training strategies, and data augmentation methods. This helps improve the robustness of the ensemble model.

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
import numpy as np

# Check whether GPU is available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Data preprocessing
transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])

train_dataset = datasets.CIFAR10(root='datasets', train=True, download=True, transform=transform)
test_dataset = datasets.CIFAR10(root='datasets', train=False, download=True, transform=transform)

train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)

# Define a simple CNN model
class SimpleCNN(nn.Module):
    def __init__(self):
        super(SimpleCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(64 * 16 * 16, 128)  # Correct the input size of the fully connected layer
        self.fc2 = nn.Linear(128, 10)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.relu(self.conv1(x))
        x = self.pool(self.relu(self.conv2(x)))
        x = x.view(x.size(0), -1)  # Flatten to [batch_size, features]
        x = self.relu(self.fc1(x))
        x = self.fc2(x)
        return x

# Define the meta-model (neural network)
class MetaModel(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(MetaModel, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, output_size)
        self.relu = nn.ReLU()

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.fc2(x)
        return x

Files already downloaded and verified
Files already downloaded and verified

# Model training function
def train_model(model, train_loader, epochs=10):
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(model.parameters(), lr=0.001)

    model.to(device)  # Move the model to GPU

    for epoch in range(epochs):
        model.train()
        for images, labels in train_loader:
            images, labels = images.to(device), labels.to(device)  # Move data to GPU

            optimizer.zero_grad()
            outputs = model(images)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

    # Evaluate on the test set
    model.eval()
    correct = 0
    total = 0
    with torch.no_grad():
        for images, labels in test_loader:
            images, labels = images.to(device), labels.to(device)  # Move data to GPU

            outputs = model(images)
            _, predicted = torch.max(outputs, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

    return correct / total

# Get the outputs of the base models
def get_predictions(models, loader):
    outputs_list = []
    labels_list = []
    with torch.no_grad():
        for images, labels in loader:
            images, labels = images.to(device), labels.to(device)  # Move data to GPU

            # Get the outputs of all base models
            model_outputs = [model(images) for model in models]
            outputs_list.append(torch.cat(model_outputs, dim=1))  # Concatenate the outputs of the base models

            labels_list.append(labels)  # Get labels

    return torch.cat(outputs_list, dim=0), torch.cat(labels_list, dim=0)

# Ensemble learning (Bagging)
def bagging_ensemble(models, test_loader):
    correct = 0
    total = 0
    with torch.no_grad():
        for images, labels in test_loader:
            images, labels = images.to(device), labels.to(device)  # Move data to GPU

            # Get predictions from all base models
            outputs = torch.stack([model(images) for model in models], dim=0)
            # Average the outputs from multiple models
            avg_outputs = torch.mean(outputs, dim=0)

            _, predicted = torch.max(avg_outputs, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

    return correct / total

# Boosting training and ensemble
def boosting_ensemble(models, train_loader, test_loader, num_epochs=10):
    model_weights = np.ones(len(models))  # Initialize the weight of each model as 1
    model_accuracies = []  # Store the accuracy of each model

    # Train each model and calculate its accuracy
    for model in models:
        accuracy = train_model(model, train_loader, epochs=num_epochs)
        model_accuracies.append(accuracy)

    # Adjust model weights according to accuracy
    total_accuracy = np.sum(model_accuracies)
    model_weights = model_accuracies / total_accuracy  # Calculate weights according to accuracy

    correct = 0
    total = 0
    with torch.no_grad():
        for images, labels in test_loader:
            images, labels = images.to(device), labels.to(device)  # Move data to GPU

            # Get predictions from all base models
            outputs = torch.stack([model(images) for model in models], dim=0)
            # Convert weights to a tensor and make sure it is on the same device
            weighted_outputs = torch.sum(outputs * torch.tensor(model_weights, device=device).view(-1, 1, 1), dim=0)
            _, predicted = torch.max(weighted_outputs, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

    return correct / total

# Stacking ensemble method
def stacking_ensemble(models, meta_model, train_loader, test_loader, epochs=20):
    # Get base-model outputs for the training and test sets
    X_train, y_train = get_predictions(models, train_loader)
    X_test, y_test = get_predictions(models, test_loader)

    # Apply softmax normalization to the outputs of the base models
    X_train = torch.softmax(X_train, dim=1)
    X_test = torch.softmax(X_test, dim=1)

    # Train the meta-model
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adam(meta_model.parameters(), lr=0.001, weight_decay=1e-4)  # Add weight-decay regularization

    meta_model.to(device)  # Move the meta-model to GPU
    X_train, y_train = X_train.to(device), y_train.to(device)
    X_test, y_test = X_test.to(device), y_test.to(device)

    # Split the training set and validation set
    dataset_size = X_train.size(0)
    indices = torch.randperm(dataset_size)
    split = int(dataset_size * 0.8)  # 80% training, 20% validation
    train_indices, val_indices = indices[:split], indices[split:]

    X_train_split, y_train_split = X_train[train_indices], y_train[train_indices]
    X_val, y_val = X_train[val_indices], y_train[val_indices]

    best_accuracy = 0.0
    for epoch in range(epochs):
        meta_model.train()
        optimizer.zero_grad()
        outputs = meta_model(X_train_split)
        loss = criterion(outputs, y_train_split)
        loss.backward()
        optimizer.step()

        # Evaluate on the validation set
        meta_model.eval()
        with torch.no_grad():
            val_outputs = meta_model(X_val)
            _, val_predicted = torch.max(val_outputs, 1)
            val_accuracy = (val_predicted == y_val).float().mean().item()

        # Save the best model
        if val_accuracy > best_accuracy:
            best_accuracy = val_accuracy
            torch.save(meta_model.state_dict(), 'best_meta_model.pth')

    # Load the best model
    meta_model.load_state_dict(torch.load('best_meta_model.pth', weights_only=True))

    # Evaluate on the test set
    meta_model.eval()
    with torch.no_grad():
        outputs = meta_model(X_test)
        _, predicted = torch.max(outputs, 1)
        accuracy = (predicted == y_test).float().mean().item()

    return accuracy

# Baseline model: train one single network
def baseline_model(train_loader, test_loader):
    model = SimpleCNN().to(device)
    accuracy = train_model(model, train_loader)
    return accuracy

# Train the baseline model
baseline_accuracy = baseline_model(train_loader, test_loader)
print(f"Baseline model accuracy: {baseline_accuracy * 100:.2f}%")

Baseline model accuracy: 68.07%

# Main program
num_models = 3  # Suppose we train 3 models
models = [SimpleCNN() for _ in range(num_models)]

# Train all models (needed for Bagging and Boosting)
for model in models:
    train_model(model, train_loader, epochs=5)

# Use Bagging for model ensemble
bagging_accuracy = bagging_ensemble(models, test_loader)
print(f"Bagging accuracy: {bagging_accuracy * 100:.2f}%")

# Use Boosting for model training
boosting_accuracy = boosting_ensemble(models, train_loader, test_loader, num_epochs=5)
print(f"Boosting accuracy: {boosting_accuracy * 100:.2f}%")

# Use Stacking for model ensemble
input_size = 10 * num_models  # Each base model outputs 10 classes, so the concatenated input size
meta_model = MetaModel(input_size, hidden_size=64, output_size=10)  # Use a neural network as the meta-model
stacking_accuracy = stacking_ensemble(models, meta_model, train_loader, test_loader, epochs=100)
print(f"Stacking accuracy: {stacking_accuracy * 100:.2f}%")

Bagging accuracy: 73.26%
Boosting accuracy: 73.35%
Stacking accuracy: 71.43%