Learning Methods of Deep Learning
created by Deepfinder
Agenda
- Learning from a Teacher: Supervised Learning
- Seeing the Big Picture from Small Clues: Unsupervised Learning
- Learning Without a Teacher: Self-supervised Learning
- Using a Few Points to Cover the Whole: Semi-supervised Learning
- Distinguishing Right from Wrong: Contrastive Learning
- Learning by Analogy: Transfer Learning
- Head-to-head Competition: Adversarial Learning
- Strength in Numbers: Ensemble Learning
- Different Paths to the Same Goal: Federated Learning
- Perseverance Through Trial and Error: Reinforcement Learning
- Eager to Learn: Active Learning
- Learning to Learn: Meta-Learning
Tutorial 03 – Learning Without a Teacher: Self-supervised Learning
Self-supervised Learning
- A form of unsupervised learning in which the data provides the supervision.
- Idea: hide part of the data, and ask the neural network to predict it from the remaining part.
Masked Autoencoders
The basic working principle of Masked Autoencoders (MAE):
Data processing stage:
The input image is split into many small image patches.
A subset of patches is randomly selected for masking, usually 75% of all patches. These patches are removed from the input.
The remaining unmasked patches are sent to the encoder for feature extraction.
Mask tokens:
After the encoder, the positions of the masked patches are filled with special “mask tokens.” A mask token is a special token used to fill the position of a masked image patch. The decoder needs a complete sequence of image patches, including both unmasked and masked patches, to reconstruct the original image. Therefore, mask tokens act as placeholders at the positions of the masked patches. The decoder processes the encoded image patches together with the mask tokens and tries to reconstruct the pixels of the original image.
In simple terms, mask tokens let the decoder distinguish which parts are known information, namely the unmasked patches provided by the encoder, and which parts are unknown information that needs to be predicted, as indicated by the masked patch positions. This helps the decoder focus on reconstructing the masked regions during training.
Usage stage:
After pretraining is finished, the decoder is discarded and only the encoder is kept.
For downstream tasks, such as image classification or object recognition, the encoder receives the full unmasked image patches as input.
The core idea of this method is to let the model learn better image representations by predicting the masked parts. The pretraining stage is similar to self-supervised learning. It improves the encoder’s feature extraction ability by reconstructing the masked regions.
- Code:
- HuggingFace: ViTMAE
- GitHub: Official PyTorch implementation (FAIR), Awesome MAE Models
Masked Language Model (MLM)
The masked language model is a key training method in BERT. It learns contextual semantics by masking part of the input tokens.
- Training process:
Random word masking:
Randomly select 15% of the words in the input text for processing:
80% of the time, replace the word with [MASK], such as "apple" → "[MASK]".
10% of the time, replace it with a random word, such as "apple" → "orange".
10% of the time, keep the original word unchanged, such as "apple" → "apple".
-
Objective:
The model predicts the masked words from the context. -
Effect:
Through bidirectional context modeling, the model understands the relationship between each word in a sentence and the words around it. This helps it learn deeper semantic representations.
Next Sentence Prediction (NSP)
What is the core difference between self-supervised learning and unsupervised learning?
from datasets import load_dataset
def load_and_split_dataset(csv_file_path, test_size=0.2, seed=42):
"""
Load data from a CSV file and split it into training and test sets.
Parameters:
-------
csv_file_path : str
CSV file path
test_size : float
Proportion of the test set (default is 0.2, or 20%)
seed : int
Random seed
Returns:
-------
dataset_dict : DatasetDict
A DatasetDict object containing two splits: 'train' and 'test'
"""
# 1. Load the CSV file. One column is named "review", and another is named "sentiment".
raw_dataset = load_dataset(
"csv",
data_files=csv_file_path
)
# Note: raw_dataset currently contains only one split named "train".
# By default, reading a single file places it under the "train" split.
# You can use raw_dataset["train"] to access all data.
# 2. Split all data into training and test sets (8:2)
# Use train_test_split to split the original raw_dataset["train"] into 'train' and 'test'
dataset_dict = raw_dataset["train"].train_test_split(
test_size=test_size,
shuffle=True,
seed=seed
)
# 3. Print the dataset structure and examples
print(f"Dataset splits: {dataset_dict}")
print(f"Train samples: {dataset_dict['train'].num_rows}")
print(f"Test samples: {dataset_dict['test'].num_rows}")
# Print the first two rows as examples
print("Train sample[0]:", dataset_dict["train"][0])
print("Test sample[0]:", dataset_dict["test"][0])
# Return a DatasetDict containing 'train' and 'test'
return dataset_dict
csv_path = "datasets/imdb/IMDB Dataset.csv" # Your CSV file path
dataset = load_and_split_dataset(
csv_file_path=csv_path,
test_size=0.2,
seed=42
)/home/arwin/anaconda3/envs/dt/lib/python3.8/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
:219: RuntimeWarning: pyarrow.lib.Tensor size changed, may indicate binary incompatibility. Expected 64 from C header, got 80 from PyObject
Dataset splits: DatasetDict({
train: Dataset({
features: ['review', 'sentiment'],
num_rows: 40000
})
test: Dataset({
features: ['review', 'sentiment'],
num_rows: 10000
})
})
Train samples: 40000
Test samples: 10000
Train sample[0]: {'review': "The film disappointed me for many reasons: first of all the depiction of a future which seemed at first realistic to me was well-built but did only feature a marginal role. Then, the story itself was a weak copy of Lost in Translation. The Middle-Eastern setting, man with family meets new girl overseas, karaoke bar, the camera movements and the imagery - all that was a very bad imitation of the excellent Lost in Translation which had also credibility. This movie tries to be something brilliant and cultural: it is not. I wonder why Tim Robbins even considered doing this film!? The female main actress is awful - did she play the precog in Minority Report? And why do you have to show the vagina in a movie like this? Lost in Translation didn't have to show excessive love scenes. R-Rated just for this? This movie isn't even worth watching it from a videostore!", 'sentiment': 'negative'}
Test sample[0]: {'review': 'Arguably the finest serial ever made(no argument here thus far) about Earthman Flash Gordon, Professor Zarkov, and beautiful Dale Arden traveling in a rocket ship to another universe to save the planet. Along the way, in spellbinding, spectacular, and action-packed chapters Flash and his friends along with new found friends such as Prince Barin, Prince Thun, and the awesome King Vultan pool their resources together to fight the evils and armies of the merciless Ming of Mongo and the jealous treachery of his daughter Priness Aura(now she\'s a car!). This serial is not just a cut above most serials in terms of plot, acting, and budget - it is miles ahead in these areas. Produced by Universal Studios it has many former sets at its disposable like the laboratory set from The Bride of Frankenstein and the Opera House from The Phantom of the Opera just to name a few. The production values across the board are advanced, in my most humble opinion, for 1936. The costumes worn by many of these strange men and women are really creative and first-rate. We get hawk-men, shark men, lion men, high priests, creatures like dragons, octasacks, orangapoids, and tigrons(oh my!)and many, many other fantastic things. Are all of them believable and first-rate special effects? No way. But for 1936 most are very impressive. The musical score is awesome and the chapter beginnings are well-written, lengthy enough to revitalize viewer memories of the former chapter, and expertly scored. Director Frederick Stephani does a great job piecing everything together wonderfully and creating a worthy film for Alex Raymond\'s phenom comic strip. Lastly, the acting is pretty good in this serial. All too often serials have either no names with no talent surrounding one or two former talents - here most everyone has some ability. Don\'t get me wrong, this isn\'t a Shakespeare troupe by any means, but Buster Crabbe does a workmanlike, likable job as Flash. He is ably aided by Jean Arden, Priscella Lawson, and the rest of the cast in general with two performers standing out. But before I get to those two let me add as another reviewer noted, it must have been amazing for this serial to get by the Hayes Office. I see more flesh on Flash and on Jean Rogers and Priscella Lawson than in movies decades later. The shorts Crabbe(and unfortunately for all of us Professor Zarkov((Frank Shannon)) wears are about as form-fitting a pair of shorts guys can wear. The girls are wearing mid drifts throughout and are absolutely beautiful Jean Rogers may have limited acting talent but she is a blonde bombshell. Lawson is also very sultry and sensuous and beautiful. But for me the two actors that make the serial are Charles Middleton as Ming: officious, sardonic, merciless, and fun. Middleton is a class act. Jack "Tiny" Lipson plays King Vultan: boisterous, rousing, hilarious - a symbol for pure joy in life and the every essence of hedonism. Lipson steals each and every scene he is in. The plot meanders here, there, and everywhere - but Flash Gordon is the penultimate serial, space opera, and the basis for loads of science fiction to follow. Excellent!', 'sentiment': 'positive'}
import torch
from datasets import load_dataset
from transformers import (
BertTokenizer,
BertForMaskedLM,
DataCollatorForLanguageModeling,
TrainingArguments,
Trainer
)
# 1. Load the IMDB dataset
# The dataset contains two splits, "train" and "test". Each item contains the "text" and "label" fields.
imdb_dataset = dataset
# 2. Initialize the tokenizer
tokenizer = BertTokenizer.from_pretrained("datasets/bert-base-uncased")
# 3. Define the tokenization function, and tokenize and numericalize the dataset
# - `padding="max_length"`: pad sentences to the same length
# - `truncation=True` : truncate sequences that exceed the specified length
# - `max_length=128` : use a fixed sequence length of 128
def tokenize_function(examples):
# Note that the column name is "review"
return tokenizer(
examples["review"],
padding="max_length",
truncation=True,
max_length=128
)
# remove_columns=["review"] means removing the original text column after processing and keeping only the fields required by the model
tokenized_imdb = imdb_dataset.map(
tokenize_function,
batched=True,
remove_columns=["review"]
)
# 4. Prepare the DataCollator
# DataCollatorForLanguageModeling automatically applies random masking to sentences in each batch
# mlm_probability=0.15 means that 15% of tokens in a sentence are randomly masked.
data_collator = DataCollatorForLanguageModeling(
tokenizer=tokenizer,
mlm=True,
mlm_probability=0.15
)
# 5. Define the BERT MLM model
model = BertForMaskedLM.from_pretrained("datasets/bert-base-uncased")
# 6. Training configuration
training_args = TrainingArguments(
output_dir="./mlm_imdb_bert", # Model output path
evaluation_strategy="epoch", # Evaluate once after each epoch
per_device_train_batch_size=8, # Batch size per GPU/CPU during training
per_device_eval_batch_size=8, # Batch size per GPU/CPU during testing
num_train_epochs=1, # Number of training epochs for demonstration. Modify as needed
logging_steps=100, # Print logs every N steps
save_steps=500 # Save the model every N steps
)
# 7. Use Trainer to wrap the training process
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_imdb["train"], # Training set
eval_dataset=tokenized_imdb["test"], # Test set
data_collator=data_collator
)
# 8. Train the model
trainer.train()
# 9. After training is complete, use trainer.evaluate() to evaluate the model on the test set
eval_results = trainer.evaluate()
print(eval_results)
[2500/2500 20:34, Epoch 1/1]
Epoch
Training Loss
Validation Loss
1
1.984600
1.933474
[625/625 02:10]
{'eval_loss': 1.9387198686599731, 'eval_runtime': 130.2856, 'eval_samples_per_second': 76.754, 'eval_steps_per_second': 4.797, 'epoch': 1.0}
Credits
- Icons made by Becris from www.flaticon.com
- Icons from Icons8.com – https://icons8.com
- Datasets from Kaggle – https://www.kaggle.com/
- Jason Brownlee – Why Initialize a Neural Network with Random Weights?
- OpenAI – Deep Double Descent
- Tal Daniel