The transformer architecture based on self-attention offers a versatile structure which has led to the definition of multiple deep learning models for various tasks or applications of natural language processing. The purpose of this work is to analyze two language models for training bidirectional encoders like BERT: the Masked Language Model (MLM) and the Conditional Masked Language Model (CMLM) for learning sentence embeddings. How sentence-level representations impact the task of sequence classification is the main focus of interest in our investigation: is there any significant difference in quality between these two pretrained language models? We evaluate via fine-tuning these pretrained models on a downstream task as sequence classification.
Fine-Tuning of BERT models for Sequence Classification
2022-12-05
358821 byte
Conference paper
Electronic Resource
English
BERT for Aviation Text Classification
TIBKAT | 2023
|BERT for Aviation Text Classification
AIAA | 2023
|Creating Capacity: Fine-tuning the classification yard
Online Contents | 2009
Automotive engineering | 1981
|