The transformer architecture based on self-attention offers a versatile structure which has led to the definition of multiple deep learning models for various tasks or applications of natural language processing. The purpose of this work is to analyze two language models for training bidirectional encoders like BERT: the Masked Language Model (MLM) and the Conditional Masked Language Model (CMLM) for learning sentence embeddings. How sentence-level representations impact the task of sequence classification is the main focus of interest in our investigation: is there any significant difference in quality between these two pretrained language models? We evaluate via fine-tuning these pretrained models on a downstream task as sequence classification.
Fine-Tuning of BERT models for Sequence Classification
05.12.2022
358821 byte
Aufsatz (Konferenz)
Elektronische Ressource
Englisch
BERT for Aviation Text Classification
TIBKAT | 2023
|BERT for Aviation Text Classification
AIAA | 2023
|Creating Capacity: Fine-tuning the classification yard
Online Contents | 2009
Kraftfahrwesen | 1981
|