Course Project / Shared-Task Participation · Feb 2024 – Mar 2024
SemEval Human Evaluation: Multi-Label Text Classification with Fine-Tuned BERT
Muhammad Talha
Introduction to Large Language Models — Habib University, SemEval Human Evaluation shared tasks at CLEF
Abstract
As part of an Introduction to Large Language Models course, this project tackled the SemEval Human Evaluation shared tasks (2023 and 2024) hosted at the CLEF conference — building a multi-label text classification pipeline to distinguish human-written from machine-generated text.
Approach
The task is framed as multi-label text classification: given a piece of text, predict which of several labels apply, spanning human vs. machine origin and related categories defined by the shared task. The pipeline covers the full supervised-learning loop — cleaning and exploring the labeled dataset with Pandas, tokenizing text with multiple subword tokenization strategies, and fine-tuning BERT-based transformer models using the Hugging Face Transformers Trainer API.
Evaluation
Model performance was measured on a held-out validation split using scikit-learn's multi-label classification metrics, with F1-score as the primary comparison point across tokenization strategies and model configurations.
Notes
This was completed across two assignments corresponding to the 2023 and 2024 editions of the shared task, building out the tokenization and preprocessing pipeline first before moving to fine-tuning and evaluation.
@misc{talha2024semeval,
title = {SemEval Human Evaluation: Multi-Label Text Classification with Fine-Tuned BERT},
author = {Talha, Muhammad},
note = {SemEval Human Evaluation shared tasks (2023, 2024), CLEF},
year = {2024}
}