Predicting language test proficiency - benchmarking a NLP classifier and a fine-tuned LLM

Tracktest provides a suite of English language proficiency tests for test takers. The objective of the task was to create an automated machine learning model for evaluating text proficiency levels, trained on historical language test data. The approach consisted of training and benchmarking two kinds of models for automatic text proficiency level prediction: a machine learning model trained on NLP features, and a fine-tuned LLM-classifier model trained on raw texts and their proficiency scores.

Designing NLP features based on POS tagging and mechanically marking frequently appearing types of grammatical errors

Training a gradient boosted tree classifier using NLP-based features

Fine-tuning a pre-trained LLM for classification

Creating train/test/validation sets, evaluating, comparing accuracy metrics and creating reports containing comparisons of various approaches

Serving trained models using an API and using local deployment