bgGLUE

The Bulgarian General Language Understanding Evaluation (bgGLUE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems in Bulgarian.

bgGLUE consists of:

• A benchmark of nine sentence- or sentence-pair language understanding tasks built on established existing datasets and selected to cover a diverse range of dataset sizes, text genres, and degrees of difficulty,
• A diagnostic dataset designed to evaluate and to analyze the model performance with respect to a wide range of linguistic phenomena found in natural language, and
• A public leaderboard for tracking the progress on the benchmark, and
• A dashboard for visualizing the performance of models on the diagnostic set.

The format of the bgGLUE benchmark is model-agnostic, and thus any system capable of processing a sentence or a pair of sentences and producing corresponding predictions is eligible to participate. The benchmark tasks are selected so as to favor models that share information across tasks using parameter sharing or other transfer learning techniques. The ultimate goal of GLUE is to drive research in the development of general and robust natural language understanding systems.