Frequently Asked Questions

To evaluate on bgGLUE, collect your system's predictions on the nine tasks. The expected format for the predictions from each dataset is a jsonl file (a valid json on each row) with two fields: `{"id": 123, "label": "label_name"}`. You may upload at most two submissions a day. A sample submission with the necessary formatting is available in our GitHub repository. You can use the training code for the baselines as a starting point. See below if you are running into issues submitting.

Our only firm rule is that you may not train or tune your systems on the test sets for the nine primary tasks. This includes the test sets for which no labels are available to the public — you may not label these yourself or use them in any other way to improve your submitted system. In addition, you may not publish any kind of detailed analysis of the test sets for the nine primary tasks. Systems associated with such work will be removed from the leaderboard.
Beyond this, you may submit results from any kind of system that is capable of producing labels for the nine target tasks and the analysis tasks. This includes systems that do not share any components across tasks or systems not based on machine learning.

No. bgGLUE is an open-ended competition with no deadline or set end date.

You can obtain the data for all tasks (except CredibleNews, see below) by following these instructions or using the HuggingFace dataset. For CredibleNews you need to agree with the Terms and Conditions listed in this form . After you submit the form, you will receive a download link to dataset. You can use the same HuggingFace class to load the data.

You can find links to the per-task training, development, and unlabeled test data from the "Tasks" page.

The primary bgGLUE tasks are built on and derived from existing datasets. We refer users to the original licenses accompanying each dataset. For each dataset the license is listed on its "Tasks" page.

Our paper contains details and citations for all data used and the processing applied to them. You can also find details about each dataset in the "Tasks" page.

We ask for seven pieces of information: A short name for your system, which will be displayed in the leaderboard. A URL for a paper or (if one is not available) website or code repository describing your system. A sentence or two describing your system. Make sure to mention any outside data or resources you use. A sentence or two explaining how you share parameters across tasks (or stating that you don't share parameters). The total number of trained parameters in your model. Do not count word or word-part embedding parameters, even if they are trained. The total number of trained parameters in your model that are shared across multiple tasks. If some parameters are shared across some but not all of your tasks, count those. Do not count word or word-part embedding parameters, even if they are trained. Whether you want your submission to be visible on the public leaderboard.

You are welcome to submit to the leaderboard without using your real name as long as you include a link to a paper describing your submission. This will often involve anonymous paper archives like OpenReview Anonymous Preprint.

We calculate scores for each of the tasks based on their individual metrics. All metrics are scaled by 100x (i.e., as percentages). These scores are then averaged to get the final score. For tasks with multiple metrics (e.g., Cinexio), the metrics are averaged. On the leaderboard, only the top scoring submission of a user is shown or ranked by default. Other submissions can be viewed under the expanded view for each user. Competitors may submit privately, preventing their results from appearing.

On the leaderboard, click on a submission. A drawer will appear from the side which will provide more information about a submission. On the side drawer, click on 'More Information' to go to an expanded submission details page.

Contact us: bulgarianglue [at] gmail [dot] com. Please report bugs to our GitHub.