Background:  Masked language modelling approaches have enjoyed success in improving benchmark performance across many general and biomedical domain natural language processing tasks, including biomedical relationship extraction (RE). However, the recent surge in both the number of novel architectures and the volume of training data they utilise may lead us to question whether domain specific pretrained models are necessary. Additionally, recent work has proposed novel classification heads for RE tasks, further improving performance. Here, we perform ablations over several pretrained models and classification heads to try to untangle the perceived benefits of each. Methods:  We use a range of string preprocessing strategies, combined with Bidirectional Encoder Representations from Transformers (BERT), BioBERT and RoBERTa architectures to perform ablations over three RE datasets pertaining to drug-drug and chemical protein interactions, and general domain relationship extraction. We explore the use of the RBERT classification head, compared to a simple linear classification layer across all architectures and datasets. Results:  We observe a moderate performance benefit in using the BioBERT pretrained model over the BERT base cased model, although there appears to be little difference when comparing BioBERT to RoBERTa large. In addition, we observe a substantial benefit of using the RBERT head on the general domain RE dataset, but this is not consistently reflected in the biomedical RE datasets. Finally, we discover that randomising the token order of training data does not result in catastrophic performance degradation in our selected tasks. Conclusions:  We find a recent general domain pretrained model performs approximately the same as a biomedical specific one, suggesting that domain specific models may be of limited use given the tendency of recent model pretraining regimes to incorporate ever broader sets of data. In addition, we suggest that care must be taken in RE model training, to prevent fitting to non-syntactic features of datasets.

/pdf/ablations-over-transformer-models-for-biomedical-2sghc29zqv.pdf

Ablations over transformer models for biomedical relationship extraction

Social media has played an important role in shaping political discourse over the last decade. It is often perceived to have increased political polarization, thanks to the scale of discussions and their public nature. Automatic political ideology detection allows us to identify the bias of information sources as well as professionals such as journalists. It has also become a relevant area due to its successful application to user behaviour analysis and prediction of malicious user versus legitimate user. This paper describes our participation at PoliticEs@IberLEF2022 shared task, whose goal is classify the political ideology of a person as well as other related information such as profession or gender. We explore several machine learning models to address the task of political ideology detection.

UC3MDeep at PoliticEs 2022: Exploring Traditional Machine Learning Algorithms for Political Ideology Detection

In this work, a system for adversarial Question Answering (QA) in Spanish is presented. Models BETO, MarIA-base, MarIA-large and RigoBERTa are tried, although finally only last 3 are used. They are first trained over a big adversarial QA corpus, AllQA. AllQA is composed of SQUAD-ES v2, a translated version of NewsQA presented in this work, and QUALES. These general QA models are then retrained over QUALES dataset. Finally, their predictions are aggregated via meta-ensembling techniques, to produce more reliable answers to the presented questions. Results in terms of F1-score are presented on the validation set of AllQA and QUALES, and complete official results on the test set of QUALES in terms of F1-score and exact match are also presented.

Adversarial Question Answering in Spanish with Transformer Models

The analysis of public affairs documents is crucial for citizens as it promotes transparency, accountability, and informed decision-making. It allows citizens to understand government policies, participate in public discourse, and hold representatives accountable. This is crucial, and sometimes a matter of life or death, for companies whose operation depend on certain regulations. Large Language Models (LLMs) have the potential to greatly enhance the analysis of public affairs documents by effectively processing and understanding the complex language used in such documents. In this work, we analyze the performance of LLMs in classifying public affairs documents. As a natural multi-label task, the classification of these documents presents important challenges. In this work, we use a regex-powered tool to collect a database of public affairs documents with more than 33K samples and 22.5M tokens. Our experiments assess the performance of 4 different Spanish LLMs to classify up to 30 different topics in the data in different configurations. The results shows that LLMs can be of great use to process domain-specific documents, such as those in the domain of public affairs.

Leveraging Large Language Models for Topic Classification in the Domain of Public Affairs

The present work proposes a legal assistance tool based on NLP and a predictive ML model adapted to the particularities of Colombian Justice system for the Legal Defense Agency of the Colombian State (ANDJE), It is developed in 4 parts associated with specific requirements: An Exploratory Data Analysis of the available information, an NLP pre-processing of large volumes of information structured in databases and not structured in scanned PDFs, development of predictive models and user assistance tool based in Transformers. Combining in an original and innovative way predictive, information presentation, similarity searching engine and summary, Question Answering modules for treatment of large legal texts, to increase the probability of success in defending the interests of the Colombian state. Considerations for future works.

Assistance System for Judicial Awards for The Colombian State Legal Defense Agency Through NLP -ANDJE-

This paper presents RigoBERTa, a State-of-the-Art Language Model for Spanish. RigoBERTa is trained over a well-curated corpus formed up from diﬀerent subcorpora with key features. It follows the DeBERTa architecture, which has several advantages over other architectures of similar size as BERT or RoBERTa. RigoBERTa performance is assessed over 13 NLU tasks in comparison with other available Spanish language models, namely, MarIA, BERTIN and BETO. RigoBERTa outperformed the three models in 10 out of the 13 tasks, achieving new ”State-of-the-Art” results.

RigoBERTa: A State-of-the-Art Language Model For Spanish

RichJackson/pytorch-transformers: supporting ablation paper v3

The objective of this work is to develop a reliable and complete Generative Question Answering (QA) System in Spanish, for the biomedical domain. The need for such kind of system for general users to clarify complex biomedical questions is noticeable, given the existing misinformation and the lack of reliable tools that join multiple sources to form a complete answer about health-related topics. Given the importance of these for society as a whole, and the lack of relevant resources in Spanish, it was considered of general interest to develop a system that could bring together the knowledge located in different sources and make it available to the Spanish-speaking community. Moreover, putting a focus on accessibility, the system should also be fully operated through voice.

Guillem García Subies

Papers

RigoBERTa: A State-of-the-Art Language Model For Spanish

RichJackson/pytorch-transformers: supporting ablation paper v3

BioMedIA: A Complete Voice-to-Voice Generative Question Answering System for the Biomedical Domain in Spanish