Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. Transformers is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at https://github.com/huggingface/transformers.

/pdf/transformers-state-of-the-art-natural-language-processing-1zozhnrntv.pdf

Transformers: State-of-the-Art Natural Language Processing

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{this https URL}.

HuggingFace's Transformers: State-of-the-art Natural Language Processing.

Recent advances in modern Natural Language Processing (NLP) research have
been dominated by the combination of Transfer Learning methods with large-scale
Transformer language models. With them came a paradigm shift in NLP with the
starting point for training a model on a downstream task moving from a blank
specific model to a general-purpose pretrained architecture. Still, creating
these general-purpose models remains an expensive and time-consuming process
restricting the use of these methods to a small sub-set of the wider NLP
community. In this paper, we present Transformers, a library for
state-of-the-art NLP, making these developments available to the community by
gathering state-of-the-art general-purpose pretrained models under a unified
API together with an ecosystem of libraries, examples, tutorials and scripts
targeting many downstream NLP tasks. Transformers features carefully crafted
model implementations and high-performance pretrained weights for two main deep
learning frameworks, PyTorch and TensorFlow, while supporting all the necessary
tools to analyze, evaluate and use these models in downstream tasks such as
text/token classification, questions answering and language generation among
others. Transformers has gained significant organic traction and adoption among
both the researcher and practitioner communities. We are committed at Hugging
Face to pursue the efforts to develop Transformers with the ambition of
creating the standard library for building NLP systems.

Transformers: State-of-the-art Natural Language Processing

We introduce Stanza, an open-source Python natural language processing toolkit supporting 66 human languages Compared to existing widely used toolkits, Stanza features a language-agnostic fully neural pipeline for text analysis, including tokenization, multi-word token expansion, lemmatization, part-of-speech and morphological feature tagging, dependency parsing, and named entity recognition We have trained Stanza on a total of 112 datasets, including the Universal Dependencies treebanks and other multilingual corpora, and show that the same neural architecture generalizes well and achieves competitive performance on all languages tested Additionally, Stanza includes a native Python interface to the widely used Java Stanford CoreNLP software, which further extends its functionality to cover other tasks such as coreference resolution and relation extraction Source code, documentation, and pretrained models for 66 languages are available at https://stanfordnlpgithubio/stanza/

/pdf/stanza-a-python-natural-language-processing-toolkit-for-many-10jne5sqxl.pdf

Stanza: A Python Natural Language Processing Toolkit for Many Human Languages

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.

/pdf/a-primer-in-bertology-what-we-know-about-how-bert-works-2v8af1zw7o.pdf

A Primer in BERTology: What We Know About How BERT Works

We present FLAIR, an NLP framework designed to facilitate training and distribution of state-of-the-art sequence labeling, text classification and language models. The core idea of the framework is to present a simple, unified interface for conceptually very different types of word and document embeddings. This effectively hides all embedding-specific engineering complexity and allows researchers to “mix and match” various embeddings with little effort. The framework also implements standard model training and hyperparameter selection routines, as well as a data fetching module that can download publicly available NLP datasets and convert them into data structures for quick set up of experiments. Finally, FLAIR also ships with a “model zoo” of pre-trained models to allow researchers to use state-of-the-art NLP models in their applications. This paper gives an overview of the framework and its functionality. The framework is available on GitHub at https://github.com/zalandoresearch/flair .

FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP

Contextual string embeddings are a recent type of contextualized word embedding that were shown to yield state-of-the-art results when utilized in a range of sequence labeling tasks. They are based on character-level language models which treat text as distributions over characters and are capable of generating embeddings for any string of characters within any textual context. However, such purely character-based approaches struggle to produce meaningful embeddings if a rare string is used in a underspecified context. To address this drawback, we propose a method in which we dynamically aggregate contextualized embeddings of each unique string that we encounter. We then use a pooling operation to distill a ”global” word representation from all contextualized instances. We evaluate these ”pooled contextualized embeddings” on common named entity recognition (NER) tasks such as CoNLL-03 and WNUT and show that our approach significantly improves the state-of-the-art for NER. We make all code and pre-trained models available to the research community for use and reproduction.

Pooled Contextualized Embeddings for Named Entity Recognition.

Data profiling is the discipline of discovering metadata about given datasets. The metadata itself serve a variety of use cases, such as data integration, data cleansing, or query optimization. Due to the importance of data profiling in practice, many tools have emerged that support data scientists and IT professionals in this task. These tools provide good support for profiling statistics that are easy to compute, but they are usually lacking automatic and efficient discovery of complex statistics, such as inclusion dependencies, unique column combinations, or functional dependencies.We present Metanome, an extensible profiling platform that incorporates many state-of-the-art profiling algorithms. While Metanome is able to calculate simple profiling statistics in relational data, its focus lies on the automatic discovery of complex metadata. Metanome's goal is to provide novel profiling algorithms from research, perform comparative evaluations, and to support developers in building and testing new algorithms. In addition, Metanome is able to rank profiling results according to various metrics and to visualize the, at times, large metadata sets.

/pdf/data-profiling-with-metanome-6kql0ikwuy.pdf

Data profiling with metanome

The provision of Linked Data about sporting results enables extensive statistics, while connections to further datasets allow enhanced and sophisticated analyses. Moreover, providing sports data as Linked Open Data may promote new applications, which are currently impossible due to the locked nature of today's proprietary sports databases. Though the sport domain is strongly underrepresented in the Linked Open Data Cloud. In this paper we present a dataset containing information about soccer entities crawled from heterogeneous sources and linked to related entities from the LOD cloud. To enable easy exploration and to illustrate the capabilities of the dataset a web interface is providing a structured overview and extensive statistics.

Generating a linked soccer dataset

The sport domain is strongly under-represented in the Linked Open Data Cloud, whereas sport competition results can be linked to already existing entities, such as events, teams, players, and more. The provision of Linked Data about sporting results enables extensive statistics, while connections to further datasets allow enhanced and sophisticated analyses. Moreover, providing sports data as Linked Open Data may promote new applications, which are currently impossible due to the locked nature of today’s proprietary sports databases. We present a dataset containing information about soccer matches, teams, players and so forth crawled from from heterogeneous sources and linked to related entities from the LOD cloud. To enable exploration and to illustrate the capabilities of the dataset a web interface is introduced providing a structured overview and extensive statistics.

Tanja Bergmann

Papers

FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP

Pooled Contextualized Embeddings for Named Entity Recognition.

Data profiling with metanome

Generating a linked soccer dataset

Linked Soccer Data.