As Transfer Learning from large-scale pre-trained models becomes more prevalent in Natural Language Processing (NLP), operating these large models in on-the-edge and/or under constrained computational training or inference budgets remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device study.

DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

羟氨苄青霉素引起大疱性类天疱疮样疹

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random token...

SpanBERT: Improving Pre-training by Representing and Predicting Spans

Thirty-three brothels in rural and small-town Nevada, which contain between 225 and 250 prostitutes, are legal or openly tolerated and strictly controlled by state statute, city and county ordinances, and local rules. Twenty-two of the brothels are in places with populations between 500 and 8,000, and the remaining eleven are in rural areas. The legal and quasi-legal restrictions placed on prostitutes severely limit their activities outside brothels. These restrictions in conjunction with historical inertia, perceived benefits of crime and venereal disease control, and the good image of madams contribute to widespread positive local attitudes toward brothel prostitution. Interactions between clients and prostitutes in brothel parlors are also restricted and limited to a few basic types which are largely determined by entrepreneurial philosophy. KEY WORDS : Nevada, Political geography, Prostitution, Restricted activity spaces.

ANNALS of the Association of American Geographers

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERT-large, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0, respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6\% F1), strong performance on the TACRED relation extraction benchmark, and even show gains on GLUE.

Recent progress in natural language processing has been driven by advances in both model architecture and model pretraining. Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks. \textit{Transformers} is an open-source library with the goal of opening up these advances to the wider machine learning community. The library consists of carefully engineered state-of-the art Transformer architectures under a unified API. Backing this library is a curated collection of pretrained models made by and available for the community. \textit{Transformers} is designed to be extensible by researchers, simple for practitioners, and fast and robust in industrial deployments. The library is available at \url{this https URL}.

HuggingFace's Transformers: State-of-the-art Natural Language Processing.

Recent advances in modern Natural Language Processing (NLP) research have
been dominated by the combination of Transfer Learning methods with large-scale
Transformer language models. With them came a paradigm shift in NLP with the
starting point for training a model on a downstream task moving from a blank
specific model to a general-purpose pretrained architecture. Still, creating
these general-purpose models remains an expensive and time-consuming process
restricting the use of these methods to a small sub-set of the wider NLP
community. In this paper, we present Transformers, a library for
state-of-the-art NLP, making these developments available to the community by
gathering state-of-the-art general-purpose pretrained models under a unified
API together with an ecosystem of libraries, examples, tutorials and scripts
targeting many downstream NLP tasks. Transformers features carefully crafted
model implementations and high-performance pretrained weights for two main deep
learning frameworks, PyTorch and TensorFlow, while supporting all the necessary
tools to analyze, evaluate and use these models in downstream tasks such as
text/token classification, questions answering and language generation among
others. Transformers has gained significant organic traction and adoption among
both the researcher and practitioner communities. We are committed at Hugging
Face to pursue the efforts to develop Transformers with the ambition of
creating the standard library for building NLP systems.

Transformers: State-of-the-art Natural Language Processing

The recent availability of data for cities has allowed scientists to exhibit scalings which present themselves in the form of a power-law dependence on population of various socio-economical and structural indicators. We propose here a stochastic theory of urban growth which accounts for some of the observed scalings and we confirm these predictions on US and OECD empirical data. In particular, we show that the dependence on population size of the total number of miles driven daily, the total length of the road network, the total traffic delay, the total consumption of gasoline, the quantity of CO2 emitted and the relation between area and population of cities, are all governed by a single parameter which characterizes the sensitivity to congestion. Our results suggest that diseconomies associated with congestion scale superlinearly with population size, implying that -despite polycentrism- cities whose transportation infrastructure rely heavily on traffic sensitive modes are unsustainable.

/pdf/how-congestion-shapes-cities-from-mobility-patterns-to-41hm2d4zbd.pdf

How congestion shapes cities: from mobility patterns to scaling

We propose a quantitative method to classify cities according to their street pattern. We use the conditional probability distribution of shape factor of blocks with a given area and define what could constitute the 'fingerprint' of a city. Using a simple hierarchical clustering method, these fingerprints can then serve as a basis for a typology of cities. We apply this method to a set of 131 cities in the world, and at an intermediate level of the dendrogram, we observe four large families of cities characterized by different abundances of blocks of a certain area and shape. At a lower level of the classification, we find that most European cities and American cities in our sample fall in their own sub-category, highlighting quantitatively the differences between the typical layouts of cities in both regions. We also show with the example of New York and its different boroughs, that the fingerprint of a city can be seen as the sum of the ones characterizing the different neighbourhoods inside a city. This method provides a quantitative comparison of urban street patterns, which could be helpful for a better understanding of the causes and mechanisms behind their distinct shapes.

https://royalsocietypublishing.org/doi/pdf/10.1098/rsif.2014.0924

A typology of street patterns

Empirical evidence suggests that most urban systems experience a transition from a monocentric to a polycentric organization as they grow and expand. We propose here a stochastic, out-of-equilibrium model of the city, which explains the appearance of subcenters as an effect of traffic congestion. We show that congestion triggers the instability of the monocentric regime and that the number of subcenters and the total commuting distance within a city scale sublinearly with its population, predictions that are in agreement with data gathered for around 9000 U.S. cities between 1994 and 2010.

/pdf/modeling-the-polycentric-transition-of-cities-3dwkg0kay2.pdf

Rémi Louf

Papers

HuggingFace's Transformers: State-of-the-art Natural Language Processing.

Transformers: State-of-the-art Natural Language Processing

How congestion shapes cities: from mobility patterns to scaling

A typology of street patterns

Modeling the Polycentric Transition of Cities