Large-scale language models show promising text generation capabilities, but users cannot easily control particular aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model, trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data via model-based source attribution. We have released multiple full-sized, pretrained versions of CTRL at this https URL.

CTRL: A Conditional Transformer Language Model for Controllable Generation

Review of summation-by-parts schemes for initial–boundary-value problems

Large-eddy simulation (LES) has proven to be a computationally tractable approach to simulate unsteady turbulent flows. However, prohibitive resolution requirements induced by near-wall eddies in high–Reynolds number boundary layers necessitate the use of wall models or approximate wall boundary conditions. We review recent investigations in wall-modeled LES, including the development of novel approximate boundary conditions and the application of wall models to complex flows (e.g., boundary-layer separation, shock/boundary-layer interactions, transition). We also assess the validity of underlying assumptions in wall-model derivations to elucidate the accuracy of these investigations, and offer suggestions for future studies.

Wall-Modeled Large-Eddy Simulation for Complex Turbulent Flows

We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.

Robust Speech Recognition via Large-Scale Weak Supervision

PyFR: An open source framework for solving advection–diffusion type problems on streaming architectures using the flux reconstruction approach

The flux reconstruction approach to high-order methods is robust, efficient, simple to implement, and allows various high-order schemes, such as the nodal discontinuous Galerkin method and the spectral difference method, to be cast within a single unifying framework. Utilizing a flux reconstruction formulation, it has been proved (for one-dimensional linear advection) that the spectral difference method is stable for all orders of accuracy in a norm of Sobolev type, provided that the interior flux collocation points are located at zeros of the corresponding Legendre polynomials. In this article the aforementioned result is extended in order to develop a new class of one-dimensional energy stable flux reconstruction schemes. The energy stable schemes are parameterized by a single scalar quantity, which if chosen judiciously leads to the recovery of various well known high-order methods (including a particular nodal discontinuous Galerkin method and a particular spectral difference method). The analysis offers significant insight into why certain flux reconstruction schemes are stable, whereas others are not. Also, from a practical standpoint, the analysis provides a simple prescription for implementing an infinite range of energy stable high-order methods via the intuitive flux reconstruction approach.

/pdf/a-new-class-of-high-order-energy-stable-flux-reconstruction-1f1ednjdto.pdf

A New Class of High-Order Energy Stable Flux Reconstruction Schemes

http://aero-comlab.stanford.edu/Papers/pev_jcp_2011.pdf

Insights from von Neumann analysis of high-order flux reconstruction schemes

The flux reconstruction (FR) approach allows various well-known high-order schemes, such as collocation based nodal discontinuous Galerkin (DG) methods and spectral difference (SD) methods, to be cast within a single unifying framework. Recently, the authors identified a new class of FR schemes for 1D conservation laws, which are simple to implement, efficient and guaranteed to be linearly stable for all orders of accuracy. The new schemes can easily be extended to quadrilateral elements via the construction of tensor product bases. However, for triangular elements, such a construction is not possible. Since numerical simulations over complicated geometries often require the computational domain to be tessellated with simplex elements, the development of stable FR schemes on simplex elements is highly desirable. In this article, a new class of energy stable FR schemes for triangular elements is developed. The schemes are parameterized by a single scalar quantity, which can be adjusted to provide an infinite range of linearly stable high-order methods on triangular elements. Von Neumann stability analysis is conducted on the new class of schemes, which allows identification of schemes with increased explicit time-step limits compared to the collocation based nodal DG method. Numerical experiments are performed to confirm that the new schemes yield the optimal order of accuracy for linear advection on triangular grids.

/pdf/a-new-class-of-high-order-energy-stable-flux-reconstruction-20mzcoj11s.pdf

A New Class of High-Order Energy Stable Flux Reconstruction Schemes for Triangular Elements

The flux reconstruction (FR) approach unifies various high-order schemes, including collocation based nodal discontinuous Galerkin (DG) methods, and all spectral difference methods (at least for a linear flux function), within a single framework. Recently a new range of linearly stable FR schemes have been identified, henceforth referred to as Vincent-Castonguay-Jameson-Huynh (VCJH) schemes. In this short note non-linear stability properties of FR schemes are elucidated via analysis of linearly stable VCJH schemes (so as to focus attention solely on issues of non-linear stability). It is shown that linearly stable VCJH schemes (at least in their standard form) may be unstable if the flux function is non-linear. This instability is due to aliasing errors, which manifest since FR schemes (in their standard form) utilize a collocation projection at the solution points to construct a polynomial approximation of the flux. Strategies for minimizing such aliasing driven instabilities are discussed within the context of the FR approach. In particular, it is shown that the location of the solution points will have a significant effect on non-linear stability. This result is important, since linear analysis of FR schemes implies stability is independent of solution point location. Finally, it is shown that if an exact L2 projection is employed to construct an approximation of the flux (as opposed to a collocation projection), then aliasing errors and hence aliasing driven instabilities will be eliminated. However, performing such a projection exactly, or at least very accurately, would be more costly than performing a collocation projection, and would certainly impact the inherent efficiency and simplicity of the FR approach. It can be noted that in all above regards, non-linear stability properties of FR schemes are similar to those of nodal DG schemes. The findings should motivate further research into the non-linear performance of FR schemes, which have hitherto been developed and analyzed solely in the context of a linear flux function.

/pdf/on-the-non-linear-stability-of-flux-reconstruction-schemes-5f9pqa5x9m.pdf

On the Non-linear Stability of Flux Reconstruction Schemes

NeMo (Neural Modules) is a Python framework-agnostic toolkit for creating AI applications through re-usability, abstraction, and composition. NeMo is built around neural modules, conceptual blocks of neural networks that take typed inputs and produce typed outputs. Such modules typically represent data layers, encoders, decoders, language models, loss functions, or methods of combining activations. NeMo makes it easy to combine and re-use these building blocks while providing a level of semantic correctness checking via its neural type system. The toolkit comes with extendable collections of pre-built modules for automatic speech recognition and natural language processing. Furthermore, NeMo provides built-in support for distributed training and mixed precision on latest NVIDIA GPUs. NeMo is open-source this https URL

Patrice Castonguay

Papers

A New Class of High-Order Energy Stable Flux Reconstruction Schemes

Insights from von Neumann analysis of high-order flux reconstruction schemes

A New Class of High-Order Energy Stable Flux Reconstruction Schemes for Triangular Elements

On the Non-linear Stability of Flux Reconstruction Schemes

NeMo: a toolkit for building AI applications using Neural Modules.