True random number generators (TRNG) sample random physical processes to create large amounts of random numbers for various use cases, including security-critical cryptographic primitives, scientific simulations, machine learning applications, and even recreational entertainment. Unfortunately, not every computing system is equipped with dedicated TRNG hardware, limiting the application space and security guarantees for such systems. To open the application space and enable security guarantees for the overwhelming majority of computing systems that do not necessarily have dedicated TRNG hardware (e.g., processing-in-memory systems), we develop QUAC-TRNG, a new high-throughput TRNG that can be fully implemented in commodity DRAM chips, which are key components in most modern systems. QUAC-TRNG exploits the new observation that a carefully-engineered sequence of DRAM commands activates four consecutive DRAM rows in rapid succession. This QUadruple ACtivation (QUAC) causes the bitline sense amplifiers to non-deterministically converge to random values when we activate four rows that store conflicting data because the net deviation in bitline voltage fails to meet reliable sensing margins. We experimentally demonstrate that QUAC reliably generates random values across 136 commodity DDR4 DRAM chips from one major DRAM manufacturer. We describe how to develop an effective TRNG (QUAC-TRNG) based on QUAC. We evaluate the quality of our TRNG using the commonly-used NIST statistical test suite for randomness and find that QUAC-TRNG successfully passes each test. Our experimental evaluations show that QUAC-TRNG reliably generates true random numbers with a throughput of 3.44 Gb/s (per DRAM channel), outperforming the state-of-the-art DRAM-based TRNG by 15.08X and 1.41X for basic and throughput-optimized versions, respectively. We show that QUAC-TRNG utilizes DRAM bandwidth better than the state-of-the-art, achieving up to 2.03X the throughput of a throughput-optimized baseline when scaling bus frequencies to 12 GT/s.

QUAC-TRNG: high-throughput true random number generation using quadruple row activation in commodity DRAM chips

Many modern workloads, such as neural networks, databases, and graph processing, are fundamentally memory-bound. For such workloads, the data movement between main memory and CPU cores imposes a significant overhead in terms of both latency and energy. A major reason is that this communication happens through a narrow bus with high latency and limited bandwidth, and the low data reuse in memory-bound workloads is insufficient to amortize the cost of main memory access. Fundamentally addressing this data movement bottleneck requires a paradigm where the memory system assumes an active role in computing by integrating processing capabilities. This paradigm is known as processing-in-memory (PIM). Recent research explores different forms of PIM architectures, motivated by the emergence of new 3D-stacked memory technologies that integrate memory with a logic layer where processing elements can be easily placed. Past works evaluate these architectures in simulation or, at best, with simplified hardware prototypes. In contrast, the UPMEM company has designed and manufactured the first publicly-available real-world PIM architecture. The UPMEM PIM architecture combines traditional DRAM memory arrays with general-purpose in-order cores, called DRAM Processing Units (DPUs), integrated in the same chip. This paper provides the first comprehensive analysis of the first publicly-available real-world PIM architecture. We make two key contributions. First, we conduct an experimental characterization of the UPMEM-based PIM system using microbenchmarks to assess various architecture limits such as compute throughput and memory bandwidth, yielding new insights. Second, we present PrIM (Processing-In-Memory benchmarks), a benchmark suite of 16 workloads from different application domains (e.g., dense/sparse linear algebra, databases, data analytics, graph processing, neural networks, bioinformatics, image processing), which we identify as memory-bound. We evaluate the performance and scaling characteristics of PrIM benchmarks on the UPMEM PIM architecture, and compare their performance and energy consumption to their modern CPU and GPU counterparts. Our extensive evaluation conducted on two real UPMEM-based PIM systems with 640 and 2,556 DPUs provides new insights about suitability of different workloads to the PIM system, programming recommendations for software designers, and suggestions and hints for hardware and architecture designers of future PIM systems.

https://ieeexplore.ieee.org/ielx7/6287639/6514899/09771457.pdf

Benchmarking a New Paradigm: Experimental Analysis and Characterization of a Real Processing-in-Memory System

Personalized recommendation systems are gaining significant traction due to their industrial importance. An important building block of recommendation systems consists of the embedding layers, which exhibit a highly memory-intensive characteristic. A fundamental primitive of embedding layers is the embedding vector gathers followed by vector reductions, exhibiting low arithmetic intensity and becoming bottlenecked by the memory throughput. To tackle such a challenge, recent proposals employ a near-data processing (NDP) solution at the DRAM rank-level, achieving impressive performance speedups. We observe that prior rank-level-parallelism-based NDP solutions leave significant performance potential on the table as they do not fully reap the abundant transfer throughput inherent in DRAM datapaths. We propose TRiM, an NDP architecture for accelerating recommendation systems. Based on the observation that the DRAM datapath has a hierarchical tree structure, TRiM augments the DRAM datapath with “in-DRAM” reduction units at the DDR4/5 rank/bank-group/bank level. We modify the interface of DRAM to provide commands effectively to multiple reduction units running in parallel. We also propose a host-side architecture with hot embedding-vector replication to alleviate the load imbalance that arises across the reduction units. An optimal TRiM design based on DDR5 achieves up to a 7.7 × and 3.9 × speedup and reduces by 55% and 50% the energy consumption of the embedding vector gather and reduction over the baseline and the state-of-the-art NDP architecture with minimal area overhead equivalent to 2.66% of DRAM chips.

https://dl.acm.org/doi/pdf/10.1145/3466752.3480080

TRiM: Enhancing Processor-Memory Interfaces with Scalable Tensor Reduction in Memory

A semiconductor memory device includes an error correction code (ECC) engine, a memory cell array, an input/output (I/O) gating circuit and a control logic circuit. The memory cell array includes a normal cell region configured to store main data and a parity cell region configured to selectively store parity data which the ECC engine generates based on the main data, and sub data received from outside of the semiconductor memory device. The control logic circuit controls the ECC engine to selectively perform an ECC encoding and an ECC decoding on the main data and controls the I/O gating circuit to store the sub data in at least a portion of the parity cell region.

Semiconductor memory devices, memory systems and methods of operating semiconductor memory devices

An error check and scrub (ECS) mode enables a memory device to perform error checking and correction (ECC) and count errors. An associated memory controller triggers the ECS mode with a trigger sent to the memory device. The memory device includes multiple addressable memory locations, which can be organized in segments such as wordlines. The memory locations store data and have associated ECC information. In the ECS mode, the memory device reads one or more memory locations and performs ECC for the one or more memory locations based on the ECC information. The memory device counts error information including a segment count indicating a number of segments having at least a threshold number of errors, and a maximum count indicating a maximum number of errors in any segment.

Memory device error check and scrub mode and error transparency

In recent years, artificial intelligence (AI) technology has proliferated rapidly and widely into application areas such as speech recognition, health care, and autonomous driving. To increase the capabilities of AI more powerful systems are needed to process a larger amount of data. This requirement has made domain-specific accelerators, such as GPUs and TPUs, popular; as they can provide orders of magnitude higher performance than state-of-the-art CPUs. However, these accelerators can only operate at their peak performance when they get the necessary data from memory as quickly as it is processed: requiring off-chip memory with a high bandwidth and a large capacity [1]. HBM has thus far met the bandwidth and capacity requirement [2] –[6], but recent AI technologies such as recurrent neural networks require an even higher bandwidth than HBM [7]–[8]. While a further increase in off-chip bandwidth can be accomplished by various techniques, it is often limited by power constraints at the chip or system level [9]. Hence, it is essential to decrease demand for off-chip bandwidth with unconventional architectures: such as processing-in-memory. In this paper, we present function-In-memory DRAM (FIMDRAM) that integrates a 16-wide single-instruction multiple-data engine within the memory banks and that exploits bank-level parallelism to provide $4 \times $ higher processing bandwidth than an off-chip memory solution. Second, we show techniques that do not require any modification to conventional memory controllers and their command protocols, which make FIMDRAM more practical for quick industry adoption. Finally, we conclude this paper with circuit- and system-level evaluations of our fabricated FIMDRAM.

25.4 A 20nm 6GB Function-In-Memory DRAM, Based on HBM2 with a 1.2TFLOPS Programmable Computing Unit Using Bank-Level Parallelism, for Machine Learning Applications

A memory device including: an error correction code (ECC) cell array; an ECC engine configured to receive write data to be written to a memory cell array and generate internal parity bits for the write data; and an ECC select unit configured to receive the internal parity bits and external parity bits and, in response to a first level of a control signal, store the internal parity bits in the ECC cell array and, in response to a second level of the control signal store the external parity bits in the ECC cell array.

Memory device having a shareable error correction code cell array

A device, system, and/or method includes an internal circuit configured to perform at least one function, an input-output terminal set and a repair circuit. The input-output terminal set includes a plurality of normal input-output terminals connected to an external device via a plurality of normal signal paths and at least one repair input-output terminal selectively connected to the external device via at least one repair signal path. The repair circuit repairs at least one failed signal path included in the normal signal paths based on a mode signal and fail information signal, where the mode signal represents whether to use the repair signal path and the fail information signal represents fail information on the normal signal paths. Using the repair circuit, various systems adopting different repair schemes may be repaired and cost of designing and manufacturing the various systems may be reduced.

Device and system including adaptive repair circuit

A semiconductor apparatus includes two or more semiconductor chips and a tester. The two or more semiconductor chips are electrically connected through one or more through-silicon vias (TSVs). The tester is on at least one of the two or more semiconductor chips and tests the state of at least one TSV based on an output signal of the TSV. The TSV is selected as a signal transmission TSV based on the state of the TSV.

Semiconductor apparatus, stack semiconductor apparatus, and test method of the stack semiconductor apparatus

A memory device including a stack semiconductor device including; an upper substrate vertically stacked on a lower substrate, the upper substrate including N upper through-silicon vias (UTSV) and upper driving circuits, and the lower substrate including N lower through-silicon vias (LTSV) and lower driving circuits, wherein each one of the upper driving circuits is stagger-connected between a Kth UTSV and a (K+1)th LTSV, where ‘N’ is a natural number greater than 1, and ‘K’ is a natural number ranging from 1 to (N−1).

Hae-Suk Lee

Papers

25.4 A 20nm 6GB Function-In-Memory DRAM, Based on HBM2 with a 1.2TFLOPS Programmable Computing Unit Using Bank-Level Parallelism, for Machine Learning Applications

Memory device having a shareable error correction code cell array

Device and system including adaptive repair circuit

Semiconductor apparatus, stack semiconductor apparatus, and test method of the stack semiconductor apparatus

Stack semiconductor device and memory device including same