DRAM has been a de facto standard for main memory, and advances in process technology have led to a rapid increase in its capacity and bandwidth. In contrast, its random access latency has remained relatively stagnant, as it is still around 100 CPU clock cycles. Modern computer systems rely on caches or other latency tolerance techniques to lower the average access latency. However, not all applications have ample parallelism or locality that would help hide or reduce the latency. Moreover, applications' demands for memory space continue to grow, while the capacity gap between last-level caches and main memory is unlikely to shrink. Consequently, reducing the main-memory latency is important for application performance. Unfortunately, previous proposals have not adequately addressed this problem, as they have focused only on improving the bandwidth and capacity or reduced the latency at the cost of significant area overhead.We propose asymmetric DRAM bank organizations to reduce the average main-memory access latency. We first analyze the access and cycle times of a modern DRAM device to identify key delay components for latency reduction. Then we reorganize a subset of DRAM banks to reduce their access and cycle times by half with low area overhead. By synergistically combining these reorganized DRAM banks with support for non-uniform bank accesses, we introduce a novel DRAM bank organization with center high-aspect-ratio mats called CHARM. Experiments on a simulated chip-multiprocessor system show that CHARM improves both the instructions per cycle and system-wide energy-delay product up to 21% and 32%, respectively, with only a 3% increase in die area.

/pdf/reducing-memory-access-latency-with-asymmetric-dram-bank-57656h2c92.pdf

Reducing memory access latency with asymmetric DRAM bank organizations

A hybrid storage system is described having a mixture of different types of storage devices comprising rotational drives, flash devices, SDRAM, and SRAM. The rotational drives are used as the main storage, providing lowest cost per unit of storage memory. Flash memory is used as a higher-level cache for rotational drives. Methods for managing multiple levels of cache for this storage system is provided having a very fast Level 1 cache which consists of volatile memory (SRAM or SDRAM), and a non-volatile Level 2 cache using an array of flash devices. It describes a method of distributing the data across the rotational drives to make caching more efficient. It also describes efficient techniques for flushing data from L1 cache and L2 cache to the rotational drives, taking advantage of concurrent flash devices operations, concurrent rotational drive operations, and maximizing sequential access types in the rotational drives rather than random accesses which are relatively slower. Methods provided here may be extended for systems that have more than two cache levels.

Multi-leveled cache management in a hybrid storage system

This paper describes a tri-modal asymmetric bidirectional differential memory interface that supports data rates of up to 20 Gbps over 3" FR4 PCB channels while achieving power efficiency of 61 mW/Gbps at full speed The interface also accommodates single-ended standard DDR3 and GDDR5 signaling at 16-Gbps and 64-Gbps operations, respectively, without package change The compact, low-power and high-speed tri-modal interface is enabled by substantial reuse of the circuit elements among various signaling modes, particularly in the wide-band clock generation and distribution system and the multi-modal driver output stage, as well as the use of fast equalization for post-cursor intersymbol interference (ISI) mitigation In the high-speed differential mode, the system utilizes a 1-tap transmit equalizer during a WRITE operation to the memory In contrast, during a memory READ operation, it employs a linear equalizer (LEQ) with 3 dB of peaking as well as a calibrated high-speed 1-tap predictive decision feedback equalizer (prDFE), while no transmitter equalization is assumed for the memory The prototype tri-modal interface implemented in a 40-nm CMOS process, consists of 16 data links and achieves more than 25 × energy-efficient memory transactions at 16 Gbps compared to a previous single-mode generation

A Tri-Modal 20-Gbps/Link Differential/DDR3/GDDR5 Memory Interface

DRAM Main memory is a performance bottleneck for many applications due to the high access latency. In-DRAM caches work to mitigate this latency by augmenting regular-latency DRAM with small-but-fast regions of DRAM that serve as a cache for the data held in the regular-latency region of DRAM. While an effective in-DRAM cache can allow a large fraction of memory requests to be served from a fast DRAM region, the latency savings are often hindered by inefficient mechanisms for relocating copies of data into and out of the fast regions. Existing in-DRAM caches have two sources of inefficiency: (1) the data relocation granularity is an entire multi-kilobyte row of DRAM; and (2) because the relocation latency increases with the physical distance between the slow and fast regions, multiple fast regions are physically interleaved among slow regions to reduce the relocation latency, resulting in increased hardware area and manufacturing complexity. We propose a new substrate, FIGARO, that uses existing shared global buffers among subarrays within a DRAM bank to provide support for in-DRAM data relocation across subarrays at the granularity of a single cache block. FIGARO has a distance-independent latency within a DRAM bank, and avoids complex modifications to DRAM. Using FIGARO, we design a fine-grained in-DRAM cache called FIGCache. The key idea of FIGCache is to cache only small, frequently-accessed portions of different DRAM rows in a designated region of DRAM. By caching only the parts of each row that are expected to be accessed in the near future, we can pack more of the frequently-accessed data into FIGCache, and can benefit from additional row hits in DRAM. Our evaluations show that FIGCache improves the average performance of a system using DDR4 DRAM by 16.3% and reduces average DRAM energy consumption by 7.8% for 8-core workloads, over a conventional system without in-DRAM caching.

FIGARO: Improving System Performance via Fine-Grained In-DRAM Data Relocation and Caching

A mechanism of booting up a system directly from a storage device and a means of initializing an embedded system prior to activating a CPU is presented. The said system is comprised of one or more CPUs, a reset controller, a storage device controller, one or more direct memory access controllers, a RAM and its controller, a ROM and its controller, a debug interface and a power-on reset (POR) sequencer. The POR sequencer controls the overall boot process of the embedded system. Said sequencer uses descriptors (POR Sequencer descriptors) which are used to update the configuration registers of the system and to enable CPU-independent data transfers with the use of DMA controllers. Using a minimal amount of non-volatile memory for booting up a system brings down costs associated with increased silicon real estate area and power consumption. Capability of pre-initializing the system even before a CPU is brought out of reset provides flexibility and system robustness. Through the use of the Power-On Reset Sequencer module, integrity of program code and user data used in the boot up process can be verified thus providing a resilient boot up sequence. The present invention provides a mechanism for booting up a system using a minimum amount of nonvolatile memory. This method also enables the embedded system to initialize all configuration registers even before any of the CPUs of the system is brought out of reset. The embedded system consists of multiple controller chips or a single controller chip. The embedded system can have a single or multiple central processing units.

Embedded system boot from a storage device

Most DRAM interfaces such as GDDR5 and DDR3 use parallel single-ended signaling due to pin-count restriction and backward compatibility. Notwithstanding poor signal and power integrity issues, GDDR5 speed reached beyond 5Gb/s in recent years by utilizing data bus inversion, error-detection coding, data training and channel equalization [1–3]. However, channel crosstalk is becoming a major barrier to further speed improvement. A common solution for channel crosstalk reduction at the system level is to use a shielding line or wide spacing between signal lines, but increasing the number of layers in a chip package and PCB increase system cost. To remove the shielding lines and increase speed, this paper presents a channel crosstalk equalizer with programmable signal ordering capability for the DRAM transmitter. In addition, this paper addresses tri-mode clocking to reduce the system jitter for better timing margin: PLL off, LC-PLL and injection-locked oscillator.

A 40nm 2Gb 7Gb/s/pin GDDR5 SDRAM with a programmable DQ ordering crosstalk equalizer and adjustable clock-tracking BW

A latency control circuit is configured to delay a read information signal in response to a CAS latency signal and an internal clock signal to generate a delayed read information signal, and is further configured to generate a latency control signal based on the delayed read information signal in response to a plurality of sampling control signals and a plurality of transfer control signals.

Latency control circuit and semiconductor memory device comprising same

As DRAM data bandwidth increases, tremendous energy is dissipated in the DRAM data bus. To reduce the energy consumed in the data bus, DRAM interfaces with asymmetric termination, such as Pseudo Open Drain (POD) and Low Voltage Swing Terminated Logic (LVSTL), have been adopted in modern DRAMs. In interfaces using asymmetric termination, the amount of termination energy is proportional to the hamming weight of the data words. In this work, we propose Bitwise Difference Encoding (BD-Encoding), which decreases the hamming weight of data words, leading to a reduction in energy consumption in the modern DRAM data bus. Since smaller hamming weight of the data words also reduces switching activity, switching energy and power noise are also both reduced.BD-Encoding exploits the similarity in data words in the DRAM data bus. We observed that similar data words (i.e. data words whose hamming distance is small) are highly likely to be sent over at similar times. Based on this observation, BD-coder stores the data recently sent over in both the memory controller and DRAMs. Then, BD-coder transfers the bitwise difference between the current data and the most similar data. In an evaluation using SPEC 2006, BD-Encoding using 64 recent data reduced termination energy by 58.3% and switching energy by 45.3%. In addition, 55% of the LdI/dt noise was decreased with BD-Encoding.

Energy efficient data encoding in DRAM channels exploiting data value similarity

An on-die termination circuit includes a termination resistor unit connected to an external pin, and a termination control unit connected to the termination resistor unit. The termination resistor unit provides termination impedance to a transmission line connected to the external pin. The termination control unit varies the termination impedance in response to a plurality of bits of strength code associated with a data rate.

On-die termination circuit, data output buffer and semiconductor memory device

Recent large-scale CNNs suffer from a severe memory wall problem as their number of weights range from tens to hundreds of millions. Processing in-memory (PIM) and binary CNN have been proposed to alleviate the number of memory accesses and footprints, respectively. By combining the two separate concepts, we propose a novel processing in-DRAM framework for binary CNN, called NID, where dominant convolution operations are processed using in-DRAM bulk bitwise operations. We first identify the problem that the bitcount operations with only bulk bitwise AND/OR/NOT incur significant overhead in terms of delay when the size of kernels gets larger. Then, we not only optimize the performance by efficiently allocating inputs and kernels to DRAM banks for both convolutional and fully-connected layers through design space explorations, but also mitigate the overhead of bitcount operations by splitting kernels into multiple parts. Partial sum accumulations and tasks of the other layers such as max-pooling and normalization layers are processed in the peripheral area of DRAM with negligible overheads. In results, our NID framework achieves 19×-36× performance and 9×-14× EDP improvements for convolutional layers, and 9×-17× performance and 1.4×-4.5× EDP improvements for fully-connected layers over previous PIM technique in four large-scale CNN models.

Ho-Seok Seol

Papers

A 40nm 2Gb 7Gb/s/pin GDDR5 SDRAM with a programmable DQ ordering crosstalk equalizer and adjustable clock-tracking BW

Latency control circuit and semiconductor memory device comprising same

Energy efficient data encoding in DRAM channels exploiting data value similarity

On-die termination circuit, data output buffer and semiconductor memory device

NID: processing binary convolutional neural network in commodity DRAM