The GCTx format and cmap{Py, R, M, J} packages: resources for optimized storage and integrated traversal of annotated dense matrices.

TLDR: The GCTx file format and a suite of open‐source packages for the efficient storage, serialization and analysis of dense two‐dimensional matrices are presented and it is anticipated that the format's generalizability will lower barriers for integrated cross‐assay analysis and algorithm development.

Abstract: Motivation Facilitated by technological improvements, pharmacologic and genetic perturbational datasets have grown in recent years to include millions of experiments Sharing and publicly distributing these diverse data creates many opportunities for discovery, but in recent years the unprecedented size of data generated and its complex associated metadata have also created data storage and integration challenges Results We present the GCTx file format and a suite of open-source packages for the efficient storage, serialization and analysis of dense two-dimensional matrices We have extensively used the format in the Connectivity Map to assemble and share massive datasets currently comprising 13 million experiments, and we anticipate that the format's generalizability, paired with code libraries that we provide, will lower barriers for integrated cross-assay analysis and algorithm development Availability and implementation Software packages (available in Python, R, Matlab and Java) are freely available at https://githubcom/cmap Additional instructions, tutorials and datasets are available at clueio/code Supplementary information Supplementary data are available at Bioinformatics online