Which Data Format to Use For Your Big Data Project?

Pickle, Parquet, CSV, Feather, HDF5, ORC, JSON: which one should you be using and why?

Choosing the right data format is crucial in Data Science projects, impacting everything from data read/write speeds to memory consumption and interoperability. This article explores seven popular serialization/deserialization formats in Python, focusing on their speed and memory usage implications.

Through the analysis, we’ll also see how we can use profiling in Python (using the cProfile built-in module) and how we can get statistics on memory usage for specific files in your filesystem (using the os Python module).

Of course, each project has its specificities, beyond just speed and memory usage. But we’ll draw some trends, that can hopefully be useful to shed light on which format we can choose for a given project.

Understanding Serialization and Deserialization

Serialization is the process of saving an object (in Python, a pandas DataFrame for example) to a format that can be saved to a file for later retrieval. Deserialization is the reverse process.

A dataframe is a Python object and cannot be persisted as is. It needs to be translated to a file to be able to load this object at a later stage.

When you save a dataframe, you “serialize” the data. And when you load it back, you “deserialize” or translate it back to a language-readable (here Python-readable) format.

Certain formats are widely used because they are human-readable, such as JSON or CSV. These two formats are also used because they are language agnostic. Just like protocol buffers, which were originally developed by Google. JSON and Protocol buffer are also popular for APIs and enable sending data between different services written in different languages.

On the other hand, some formats, like Python’s pickle, are language-specific and not ideal for transferring data between services in different programming languages. For example, for a machine learning use case, if a repository trains a model and serializes it in pickle, this file will only be able to be read from Python. So if the API that…