MLA 003 Storage: HDF, Pickle, Postgres
Practical workflow of loading, cleaning, and storing large datasets for machine learning, moving from ingesting raw CSVs or JSON files with pandas to saving processed datasets and neural network weights using HDF5 for efficient numerical storage. It clearly distinguishes among storage options - explaining when to use HDF5, pickle files, or SQL databases - while highlighting how libraries like pandas, TensorFlow, and Keras interact with these formats and why these choices matter for production pipelines.
Sitting for hours drains energy and focus. A walking desk boosts alertness, helping you retain complex ML topics more effectively.Boost focus and energy to learn faster and retain more.Discover the benefitsDiscover the benefits
Data Ingestion and Preprocessing
-
Data Sources and Formats:
- Datasets commonly originate as CSV (comma-separated values), TSV (tab-separated values), fixed-width files (FWF), JSON from APIs, or directly from databases.
- Typical applications include structured data (e.g., real estate features) or unstructured data (e.g., natural language corpora for sentiment analysis).
-
Pandas as the Core Ingestion Tool:
- Pandas provides versatile functions such as
read_csv,read_json, and others to load various file formats with robust options for handling edge cases (e.g., file encodings, missing values). - After loading, data cleaning is performed using pandas: dropping or imputing missing values, converting booleans and categorical columns to numeric form.
- Pandas provides versatile functions such as
-
Data Encoding for Machine Learning:
- All features must be numerical before being supplied to machine learning models like TensorFlow or Keras.
- Categorical data is one-hot encoded using
pandas.get_dummies, converting strings to binary indicator columns. - The underlying NumPy array of a DataFrame is accessed via
df.valuesfor direct integration with modeling libraries.
Numerical Data Storage Options
-
HDF5 for Storing Processed Arrays:
- HDF5 (Hierarchical Data Format version 5) enables efficient storage of large multidimensional NumPy arrays.
- Libraries like h5py and built-in pandas functions (
to_hdf) allow seamless saving and retrieval of arrays or DataFrames. - TensorFlow and Keras use HDF5 by default to store neural network weights as multi-dimensional arrays for model checkpointing and early stopping, accommodating robust recovery and rollback.
-
Pickle for Python Objects:
- Python's pickle protocol serializes arbitrary objects, including machine learning models and arrays, into files for later retrieval.
- While convenient for quick iterations or heterogeneous data, pickle is less efficient with NDarrays compared to HDF5, lacks significant compression, and poses security risks if not properly safeguarded.
-
SQL Databases and Spreadsheets:
- For mixed or heterogeneous data, or when producing results for sharing and collaboration, relational databases like PostgreSQL or spreadsheets such as CSVs are used.
- Databases serve as the endpoint for production systems, where model outputs - such as generated recommendations or reports - are published for downstream use.
Storage Workflow in Machine Learning Pipelines
-
Typical Process:
- Data is initially loaded and processed with pandas, then converted to numerical arrays suitable for model training.
- Intermediate states and model weights are saved using HDF5 during model development and training, ensuring recovery from interruptions and facilitating early stopping.
- Final outputs, especially those requiring sharing or production use, are published to SQL databases or shared as spreadsheet files.
-
Best Practices and Progression:
- Quick project starts may involve pickle for accessible storage during early experimentation.
- For large-scale, high-performance applications, migration to HDF5 for numerical data and SQL for production-grade results is recommended.
- Alternative options like Feather and PyTables (an interface on top of HDF5) exist for specialized needs.
Summary
- HDF5 is optimal for numerical array storage due to its efficiency, built-in compression, and integration with major machine learning frameworks.
- Pickle accommodates arbitrary Python objects but is suboptimal for numerical data persistence or security.
- SQL databases and spreadsheets are used for disseminating results, especially when human consumption or application integration is required.
- The selection of a storage format is determined by data type, pipeline stage, and end-use requirements within machine learning workflows.