MLA 005 Shapes and Sizes: Tensors and NDArrays

Definitions

• Tensor: A general term for an array of any number of dimensions.

  • 0D Tensor (Scalar): A single number (e.g., 5).
  • 1D Tensor (Vector): A simple list of numbers.
  • 2D Tensor (Matrix): A grid of numbers (rows and columns).
  • 3D+ Tensors: Higher-dimensional arrays, such as images or batches of images.

• NDArray (NumPy): Stands for "N-dimensional array," the foundational array type in NumPy, synonymous with "tensor."

Tensor Properties

Dimensions

• Number of nested levels in the array (e.g., a matrix has two dimensions: rows and columns).
• Access in NumPy: Via .ndim property (e.g., array.ndim).

Size

• Total number of elements in the tensor.
• Examples:
  • Scalar: size = 1
  • Vector: size equals number of elements (e.g., 5 for [1, 2, 3, 4, 5])
  • Matrix: size = rows × columns (e.g., 10×10 = 100)
• Access in NumPy: Via .size property.

Shape

• Tuple listing the number of elements per dimension.
• Example: An image with 256×256 pixels and 3 color channels has shape = (256, 256, 3).

Common Scenarios & Examples

Data Structures in Practice

• CSV/Spreadsheet Example: Dataset with 1 million housing examples and 50 features:
  • Shape: (1_000_000, 50)
  • Size: 50,000,000
• Image Example (RGB): 256×256 pixel image:
  • Shape: (256, 256, 3)
  • Dimensions: 3 (width, height, channels)
• Batching for Models:
  • For a convolutional neural network, shape might become (batch_size, width, height, channels), e.g., (32, 256, 256, 3).

Conceptual Clarifications

• The term "dimensions" in data science often refers to features (columns), but technically in tensors it means the number of structural axes.
• The "curse of dimensionality" often uses "dimensions" to refer to features, not tensor axes.

Reshaping and Manipulation in NumPy

Reshaping Tensors

• Adding Dimensions:

  • Useful when a model expects higher-dimensional input than currently available (e.g., converting grayscale image from shape (256, 256) to (256, 256, 1)).
  • Use np.expand_dims or array.reshape.

• Removing Singleton Dimensions:

  • Occurs when, for example, model output is (N, 1) and single dimension should be removed to yield (N,).
  • Use np.squeeze or array.reshape.

• Wildcard with -1:

  • In reshaping, -1 is a placeholder for NumPy to infer the correct size, useful when batch size or another dimension is variable.

• Flattening:

  • Use np.ravel to turn a multi-dimensional tensor into a contiguous 1D array.

Axis Reordering

• Transposing Axes:
  • Needed when model input or output expects axes in a different order (e.g., sequence length and embedding dimensions in NLP).
  • Use np.transpose for general axis permutations.
  • Use np.swapaxes to swap two specific axes but prefer transpose for clarity and flexibility.

Practical Example

• In NLP sequence models:
  • 3D tensor with (batch_size, sequence_length, embedding_dim) might need to be reordered to (batch_size, embedding_dim, sequence_length) for certain models.
  • Achieved using: array.transpose(0, 2, 1)

Core NumPy Functions for Manipulation

• reshape: General function for changing the shape of a tensor, including adding or removing dimensions.
• expand_dims: Adds a new axis with size 1.
• squeeze: Removes axes with size 1.
• ravel: Flattens to 1D.
• transpose: Changes the order of axes.
• swapaxes: Swaps specified axes (less general than transpose).

Summary Table of Operations

Operation	NumPy Function	Purpose
Add dimension	np.expand_dims	Convert (256,256) to (256,256,1)
Remove dimension	np.squeeze	Convert (N,1) to (N,)
General reshape	np.reshape	Any change matching total size
Flatten	np.ravel	Convert (a,b) to (a*b,)
Swap axes	np.swapaxes	Exchange positions of two axes
Permute axes	np.transpose	Reorder any sequence of axes

Closing Notes

• A deep understanding of tensor structure - dimensions, size, and shape - is vital for preparing data for machine learning models.
• Reshaping, expanding, squeezing, and transposing tensors are everyday tasks in model development, especially for adapting standard datasets and models to each other.