Snapshotting functions

​

snapshot_records

A function for managing data snapshots by updating existing snapshots or creating new ones. Handles both CSV and Parquet formats with support for type coercion.

​

Installation

from gluestick.etl_utils import snapshot_records

​

Basic Usage

# Create or update a snapshot for the users table
updated_data = snapshot_records(
    stream_data,
    "users",
    SNAPSHOT_DIR,
    pk="user_id"
)

​

Parameters

• stream_data (DataFrame): New data to be included in snapshot
• stream (str): Name of the stream
• snapshot_dir (str): Directory for storing snapshots
• pk (str|list): Primary key(s) for deduplication
• just_new (bool): Adds only new records if True
• use_csv (bool): Use CSV format instead of Parquet
• coerce_types (bool): Force snapshot types to match stream_data
• **kwargs: Additional pandas read options

​

Returns

pandas DataFrame containing either complete snapshot or just new records

​

read_snapshots

Read existing snapshot data into a dataframe.

​

Usage

from gluestick.etl_utils import read_snapshots

# Read existing snapshot
snapshot_df = read_snapshots(
    "users",
    SNAPSHOT_DIR
)

​

Parameters

• stream (str): Name of the stream to read
• snapshot_dir (str): Directory containing snapshots
• **kwargs: Additional pandas read options

​

Returns

pandas DataFrame containing snapshot data, or None if no snapshot exists

​

drop_redundant

Removes duplicate rows based on content hashing, maintaining a snapshot of processed data.

​

Usage

from gluestick.etl_utils import drop_redundant

# Basic deduplication
clean_df = drop_redundant(
    df,
    "users",
    OUTPUT_DIR,
    pk="user_id"
)

# Track updated records
clean_df = drop_redundant(
    df,
    "users",
    OUTPUT_DIR,
    pk=["user_id", "email"],
    updated_flag=True
)

​

Parameters

• df (DataFrame): DataFrame to check for duplicates
• name (str): Name for snapshot hash file
• output_dir (str): Directory to save state files
• pk (str|list): Primary key(s) for state tracking
• updated_flag (bool): Add flag for new/updated rows
• use_csv (bool): Use CSV format instead of Parquet

​

Returns

pandas DataFrame with redundant rows removed

​

Common Patterns

​

Incremental Processing

import gluestick as gs

# Process only new records
reader = gs.Reader()
for stream in eval(str(reader)):
    # Get new data
    input_df = reader.get(stream)
    
    # Combine new data with previously snapshotted data
    all_records = gs.snapshot_records(
        input_df,
        stream,
        SNAPSHOT_DIR,
        pk=reader.get_pk(stream)
    )
    row_count = all_records.shape[0]
    print(f"Total number of records across jobs: {row_count}")