Data Export & Error Handling

​

to_export

​

Installation

import gluestick as gs

​

Basic Usage

# Basic export to default format (Singer)
gs.to_export(df, "users", OUTPUT_DIR, keys=['id'])

# Export to specific format
gs.to_export(
    df,
    "users",
    OUTPUT_DIR,
    export_format="parquet",
    stringify_objects=True
)

​

Parameters

• data (pd.DataFrame): DataFrame to export
• name (str): Output file name/stream name
• output_dir (str): Directory for output files
• keys (list): Primary key fields
• unified_model (pydantic.BaseModel): Pydantic model for schema validation
• export_format (str): Output format (‘singer’, ‘parquet’, ‘json’, ‘jsonl’, ‘csv’)
• output_file_prefix (str): Optional prefix for output files
• schema (dict): Custom schema for Singer format
• stringify_objects (bool): Convert complex objects to strings for Parquet
• target_state_fields (list | str): Column names from export records to persist in target state customData (Singer export only)
• target_state_include_hash (bool): Persist the target SDK record hash in target state customData (Singer export only)

​

Returns

​

Notes

• Uses environment variables for format defaults
• Supports prefix override per stream
• Handles complex data types appropriately per format
• Validates against Pydantic models when provided

​

to_singer

​

Usage

import gluestick as gs

# Basic Singer export
gs.to_singer(df, "users", OUTPUT_DIR, keys=['id'])

# With schema validation
gs.to_singer(
    df,
    "users",
    OUTPUT_DIR,
    keys=['id'],
    allow_objects=True,
    schema=custom_schema
)

# Persist ETL fields in target state customData
gs.to_singer(
    df,
    "contacts",
    OUTPUT_DIR,
    keys=["externalId"],
    target_state_fields=["email", "status"],
    target_state_include_hash=True,
)

​

Parameters

• df (pd.DataFrame): DataFrame to export
• stream (str): Singer stream name
• output_dir (str): Output directory
• keys (list): Primary key fields
• filename (str): Output filename (default: ‘data.singer’)
• allow_objects (bool): Enable complex object handling
• schema (dict): Custom schema definition
• unified_model (pydantic.BaseModel): Pydantic model for validation
• target_state_fields (list | str): Column names from export records to persist in target state customData
• target_state_include_hash (bool): Persist the target SDK record hash in customData

​

Notes

• Handles datetime conversions automatically
• Supports catalog-based schemas
• Validates data against provided schemas
• Creates Singer-compliant output files

​

Target state metadata

• target_state_fields: columns to persist in target state customData. Field names in target_state_fields refer to columns on records as your ETL sends them to the target. A single string is treated as one field.
• target_state_include_hash: persist the target SDK record hash in target state customData.

import json
import os

import gluestick as gs
import pandas as pd

snapshot_df = gs.read_snapshots(f"contacts_{os.environ['FLOW']}", f"{ROOT_DIR}/snapshots")
if snapshot_df is not None and not snapshot_df.empty:
    rows = snapshot_df.dropna(subset=["CustomData"])
    meta = pd.DataFrame(
        rows["CustomData"].map(json.loads).tolist(),
        index=rows["InputId"],
    )
    df = df.merge(meta[["status", "hash"]], left_on="externalId", right_index=True, how="left")

​

Common Patterns

​

Handling Complex Exports

import gluestick as gs

# Multiple format export
def multi_format_export(df, name):
    for format in ['singer', 'parquet', 'json']:
        try:
            gs.to_export(
                df,
                name,
                OUTPUT_DIR,
                export_format=format
            )
        except Exception as e:
            gs.exception(
                e,
                ROOT_DIR,
                f"Failed {format} export for {name}"
            )