Data Reading Functions

Reader

Recommended: A class for reading ETL files from the sync-output directory. This is the recommended way to read ETL files as it provides consistent error handling and metadata access.

Installation

from gluestick.reader import Reader

Basic Usage

# Initialize with default directories
reader = Reader()  # Uses ROOT_DIR/sync-output

# Get available streams
print(reader)  # Shows list of available streams

Key Methods

get(stream, default=None, catalog_types=False, **kwargs)

Reads the selected file into a pandas DataFrame.

# Basic reading
df = reader.get("users")

# Using catalog types
df = reader.get("users", catalog_types=True)

get_metadata(stream)

Retrieves metadata from parquet files.

metadata = reader.get_metadata("users")
# Returns dict of metadata key-value pairs

get_pk(stream)

Gets primary key(s) from parquet file metadata.

primary_keys = reader.get_pk("users")
# Returns list of primary key column names

get_catalog_schema

Retrieves Singer schema from catalog file.

Usage

from gluestick.singer import get_catalog_schema, to_singer


# Get schema for specific stream
schema = get_catalog_schema("users")

# Use schema with Singer export
to_singer(df, "users", OUTPUT_DIR, schema=schema)

Returns

Dictionary containing the stream’s schema definition

Notes

Requires catalog.json in root directory
Raises exception if stream not found in catalog
Filters schema to include only type and properties
Ensures array types have items dictionary

Common Patterns

Iterating through multiple streams

etl.py

import gluestick as gs

# iterate through each stream in the input directory
reader = gs.Reader()
for key in eval(str(reader)):

    # define a dataframe and apply transformations
    input_df = reader.get(key, catalog_types=True)
    input_df["tenant"] = TENANT_ID

    # define the primary keys (assuming the data is Parquet)
    key_properties = reader.get_pk(key)

    # write the data out to the output directory
    gs.to_export(input_df, key, OUTPUT_DIR, keys=key_properties)

Getting Started

Key Concepts

Managed authentication

Transformation

Custom Connectors

Unified Schema

Environment Settings

CLI

Connectors

Reader

Installation

Basic Usage

Key Methods

get(stream, default=None, catalog_types=False, **kwargs)

get_metadata(stream)

get_pk(stream)

get_catalog_schema

Usage

Returns

Notes

Common Patterns

Iterating through multiple streams

​Reader

​Installation

​Basic Usage

​Key Methods

​get(stream, default=None, catalog_types=False, **kwargs)

​get_metadata(stream)

​get_pk(stream)

​get_catalog_schema

​Usage

​Returns

​Notes

​Common Patterns

​Iterating through multiple streams

Reader

Installation

Basic Usage

Key Methods

get(stream, default=None, catalog_types=False, **kwargs)

get_metadata(stream)

get_pk(stream)

get_catalog_schema

Usage

Returns

Notes

Common Patterns

Iterating through multiple streams