Data Reading Functions

​

Reader

Recommended: A class for reading ETL files from the sync-output directory. This is the recommended way to read ETL files as it provides consistent error handling and metadata access.

​

Installation

from gluestick.reader import Reader

​

Basic Usage

# Initialize with default directories
reader = Reader()  # Uses ROOT_DIR/sync-output

# Get available streams
print(reader)  # Shows list of available streams

​

Key Methods

​

get(stream, default=None, catalog_types=False, **kwargs)

Reads the selected file into a pandas DataFrame.

# Basic reading
df = reader.get("users")

# Using catalog types
df = reader.get("users", catalog_types=True)

​

get_metadata(stream)

Retrieves metadata from parquet files.

metadata = reader.get_metadata("users")
# Returns dict of metadata key-value pairs

​

get_pk(stream)

Gets primary key(s) from parquet file metadata.

primary_keys = reader.get_pk("users")
# Returns list of primary key column names

​

read_csv_folder

Legacy function for reading multiple CSV files. Consider using Reader class instead.

​

Usage

from gluestick.etl_utils import read_csv_folder

# Basic usage
entity_data = read_csv_folder(INPUT_DIR)

​

Parameters

• path (str): Directory containing CSV files or path to single CSV file
• converters (dict): Entity-specific column converters
• index_cols (dict): Entity-specific index columns
• ignore (list): Files to ignore

​

Returns

Dictionary of pandas DataFrames, keyed by entity name

​

read_parquet_folder

Legacy function for reading multiple parquet files. Consider using Reader class instead.

​

Usage

from gluestick.etl_utils import read_parquet_folder

# Basic usage
entity_data = gs.read_parquet_folder(INPUT_DIR)

​

Parameters

• path (str): Directory containing parquet files or path to single file
• ignore (list): Files to ignore

​

Returns

Dictionary of pandas DataFrames, keyed by entity name

​

get_catalog_schema

Retrieves DataFrame schema from Singer catalog file.

​

Usage

from gluestick.singer import get_catalog_schema, to_singer


# Get schema for specific stream
schema = get_catalog_schema("users")

# Use schema with Singer export
to_singer(df, "users", OUTPUT_DIR, schema=schema)

​

Returns

Dictionary containing the stream’s schema definition

​

Notes

• Requires catalog.json in root directory
• Raises exception if stream not found in catalog
• Filters schema to include only type and properties
• Ensures array types have items dictionary

​

Common Patterns

​

Iterating through sultiple streams

import gluestick as gs

# iterate through each stream in the input directory
reader = gs.Reader()
for key in eval(str(reader)):

    # define a dataframe and apply transformations
    input_df = reader.get(key, catalog_types=True)
    input_df["tenant"] = TENANT_ID

    # define the primary keys (assuming the data is Parquet)
    key_properties = reader.get_pk(key)

    # write the data out to the output directory
    gs.to_export(input_df, key, OUTPUT_DIR, keys=key_properties)