​
toExport

A function for exporting Polars DataFrames to various file formats with configurable options. This is the recommended way to export ETL data as it provides consistent formatting and multiple export format support.

​
Basic Usage

import * as gs from '@hotglue/gluestick-ts';

const df = pl.DataFrame({...});
gs.toExport(df, "users", "./etl-output");

​
Key Methods

​
toExport(data, name, outputDir, options)

Exports a Polars DataFrame to the specified output directory in the chosen format.

​
Usage

// Basic singer export
gs.toExport(df, "users", "./etl-output");

// Export with custom options
gs.toExport(df, "users", "./etl-output", {
    exportFormat: 'parquet',
    keys: ['user_id'],
    outputFilePrefix: 'processed_'
});

​
Parameters

  • data (pl.DataFrame): Polars DataFrame to export
  • name (string): Name for the output file
  • outputDir (string): Directory path for output files
  • options (ExportOptions): Optional configuration object

​
ExportOptions Interface

Configuration options for the export function.

​
Properties

  • keys? (string[]): Primary keys for the dataset (used in singer format)
  • exportFormat? (‘csv’ | ‘json’ | ‘jsonl’ | ‘parquet’ | ‘singer’): Output format (defaults to ‘singer’ or DEFAULT_EXPORT_FORMAT env var)
  • outputFilePrefix? (string): Prefix to add to output filenames
  • stringifyObjects? (boolean): Whether to stringify object values
  • reservedVariables? (Record<string, string>): Custom variables for prefix formatting
  • allowObjects? (boolean): Whether to allow object values in singer format (defaults to true)
  • schema? (SingerHeaderMap): Singer schema definition

​
Common Patterns

​
Iterating through multiple streams and exporting

import * as gs from '@hotglue/gluestick-ts';

// iterate through each stream in the input directory
const input = new gs.Reader();
const availableStreams = input.keys();

for (const key of availableStreams) {
    const inputDf = input.get(key, { catalogTypes: true });
    
    // Export the transformed data
    gs.toExport(inputDf, key, "./etl-output");
}