Custom Plugins

What are custom plugins?

Custom plugins let you change what a unified schema connector exports without replacing its transformation script. A plugin is a Python file that receives one unified table as a pandas DataFrame just before hotglue exports it. It returns the DataFrame, with any changes you made. Common uses include:

Adding a column, such as a tenant-specific label
Removing rows that a tenant does not want to sync
Filling in a default value from the connector’s configuration
Changing the primary keys sent with the exported table

The Custom Plugins tab is only shown for connectors that use the unified schema, and only when custom plugins are enabled for your organization. Contact hotglue support if you don’t see the tab.

How plugins run

When a job runs, hotglue downloads the connector’s plugins into a plugins folder beside the transformation script. Each time the unified script exports a table, it looks for a plugin named after that table and calls the plugin’s main function:

sync-output → unified mapping → Accounts DataFrame → plugins/Accounts_post_process.py → etl-output

A plugin runs after the source data has been mapped to the unified model, and before the table is written to etl-output.
A table without a matching plugin is exported unchanged.
If a plugin raises an error, the export stops and the job fails with ETL_FAILED. The error appears in the job logs.

Name the plugin file

Name each plugin {Table}_post_process.py, where {Table} is the unified table name, such as Accounts, Vendors, or Bills. One file handles one table.

Write the `main` function

Each plugin must define a top-level main function with exactly this signature:

def main(context, df, model, key_properties, tenant_config={}):
    ...
    return df, model, key_properties

hotglue checks this signature when you upload or save the file. Type annotations are allowed, but the parameter names, their order, and the tenant_config={} default must match.

Arguments

Argument	Description
`context`	A dictionary with job details and paths. See Context.
`df`	A pandas DataFrame with the table’s unified records.
`model`	The unified Pydantic model used to build the exported schema.
`key_properties`	The table’s primary keys, such as `["id", "subsidiaryId"]`.
`tenant_config`	The tenant’s `tenant-config.json`, or `{}` if the tenant has none.

Return value

main must return a tuple of three values: (df, model, key_properties).

df must be a pandas DataFrame. Returning None or a DataFrame on its own fails the job.
Return model and key_properties unchanged unless you want to change them. The values you return are used for the export.
If you add columns to df, hotglue adds them to the exported schema as optional fields. You don’t need to change model.

Context

Key	Value
`ROOT_DIR`	The job’s working directory. Configuration files are in this directory.
`INPUT_DIR`	`{ROOT_DIR}/sync-output`
`OUTPUT_DIR`	The directory the table is exported to, usually `{ROOT_DIR}/etl-output`
`SNAPSHOT_DIR`	`{ROOT_DIR}/snapshots`
`tenant_id`	The ID of the tenant the job is running for
`flow_id`	The ID of the flow

Read configuration files

A plugin can read the same configuration files as a transformation script. Build paths from context["ROOT_DIR"] and context["SNAPSHOT_DIR"]:

File	Contents	Available
`tenant-config.json` in `SNAPSHOT_DIR`	The tenant config. Also passed to `main` as `tenant_config`.	When the tenant has a tenant config
`source-config.json` in `ROOT_DIR`	Settings and custom params of the source. In a read job, this is the connector’s config.	Read and write jobs
`target-config.json` in `ROOT_DIR`	Settings of the target. In a write job, this is the connector’s config.	Write jobs, and read jobs whose flow has a target
`config.json` in `ROOT_DIR`	The connector’s own config: the same file as `source-config.json` in a read job and `target-config.json` in a write job.	Read and write jobs

These files can contain credentials. Don’t print or log their values. Always check that a file exists before you read it. This lets the same plugin run in both read and write jobs.

Examples

Return the table unchanged

The smallest valid plugin.

Accounts_post_process.py

def main(context, df, model, key_properties, tenant_config={}):
    return df, model, key_properties

Add a column from the tenant config

This plugin reads region from the tenant config and adds it to every account as a new tenantRegion column. The new column is added to the exported schema automatically.

Accounts_post_process.py

def main(context, df, model, key_properties, tenant_config={}):
    df["tenantRegion"] = tenant_config.get("region")
    return df, model, key_properties

With the tenant config {"region": "EMEA"}, every exported account has "tenantRegion": "EMEA".

Read the tenant config file from snapshots

tenant_config contains the same data as tenant-config.json. You can also read the file directly, which is useful when you reuse code from your transformation scripts:

Accounts_post_process.py

import json
import os


def main(context, df, model, key_properties, tenant_config={}):
    path = os.path.join(context["SNAPSHOT_DIR"], "tenant-config.json")
    if os.path.exists(path):
        with open(path) as f:
            hidden_ids = json.load(f).get("hidden_account_ids", [])
        df = df[~df["id"].isin(hidden_ids)]
    return df, model, key_properties

Use `config.json` and change the primary keys

config.json is always the connector’s own config, in both read and write jobs. This plugin adds the connector’s company_id setting as a column and makes it part of the primary key:

Accounts_post_process.py

import json
import os


def main(context, df, model, key_properties, tenant_config={}):
    path = os.path.join(context["ROOT_DIR"], "config.json")
    if not os.path.exists(path):
        return df, model, key_properties
    with open(path) as f:
        config = json.load(f)
    df["companyId"] = config.get("company_id")
    return df, model, [*key_properties, "companyId"]

Upload and manage plugins

In the dashboard

Open your flow and select the connector.
Select the Custom Plugins tab.
Click Upload, or drag .py files into the tab, then click Upload again to save them.

Click a plugin to view or edit it in the browser, then click Save. Only Developers and Admins can upload, edit, or delete a plugin.

With the API

The plugin endpoints use your environment’s API key in the x-api-key header. Use default as the tenant to manage the plugins shared by every tenant. Use connectors as the entity type for v2 flows, and taps for v1 flows.

https://api.hotglue.com/{env_id}/{flow_id}/{tenant}/{entity_type}/{connector_id}/etl/plugins

curl --request POST \
  --url 'https://api.hotglue.com/{env_id}/{flow_id}/default/connectors/{connector_id}/etl/plugins' \
  --header 'x-api-key: <API_KEY>' \
  --form 'Accounts_post_process.py=@Accounts_post_process.py'

curl --request GET \
  --url 'https://api.hotglue.com/{env_id}/{flow_id}/default/connectors/{connector_id}/etl/plugins' \
  --header 'x-api-key: <API_KEY>'

curl --request GET \
  --url 'https://api.hotglue.com/{env_id}/{flow_id}/default/connectors/{connector_id}/etl/plugins?file=Accounts_post_process.py' \
  --header 'x-api-key: <API_KEY>'

curl --request PUT \
  --url 'https://api.hotglue.com/{env_id}/{flow_id}/default/connectors/{connector_id}/etl/plugins' \
  --header 'x-api-key: <API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{"filename": "Accounts_post_process.py", "content": "def main(context, df, model, key_properties, tenant_config={}):\n    return df, model, key_properties\n"}'

curl --request DELETE \
  --url 'https://api.hotglue.com/{env_id}/{flow_id}/default/connectors/{connector_id}/etl/plugins' \
  --header 'x-api-key: <API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{"file": "Accounts_post_process.py"}'

When you upload with the API, hotglue uses the multipart field name as the plugin’s file name. Name each form field after its file, as in --form 'Accounts_post_process.py=@Accounts_post_process.py'. A generic field name such as --form 'file=@Accounts_post_process.py' is rejected.

Calling DELETE with an empty body ({}) deletes all plugins for the connector and tenant.

Getting Started

Key Concepts

Managed authentication

Transformation

Custom Connectors

Composite MCP

Unified Schema

Environment Settings

CLI

Connectors

What are custom plugins?

How plugins run

Name the plugin file

Write the `main` function

Arguments

Return value

Context

Read configuration files

Examples

Return the table unchanged

Add a column from the tenant config

Read the tenant config file from snapshots

Use `config.json` and change the primary keys

Upload and manage plugins

In the dashboard

With the API

​What are custom plugins?

​How plugins run

​Name the plugin file

​Write the main function

​Arguments

​Return value

​Context

​Read configuration files

​Examples

​Return the table unchanged

​Add a column from the tenant config

​Read the tenant config file from snapshots

​Use config.json and change the primary keys

​Upload and manage plugins

​In the dashboard

​With the API

What are custom plugins?

How plugins run

Name the plugin file

Write the `main` function

Arguments

Return value

Context

Read configuration files

Examples

Return the table unchanged

Add a column from the tenant config

Read the tenant config file from snapshots

Use `config.json` and change the primary keys

Upload and manage plugins

In the dashboard

With the API