etl.py
file or a etl.ipynb
.
If you are developing your transformation script as a JupyterLab script, you can develop this script directly in the hotglue UI:
Start the JupyterLab workspace
To edit the transformation script for a connector, you can start a JupyterLab workspace directly from hotglue. Start by opening the settings for the connector you’d like to update. From here, launch the JupyterLab workspace by selecting the Python icon:Generally, when updating the transformation script you should launch the JupyterLab workspace from the admin view – not the tenant view. Launching JupyterLab as a tenant create a custom forked script for that tenant.

Connector configuration screen

Connect to JupyterLab workspace

JupyterLab
Note that JupyterLab workspaces will timeout after 30 minutes of inactivity, at which point you’ll need to start a new workspace. When your session times out, your script is autosaved.
Define dependencies
Therequirements.txt
file in the root directory is where you can specify any Python modules you wish to use as dependencies. In this example, my requirements.txt
contains the following:
requirements.txt
requirements.txt
we created in Jupyter below:

Edit the script
Inside theetl
folder, you will find a file titled etl.ipynb
containing a default transformation script

Default transformation script
gluestick
package and manipulate it using pandas
. You can also find more sample scripts available on GitHub.
Deploy the script
Once you have written your transformation script, you must deploy it using the hotglue tab in Jupyter.
Deploy the transformation script

Confirm transformation script deployment
cached
folder.

Note that the CLI only downloads the deployed version of the script. If you have a script that is not deployed, you will need to load a Jupyter notebook to deploy your cached edits