Debugging a script

Before you start

This guide assumes you have already launched a JupyterLab workspace, as described in the Transformations Overview.

Access prior job logs and data

If you're debugging why a specific job failed, you can learn more about what caused a job to fail by reading the logs.

To start, navigate to the job that failed (this could be under a specific tenant):

By clicking the download icon next to the job, a zip file will be generated containing the raw data that was was generated during that job. The directory structure is exactly the same as described on the transformation script docs.

By clicking the job name, you can access the logs and understand any errors that may have occurred:

Clone a job

While in a JupyterLab workspace, you can clone the data from a prior job. This is especially useful if you're debugging a specific job that failed, or you need some testing data to develop a script.

Inside of Jupyter, select the hotglue tab and press clone job.

Jupyter will present you with a list of the most recent jobs. Press select on the one you would like to clone.

Once the data is cloned, you'll see a success message as below, and the data will be populated in the sync-output folder.

You can preview the data that was cloned in JupyterLab by opening the sync-output folder. As you can see, my subscribe_list data was cloned:

Test the script

Now that your data is cloned, you can run your transform script normally and it should pick up the data automatically.

If your script fails, you will see the error in the notebook (etl.ipynb). From there, you can change your script and run against the data again.

When your script works correctly, you should see the output files in the etl-output folder:

That's all there is to testing your script in hotglue!