Learn how to create custom taps using the Connector Builder
The Connector Builder is a great way to build taps without writing any code! We highly recommend starting with the walkthrough below:
To open the Connector Builder, head to your Environment Settings -> Connectors -> Build a Connector, as shown below:
From there, click Create New to start building a new tap.
Now we can define our connector’s basic details:
sage-intacct
or chargebee
intacct.com
). This is only used to load a default logoThe first step of configuring authentication is to define the base API URL. This can include variables.
For example, for Shopify we could define the following, where store_name
is a parameter the tenant will configure:
In most cases, the base URL is a static string, such as:
The Connector Builder currently supports the following authentication methods:
For API Key based authentication, you need to specify the following:
{config.api_key}
For example, if I wanted to include my API Key in the x-api-key
header, I’d define the following:
Or, if I wanted to include my API Key in the api_key
query parameter (meaning the request will have ?api_key={api_key}
in the URL), I’d define:
For Basic authentication, you need to define the Username and Password. Following standard Basic auth convention, these
will be concatenated with a colon username:password
and then base64 encoded.
In many cases, Password will be an empty string, in which case you can leave Password blank. See example below:
In the parameters section, you can define the fields tenants will need to fill in when they link this connector.
Any of the parameters you define can be used in other sections (such as the Authentication step) by using the variable syntax. Such as:
For example, if I define the following:
then in the widget tenants will see:
The final thing to configure are the Streams. This is where you define the objects or tables this connector supports.
For my example (Aircall), the stream I’ll be configuring is Calls
.
To start, we’ll define the following:
Calls
calls
/calls
id
GET
or POST
. In almost all cases this will be GET
application/json
In the query parameters section, you can specify fields that should be added to the request URL. This is commonly used for filtering data.
For example, you may need to add a query parameter like ?status=Paid
. This can even reference config parameters if this behavior is configurable
by users:
In the pagination section we define how to page through the API. We support two methods:
Page Increment
Page increment should be used if the API expects some sort of page parameter in the requests, typically in the URL.
To use this, you specify the following:
page
)1
)In the example below we are defining a page
URL parameter, so requests will start from ?page=1
and continue until there is no more data.
Offset
With offset pagination, we expect the API to specify some parameter which we can use to get the next page of data.
To use this, you specify the following:
offset
)next_offset
)limit
)100
)The below example implements a typical offset setup (it’s actually the format Chargebee’s API uses!)
This section is optional, and is used to specify a replication key (a datetime field) that can be used as a filte rin the API request, so we only sync new or updated data since the last job.
To use this you will specify:
updated_at
)updated_at[after]
)timestamp
for UNIX timestamps, or a standard date format like %Y-%m-%d %H:%M:%S
Here’s an example for Aircall, which has a started_at
timestamp in the response data, and expects a filter in the URL like: &from=<timestamp>
First we specify the JSON path to the records within the response. For example, if I have a response payload that looks like:
Then my JSON selector should be:
Finally, we specify the JSON Schema of the records. You can do this using the infer schema tool within hotglue, or manually in JSON.
To use the infer schema tool, you can simply paste a sample JSON record and click infer:
This generates the JSON Schema automatically, which you can then modify as needed. For example, I modified the started_at
field in my
schema to be formatted as a date-time
beause I am using it as the Replication Key:
Sample JSON Schema
That’s it! You’ve now configured a custom tap. From here you can add more streams, or click Publish to start adding it to flows and running jobs.
In some cases streams are nested and require a parent-child relationship to work properly. For example, if you have the following streams:
/clients
/clients/{client_id}/medical-forms
you will need to fetch the medical forms for each client. This is a great use case for parent-child streams.
In the parent stream, we need to define the child_context
– essentially a list of parameters the child streams will have access to in order to make their requests.
In the child stream, we simply need to define the parent_stream
:
Watch the video below for a full walkthrough and demo of using parent-child streams:
Learn how to create custom taps using the Connector Builder
The Connector Builder is a great way to build taps without writing any code! We highly recommend starting with the walkthrough below:
To open the Connector Builder, head to your Environment Settings -> Connectors -> Build a Connector, as shown below:
From there, click Create New to start building a new tap.
Now we can define our connector’s basic details:
sage-intacct
or chargebee
intacct.com
). This is only used to load a default logoThe first step of configuring authentication is to define the base API URL. This can include variables.
For example, for Shopify we could define the following, where store_name
is a parameter the tenant will configure:
In most cases, the base URL is a static string, such as:
The Connector Builder currently supports the following authentication methods:
For API Key based authentication, you need to specify the following:
{config.api_key}
For example, if I wanted to include my API Key in the x-api-key
header, I’d define the following:
Or, if I wanted to include my API Key in the api_key
query parameter (meaning the request will have ?api_key={api_key}
in the URL), I’d define:
For Basic authentication, you need to define the Username and Password. Following standard Basic auth convention, these
will be concatenated with a colon username:password
and then base64 encoded.
In many cases, Password will be an empty string, in which case you can leave Password blank. See example below:
In the parameters section, you can define the fields tenants will need to fill in when they link this connector.
Any of the parameters you define can be used in other sections (such as the Authentication step) by using the variable syntax. Such as:
For example, if I define the following:
then in the widget tenants will see:
The final thing to configure are the Streams. This is where you define the objects or tables this connector supports.
For my example (Aircall), the stream I’ll be configuring is Calls
.
To start, we’ll define the following:
Calls
calls
/calls
id
GET
or POST
. In almost all cases this will be GET
application/json
In the query parameters section, you can specify fields that should be added to the request URL. This is commonly used for filtering data.
For example, you may need to add a query parameter like ?status=Paid
. This can even reference config parameters if this behavior is configurable
by users:
In the pagination section we define how to page through the API. We support two methods:
Page Increment
Page increment should be used if the API expects some sort of page parameter in the requests, typically in the URL.
To use this, you specify the following:
page
)1
)In the example below we are defining a page
URL parameter, so requests will start from ?page=1
and continue until there is no more data.
Offset
With offset pagination, we expect the API to specify some parameter which we can use to get the next page of data.
To use this, you specify the following:
offset
)next_offset
)limit
)100
)The below example implements a typical offset setup (it’s actually the format Chargebee’s API uses!)
This section is optional, and is used to specify a replication key (a datetime field) that can be used as a filte rin the API request, so we only sync new or updated data since the last job.
To use this you will specify:
updated_at
)updated_at[after]
)timestamp
for UNIX timestamps, or a standard date format like %Y-%m-%d %H:%M:%S
Here’s an example for Aircall, which has a started_at
timestamp in the response data, and expects a filter in the URL like: &from=<timestamp>
First we specify the JSON path to the records within the response. For example, if I have a response payload that looks like:
Then my JSON selector should be:
Finally, we specify the JSON Schema of the records. You can do this using the infer schema tool within hotglue, or manually in JSON.
To use the infer schema tool, you can simply paste a sample JSON record and click infer:
This generates the JSON Schema automatically, which you can then modify as needed. For example, I modified the started_at
field in my
schema to be formatted as a date-time
beause I am using it as the Replication Key:
Sample JSON Schema
That’s it! You’ve now configured a custom tap. From here you can add more streams, or click Publish to start adding it to flows and running jobs.
In some cases streams are nested and require a parent-child relationship to work properly. For example, if you have the following streams:
/clients
/clients/{client_id}/medical-forms
you will need to fetch the medical forms for each client. This is a great use case for parent-child streams.
In the parent stream, we need to define the child_context
– essentially a list of parameters the child streams will have access to in order to make their requests.
In the child stream, we simply need to define the parent_stream
:
Watch the video below for a full walkthrough and demo of using parent-child streams: