Learn how to build custom targets using the Singer SDK
cookiecutter
Python package via pip
destination_name
with the target name (capitalized), the author name, select “Per record” as the serialization method, and choose the authorization method (OAuth2 or Bearer token).
target.py
define all required values in the config file, such as username, password, api_key, client_id, etc
client.py
contains all the common functions and properties, such as the api base url and the authentication logic within a base sink.
You can overwrite any of the prebuilt functions here and they will be applied to all sinks
The base sink should be a child class from either of these Hotglue sinks:
HotglueSink
HotglueBatchSink
base_url
: The base url of the APIhttp_headers
: Any header that should be sent when making the request.MAX_SIZE_DEFAULT
: When making a target with per batch serialization methos, this attribute defines the amount of records that will be processed and passed into the make_batch_request
function.auth_headers
: Inside we can validate if the token is valid, call the function to request a new token and structure the access token. It should return a dict with the Authorization key and the access_token to be sent in the requests made to the API.
oauth_request_body
: It builds the payload to be sent when making the request to obtain a new access token.
is_token_valid
: It validates if the token is still valid or not to be used. It should return a boolean.
update_access_token
: It makes a request to generate a new access token.sinks.py
target-hotglue
sdk, in most cases only 2 functions logic need to be defined.
preprocess_record
: This functions reads each record from the data.singer
and builds the payload that will be sent to the API. Here is where data can be mapped or customized logic to build the payload can be added.upsert_record
: It receives the payload returned by preprocess_record
function and it sends it to the API.target-state.json
commonly an empty dict.process_record
function won’t stop the target, the error will be written in the target-state.json
file together with the externalId
provided in the data.singer
, and the target will continue and read the next record.
preprocess_record
to the ContactsSink class, this will take the input data and transform it into the payload we pass to the endpoint
upsert_record
to the ContactsSink class, here the payload created from preprocess_record
will be sent to the api
upsert_record
.
It holds the id of the newly created or updated records, or if an error ocurred it contains the reason of the error and the externalId
data to recognize which record has failed.
It stores the summary which is a count of 4 types of records:
upsert_record
function.externalId
to determine this.upsert_record
.jsonl
(JSON lines) input, we typically call it data.singer
which should be Singer formatted records.
Note: Each record should have an externalId
field, this will serve as a primary key to update the target-state if errors happen.
Example:
launch.json
by setting a definition that looks like this: