What snapshotting is, and how to use it
snapshots
directory. Any data saved to that directory will be persisted across job runs.
Let’s walk through a quick sample using pandas:
update_snapshot
is designed to take data new data from a job sync, and append it to any existing snapshot. This will ensure every job run has access to the full history of data that has been synced by this tenantget_snapshot
reads the snapshot directory to get the snapshot for the stream
id passed (you can think of a stream
as the name of a table)etl.ipynb
to generate a snapshot of all the Account
data from an integration (like Quickbooks):
config.json
in the snapshot to store some metadata such as an API key programmatically.