Database
MySQL
Connector Details
Name | Value |
---|---|
Platform | MySQL |
Auth Type | API Keys |
Direction | Bidirectional |
Tap Repo | https://github.com/transferwise/pipelinewise-tap-mysql |
Target Repo | https://github.com/hotgluexyz/target-mysql-v2 |
Tap Metrics | Usage: |
Target Metrics | Usage: |
Target MySQL
Config
target-mysql
requires the standard 5 connection parameters to be specified in the config:
Example ETL Script
Optional config flags
Property | Description | Default |
---|---|---|
table_name_pattern | MySQL table name pattern to use when creating tables | ”${TABLE_NAME}“ |
lower_case_table_names | Use lowercase for table names or not | true |
allow_column_alter | Allow column alterations or not | false |
replace_null | Replace null values with others or not | false |
table_config | Dictionary with specifications on insert/upsert/truncate |
The replace_null
Option (Experimental)
By enabling the replace_null
option, null values are replaced with ‘empty’ equivalents based on their data type. Use with caution as it may alter data semantics.
When replace_null
is true
, null values are replaced as follows:
JSON Schema Data Type | Null Value Replacement |
---|---|
string | Empty string("" ) |
number | 0 |
object | Empty object({} ) |
array | Empty array([] ) |
boolean | false |
null | null |
The table_config
option
By default, target-mysql will upsert your payload into each table. This means it will insert all new data into your table, but perform an update on primary key collision.
The table_config
allows you to override this behavior with one of two insertion methods:
truncate
: Delete the existing table and replace it with the new datainsert
: Attempt to insert all records as new rows, throwing an error on primary key collision
If you want to use the truncate
or insert
methods, you can specify which tables should use the method using the table_config
flag:
Tap Changelog
Tap Changelog
Tap Changelog
Version | Notes |
---|---|
1.6.0 (2024-05-09) | ## What’s Changed * Prefer SSL connection to source by @louis-pie in https://github.com/transferwise/pipelinewise-tap-mysql/pull/181 Full Changelog: https://github.com/transferwise/pipelinewise-tap-mysql/compare/v1.5.6…v1.6.0 |
1.5.6 (2023-08-10) | ## What’s Changed * AP-1531 Zero-pad fixed-length binary fields by @Samira-El in https://github.com/transferwise/pipelinewise-tap-mysql/pull/162 Full Changelog: https://github.com/transferwise/pipelinewise-tap-mysql/compare/v1.5.5…v1.5.6 |
1.5.5 (2023-07-05) | ## What’s Changed * LookupError: unknown encoding: utf8mb3 by @Samira-El in https://github.com/transferwise/pipelinewise-tap-mysql/pull/163 * Bump plpygis from 0.2.0 to 0.2.1 by @dependabot in https://github.com/transferwise/pipelinewise-tap-mysql/pull/142 Full Changelog: https://github.com/transferwise/pipelinewise-tap-mysql/compare/v1.5.4…v1.5.5 |
1.5.4 (2023-05-22) | ## What’s Changed * AP-1477: Bump python-mysql-replication by @Samira-El in https://github.com/transferwise/pipelinewise-tap-mysql/pull/156 Full Changelog: https://github.com/transferwise/pipelinewise-tap-mysql/compare/v1.5.3…v1.5.4 |
1.5.3 (2023-04-25) | * LOG_BASED: Set mariadb slave capability to 4 to mitigate bug in Mariadb 10.6.12 (https://github.com/transferwise/pipelinewise-tap-mysql/pull/149) |
1.5.2 (2022-08-12) | * Bump mysql-replication to 0.30 |
1.5.1 (2022-04-05) | Fix: Handle case when BINLOG_GTID_POS returns multiple comma separated GTIDs |
1.5.0 (2022-03-11) | - Support logical replication using GTID, for both Mariadb & MySql - Log error message when session sqls fail - Bump depenedencies to support Mysql 8 - Migrate CI to Github Actions. |
1.4.3 (2021-04-09) | * Fix in LOG_BASED method: re-discovery constantly running when table has unsupported column type. * Add support for tinytext column type * Bump pendulum to 1.5.1 * Add unit tests and re-arrange tests folder |
1.4.2 (2021-03-15) | Fixing typo introduced by a previous change. |
1.4.1 (2021-03-12) | Fix data loss during log_based replication. |
1.4.0 (2020-11-09) | Support MySQL spatial types |
1.3.8 (2020-10-16) | Fix mapping bit to boolean values |
1.3.7 (2020-09-04) | Fix an issue with converting row to singer record where encountering a time type column causes the override of the whole row. |
1.3.6 (2020-09-02) | Fixed an issue when every extracted row was logged at INFO level and produced huge log files |
1.3.5 (2020-08-27) | Fix issue with time sql type not properly processed.Previously, it used to be turned into a datetime in the form of 1970-01-01 HH:MM:SS , and now it’s formatted to only be HH:MM:SS |
1.3.4 (2020-08-05) | Fix few issues with new discovered schema after changes are detected during LOG_BASED runtime. |
1.3.3 (2020-07-23) | During LOG_BASED runtime, detect new columns, incl renamed ones, by comparing the columns in the binlog event to the stream schema, and if there are any additional columns, run discovery and send a new SCHEMA message to target. This helps avoid data loss. |
1.3.2 (2020-06-15) | - Revert pymysql back to 0.7.11 .pymysql >= 0.8.1 introducing some not expected and not backward compatible changes how it’s dealing withinvalid datetime columns. |
1.3.1 (2020-06-15) | - Fix dependency issue by removing attrs from setup.py - Bump pymysql to 0.9.3 |
1.3.0 (2020-05-18) | - Add optional session_sqls connection parameter- Support JSON column types |
Make logging customizable | |
Update bookmark only if binlog position is valid | |
Update bookmark when reading bookmark finished | |
Update bookmark only before writing state message | |
Handle null bytes in Binary type columns using SQL | |
Handle padding 0s in Binary type columns | |
Support binary and varbinary columns | |
Upgrade mysql-replication package | |
Add license classifier | |
Remove BINARY and VARBINARY support |