Name | Value |
---|---|
Platform | MySQL |
Auth Type | API Keys |
Direction | Bidirectional |
Tap Repo | https://github.com/transferwise/pipelinewise-tap-mysql |
Target Repo | https://github.com/hotgluexyz/target-mysql-v2 |
Tap Metrics | Usage: |
Target Metrics | Usage: |
target-mysql
requires the standard 5 connection parameters to be specified in the config:
Property | Description | Default |
---|---|---|
table_name_pattern | MySQL table name pattern to use when creating tables | ”${TABLE_NAME}“ |
lower_case_table_names | Use lowercase for table names or not | true |
allow_column_alter | Allow column alterations or not | false |
replace_null | Replace null values with others or not | false |
table_config | Dictionary with specifications on insert/upsert/truncate |
replace_null
Option (Experimental)replace_null
option, null values are replaced with ‘empty’ equivalents based on their data type. Use with caution as it may alter data semantics.
When replace_null
is true
, null values are replaced as follows:
JSON Schema Data Type | Null Value Replacement |
---|---|
string | Empty string("" ) |
number | 0 |
object | Empty object({} ) |
array | Empty array([] ) |
boolean | false |
null | null |
table_config
optiontable_config
allows you to override this behavior with one of two insertion methods:
truncate
: Delete the existing table and replace it with the new datainsert
: Attempt to insert all records as new rows, throwing an error on primary key collisiontruncate
or insert
methods, you can specify which tables should use the method using the table_config
flag:
Tap Changelog
Version | Notes |
---|---|
1.6.0 (2024-05-09) | ## What’s Changed * Prefer SSL connection to source by @louis-pie in https://github.com/transferwise/pipelinewise-tap-mysql/pull/181 Full Changelog: https://github.com/transferwise/pipelinewise-tap-mysql/compare/v1.5.6…v1.6.0 |
1.5.6 (2023-08-10) | ## What’s Changed * AP-1531 Zero-pad fixed-length binary fields by @Samira-El in https://github.com/transferwise/pipelinewise-tap-mysql/pull/162 Full Changelog: https://github.com/transferwise/pipelinewise-tap-mysql/compare/v1.5.5…v1.5.6 |
1.5.5 (2023-07-05) | ## What’s Changed * LookupError: unknown encoding: utf8mb3 by @Samira-El in https://github.com/transferwise/pipelinewise-tap-mysql/pull/163 * Bump plpygis from 0.2.0 to 0.2.1 by @dependabot in https://github.com/transferwise/pipelinewise-tap-mysql/pull/142 Full Changelog: https://github.com/transferwise/pipelinewise-tap-mysql/compare/v1.5.4…v1.5.5 |
1.5.4 (2023-05-22) | ## What’s Changed * AP-1477: Bump python-mysql-replication by @Samira-El in https://github.com/transferwise/pipelinewise-tap-mysql/pull/156 Full Changelog: https://github.com/transferwise/pipelinewise-tap-mysql/compare/v1.5.3…v1.5.4 |
1.5.3 (2023-04-25) | * LOG_BASED: Set mariadb slave capability to 4 to mitigate bug in Mariadb 10.6.12 (https://github.com/transferwise/pipelinewise-tap-mysql/pull/149) |
1.5.2 (2022-08-12) | * Bump mysql-replication to 0.30 |
1.5.1 (2022-04-05) | Fix: Handle case when BINLOG_GTID_POS returns multiple comma separated GTIDs |
1.5.0 (2022-03-11) | - Support logical replication using GTID, for both Mariadb & MySql - Log error message when session sqls fail - Bump depenedencies to support Mysql 8 - Migrate CI to Github Actions. |
1.4.3 (2021-04-09) | * Fix in LOG_BASED method: re-discovery constantly running when table has unsupported column type. * Add support for tinytext column type * Bump pendulum to 1.5.1 * Add unit tests and re-arrange tests folder |
1.4.2 (2021-03-15) | Fixing typo introduced by a previous change. |
1.4.1 (2021-03-12) | Fix data loss during log_based replication. |
1.4.0 (2020-11-09) | Support MySQL spatial types |
1.3.8 (2020-10-16) | Fix mapping bit to boolean values |
1.3.7 (2020-09-04) | Fix an issue with converting row to singer record where encountering a time type column causes the override of the whole row. |
1.3.6 (2020-09-02) | Fixed an issue when every extracted row was logged at INFO level and produced huge log files |
1.3.5 (2020-08-27) | Fix issue with time sql type not properly processed.Previously, it used to be turned into a datetime in the form of 1970-01-01 HH:MM:SS , and now it’s formatted to only be HH:MM:SS |
1.3.4 (2020-08-05) | Fix few issues with new discovered schema after changes are detected during LOG_BASED runtime. |
1.3.3 (2020-07-23) | During LOG_BASED runtime, detect new columns, incl renamed ones, by comparing the columns in the binlog event to the stream schema, and if there are any additional columns, run discovery and send a new SCHEMA message to target. This helps avoid data loss. |
1.3.2 (2020-06-15) | - Revert pymysql back to 0.7.11 .pymysql >= 0.8.1 introducing some not expected and not backward compatible changes how it’s dealing withinvalid datetime columns. |
1.3.1 (2020-06-15) | - Fix dependency issue by removing attrs from setup.py - Bump pymysql to 0.9.3 |
1.3.0 (2020-05-18) | - Add optional session_sqls connection parameter- Support JSON column types |
Make logging customizable | |
Update bookmark only if binlog position is valid | |
Update bookmark when reading bookmark finished | |
Update bookmark only before writing state message | |
Handle null bytes in Binary type columns using SQL | |
Handle padding 0s in Binary type columns | |
Support binary and varbinary columns | |
Upgrade mysql-replication package | |
Add license classifier | |
Remove BINARY and VARBINARY support |