Awesome

Description

This project replicates Mysql to Elasticsearch, reading binlog events and converting them to python objects to jsons.

Note that the project does not supoort HTTPS when migrating.

The project is a modified version of py-mysql-elasticsearch-sync, using library named python-mysql-replication.

Give it a try!

make containers by:

docker-compose up

stop the elasticsearch container and modify /usr/share/elasticsearch/config/elasticsearch.yml as:

(edit true to false) xpack.security.enabled: false

(at the end of the file add this) action.destructive_requires_name: false

restart the elasticsearch container, and in the pymyelarepl container,

(for test) cd pymyelarepl && python test/test_basic.py


(after executing the following sql in the mysql container as root without password, for example) cd pymyelarepl && python example/run.py

CREATE DATABASE test;
use test;
CREATE TABLE test4 (id int NOT NULL AUTO_INCREMENT, data VARCHAR(255), data2 VARCHAR(255), PRIMARY KEY(id));
INSERT INTO test4 (data, data2) VALUES ("Hello", "World");
UPDATE test4 SET data="World", data2="Hello" WHERE id=1;
DELETE FROM test4 WHERE id=1;

the results of the two cases are similar to the below, which means data is replicated successfully:

{'errors': False, 'took': 12, 'items': [{'create': {'_index': 'basic_replication', '_id': '1', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 6, '_primary_term': 1, 'status': 201}}, {'create': {'_index': 'basic_replication', '_id': '2', '_version': 1, 'result': 'created', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 7, '_primary_term': 1, 'status': 201}}]}
{'errors': False, 'took': 11, 'items': [{'update': {'_index': 'basic_replication', '_id': '1', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 8, '_primary_term': 1, 'status': 200}}, {'update': {'_index': 'basic_replication', '_id': '2', '_version': 2, 'result': 'updated', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 9, '_primary_term': 1, 'status': 200}}]}
{'errors': False, 'took': 2, 'items': [{'delete': {'_index': 'basic_replication', '_id': '1', '_version': 3, 'result': 'deleted', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 10, '_primary_term': 1, 'status': 200}}, {'delete': {'_index': 'basic_replication', '_id': '2', '_version': 3, 'result': 'deleted', '_shards': {'total': 2, 'successful': 1, 'failed': 0}, '_seq_no': 11, '_primary_term': 1, 'status': 200}}]}

(blocked for the example)

License

MIT