Awesome
Syncer: MySQL/MongoDB => Elasticsearch/MySQL/Kafka/HBase
中文文档
Features
- Sync data consistently: make sure data consistency
- Sync data async: little effect on origin server
- Sync data and manipulate: write Java code to customize sync
- Sync data from MySQL/MongoDB to Kafka/ES/MySQL/HBase
- ETL and real-time data sync combined: missing no data, sample config
Use Syncer
Preparation
- MySQL config
- binlog_format: row
- binlog_row_image: full
- MongoDB config:
- (optional) update
bind_ip
to allow listens for connections from applications on configured addresses. - enable replication set:
mongod --replSet myapp
- Or use docker:
docker run -d --name mongodb -p 27017:27017 -v /root/mongodb-container/db:/data/db mongo:3.2 mongod --replSet chat
- init replication set in shell:
rs.initiate()
- (optional) update
Run
git clone https://github.com/zzt93/syncer
cd syncer/ && mvn package
# /path/to/config/: producer.yml, consumer.yml, password-file
# use `-XX:+UseParallelOldGC` if you have less memory and lower input pressure
# use `-XX:+UseG1GC` if you have at least 4g memory and event input rate larger than 2*10^4/s
java -server -XX:+UseG1GC -jar ./syncer-core/target/syncer-core-1.0-SNAPSHOT.jar [--debug] [--port=40000] [--config=/absolute/path/to/syncerConfig.yml] --producerConfig=/absolute/path/to/producer.yml --consumerConfig=/absolute/path/to/consumer1.yml,/absolute/path/to/consumer2.yml
Full and usable sample config can be found under test/config/
, like test/config/simplest
How to ?
If you have any problems with how to use Syncer
or bugs of it, write an issue.
I will handle it as soon as I can.
FAQ
- Q: "Got error produce response in correlation id xxx on topic-partition xxx.xxPartition-0, splitting and retrying (5 attempts left). Error: MESSAGE_TOO_LARGE"?
- A: Adjust message
batch.size
to smaller number or configkafka
to receive large message
- A: Adjust message
Used In Production
- Search system: search data sync
- Micro-service: auth/recommend/chat data sync
- Sync Requirement: low latency, high availability
- Join table: avoid join in production env, use space for speed by joining table
- Sync Requirement: low latency, high availability
- Kafka: sync data to kafka, for other heterogeneous system to use
- For data recovery: In case of drop entity mistakenly, or you know where to start & end
- For alter table sync:
- For data warehouse sync
TODO
Implementation
Implementation detail can be found in doc