Awesome
<div align="center"> <h1 align="center"> <img src="https://raw.githubusercontent.com/PKief/vscode-material-icon-theme/ec559a9f6bfd399b82bb44393651661b08aaf7ba/icons/folder-markdown-open.svg" width="100" /> <br> FlinkFlow </h1> <h3 align="center">๐ Real-time stream processing wiht PyFlink.</h3> <h3 align="center">๐ Developed with the software and tools below.</h3> <p align="center"> <img src="https://img.shields.io/badge/Apache%20Flink-E6526F.svg?style=for-the-badge&logo=Apache-Flink&logoColor=white" alt="py" /> <img src="https://img.shields.io/badge/Apache%20Kafka-231F20.svg?style=for-the-badge&logo=Apache-Kafka&logoColor=white" alt="pyflink" /> <img src="https://img.shields.io/badge/AIOHTTP-2C5BB4.svg?style=for-the-badge&logo=AIOHTTP&logoColor=white" alt="" /> <img src="https://img.shields.io/badge/Python-3776AB.svg?style=for-the-badge&logo=Python&logoColor=white" alt="aioresponses" /> <img src="https://img.shields.io/badge/pandas-150458.svg?style=for-the-badge&logo=pandas&logoColor=white" alt="aiohttp" /> <img src="https://img.shields.io/badge/GNU%20Bash-4EAA25.svg?style=for-the-badge&logo=GNU-Bash&logoColor=white" alt="asyncio" /> <img src="https://img.shields.io/badge/Markdown-000000.svg?style=for-the-badge&logo=Markdown&logoColor=white" alt="pack" /> </p> </div>๐ Table of Contents
- ๐ Table of Contents
- ๐Overview
- ๐ฎ Feautres
- โ๏ธ Project Structure
- ๐ป Modules
- ๐ Getting Started
- ๐ Future Development
- ๐ค Contributing
- ๐ชช License
- ๐ Acknowledgments
๐Overview
FlinkFlow is a repository for building real-time data processing apps with PyFlink.
๐ฎ Feautres
[๐ INSERT-PROJECT-FEATURES]
<img src="https://raw.githubusercontent.com/PKief/vscode-material-icon-theme/ec559a9f6bfd399b82bb44393651661b08aaf7ba/icons/folder-github-open.svg" width="80" />
โ๏ธ Project Structure
.
โโโ README.md
โโโ conf
โย ย โโโ conf.toml
โย ย โโโ flink-config.yaml
โโโ data
โย ย โโโ data.csv
โโโ requirements.txt
โโโ scripts
โย ย โโโ clean.sh
โย ย โโโ run.sh
โโโ setup
โย ย โโโ setup.sh
โโโ setup.py
โโโ src
โโโ alerts_handler.py
โโโ consumer.py
โโโ logger.py
6 directories, 12 files
<img src="https://raw.githubusercontent.com/PKief/vscode-material-icon-theme/ec559a9f6bfd399b82bb44393651661b08aaf7ba/icons/folder-src-open.svg" width="80" />
๐ป Modules
<details closed><summary>Scripts</summary>File | Summary |
---|---|
run.sh | This code is a Bash script that starts a Flink cluster, submits a PyFlink job, and then stops the Flink cluster. |
clean.sh | This code is a Bash script that cleans up files and directories related to Python, Jupyter Notebooks, and pytest. It deletes Python cache files, build artifacts, Jupyter notebook checkpoints, and log files. |
File | Summary |
---|---|
alerts_handler.py | This code is a REST API alert handler for the Flink consumer. It buffers alerts and sends them to the API in batches using aiohttp, and serializes them using Apache Avro. |
logger.py | Logger is a class for the project that provides logging capabilities with colored output and different log levels. |
consumer.py | This code is a Python script that uses Apache Flink to process streaming data. It creates a StreamExecutionEnvironment, sets the parallelism, time characteristic, and checkpointing mode, and creates a StreamTableEnvironment. |
๐ Getting Started
โ Prerequisites
Before you begin, ensure that you have the following prerequisites installed:
[๐ INSERT-PROJECT-PREREQUISITES]
๐ป Installation
- Clone the FlinkFlow repository:
git clone https://github.com/eli64s/FlinkFlow
- Change to the project directory:
cd FlinkFlow
- Install the dependencies:
pip install -r requirements.txt
๐ค Using FlinkFlow
python main.py
๐งช Running Tests
#run tests
<hr />
๐ Future Development
- [๐ COMPLETED-TASK]
- [๐ INSERT-TASK]
- [๐ INSERT-TASK]
๐ค Contributing
Contributions are always welcome! Please follow these steps:
- Fork the project repository. This creates a copy of the project on your account that you can modify without affecting the original project.
- Clone the forked repository to your local machine using a Git client like Git or GitHub Desktop.
- Create a new branch with a descriptive name (e.g.,
new-feature-branch
orbugfix-issue-123
).
git checkout -b new-feature-branch
- Make changes to the project's codebase.
- Commit your changes to your local branch with a clear commit message that explains the changes you've made.
git commit -m 'Implemented new feature.'
- Push your changes to your forked repository on GitHub using the following command
git push origin new-feature-branch
- Create a pull request to the original repository. Open a new pull request to the original project repository. In the pull request, describe the changes you've made and why they're necessary. The project maintainers will review your changes and provide feedback or merge them into the main branch.
๐ชช License
This project is licensed under the [๐ INSERT-LICENSE-TYPE]
License. See the LICENSE file for additional info.
๐ Acknowledgments
[๐ INSERT-DESCRIPTION]