Home

Awesome

<div align="center"> <h1 align="center"> <img src="https://raw.githubusercontent.com/PKief/vscode-material-icon-theme/ec559a9f6bfd399b82bb44393651661b08aaf7ba/icons/folder-markdown-open.svg" width="100" /> <br> FlinkFlow </h1> <h3 align="center">๐Ÿ“ Real-time stream processing wiht PyFlink.</h3> <h3 align="center">๐Ÿš€ Developed with the software and tools below.</h3> <p align="center"> <img src="https://img.shields.io/badge/Apache%20Flink-E6526F.svg?style=for-the-badge&logo=Apache-Flink&logoColor=white" alt="py" /> <img src="https://img.shields.io/badge/Apache%20Kafka-231F20.svg?style=for-the-badge&logo=Apache-Kafka&logoColor=white" alt="pyflink" /> <img src="https://img.shields.io/badge/AIOHTTP-2C5BB4.svg?style=for-the-badge&logo=AIOHTTP&logoColor=white" alt="" /> <img src="https://img.shields.io/badge/Python-3776AB.svg?style=for-the-badge&logo=Python&logoColor=white" alt="aioresponses" /> <img src="https://img.shields.io/badge/pandas-150458.svg?style=for-the-badge&logo=pandas&logoColor=white" alt="aiohttp" /> <img src="https://img.shields.io/badge/GNU%20Bash-4EAA25.svg?style=for-the-badge&logo=GNU-Bash&logoColor=white" alt="asyncio" /> <img src="https://img.shields.io/badge/Markdown-000000.svg?style=for-the-badge&logo=Markdown&logoColor=white" alt="pack" /> </p> </div>

๐Ÿ“š Table of Contents


๐Ÿ“Overview

FlinkFlow is a repository for building real-time data processing apps with PyFlink.

๐Ÿ”ฎ Feautres

[๐Ÿ“Œ INSERT-PROJECT-FEATURES]


<img src="https://raw.githubusercontent.com/PKief/vscode-material-icon-theme/ec559a9f6bfd399b82bb44393651661b08aaf7ba/icons/folder-github-open.svg" width="80" />

โš™๏ธ Project Structure

.
โ”œโ”€โ”€ README.md
โ”œโ”€โ”€ conf
โ”‚ย ย  โ”œโ”€โ”€ conf.toml
โ”‚ย ย  โ””โ”€โ”€ flink-config.yaml
โ”œโ”€โ”€ data
โ”‚ย ย  โ””โ”€โ”€ data.csv
โ”œโ”€โ”€ requirements.txt
โ”œโ”€โ”€ scripts
โ”‚ย ย  โ”œโ”€โ”€ clean.sh
โ”‚ย ย  โ””โ”€โ”€ run.sh
โ”œโ”€โ”€ setup
โ”‚ย ย  โ””โ”€โ”€ setup.sh
โ”œโ”€โ”€ setup.py
โ””โ”€โ”€ src
    โ”œโ”€โ”€ alerts_handler.py
    โ”œโ”€โ”€ consumer.py
    โ””โ”€โ”€ logger.py

6 directories, 12 files

<img src="https://raw.githubusercontent.com/PKief/vscode-material-icon-theme/ec559a9f6bfd399b82bb44393651661b08aaf7ba/icons/folder-src-open.svg" width="80" />

๐Ÿ’ป Modules

<details closed><summary>Scripts</summary>
FileSummary
run.shThis code is a Bash script that starts a Flink cluster, submits a PyFlink job, and then stops the Flink cluster.
clean.shThis code is a Bash script that cleans up files and directories related to Python, Jupyter Notebooks, and pytest. It deletes Python cache files, build artifacts, Jupyter notebook checkpoints, and log files.
</details> <details closed><summary>Src</summary>
FileSummary
alerts_handler.pyThis code is a REST API alert handler for the Flink consumer. It buffers alerts and sends them to the API in batches using aiohttp, and serializes them using Apache Avro.
logger.pyLogger is a class for the project that provides logging capabilities with colored output and different log levels.
consumer.pyThis code is a Python script that uses Apache Flink to process streaming data. It creates a StreamExecutionEnvironment, sets the parallelism, time characteristic, and checkpointing mode, and creates a StreamTableEnvironment.
</details> <hr />

๐Ÿš€ Getting Started

โœ… Prerequisites

Before you begin, ensure that you have the following prerequisites installed:

[๐Ÿ“Œ INSERT-PROJECT-PREREQUISITES]

๐Ÿ’ป Installation

  1. Clone the FlinkFlow repository:
git clone https://github.com/eli64s/FlinkFlow
  1. Change to the project directory:
cd FlinkFlow
  1. Install the dependencies:
pip install -r requirements.txt

๐Ÿค– Using FlinkFlow

python main.py

๐Ÿงช Running Tests

#run tests
<hr />

๐Ÿ›  Future Development


๐Ÿค Contributing

Contributions are always welcome! Please follow these steps:

  1. Fork the project repository. This creates a copy of the project on your account that you can modify without affecting the original project.
  2. Clone the forked repository to your local machine using a Git client like Git or GitHub Desktop.
  3. Create a new branch with a descriptive name (e.g., new-feature-branch or bugfix-issue-123).
git checkout -b new-feature-branch
  1. Make changes to the project's codebase.
  2. Commit your changes to your local branch with a clear commit message that explains the changes you've made.
git commit -m 'Implemented new feature.'
  1. Push your changes to your forked repository on GitHub using the following command
git push origin new-feature-branch
  1. Create a pull request to the original repository. Open a new pull request to the original project repository. In the pull request, describe the changes you've made and why they're necessary. The project maintainers will review your changes and provide feedback or merge them into the main branch.

๐Ÿชช License

This project is licensed under the [๐Ÿ“Œ INSERT-LICENSE-TYPE] License. See the LICENSE file for additional info.


๐Ÿ™ Acknowledgments

[๐Ÿ“Œ INSERT-DESCRIPTION]