Home

Awesome

PYTSDA-TDengine

Build Status Coverage Status Documentation Status Codacy Badge PyPI version

Official Website: https://www.taosdata.com/cn/

TDengine Version:1.6.5.9 , Any question about TDengine, wechat : 13720014098

PYTSDA-TDengine is an end-to end time series data analysis Python system with TDengine. PYTSDA-TDengine provides algorithms which meet the demands for users in time series data analysis fields, w/wo data science or machine learning background. PYTSDA-TDengine gives the ability to execute machine learning algorithms in-database without moving data out of the database server or over the network. It also provides access to a wide range of time series data analysis algorithms, including statistical analysis and more recent deep learning based approaches.

PYTSDA-TDengine is featured for:

The Full API Reference can be found in handbook.

API Demo:

from utils.import_algorithm import algorithm_selection
from utils.utilities import output_performance,connect_server,query_data

# connect to the database
conn,cursor=connect_server(host, user, password)

# query data from specific time range
data = query_data(database_name,table_name,start_time,end_time)

# train the anomaly detection algorithm
clf = algorithm_selection(algorithm_name)
clf.fit(X_train)

# get outlier result and scores
prediction_result = clf.predict(X_test)
outlierness_score = clf.decision_function(test)

#visualize the prediction_result
visualize_distribution(X_test,prediction_result,outlierness_score)

Quick Start

python demo.py --ground_truth --visualize_distribution

Results are shown as

connect to TDengine success
Load dataset and table
Loading cost: 0.151061 seconds
Load data successful
Start processing:
100%|████████████████████████████████████| 10/10 [00:00<00:00, 14.02it/s]
==============================
Results in Algorithm dagmm are:
accuracy_score: 0.98
precision_score: 0.99
recall_score: 0.99
f1_score: 0.99
roc_auc_score: 0.99
processing time: 15.330137 seconds
==============================
connection is closed

<img src="https://github.com/datamllab/PyODDS/blob/master/output/img/Result.png" width="50%" height="45%">

Installation

To install the package, please use the pip installation as follows:

pip install pyodds
pip install git+git@github.com:datamllab/PyODDS.git

Note: PYTSDA-TDengine is only compatible with Python 3.6 and above.

Required Dependencies

- pandas>=0.25.0
- taos==1.4.15
- tensorflow==2.0.0b1
- numpy>=1.16.4
- seaborn>=0.9.0
- torch>=1.1.0
- luminol==0.4
- tqdm>=4.35.0
- matplotlib>=3.1.1
- scikit_learn>=0.21.3

目前测试的环境:

pandas =1.0.3
tensorflow =2.1.0
numpy=1.18.1
seaborn=0.10.1
torch=1.4.0+cpu
luminol=0.4
hyperopt=0.2.4
tqdm=4.46.0
matplotlib=3.1.3
scikit_learn=0.23.1

To compile and package the JDBC driver source code, you should have a Java jdk-8 or higher and Apache Maven 2.7 or higher installed. To install openjdk-8 on Ubuntu:

sudo apt-get install openjdk-8-jdk

To install Apache Maven on Ubuntu:

sudo apt-get install maven

To install the TDengine as the back-end database service, please refer to this instruction.

To enable the Python client APIs for TDengine, please follow this handbook.

To insure the locale in config file is valid:

sudo locale-gen "en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
locale

To start the service after installation, in a terminal, use:

taosd

Implemented Algorithms

Statistical Based Methods

MethodsAlgorithmClass API
CBLOFClustering-Based Local Outlier Factor:class:algo.cblof.CBLOF
HBOSHistogram-based Outlier Score:class:algo.hbos.HBOS
IFORESTIsolation Forest:class:algo.iforest.IFOREST
KNNk-Nearest Neighbors:class:algo.knn.KNN
LOFLocal Outlier Factor:class:algo.cblof.CBLOF
OCSVMOne-Class Support Vector Machines:class:algo.ocsvm.OCSVM
PCAPrincipal Component Analysis:class:algo.pca.PCA
RobustCovarianceRobust Covariance:class:algo.robustcovariance.RCOV
SODSubspace Outlier Detection:class:algo.sod.SOD

Deep Learning Based Methods

MethodsAlgorithmClass API
autoencoderOutlier detection using replicator neural networks:class:algo.autoencoder.AUTOENCODER
dagmmDeep autoencoding gaussian mixture model for unsupervised anomaly detection:class:algo.dagmm.DAGMM

Time Serie Methods

MethodsAlgorithmClass API
lstmadLong short term memory networks for anomaly detection in time series:class:algo.lstm_ad.LSTMAD
lstmencdecLSTM-based encoder-decoder for multi-sensor anomaly detection:class:algo.lstm_enc_dec_axl.LSTMED
luminolLinkedin's luminol:class:algo.luminol.LUMINOL

APIs Cheatsheet

The Full API Reference can be found in handbook.

License

<!-- Biblatex entry: -->

You may use this software under the MIT License.

Environment configuration

<!-- Biblatex entry: -->

Linux:TDengine Server 1.6.5.9

Windows:TDengine 1.5.5.9 client & Python Connector https://www.taosdata.com/cn/documentation/connector/#Python-Connector and follows:

1 anaconda 安装 清华Anaconda镜像地址:https://mirrors.tuna.tsinghua.edu.cn/help/anaconda/

2 全部装上吧 https://github.com/Shawshank-Smile/pyodds pandas =1.0.3 tensorflow =2.1.0 numpy=1.18.1 seaborn=0.10.1 torch=1.4.0+cpu luminol=0.4 hyperopt=0.2.4 tqdm=4.46.0 matplotlib=3.1.3 scikit_learn=0.23.1

3 https://blog.csdn.net/ANNILingMo/article/details/88032599 https://www.cnblogs.com/andrew-address/p/12733669.html (这个是可以的) Torch 没有安装成功,先不搞了。如果需要安装,参考以上的链接吧。

4 安装vscode

5 vscode 使用anaconda的环境 https://blog.csdn.net/m0_45161766/article/details/105729025 https://www.cnblogs.com/lataku/p/10743257.html

6 vscode 配置 git https://www.cnblogs.com/ostrich-sunshine/p/11329444.html

7 Git安装教程(windows) https://www.cnblogs.com/wj-1314/p/7993819.html

8 git配置 https://www.cnblogs.com/lucy-xyy/p/11733317.html https://www.cnblogs.com/ashidamana/p/6122619.html https://www.cnblogs.com/wj-1314/p/7992543.html