Awesome

Adanos-alert

Adanos-alert is an alert notification system developed using Golang. It focuses on notifications, offering rich event aggregation and notification mechanisms, addressing the shortcomings of common monitoring alert systems in terms of notifications.

Features

• Customize the formatting of alarm events using Go template syntax.
  • Provide multiple built-in templates for alarm notification events.
  • Support custom alarm notification templates.
• Grouping of events based on custom expressions.
• Support for ignoring certain events based on custom expressions.
  • Supports a maximum number of ignored events, after which an alarm is issued.
• Support for establishing relationships between events using custom expressions, for easy tracing.
• Aggregate events based on multiple time dimensions.
  • Aggregation at fixed time intervals.
  • Aggregation at multiple fixed points in time.
  • Aggregation at multiple fixed time ranges, with different aggregation periods for each range.
• Support for multiple alarm event delivery methods.
  • Logstash error log output.
  • HTTP API interfaces.
  • Command-line pipelines (using adanos-proxy).
  • Grafana alarm event delivery.
  • Prometheus alarm event delivery.
  • Prometheus Alertmanager alarm event delivery.
  • Openfalcon alarm event delivery.
• Multiple notification channel support.
  • DingTalk group messages.
  • Jira issue creation.
  • Alibaba Cloud voice notifications.
  • HTTP interface calls.
  • Email notifications.
  • WeChat notifications.
  • Alibaba Cloud SMS.
  • NetEase Cloud SMS.
• Implement alarm event reception and dispatching separation through adanos-alert-agent for high availability of event alarm notifications.

Environment

Adanos-alert uses MongoDB as its backend data storage. All data (alert rules, alert events, event groups, etc.) are stored in MongoDB. Therefore, it is necessary to install the MongoDB database, with a recommended version of 4.0 or higher (lower versions have not been tested and may have compatibility issues).

Installation

The following configuration files are available for reference:

• The ./systemd/ directory contains startup commands for the server and agent managed under systemd
• server.config.yaml is an example configuration file for adanos-alert-server
• agent.config.yaml is an example configuration file for adanos-alert-agent
• logstash.example.conf is an example of Logstash outputting error logs to adanos
• prometheus.example.yaml is an example of Prometheus integrating alert information into adanos
• prometheus.rules.example.conf provides examples of Prometheus alert rules

adanos-alert-server command line options (config file)

• conf: configuration file path
• shutdown_timeout: set a shutdown timeout for each service (default: 5s) [$GLACIER_SHUTDOWN_TIMOUT]
• listen: http listen addr (default: ":19999")
• grpc_listen: GRPC Server listen address (default: ":19998")
• grpc_token: GRPC Server token (default: "000000")
• preview_url: Alert preview page url (default: "http://localhost:19999/ui/groups/%s.html") [$ADANOS_PREVIEW_URL]
• report_url: Alert report page url (default: "http://localhost:19999/ui/reports/%s.html") [$ADANOS_REPORT_URL]
• mongo_uri: Mongodb connection uri，refer to https://docs.mongodb.com/manual/reference/connection-string/ (default: "mongodb://localhost:27017") [$MONGODB_HOST]
• mongo_db: Mongodb database name (default: "adanos-alert") [$MONGODB_DB]
• api_token: API Token for api access control [$ADANOS_API_TOKEN]
• use_local_dashboard: whether using local dashboard, this is used when development
• enable_migrate: whether enable database migrate when app run
• re_migrate: whether re-execute migrate, re-migrate will remove existing predefined templates
• aggregation_period: aggregation job execute period (default: "5s") [$ADANOS_AGGREGATION_PERIOD]
• action_trigger_period: action trigger job execute period (default: "5s") [$ADANOS_ACTION_TRIGGER_PERIOD]
• queue_job_max_retry_times: set queue job max retry times (default: 3) [$ADANOS_QUEUE_JOB_MAX_RETRY_TIMES]
• keep_period: how long to keep the alarm. If all are kept, set it to 0. The unit is the day. Adanos-Alert automatically cleans up alarms that exceed the keep_period day (default: 0) [$ADANOS_KEEP_PERIOD]
• syslog_keep_period: how long to keep system logs. If all are kept, set it to 0. The unit is the day. Adanos-Alert automatically cleans up system logs that exceed the syslog_keep_period day (default: 0) [$ADANOS_SYSLOG_KEEP_PERIOD]
• queue_worker_num: set queue worker numbers (default: 3) [$ADANOS_QUEUE_WORKER_NUM]
• query_timeout: query timeout for backend service (default: "30s") [$ADANOS_QUERY_TIMEOUT]
• aliyun_access_key: Aliyun voice notification interface Access Key ID [$ADANOS_ALIYUN_ACCESS_KEY]
• aliyun_access_secret: Aliyun voice notification interface Access Secret [$ADANOS_ALIYUN_ACCESS_SECRET]
• aliyun_voice_called_show_number: Aliyun voice notification called display number
• aliyun_voice_tts_code: Aliyun voice notification template. This is the template ID. The template content is applied for in Aliyun. The recommended content is: "You have an alarm notification named ${title}. Please handle it in time!"
• aliyun_voice_tts_param: Aliyun voice notification template variable name (default: "title")
• log_path: The log file output directory (not the file name). The default is empty, which outputs to the standard output
• jira_url: Jira server address, such as http://127.0.0.1:8080 [$ADANOS_JIRA_URL]
• jira_username: Jira connection account [$ADANOS_JIRA_USERNAME]
• jira_password: Jira connection password [$ADANOS_JIRA_PASSWORD]
• no_job_mode: After this flag is enabled, the event aggregation and queue task processing will be stopped, which is used for development and debugging

adanos-alert-agent CLI options (config file)

• conf: configuration file path
• shutdown_timeout: set a shutdown timeout for each service (default: 5s) [$GLACIER_SHUTDOWN_TIMOUT]
• server_addr: server grpc listen address (default: "127.0.0.1:19998") [$ADANOS_SERVER_ADDR]
• server_token: API Token for grpc api access control (default: "000000") [$ADANOS_SERVER_TOKEN]
• data_dir: local database storage dir (default: "/tmp/adanos-agent")
• listen: listen address (default: "127.0.0.1:29999") [$ADANOS_AGENT_LISTEN_ADDR]
• log_path: log file output dir (not filename), empty for stdout by default

Manual compilation

During compilation, the following tools are required:

• esc is a tool library used to bundle static files into Go programs.

Execute the following command to compile and run directly:

make run

Access the dashboard at http://localhost:19999

Architecture

The relationship between the Adanos Alert platform and other systems

The relationship between the components of the Adanos Alert platform

The internal structure of Adanos Alert Server

Related Projects

• adanos-mail-connector can pretend to be an SMTP server, convert emails into Adanos events and send them to Adanos-alert Server
• Glacier Framework Go language application development framework
• Go-IOC Go language runtime dependency injection framework
• Asteria Structured logging library