– Open-source system for monitoring and alerting. Also allow to analyse application and infrastructure with metrics discovered from targets.
– Use very simple and flexible data query language is PromSQL. PromSQL is read-only query language mean only get date from database and cannot write data back to database.
– Prometheus not store data on cluster storage only use single server for storing database on memory and local disk.
– Default library for get metric from Linux,Window,MySQL and some popular services. To monitor custom service/application can add Prometheus client library to your code support Go, Java, Python, Ruby,…
– Can adjust retention for keeping metric on database by using retention.time (number of days to store the metrics by default to 15d) and retention.size (limit of size to be used).
– Generate alert by rule and send to Alert Manager to handles with some actions as deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, Slack, or OpsGeni.
– Target is defination of an object for scrapping metric (object need to be monitored).
– Job and isntance: endpoint you can scrape is called an instance, usually corresponding to a single process. A collection of instances with the same purpose, a process replicated for scalability or reliability for example, is called a job. Example: Job iclude all api server running same purpose.
*Architecture of prometheus:
– Prometheus server pull metric over HTTP from target system (target system not push metric to prometheus server).
– Prometheus Pushgateway using in case some short-lived jobs push time series metric when exit and Prometheuse server will pull metric from Pushgateway.
– Prometheus using Service Discovery for dynamic locating target instead of using hard code configuration target. Example when adding more node to K8s cluster , prometheuse automatically discovery new target and start scraping it.
– Using PromQL on Prometheus Web UI can request raw data stored from database and support some simple graph and dashboard for metrics.
– Integrated with Grafana to create more nice and flexible dashboard for montioring metric.
– Alert Manager handles metric with some actions as deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, Slack, or OpsGeni.
1/Prepare Prometheus VM:
– Install VM using Ubuntu 20.4 LTS
– IP address: 192.168.129.19/24
– Change timezone correctly for VM
– Download prometheus at link: https://prometheus.io/download/
– Extract source on prometheus server
– Rename source prometheus folder to simple name (Example prometheus)
3/Overview about Prometheus UI:
– Run prometheus binary file
– Using web browser access Prometheus Web UI on port 9090
Using Expression for input PromQL query to get data form time series database
Example get metric name for total http request to prometheus server with different URL (value of key handler)
– Using tab grap for visualize metric as graph (Check use local time for displaying graph with local prometheus server instead of using UTC)
4/Understand Prometheus Configuration File:
– Location of prometheus configuration file: <path source>/ prometheus.yml
– This configruation file define anything about time for scraping job, target and instances,rule to load,…
+ scrape_interval: control how often prometheus will scrape metric from target. Default this configuration define on global section will apply for all target. If want setup interval different for each target can override global value by defining this parameter in scrape_configs section.
+ evaluation_interval: how often prometheus will evaluate rule from rule files.
– Default not setup anyting for push alerting to AlertManager
– Rules file block for path of any file using for definition rule load to prometheus server. These rule used for create time series and generate alert.
– scrape_configs block contain information for all targets need to be scrape. Default prometheus expose metric of itself at HTTP endpoint port 9090 (define static instance localhost:9090)
– Prometheus stores its on-disk time series data under the directory specified by the flag storage.local.path. The default path is ./data (relative to the working directory). Can change location of time series database by using option storage.local.path when running prometheus server. Example: # prometheus -storage.local.path /var/lib/prometheus/
– Exportes is a software installed on servers (Linux,Window) for exporting existing metrics in same format as Prometheus metrics.
– Support many exporters for converting metric for many application/database as HAProxy, MySQL, ElasticSearch,MongoDB…or hardware reporter like disk usage, Window exporter.
2/Installing Node Exporter For Monitoring Linux System:
– Node Exporter expose kernel-level and machine-level metric on Unix system such as CPU,Memory, disk I/O, network bandwitch,…).
– Download Node Exporter on Prometheus.io page: https://prometheus.io/download/#node_exporter
– Upload and extract on Linux machine need expose metric to Prometheus server
– Running node exporter binary file
– Default node exporters will expose metric at HTTP endpoint port 9100
– Add Linux machine as new target to prometheus server: Define new job_name in block scrape_configs of prometheus.yml file
– Restart Prometheus Server for reload new configurations
– Check target menu on Prometheus Web UI already display new target Linux machine using Node Exporter with status is Up
– Using simple PromQL query number inode free metric expose from Linux machine
– Using graph to visualize free memory on this machien over the time
3/Installing WMI Exporter For Monitoring Window System:
– WMI Export be used to expose hardware and OS metric fro Window machine (WMI mean as Window Management Instrumentation).
– Download WMI Exporter at link: https://github.com/prometheus-community/windows_exporter
– Upload WMI Export installation file to Window machine
– Click running exporter will expose metric on port 9182
– Add Window machine as new target to prometheus server: Define new job_name in block scrape_configs of prometheus.yml file
– Restart Prometheus Server for reload new configurations
– Check target menu on Prometheus Web UI already display new target Window machine using WMI Exporter with status is Up
– Query free disk space of Window machine using PromQL