Red Hat Enterprise Linux 7 Performance Co-Pilot : Overview
Agenda: ➢ Objective of performance monitoring ➢ How is this typically done ? ➢ Introduction to Performance Co-Pilot ➢ Component and Architecture ➢ Useful performance monitoring tools ➢ Demo ➢ Knowledge sources
Objective of performance monitoring ➢ Early detection of a ( potential ) problem. ➢ Rapidly drill down & pinpoint issue in specific program.
How is this typically done ? ➢ System log files ( rsyslog/syslog-ng/journald ) ➢ Native performance monitoring tools ( top/iostat/vmstat/ps ) ➢ Mixture of scripting languages ( bash/perl/python ) ➢ Specific tools vary per platform
Why we need another Performance Monitoring Tool ? ➢ Ability to monitor performance statistics of a local or remote machines in a complex network. ➢ Ability to record and replay performance statistics. ➢ Ability to monitor multiple hosts performance statistics concurrently. ➢ Customization of clients application to collect specific metrics. ➢ Performance regression detection. ➢ Evaluation of effects of an operating system upgrade etc.
Introduction to Performance Co-Pilot ➢ PCP is a system level performance monitoring and performance management toolkit. ➢ PCP provides a range of services that may be used for collection, monitoring and analysis of system metrics. ➢ The Performance Co-Pilot includes many facilities for creating and replaying archive logs that capture performance information. ➢ Cross platform support : Linux, Mac OS and Windows. ➢ End-to-end : Hardware, Core OS, services and applications.
Introduction: Performance Co-Pilot ➢ Distributed architecture Monitoring of local and remote nodes. ➢ Real-time or retrospective Live system or archive. ➢ Pluggable New agents system metrics within PCP.
Performance Co-Pilot - Components Collectors ● Collect and export performance metrics ● Performance Metric Domain Agents (PMDA) ● Performance Metric Collection Daemon (PMCD) Consumers ● Record, visualise, monitor and analyse performance data ● Consume data either in realtime or replay historical data from archive logs Note: Hosts may operate as collectors, consumers or both ● Multiple consumers may connect with one or more collectors
Performance Co-Pilot - Collectors Performance Metric Domain Agents (PMDA) ● Extracts & exports metric data from a system component ● Communicates with pmcd on local system Performance Metric Collection Daemon (PMCD) ● One pmcd process per host ● Coordinates handling of fetch requests between consumer applications and agents ● Listens for connections from localhost & remote clients ● Authenticated & encrypted connection options
Performance Co-Pilot : Consumers ● pmlogger ● Utility to capture and store metrics exported by PMCD ● Concurrent logging of data from local and remote hosts ● Archive playback by other consumer tools (eg pmchart, pmval) ● Tools for log archival, log rotation etc ● Operates on data in realtime only ● pminfo - display PCP metrics available on a host or in an archive ● pmchart - GUI utility providing graphical display of PCP data ● pmstat - vmstat-like utility ● pmatop - top-like utility ● pmie - inference engine and alerting utility ● pmval - display a metric
Performance Co-Pilot - Architecture PMDAs Consumers Kernel pmchart Webserver pmstat PMCD pmlogger DBMS Network Archive C Cli on en su ts mers
Performance Co-Pilot – Distributed Architecture Host 1 PMC M D C A P gen MD ts A's C Cli on en su ts mers Host 2 PMCD Host 4 PMDA's C Cli on en su ts mers Host 3 PMCD PMDA's
Installation: ➢ To install pcp and pcp-gui package, type the following command: # yum install pcp pcp gui “pcp” package provides a framework and services to support system-level performance monitoring and performance management. “pcp-gui” package provides a visualization tools for the Performance Co-Pilot toolkit.
Services: To start pmlogger and pcp services, type the following command: # chkconfig pmlogger on # service pmlogger start Starting pmlogger ... # service pcp start Starting pmcd ... Starting pmlogger ... To check the status of pcp service, type the following command: # service pcp status Checking for pmcd: running Checking for pmlogger: running Note: Bug 1044682 - pcp should use systemd
Verification: pcp - Summary of PCP installation # pcp Performance Co Pilot configuration on localhost.localdomain: platform: Linux localhost.localdomain 3.10.0 121.el7.x86_64 #1 SMP Tue Apr 8 10:48:19 EDT 2014 x86_64 hardware: 2 cpus, 1 disk, 1 node, 1840MB RAM timezone: IST 5:30 pmcd: Version 3.8.10 1, 6 agents pmda: pmcd proc xfs linux mmv jbd2 # pcp h `hostname` Performance Co Pilot configuration on localhost.localdomain: platform: Linux localhost.localdomain 3.10.0 121.el7.x86_64 #1 SMP Tue Apr 8 10:48:19 EDT 2014 x86_64 hardware: 2 cpus, 1 disk, 1 node, 1840MB RAM timezone: IST 5:30 pmcd: Version 3.8.10 1, 6 agents pmda: pmcd proc xfs linux mmv jbd2
Configuration files: Main configuration file for pcp: /etc/pcp.conf Main configuration file for pmcd: /etc/pcp/pmcd/pmcd.conf Default pmlogger config file: /etc/pcp/pmlogger/config.default PCP archive logging configuration/control: /etc/pcp/pmlogger/control Note: pmlogger is utility to capture and store metrics exported by PMCD
Performance monitoring tools: ➢ For all PCP monitoring tools, metrics values may come from a real- time feed (i.e. from pmcd on some host), or from an archive log. ➢ Performance monitoring tools available in Performance Co-Pilot (PCP). pmstat pmatop pmcollectl pmval pmchart pminfo
Performance monitoring tools: pmstat - vmstat-like utility, intended to monitor system performance at the highest level. # pmstat # pmstat t 1 T 3 # pmstat a /var/log/pcp/pmlogger/localhost.localdomain/20140607.20.20.0
Performance monitoring tools: pmatop - top-like utility #pmatop Record Mode: pmatop w rawfile [ interval [ samples ]] Playback Mode: pmatop r [ rawfile ] [ g| m] [ L linelen] [ h host] Eg: # pmatop w atop.out 1 4 # pmatop r atop.out
Performance monitoring tools: pmcollectl - System-level performance monitoring utility that records or displays specific operating system data for one or more sets of subsystems. Basic System level performance monitoring: # pmcollectl Subsystem Specific performance monitoring: # pmcollectl sm # pmcollectl sm –verbose # pmcollectl smcdn –verbose Record Mode: # pmcollectl f perf data c 3 Playback Mode: # pmcollectl p perf data
Performance monitoring tools: pminfo - Display PCP metrics available on a host or in an archive. Full list of all available metrics: # pminfo F Fetch and print values for all or specific metrics: # pminfo f # pminfo f <metric> # pminfo F a /var/log/pcp/pmlogger/localhost.localdomain/20140607.20.56.0 Eg: # pminfo f proc.nprocs disk.dev.read filesys.free Descriptions of all or specific metrics: # pminfo T d # pminfo T d <metric> Eg:
# pminfo T d mem.util.cached # pminfo T d proc.nprocs disk.dev.read filesys.free
Performance monitoring tools: pmval - Dumps the current values for the named performance metrics. # pmval <metric> Eg: # pmval proc.nprocs # pmval kernel.all.load # pmval t 2sec s 4 kernel.percpu.cpu.idle
Performance monitoring tools: pmchart - GUI utility providing graphical display of PCP data