MySQL 5.6: InnoDB NoSQL Key Value Access to InnoDB Same app can leverage: Key-value access to InnoDB via familiar Memcached API SQL for rich queries, JOINs, FKs, etc. Fully transactional Up to 9x performance boost for updates Great for fast data ingestion in Big Data pipeline
Defining Performance • Simple word but many meanings • Main objective: • Users (direct or indirect) should be satisfied • Most typical performance metrics • Throughput • Latency / Response time • Scalability • Combined metrics transactions/time Starvation
Queuing Theory • Multi User applications • Request waits in queue before being processed • User response time = queueing delay + service time • Non high tech example – support cal center. • “Hockey Stick” - queuing delay grows rapidly when system R getting close to saturation es • p Need to improve queueing delay or service time on to improve performance se T • Improving service time reduces queuing delay im e Connections
Service Time: Key to the hotspot • Main Question – where does service time comes from ? • network, cpu, disk, locks... • Direct Measurements • Sum up al query times from web pages • Indirect measurements • CPU usage • Disk IO latency • Network traffic • Load Average • Number of running queries • etc.
Benchmark Tests • Great tool to: • Quantify application performance • Measure performance effect of the changes • Validate Scalability • Plan deployment • But • Can be very misleading if done wrong • Need to be able to read results correctly • Typical Errors • Testing with 1GB size with 100G in production • Using uniform distribution • “Harry Potter” ordered as frequent as Zulu dictionary • Testing in single user scenario
Business side of optimization • Performance costs money, whatever road you take • Investigate different possibilities • Better hardware could be cheaper than a major rewrite • How much performance/scalability/reliability do you need ? • 99.999% could be a lot more expensive than 99.9% • Is peak traffic requirements 100x average or just 3x ? • Take a look at whole picture • Is this the largest risk/bottleneck ? • Identify which optimizations are critical for business • Optimization of “everything” is often waste of resources • What is the cost of suboptimal performance ?
Basic1: Checking Server Configurations • The MySQL server is controlled by “System Variables” • Set via: • Option File: my.cnf / my.ini • Temporary change: SET [GLOBAL] <vriable>=<value> • Can be per connection (LOCAL) or server wide(GLOBAL)
MySQL Server Architecture Client1 Client2 ClientN MySQL Server Connection Thread Pool Query Cache InnoDB Storage Engines Parser Query 101101 MyISAM MERGE MEMORY Optimizer ARCHIVE
Server Connections & Threads • max_connections (151) • number of connections server will Client1 Client2 ClientN allow. May run out of memory if too high, because of per connections Connection Thread Pool memory usage • thread_cache_size (8) mysql> show status; • • Keep up to this amount of threads Max_used_connections • check if it matches “cached” after disconnect max_connections, too low value or • Typical setting sign of overload • Threads_created max_connections/3 • thread_cache misses • should be low.
Connection Thread Work Buffers • sort_buffer_size (2M) • Memory to allocate for sort. Wil use Client1 Client2 ClientN disk based sort for larger data sets. Often fine at 512K or 1M • other buffers, read, read_rnd, Connection Thread Pool etc… smal er defaults often OK • You can change dynamically if mysql> show status; • Sort_merge_passes - large sort operation is needed • number of passes made in batch operation etc during file merge sort. • check if file sort needs to be done at all • use index if possible
Server Query Cache • query_cache_size (0) • Amount of memory to use for query cache • Typically 32M is fine, some databases Connection Thread Pool need 128M • query_cache_type (ON) Query Cache • Worst case performance overhead is about 15%-20% Parser Query 101101 • Favors servers with higher SELECT/WRITE ratios mysql> show status; • Best Practice • Qcache_hits, Qcache_inserts • Set to DEMAND • hits/inserts cache hit ratio, if small, maybe • Add SQL_CACHE to appropriate queries disable query cache • Qcache_lowmem_prunes • times older queries were removed due to low memory, need to increase query_cache_size if high
InnoDB Performance Tips • innodb_buffer_pool_size • 80% of memory on Innodb only system InnoDB • caches data & indexes unlike MyISAM Storage Engines MyISAM • innodb_log_file_size MERGE • A key parameter for write performance MEMORY • Recovery time is no more an issue. ARCHIVE • Bigger is better for write QPS stability • innodb_flush_log_at_trx_commit mysql> SHOW ENGINE • 1 (slow) will flush (fsync) log at each commit. Truly ACID INNODB STATUS; • 2 (fast) will only flush log buffer to OS cache on commit, sync to disk once/sec. Great way to see what is • 0 (fastest) will flush (fsync) log every second or so going on inside InnoDB, • innodb_file_per_table hard to parse • Always good choice to distribute i/o • File IO • Default ON from 5.6 • Buffer Pool • Log activity • Row activity
InnoDB Performance Tips (Cont.) • innodb_flush_method = O_DIRECT InnoDB • Not to consume OS cache Storage Engines MyISAM • innodb_buffer_pool_instances (5.5+) MERGE • To avoid mutex contention MEMORY • 2 or more in can ARCHIVE • innodb_io_capacity (5.5+) mysql> SHOW ENGINE • Enlarge if you have fast disks INNODB STATUS; • Default 200 is good for 2 disks striped Great way to see what is • innodb_read_io_threads (5.5+) going on inside InnoDB, • innodb_write_io_threads (5.5+) hard to parse • • File IO Enlarge if you have fast disks • Buffer Pool • Default 4 is usually good enough • Log activity • Row activity
Semi-in-memory Database with InnoDB DBT-2 (W200) Transactions per Minute %user %iowait Buffer pool 1G 1125.44 2% 30% Buffer pool 2G 1863.19 3% 28% Buffer pool 5G 4385.18 5.5% 33% Buffer pool 30G (All data in cache) 36784.76 36% 8% • DBT-2 benchmark (write intensive) • 20-25GB hot data (200 warehouses, running 1 hour) • Nehalem 2.93GHz x 8 cores, MySQL 5.5.2, 4 RAID1+0 HDDs • RAM size affects everything. Not only for SELECT, but also for INSERT/UPDATE/DELETE • INSERT: Random reads/writes happen when inserting into indexes in random order • UPDATE/DELETE: Random reads/writes happen when modifying records
Schema Design • Normalization • Data types • good for OLTP, writes – use tinyint, smallint, mediumint, • data redundancies eliminated save space! • join performance penalty – join columns same data type • smal er total data set – • E/R diagram clean translation varchar(64) instead of char(64) – declare NOT NULL if true • Denormalization – varchar(64) instead of varchar(255) • good for OLAP, reporting – INT not DECIMAL(9) • data redundancies across tables for better indexing • Indexing • maybe eliminate joins – multi-column
– ordered BTREE – index prefixes – covering
Monitoring Queries - Slow Query Log #Time: 08073101 16:25:24 #User@Host: root[root] @ localhost [127.0.0.1] #Query_time: 8 Lock_time: 0 Rows_sent: 20 Rows_examined: 243661 SELECT part_num FROM `inventory`.`parts` WHERE (`ven` = "foo") ORDER BY `delivery_datetime` DESC LIMIT 100; Pros • Logs queries that took longer than X (user defined) • Logs queries that do not use indexes (5.0 and higher) • Includes data needed to trace offending queries Cons • Requires MySQL restart (5.0 and lower) • Growth must be managed using FLUSH LOGS • Entries must be parsed/sorted for relevance • mysqldumpslow helps, but stil tedious, takes time
Monitoring Queries – SHOW PROCESSLIST; Pros • Shows current processes mysql> SHOW FULL PROCESSLIST\G • Shows status of executing ******** 1. row ***************** queries Id: 1 • Includes data needed to trace User: MyUser offending queries Host: localhost db: inventory Con Command: Query Scripting needed to: Time: 1030455 State: Sending Data • automate, Info: SELECT part_num from ‘inv’; • integrate with Slow Query Log, ….. • Aggregate/parse results for 2 rows in set (0.00 sec) analysis, • notify DBA of problem
Fixing Problem Queries – EXPLAIN; EXPLAIN SELECT part_num FROM `inventory`.`parts` WHERE (`ven` = "foo") Analyze ORDER BY `delivery_datetime` • How indexes are being used (or DESC LIMIT 100;\G
not…) ******** 1. row ************* • required filesorts ID: 1 select_type: SIMPLE • What tables, columns are being table: parts queried type: ref possible_keys: ven, part# Fix/Tune - involves iterations of: key: ven key_len: 3 • Add/alter indexes ref: null • Alter tables, columns, datatypes rows: 872 Extra: Using WHERE • Alter query structure 1 row in set (0.00 sec) • Test, 10 GOTO 10 until done
MySQL Enterprise Scalability New! MySQL Thread Pool • MySQL default thread-handling – excel ent performance, can limit scalability as connections grow • MySQL Thread Pool improves sustained performance/scale as user connections grow • Thread Pool API
MySQL Enterprise Scalability Configuration MySQL 5.6.11 Oracle Linux 6.3, Unbreakable Kernel 2.6.32 4 sockets, 24 cores, 48 Threads Intel(R) Xeon® E7540 2GHz CPUs 512GB DDR RAM