Unveiling the Early Universe with Intel® Xeon® Processors and Intel® Xeon Phi Copr ™ ocessors James Briggs, John Fu, Karl Feind, Mike John Pennycook, Jim Jeffers Carlos Martins, Paul Shellard Woodacre Intel COSMOS (University of SGI Cambridge)
COSMOS @ DiRAC - University of Cambridge Supercomputing facility originally dedicated to cosmology research: Founded in January 1997, now part of the UK DiRAC facility Consortium brought together by Prof. Stephen Hawking History of large, shared-memory Intel® Architecture machines COSMOS-IX: SGI Altix UV2000, delivered in July 2012 1856 Intel® Xeon® processor cores (Sandy Bridge) 14.5 TB of globally shared memory 31 Intel® Xeon Phi ™ coprocessors introduced in December 2012 2
What is the SGI UV? The most flexible compute platform in the industry SSI PGAS Cluster SuperPC UPC, SHMEM, etc. MPI Huge Coherent 8 PB Global Address Connection- Memory Space less Scalable Optimized for Small Small IO/Coprocessors Transfers message Rich Synchronization optimized 3
The Code – “Walls” Simulates the evolution of domain wall networks in the early universe: Adjacent regions misalign over time (Higgs) Domain ‘energy’ walls form to separate them ‘Observable” analogy is ferromagnet domains Press-Ryden-Spergel algorithm  Compiled for 3D or 4D simulations Used at COSMOS for over 10 years, developed for complex hybrid networks Stencil code, targeting SMP with OpenMP Benchmark for acceptance testing on previous machines To find out more about domain walls see the Hawking Cosmology Centre public pages: www.ctc.cam.ac.uk/outreach/origins/cosmic_structures_two.php Video courtesy of Dr. Carlos Martins, University of Porto. 4
The Algorithm (in 3D) 1.
Laplacian stencil operation (radius 1): 2. Timestepping (leap-frog integration): , where 3. Calculate area of domain walls: 5
Baseline Performance “Out-of-the-Box” Comparison: 1.2 Processor is ~2x faster than coprocessor! 1 Why? Poor vectorization 0.8 Poor memory behavior etc 0.6 Experimental Setup: 0.4 480 x 480 x 480 problem 0.2 2 x Intel® Xeon® E5-4650L processor 1 x Intel® Xeon Phi ™ 5110P coprocessor 0 icc version 14.0.0 2 x Processor 1 x Coprocessor 6
Optimization and Modernization (1/3) The Strategy: Use straightforward parallel tuning techniques vectorize, scale, memory Use tools/compiler guidance features Maintain readability and platform portability The Result: Significant performance improvements in ~3-4 weeks Single, clear, readable code-base “Template” stencil code transferable to other simulations 7
Optimization and Modernization (2/3) Optimizations Improve auto-vectorization (using -vec-report3 and -guide-vec) int ip1 = (i+1) % Nx; loop was not vectorized: operator unsuited for vectorization. int ip1 = (i < Nx-1) ? i+1 : 0; LOOP WAS VECTORIZED 8
Optimization and Modernization (2/3) Optimizations Improve auto-vectorization (using -vec-report3 and -guide-vec) Introduce halo regions, to improve cache behavior (and remove gathers) int ip1 = i+1; Data from i = 0 is replicated at i = Nx; all loads become contiguous 9
Optimization and Modernization (2/3) Optimizations Improve auto-vectorization (using -vec-report3 and -guide-vec) Introduce halo regions, to improve cache behavior (and remove gathers) Swap division by constants for multiplication by pre-computed reciprocals P2[i][j][k][l] = … / (1-delta); One division per stencil point P2[i][j][k][l] = … * i1mdelta; One division reused for all stencil points 10
Optimization and Modernization (3/3) M odernizations Reduce memory footprint (2x) by removing redundant arrays Rewrite as ; no need to store time derivative in addition to solution at two timesteps 11
Optimization and Modernization (3/3) Modernizations Reduce memory footprint (2x) by removing redundant arrays Remove unnecessary 4D calculations and array lookups from 3D simulations Lphi = P[i-1][j][k][l] + P[i+1][j][k][l] + P[i][j-1][k][l] + … - 8 * P[i][j][k][l]; Lphi = P[i][j-1][k][l] + P[i][j+1][k][l] + … - 6 * P[i][j][k][l]; Saves two reads/additions per stencil point 12
Optimization and Modernization (3/3) Modernizations Reduce memory footprint (2x) by removing redundant arrays Remove unnecessary 4D calculations and array lookups from 3D simulations Combine three algorithmic stages into a single loop Before: solve(t), timestep(), area(t+1) After: solve(t), area(t), timestep() Allows for one pass through the data each timestep.
Final Performance 8 7 6 5 4 3 2 1 0 2 x Processor 1 x Coprocessor 14
Final Performance 8 1.38 7 x 6 1.44 5 x 1.62 4 1.52 x 3 x 1.28 2 1.18 1.12 x 1 x x 0 0.45 x 2 x Processor 1 x Coprocessor 15
Future Work Exploration of other optimizations (e.g. cache blocking) Stream larger problems through coprocessors Work sharing between multiple processors and coprocessors Incorporate stencil template into other key codes
Conclusions Modernizing Code for Parallelism Works! Straightforward code changes -> Dramatic Performance Impact Dual-tuning advantage –> Single Source Processor ~6x ; Coprocessor ~18x over baseline Future benefits -> ready to take advantage of increasing parallelism We W lc e o lc m o e m t e o t t o h t e P e ara a lle ra l Un lle iv l Un ers e e! e 17
Intel® Xeon Phi™ Coprocessor Starter Kits Go paral el today with a ful y-configured system starting below $5K* 3120A OR 5110P software.intel.com/xeon-phi-starter-kit Other brands and names are the property of their respective owners. *Pricing and starter kit configurations will vary. See software.intel.com/xeon-phi-starter-kit and provider websites for full details and disclaimers. Stated currency is US Dollars.
References  W.H. Press, B.S. Ryden and D.N. Spergel, “Dynamical Evolution of Domain Walls in an Expanding Universe”, Astrophys. J. 347 (1989)  A.M.M. Leite and C.J.A.P Martins, “Scaling Properties of Domain Wall Networks”, Physical Review D 84 (2011)  A.M.M. Leite, C.J.A.P Martins and E.P.S. Shellard, “Accurate Calibration of the Velocity-Dependent One-Scale Model for Domain Walls”, Physics Letters B 7 18 (2013) 19
Risk Factors The above statements and any others in this document that refer to plans and expectations for the third quarter, the year and the future are forward-looking statements that involve a number of risks and uncertainties. Words such as “anticipates,” “expects,” “intends,” “plans,” “believes,” “seeks,” “estimates,” “may,” “will,” “should” and their variations identify forward-looking statements. St atements that refer to or are based on projections, uncertain events or assumptions also identify forward-looking statements. M any factors could afect Intel’s actual results, and variances from Intel’s current expectations regarding such factors could cause actual results to difer materially from those expressed in these forward-looking statements. Intel presently considers the following to be the important factors that could cause actual results to difer materially from the company’s expectations. Demand could be diferent from Intel's expectations due to factors including changes in business and economic conditions; customer acceptance of Intel’s and competitors’ products; supply constraints and other disruptions afecting customers; changes in customer order patterns including order cancellations; and changes in the level of inventory at customers. Uncertainty in global economic and financial conditions poses a risk that consumers and businesses may defer purchases in response to negative financial events, which could negatively afect product demand and other related matters. Intel operates in intensely competitive industries that are characterized by a high percentage of costs that are fixed or difficult to reduce in the short term and product demand that is highly variable and difficult to forecast. Revenue and the gross margin percentage are afected by the timing of Intel product introductions and the demand for and market acceptance of Intel's products; actions taken by Intel's competitors, including product oferings and introductions, marketing programs and pricing pressures and Intel’s response to such actions; and Intel’s ability to respond quickly to technological developments and to incorporate new features into its products. The gross margin percentage could vary significantly from expectations based on capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; changes in revenue levels; segment product mix; the timing and execution of the manufacturing ramp and associated costs; start-up costs; excess or obsolete inventory; changes in unit costs; defects or disruptions in the supply of materials or resources; product manufacturing quality/yields; and impairments of long-lived assets, including manufacturing, assembly/test and intangible assets. Intel's results could be afected by adverse economic, social, political and physical/infrastructure conditions in countries where Intel, its customers or its suppliers operate, including military confict and other security risks, natural disasters, infrastructure disruptions, health concerns and fuctuations in currency exchange rates. Expenses, particularly certain marketing and compensation expenses, as well as restructuring and asset impairment charges, vary depending on the level of demand for Intel's products and the level of revenue and profits. Intel’s results could be afected by the timing of closing of acquisitions and divestitures. Intel's results could be afected by adverse efects associated with product defects and errata (deviations from published specifications), and by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust, disclosure and other issues, such as the litigation and regulatory matters described in Intel's SEC reports. An unfavorable ruling could include monetary damages or an injunction prohibiting Intel from manufacturing or selling one or more products, precluding particular business practices, impacting Intel’s ability to design its products, or requiring other remedies such as compulsory licensing of intellectual property. A detailed discussion of these and other factors that could afect Intel’s results is included in Intel’s SEC filings, including the company’s most recent reports on Form 10-Q, Form 10-K and earnings release. Rev. 7/17/13