Lecture 5-0: High Performance Computing (HPC)

Introduction to ANSYS HFSS

1 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

Leveraging High Performance Computing Hardware*

Faster
HFSS-Hybrid DDM
Distributed Parallel Solvers Hybrid HFSS/HFSS-IE Domain Solver

HFSS Distributed Direct

HFSS Direct Solver Memory

Spectral Domain Method

Distributed Frequency Sweeps
HFSS-IE DDM
Matrix based Domain Solver

Multi-Threading HFSS Periodic Domains

Finite Array Domain Solver
Bigger
HFSS DDM
Mesh and Matrix based Domain Solver

*all HPC technologies are enabled by HPC license

2 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0
HFSS with HPC

Faster - Solver technology targeted at utilizing multiple processor/cores to accelerate the

solution process.

Faster

Multi-Threading

Spectral Domain Method

Distributed HFSS-Transient

3 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

 HPC: Multi-Threading (MT)

• Multi-Threading (HPC-MT)
• Single workstation solution
• Many tasks of the solution process Thread 1 Thread 3
Thread 2 Thread 4
–TAU Initial Mesh Generation
–Direct Matrix Solver
–Iterative Solver
–Field Recovery
• Available in HFSS 3D / Layout, HFSS-IE, and HFSS-Transient
ANSYS HFSS Layout with HPC
12 5.61

8 4.68
Cores

4 3.08
HPC
2 1.87

1 1.00

0.00 1.00 2.00 3.00 4.00 5.00 6.00

Speed-up
*HFSS Direct Matrix Solver

4 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

HPC: Spectral Domain Method (SDM)

• Spectral Decomposition Method

(HPC-SDM)
• Distribute frequency points in frequency sweeps
across a network of processors Frequency 3
–Uses RSM
• My be combined with HPC-MT Frequency 2

• Scalable to large numbers of cores

• Available in HFSS 3D / Layout and HFSS-IE
Frequency 4

Frequency 1

5 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

HFSS: HPC-SDM for Discrete and Interpolating Sweeps

• HPC setup to maximize SDM Factor: Frequency Points vs. Multi-Threading

• Total # of cores = # of parallelized frequency points times # of multi-threaded cores per frequency
point
Discrete sweep: Interpolating sweep:
• Best setup is without multi-threading • Total core count is only factor that impacts
• Each frequency point is solved with performance
single core • Number of cores per frequency point does not
matter

SDM4 SDM4

SDM2 SDM2

SDM1 SDM1

Local Local

0.00 1.00 2.00 3.00 4.00 5.00 0.00 1.00 2.00 3.00 4.00 5.00
SDM Factor SDM Factor
SDM1: 32 Freq
SDM2: 16 Freq/2 MT
SDM4: 8 Freq/4 MT

6 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

HPC: HFSS-Transient Distributed Parallel Solver

• HFSS-Transient Distributed Parallel (HPC-DP)

• Accelerates HFSS-Transient solutions by distributing the excitations across a network of processors
• Increases simulation speed
–Combines with HPC-MT
Excitation 3
• Available in HFSS-Transient
Excitation 2

Excitation 4

Excitation 1

7 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

HFSS with HPC

Bigger - Solver technology targeted at distributing the simulation memory across multiple
computers. The distributed nature of the solution may also result in faster simulations, but it is
primarily intended to increase capacity.

Bigger

8 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

HPC: HFSS-DDM (Mesh Based)

• Domain Decomposition Method: Meshed Based

• Distributes mesh sub-domains to network of processors/RAM
• Highly scalable to large numbers of processors Domain 3
• Automatic generation of domains / mesh partitioning
• Combines iterative & direct solver Domain 2
• Available in HFSS 3D

Domain 4

Domain 1

Solution Total RAM Elapsed Distributed

Size (GB) Time Engines
(hours)
33,750 λ3 300 GB 5 72
(DDM)

9 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

HFSS: HPC-Periodic Domains: Snowflake Array
E-field 5mm above aperture
Circularly polarized elements

10 GHz RAM Elapsed Time

529 circular WG elements, 1058 modes
HFSS 62G 27min
Array Mask HPC-PDM
Composite Excitation

13 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

HPC Setup
• HPC Setup
• Select the menu item Tools > Options > HPC and Analysis Options
– Design Type: HFSS 3D Layout Design
• Configurations Tab
– Supports multiple Configuration definitions. Only one Active configuration at a time.
• Options Tab
– Queue all simulations: Single user First-In, First-Out simulation queue
– Defines the HPC License Type: Pack or Pools

14 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

HPC Setup: Multi-Threading

• HPC for Multi-Threading

• Machine Details:
–Identification: localhost, IP Address, DNS or
UNC name
–Cores: Total number of cores on the machine
–Only first machine will be used
• Job Distribution:
–Tasks: 1 Multi-Thread Solver
Localhost: 4 Cores

Matrix Solver

Core 1
Core 2
Core 3
Core 4

15 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

HPC Setup: Frequency Sweeps
• HPC for Frequency Sweeps
• Machine Details:
– Cores: Total number of cores on the machine
– Supports multiple machines
• Job Distribution:
– Tasks: How many distributed frequency points will be launched per
machine
• If Tasks < Cores, the remaining cores will be used to accelerate
the solver
– Distribution Type: Frequencies
• Adaptive Mesh Process
– Since the adaptive mesh process only supports a single machine, the
solve will automatically be run on the first machine in the list. All of the
available cores on that machine will be used for multi-treading during adaptive
process. Task 2
Task 1

Frequency 1 Frequency 2
Distributed Frequency Sweep
Localhost: 4 Cores
2 Distributed Frequencies
2 Cores/Frequency

Core 1 Core 3
Core 2 Core 4

16 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

Analysis Configuration: Manual vs. Automatic
• Automatic Settings of Analysis configurations
• Indicate machines and total number of cores
per machine to use in simulations
• Default Settings of Analysis configurations
• Indicate machines, tasks and total number
of cores per machine to use in simulations
• Indicate Job Distribution

17 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

 Multi-level HPC for Speed and Scale
Level 1
Distributed
Variations
Level 2
Distributed
Memory

~4X faster
32 core DDM per variation
Time for 8 variations, serial: 14:52:57

128 core ‘two level’, 32 core DDM per variation

Time for 8 variations, four variations in parallel: 3:39:38

18 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0

Distributed Simulation Technologies Installation

RSM and MPI:

Manage communications between
local and remote computers for
HFSS simulations

Use RSM

Use MPI

19 © 2016 ANSYS, Inc. December 22, 2015 Release 2016.0