Documente Academic
Documente Profesional
Documente Cultură
Eric Grancher
eric.grancher@cern.ch
CERN IT
Anton Topurov
CERN IT Department
anton.topurov@cern.ch
CH-1211 Genève 23
Switzerland openlab, CERN IT
www.cern.ch/it
Outline
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
LHC gets ready …
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it Jürgen Knobloch / CERN
The LHC Computing
Challenge
• Signal/Noise 10-9
• Data volume
– High rate * large number of
channels * 4 experiments
15 PetaBytes of new data
each year
• Compute power
– Event complexity * Nb.
events * thousands users
100 k of (today's) fastest
CPUs
• Worldwide analysis &
funding
– Computing funding locally
in major regions &
countries
CERN IT Department
CH-1211 Genève 23 – Efficient analysis
Switzerland
www.cern.ch/it everywhereJürgen Knobloch / CERN
GRID technology
WLCG Collaboration
• The Collaboration
– 4 LHC experiments
– ~250 computing centres
– 12 large centres
(Tier-0, Tier-1)
– 38 federations of smaller
“Tier-2” centres
• Growing to ~40 countries
– Grids: EGEE, OSG, Nordugrid
• Technical Design Reports
– WLCG, 4 Experiments: June 2005
• Memorandum of Understanding
– Agreed in October 2005
• Resources
– 5-year forward look
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it Jürgen
Jürgen Knobloch / CERN
Knobloch/CERN Slide 5
Centers around the world
form a Supercomputer
CERN IT Department
CH-1211 Genève 23
Switzerland Inter-operation between Grids is working!
www.cern.ch/it Jürgen Knobloch / CERN
CPU & Disk
Requirements 2006
350 140
300 120
LHCb-Tier-2
CMS-Tier-2
250
CPU ATLAS-Tier-2 100 Disk
ALICE-Tier-2
LHCb-Tier-1
200 80
MSI2000
CMS-Tier-1
PB
ATLAS-Tier-1
150 ALICE-Tier-1 60
LHCb-CERN
CMS-CERN
100 ATLAS-CERN 40
ALICE-CERN
50 20
CERN:
0
~ 10% 0
CERN IT Department
CH-12112007
Genève 23 2008 2009 2010 2007 2008 2009 2010
Switzerland
www.cern.ch/it Year Jürgen Knobloch/CERNYear
Jürgen Knobloch / CERN Slide 7
Oracle databases at CERN
• Today:
Oracle RAC for most demanding services:
– CASTOR mass storage system (15 PB / year)
– Administrative applications (AIS)
– Accelerators and controls etc.
– LHC Computing Grid (LCG)
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
(our view of) RAC basics
Clients
CERN IT Department DB Servers Storage
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Linux RAC deployment example
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/i
Frits t
Ahlefeldt-Laurvig / http://downloadillustration.com/
RAC Scalability (1)
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
RAC Scalability (2)
Why?
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Reason 1
Clients
CERN IT Department DB Servers Storage
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Reason 2
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Reason 3
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Examples
• PVSS
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
CASTOR Name Server
• CASTOR
– CERN Advanced STORage manager
– Store physics production files and user files
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Stress Test Application
• Multithreaded
• Used with up to 40 threads
• Each thread loops 5000 times on
– Creating a file
– Checking it’s parameters
– Changing size of the file
• Test Made:
– Single instance vs. 2 nodes RAC
– No changes in schema and application code
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Result
500
400
ops/s
300
200
100
0
1 2 5 7 10 12 14 16 20 25 30 35 40
Single instance
RAC, 2 instances
Threads
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Analysis 1/2
Problem:
• Contention on CNS_FILE_METADATA table
Change:
• Hash partition with local PK index
Result:
10% gain, but still worse than single instance
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Analysis (2/2)
• Top event:
– enq: TX - row lock contention
– Again on the CNS_FILE_METADATA
• Findings:
– Application logic causes row lock contention
– Table structure reorganization can’t help
• Follow – up
– No simple solution
– Work in progress now
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
PVSS II
• CERN needs:
150 000 changes per second = 1500 times faster!
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
The Tuning Process
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
PVSS Tuning (1/6)
Table
event trigger on
lastval update eventlast
Update eventlastval set …
… merge (…)
• Shared resource:
EVENTS_HISTORY (ELEMENT_ID, VALUE…)
• Each client “measures” input and registers history with a
“merge” operation in the EVENTS_HISTORY table
Performance:
CERN IT Department
• 100 “changes” per second
CH-1211 Genève 23
Switzerland
www.cern.ch/it
PVSS Tuning (2/6)
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
PVSS Tuning (3/6)
Changes:
• bulk insert to a temporary table with OCCI, then call PL/SQL
to load data into history table
Performance:
• 2000 changes per second
rate75000_awrrpt_2_872_873.html
Avg
Event Waits Time(s) % Total Call Time Wait Class
Wait(ms)
row cache lock 813 665 818 27.6 Concurrency
gc current multi block request 7,218 155 22 6.4 Cluster
CPU time 123 5.1
CERN IT Department
log file parallel
CH-1211 Genève 23
write 1,542 109 71 4.5 System I/O
Switzerland
www.cern.ch/i
undo segmentt extension 785,439 88 0 3.6 Configuration
PVSS Tuning (6/6)
Problem investigation:
• Link between foreground process and ASM processes
• Difficult to interpret ASH report, 10046 trace
Problem identification:
• ASM space allocation is blocking some operations
Changes:
• Space pre-allocation, background task.
Result:
• Stable 150 000 “changes” per second.
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
PVSS Tuning Schema
Table
event trigger on
lastval update eventlast
Update eventlastval set …
… merge (…)
Conclusion:
• from 100 changes per second to 150 000
“changes” per second
• 6 nodes RAC (dual CPU, 4GB RAM), 32
disks SATA with FCP link to host
• 4 months effort:
– Re-writing of part of the application with
changes interface (C++ code)
– Changes of the database code (PL/SQL)
– Schema change
– Numerous work sessions, joint work with other
CERN IT groups
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Scalability Conclusions
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it
Recommendations
CERN IT Department
CH-1211 Genève 23
Switzerland
• CERN IT-DES (CERN IT-DES group web
www.cern.ch/it
site)
Q&A
CERN IT Department
CH-1211 Genève 23
Switzerland
www.cern.ch/it