Documente Academic
Documente Profesional
Documente Cultură
Agenda
Latency
-
What is it and why is it such a big deal? How do I test for it? How to use this information to demonstrate performance for your users What are they How to read the logs How to let your AR Server tell you how many it needs What types there are
Plugins
-
To understand that no matter how many CPUS and RAM you have, a poor network can bring an application to its knees Demystify the confusion about queues and threads Understand the black boxes called plugins Understand what these are and to give you the tools to understand your own AR System To sit through a very technical and possibly boring tech talk and live to talk about it
Results
-
Skills developed
-
The knowledge to understand the not so well documented parts of the AR System
Latency
Higher latency decreases app response time, user performance and perceived app quality
2013 WWRUG 2012Canada WWRUG Inc. Canada All Rights Inc.Reserved All Rights Reserved 4
In a network, latency, a synonym for delay, is an expression of how much time it takes for a packet of data to get from one designated point to another. There are two typical types:
One way
The time from the source sending a packet to the destination receiving it
Round trip
The one-way latency from source to destination plus the one-way latency from the destination back to the source
Two key elements of network performance are bandwidth and latency. The average person is more familiar with the concept of bandwidth as that is the one advertised by manufacturers of network equipment. However, latency matters equally to the end user experience
2013 WWRUG Canada Inc. All Rights Reserved
5
ITSM 7.6.04 SP2, load balanced environment as the diagram. Browser was Firefox v21.0. URL is HTTPS From the incident console, double clicking an incident. Took a measurement on the current network to get the number of trips, amount of data and response time in seconds With added latency of 50 ms from the client (browser on desktop) through the load balancer and to one of the Mid Tier servers
Test
-
50ms
50ms
Test summary Just 50ms of network latency in just one piece of the BMC architecture, from the browser to the Mid Tier, can add to your end user response times!
2013 WWRUG Canada Inc. All Rights Reserved
8
What happens when you type the Mid Tier address in the URL bar
Your browser will use your desktop network configuration to get the network details of your Mid Tier. First is will look at your local hosts file for the Mid Tier host name. If it is not in there it will ask the Domain Name Service (DNS) If your Mid Tier has an IP address of 192.168.0.5 and it is configured in your DNS database, then the browser will connect to it and everything works, you see the application etc BUT if you added a line in your local hosts file (c:\windows\system32\drivers\etc\hosts) so that your desktop believes its not that IP address but a different one e.g. 127.0.0.1 then the browser will try and connect to that. 127.0.0.1 is something called a loopback adaptor and its basically means your own machine you are typing on. And unless you have a Mid Tier running on your machine, it will fail.
2013 WWRUG Canada Inc. All Rights Reserved
9
What happens when you type the Mid Tier address in the URL bar (part 2)
What if you had a Mid tier on your desktop and you added the same Mid Tier host name to your local hosts file with 127.0.0.1? Your browser would still say the correct URL address but you would be connecting to the Mid Tier on your desktop and not the one on the network.
If you installed a piece of software on your desktop that wasnt a Mid Tier but something that connected to your REAL Mid Tier on the network, 195.168.0.5, BUT delayed all connections, adding latency . Then this is called a Proxy and this is what I used.
2013 WWRUG Canada Inc. All Rights Reserved
10
11
Add the Mid Tier URL host name to the loopback address in your local hosts file
Click Start on the proxy. Use your browser as before and record the timings.
2013 WWRUG Canada Inc. All Rights Reserved
12
Open Incident
Houston
484
2013 WWRUG Canada Inc. All Rights Reserved
13
Using ping, which uses ICMP but sometimes turned off on network equipment Those times are round trip, so the time its taken from client to server AND back from server to client. Therefore when testing, half those values!
14
Using http-ping, which uses the actual web server and this will always be slower than a ping due to the fact it waits for the web server to respond with a HTTP code
Those times are round trip, so the time its taken from client to server AND back from server to client. Therefore when testing, half those values!
15
16
17
Some network equipment can prioritise certain protocols. The Mid Tier uses either HTTP typically on port 80 or HTTPS typically on port 443
Make sure your architecture has as little latency as possible between the Mid Tiers and AR Servers and more importantly between the AR Server and the database.
-
ITSM 7.6.04 with approx 900 concurrent users fires approx 127 SQL statements per second at the database. Latency would bring the app to its knees!! There is more traffic between the browser and the Mid Tier than there is from the Mid Tier to the AR Server.
2013 WWRUG 2012Canada WWRUG Inc. Canada All Rights Inc.Reserved All Rights Reserved 19
They are identified by a number, and sometimes referred to as an RPC queue. Here are some examples
390600 = Admin 390603 = Escalation 390620 = Fast API calls (just a name without intending to indicate performance) 390635 = List API calls (just a name as well but was aimed at things that search and return lists/large amount of data)
A queue can have one or more threads defined for them. On startup, each thread creates a connection to the database that it uses throughout its existence. Threads only close when you shutdown the server or it cannot connect to the database One queue has one or more threads
20
2013 WWRUG Canada Inc. All Rights Reserved
20
If the list queue is at max resource, it will put the work on fast queue and vice versa
If both are full, it will move the work to the admin thread So, the system has a set of queues -- some pre-defined, some private and defined per instance -- and each of them has processing threads as configured in the ar.cfg/ar.conf
21
2013 WWRUG Canada Inc. All Rights Reserved
21
Fast threads At minimum, the same number as you have CPU cores; at maximum, 3 times the number of CPU cores, but no exceeding 32 List threads At minimum, the same number as you have CPU cores; at maximum, 5 times the number of CPU cores, but not exceeding 32 CPU x 1.5 for the Private queue.
Page 2048
SW00427239 - Fast and List threads are not set as per the recommended Queue settings.
22
Doug Mueller ARSList post - In theory, there is no reason you cannot have 10s or even 100s of threads in a queue.
2013 WWRUG Canada Inc. All Rights Reserved
22
Sure it can handle more if the rest of the car components and driver can handle it!
2013 WWRUG Canada Inc. All Rights Reserved
23
23
Therefore after 14 years of experience, researching hours and hours, googling the WHOLE internet on system architecture, posting on so many forums my answer is
24
2013 WWRUG Canada Inc. All Rights Reserved
24
Every environment is different so suck it and see! Get the system to tell you!
And heres how I do it...
If you have a high number of threads already, and these threads are taking too long to complete. E.g time to look at the queries to the database and work with your DBA to speed them up If the above doesnt work, or the DBA doesnt want to play ball, then its time to increase the Max Thread count in the AR System Administration Console
Each queue and thread takes system resources such as CPU power and memory. If those resources are maximised, then its time for an upgrade or another AR Server in the server group
25
2013 WWRUG Canada Inc. All Rights Reserved
25
Run this log in your peak periods if you can. Or if you are not live, then use a volume and performance testing application
This log will get large so make sure you have enough space
26
2013 WWRUG Canada Inc. All Rights Reserved
26
ARLogAnalyzer
https://communities.bmc.com/docs/DOC-2973 http://www.missingpiecessoftware.com/products/ar-log-file-analyser
27
2013 WWRUG Canada Inc. All Rights Reserved
27
Green -- The same user Purple -- The same RPC 390620 which is the Fast queue Red -- TWO RPC IDs meaning two different API calls being processed
Every call that the dispatcher thread receives is assigned an RPC ID that can be used to identify the call from the time the call is placed into the queue until a response is sent back to the client
Cyan -- One thread executing both API calls one after the other
28
2013 WWRUG Canada Inc. All Rights Reserved
28
ARLogAnalyzer
29
Both results show : Fast Max is 30 but 27 being used List Max is 40 but 35 being used
2013 WWRUG Canada Inc. All Rights Reserved
29
Remember that this will probably spike during the working day but this is truly the best way to monitor when your busy periods are and how stressed your system is
There is no such thing as 0 idle time. Even getting work from the dispatcher takes at least some time
-
Look for very small numbers on the AVE idle time column
30
2013 WWRUG Canada Inc. All Rights Reserved
30
Looking through the log, we can see the thread number synchronously increment for the first 6, 28 to 33, then we see 130
31
2013 WWRUG Canada Inc. All Rights Reserved
31
The above screenshot of the log entry is on one line but I had to cut it to fit on the slide Find the first instance of the thread id (TID) and note the time in this example is 13:08. This is when the AR Server decided it needed to create a new thread on the 390626 queue
32
2013 WWRUG Canada Inc. All Rights Reserved
32
AR System Administration Console > System > General > Review Statistics
33
2013 WWRUG Canada Inc. All Rights Reserved
33
Plugins
2013 WWRUG 2012Canada WWRUG Inc. Canada All Rights Inc.Reserved All Rights Reserved 34
AREA External Authentication ARF Filter plugin which are used to extend actions of filters ARDBC Access data outside of forms but mimic the behaviour of forms Which obviously relates to the programming language they are built with Runs through an executable file arplugin.exe (windows), arplugin (*NIX) Surprisingly runs from separate java processes, or java virtual machines
C
-
Java
-
arplugin.exe / arplugin started via armonitor and configured to run in armonitor.cfg / armonitor.conf
Configured through the ar.cfg / ar.conf Plugin:, Plugin-Path: & Plugin-Port: apply only to the C plugin daemon. How to identify them? In the ar.cfg/ar.conf
36
2013 WWRUG Canada Inc. All Rights Reserved
36
A Java process is started via armonitor and configured to run in armonitor.cfg / armonitor.conf
E.g. /usr/java/jdk1.6.0_06/jre/bin/java -Xmx512m -classpath /opt/bmc/ARSystem/pluginsvr:/opt/bmc/ARSystem/pluginsvr/arpluginsvr 75.jar com.bmc.arsys.pluginsvr.ARPluginServerMain -x svr1 -i /opt/bmc/ARSystem Primary plugin server Full Text Search Engine 2 CMDB plugin servers
How to identify them? Within pluginsvr_config.xml files and some are aliased in the ar.cfg/ar.conf
Logging is controlled through each log4j_pluginsvr.xml files
2013 WWRUG Canada Inc. All Rights Reserved
37
37
1-Way is the AR Server calls the plugin and the plugin returns a response
2-Way is the AR Server calls the plugin but in order to complete that request, the plugin must connect back to the AR Server and lookup some data, and then returns a response
The two way plugins are typically the ones to look out for with regards to performance
-
E.g. REMEDY.ARDBC.APPQUERY
38
2013 WWRUG Canada Inc. All Rights Reserved
38
E.g. If the a plugin is configured on 390624 then what API and SQL the plugin fires against the AR Server, will be in the API and SQL logs with:
39
2013 WWRUG Canada Inc. All Rights Reserved
39
Server-Plugin-Alias: REMEDY.ARDBC.APPQUERY REMEDY.ARDBC.APPQUERY srv1:9999 See port number as :9999 so if Plugin-Port: 9999 then its a C plugin. Otherwise you can tell its a Java plugin.
ITSM OOTB, this plugin is not configured to run on its own queue. Add the line in the pluginsvr_config.xml
41
2013 WWRUG Canada Inc. All Rights Reserved
41
Plugins :: Summary
Plugins that connect back to the AR Server are clients just like the User Tool and Mid Tier etc Most, if not all, are not configured to run on queues OOTB
No better log than a real users, working system, filled one Have some automated scripts for log rotation with email notification Just to get a weeks worth of usage statistics E.g. If we add 1000 users, will my need to increase my thread count?
42
2013 WWRUG Canada Inc. All Rights Reserved
42
2013 WWRUG 2012Canada WWRUG Inc. Canada All Rights Inc.Reserved All Rights Reserved 43
Conclusion
Add your conclusions here.
44
Question/Discussions
Add your discussion points here
45
Wrap-up
Contact information
46