Scalability and robustness tests (integrate JMeter to TT)

[Jani Laakso]

Idea on how to perform real world scalability teests (simulate very closely e.g. ten thousands of concurrent web browser clients):

1. record test case using TT Server
2. schedule scalability (or robustness) test
   • do not use "Testing Hosts", use massive amounts of Java threads that shoot UIDL directly to "Testing target"

=> brings testing target down to it's knees very easily

Benefits: => brings out concurrency issues very easily => demonstrates how much of load your server can handle => pinpoints application scalability issues together with profiler

NOTE: I have done this kind of Java application (for TK4) which opens hundreds of threads simulating very closely Firefox socket connections.

[Jani Laakso]

Read this: Integrate JMeter (Apache project) to TT. It's excellent tool for this kind of testing..

Integration can be very lightweight, it works as-is (without TT), but for license / income reasons we should do some kind of lean integration for TT.

Possible tasks:

• package into TT (it's Java application with optional swing IDE)
• better asserts, these are quite hard to do with vanilla JMeter
• TT could startup JMeter performance tests

Hours do not contain documentation.

[Jani Laakso]

Please see the graphs that JMeter can produce, some of them could be shown inside TT (without JMeter cryptic GUI). http://jakarta.apache.org/jmeter/usermanual/component_reference.html#listeners

JMeter is driven by quite simple XML test plan files, results are written directly to disk (csv, text files, image files containing graphs..)

[Jani Laakso]

I'd feel that providing two graphs in a very simple "mac way" through TT is very good feature:

1. JMeter throghput results

• see http://jakarta.apache.org/jmeter/usermanual/component_reference.html#Graph_Results
  • use case: I want to know how many users my server / application combo can handle when maximum allowed latency for any user is 10 seconds to execute test case X

2. Memory consumption graphs

• http://jakarta.apache.org/jmeter/usermanual/component_reference.html#Monitor_Results
  • use case: I want to find out maximum memory consumption with 5000 concurrent users
  • use case: I want to ensure that memory growth stops after maximum concurrent sessions is reached
    • e.g. set session lifetime to 1 minute, let hundreds of threads to execute test case with unique sessions

[Joonas Lehtinen]

In brief, IMO this scalability feature would (from UI perspective) be:

1. Select (already recorded) test from Testing Tools ui
2. Press "Scalability Test" button
3. Up comes a dialog. Select:
   • Number of users (default 100)
   • Number of test rounds (default 5)
   • Ramp-up time (default 5sec)
4. UI shows progressbar while testing (hidden, under the hood TT sets up proxy to one host, plays the normal test there creating JMeter test profile and then runs JProfile test with this profile).
5. Up comes a graph with: all the info JMeter can give

[Joonas Lehtinen]

Furthermore, we could also create graph about JVM memory usage during the scalability test and include it to results.

[Jani Laakso]

When thinking of use cases (and later designing UI) Must remember "higher goal" here:

• keep it simple
• testing for dummies, anyone can do this
• testing / scalability / robustness tests is dull and troublesome => many (smaller) companies skip this because "time to market" hits to their faces

Note, JVM memory usage exists already in JMeter (Tomcat specific), it might be one graph that we could shown in TT.

[Marc Englund]

Also see #707 (doing this should fix #707 almost as a side-effect)

[Joonas Lehtinen]

Milestone Testing Tools 2.0.0 deleted