Benchmarking Aida/Web

Benchmarking on Aida 5.6 with Swazoo 2.2beta1

This is an update of previous benchmark on Swazoo 1. See below for details of actual benchmark.

Throughput which Aida can sustain is now up to 30 generated and 170 cached requests/s for a 4KB dynamically generated test page.

Smalltalk generated req/s

cached req/s
VisualWorks 7.6
 30 170
Squeak 3.9
 20
Gemstone64 2.2.5 (Hyper!), commits
 12
Gemstone64 2.2.5 (Hyper!), no commits
 18
  • generated
  • httperf --server localhost --port 8888 --uri "/admin.html?view=login&id=123456" --rate 30 --num-conn 400
  • Connection time ms: min 15.9 avg 30.2 max 86.7 median 26.5 stddev 12.4
  • Request rate: 30.0 req/s (33.3 ms/req)
  • Reply rate replies/s: min 29.8 avg 29.8 max 29.8 stddev 0.0
  • Net I/O: 125.7 KB/s
  • cached
  • httperf --server localhost --port 8888 --uri "/admin.html?view=login&id=123456" --rate 170 --num-conn 1700
  • Connection time ms: min 0.3 avg 29.0 max 466.3 median 3.5 stddev 72.2
  • Request rate: 170.0 req/s (5.9 ms/req)
  • Reply rate replies/s: min 169.7 avg 169.9 max 170.0 stddev 0.2
  • Net I/O: 702.9 KB/s

Benchmarking on Aida 5.2 with Swazoo 1.0

Conclusions

I measured memory consumption and throughput on an Aida/Web dynamically generated login web page (http://demo.aidaweb.si/admin.html?view=login), which has 4KB of HTML.

Tool named httperf was used for generating web requests. First I generated 10.000 requests, which generated 10.000 new sessions and then measured memory consumption:

  • 220MB, that is 23KB per session

After optimizing memory consumption (gradually deleting more and more session state, which is normally done every night) I achieved:

  • 17KB per session, when last requests were removed from every session
  • 400B per session when all WebApplication instances were removed.

Above page create 3 WebAplication instances, which hold session state for representing domain objects and each such instance consumes 1KB and above. WebAdminApp which generates that web page consumes 14KB, but with some simple optimization it can be decreased to 1KB too.

Throughput which Aida can sustain is up to 30 requests/s for above test page.

Benchmarking was done on P4 3.20GHz with 2GB memory running SuSE Linux 10.0 64bit, on VisualWorks 7.4.1.

 

Procedure

  1. load AIDAWeb parcel into virgin VW7.4.1 image (from Application Server in Parcel mgr)
  2. SwazooServer demoStart, test http://localhost:8888 in browser
  3. page to benchmark: login page http://localhost:8888/admin.html?view=login
    • FireBug 2.0 Net statistics:
    • 18 requests, 108KB total, 700ms (on same machine!)
      • admin.html?view=login 4KB 33ms
      • CSS 2 requests 21KB
      • JavaScript 1 request 76KB
      • Images 14 requests 5KB
    • cached: 5 requests 103KB, 500ms
  4. checking - 1 request only

    • httperf --server localhost --port 8888 --uri /admin.html?view=login --rate 1 --num-conn 1 --num-call 1 -- timeout 5

    • Connection time ms: 45.0
    • Request size B: 81.0
    • Reply size B: header 268.0 content 3724.0 (total 3992.0)
  5. memory consumption
    • collect garbage, after: 26.6 Mbytes total; 16.42 Mbytes used, 10.17 Mbytes free.
    • sessions: 5 (WebSession allInstances size)
    • generate 1000 new sessions:

      • httperf --server localhost --port 8888 --uri /admin.html?view=login --rate 10 --num-conn 1000 --num-call 1

      • Connection time ms: min 17.6 avg 25.2 max 53.5 median 24.5 stddev 3.3
    • sessions: 1005
    • collect garbage, after: 46.6 Mbytes total; 39.44 Mbytes used, 7.16 Mbytes free.
    • generate another 9000 to have 10.000 sessions:

      • httperf --server localhost --port 8888 --uri /admin.html?view=login --rate 40 --num-conn 9000 --num-call 1

      • httperf: Connection time ms: min 46.1 avg 19568.4 max 180994.3 (?) only 4900 successfull requests!
      • long pauses - long garbage collections?
    • sessions: 5904
    • collect garbage, after: 156.63 Mbytes total; 152.27 Mbytes used, 4.36 Mbytes free.
    • generate less fast another 4100 to have 10.000:

      • httperf --server localhost --port 8888 --uri /admin.html?view=login --rate 10 --num-conn 4100 --num-call 1

    • sessions 9768
    • collect garbage, after: 246.65 Mbytes total; 241.26 Mbytes used, 5.38 Mbytes free.
    • memory per session: 23KB ! (241.260KB-16.420KB)/9763)
  6. optimizing memory consumption
    • removing lastRequest in sessions
      • WebSession allInstances do: [:each | each initRequests; nilLastRequest]]
    • collect garbage, after: 196.64 Mbytes total; 180.75 Mbytes (-61M) used, 15.89 Mbytes free.
    • memory per session: 17KB ! (180.750KB-16.420KB)/9763)
    • removing WebApplication instances
      • apps per session: 3 (WebAdminApp, WebDemoApp, WebDiscussionsApp)
      • WebSession allInstances do: [:each | each initAppsForObjects; nilLastApp]
    • collect garbage, after: 186.64 Mbytes total; 159.92 Mbytes (-22M) used, 26.71 Mbytes free.
    • memory per session: 15KB ! (159.920KB-16.420KB)/9763)
    • memory per app instance: 1KB ! (180.750KB-159.920KB)/(9763x2) only WebDemoApp and WebDiscussionsApp
      • WebSession allInstances do: [:each | each nilLastApp] (this will finally remove WebAdminApp)
    • collect garbage, after: 36.6 Mbytes total; 20.22 Mbytes (-140M !!!) used, 16.37 Mbytes free.
    • memory per session: 400B ! (20.220KB-16.420KB)/9763)
    • memory per admin app: 14KB !!! (159.920KB-20.220KB)/9763
      • full web page object tree stays in memory?
  7. throughput
    • 100 sessions, 10 requests per session, cookies used(?), for same web page

      • httperf --server localhost --port 8888 --uri /admin.html?view=login --rate 1 --wsess=100,10,0

      • Reply time ms: response 31.2
      • httperf didn't use cookies, but it should!
    • 1000 requests per same session (session id added in url)

      • httperf --server localhost --port 8888 --uri "/admin.html?view=login&id=123456" --rate 10 --num-conn 1000

      • Total: connections 1000 requests 1000 replies 1000 test-duration 99.931 s
      • Connection time ms: min 10.3 avg 25.9 max 73.9 median 24.5 stddev 4.6
    • rate 20 req/s, 1000 requests per same session

      • httperf --server localhost --port 8888 --uri "/admin.html?view=login&id=123456" --rate 20 --num-conn 1000

      • Total: connections 1000 requests 1000 replies 1000 test-duration 49.981 s
      • Connection time ms: min 7.1 avg 24.3 max 56.4 median 23.5 stddev 3.9
      • Request rate: 20.0 req/s (50.0 ms/req)
      • Reply rate replies/s: min 20.0 avg 20.0 max 20.0 stddev 0.0 (9 samples)
      • Net I/O: 79.0 KB/s
    • rate 25 req/s, 1000 requests per same session
      • Total: connections 1000 requests 1000 replies 1000 test-duration 39.994 s
      • Connection time ms: min 12.4 avg 26.5 max 64.8 median 24.5 stddev 5.5
      • Request rate: 25.0 req/s (40.0 ms/req)
      • Reply rate replies/s: min 25.0 avg 25.0 max 25.0 stddev 0.0 (7 samples)
      • Net I/O: 98.8 KB/s
    • rate 30 req/s, 1000 requests per same session
      • Total: connections 1000 requests 1000 replies 1000 test-duration 34.571 s
      • Connection time ms: min 17.5 avg 258.2 max 1283.1 median 32.5 stddev 359.0
      • Request rate: 28.9 req/s (34.6 ms/req)
      • Reply rate replies/s: min 27.0 avg 29.3 max 30.0 stddev 1.2 (6 samples)
      • Net I/O: 114.3 KB/s
    • rate 35 req/s, 1000 requests per same session
      • Total: connections 1000 requests 1000 replies 1000 test-duration 60.185 s
      • Connection time ms: min 19.5 avg 6028.8 max 45291.3 median 3270.5 stddev 6897.0
      • Request rate: 16.6 req/s (60.2 ms/req)
      • Reply rate replies/s: min 0.0 avg 16.6 max 28.8 stddev 11.0 (12 samples)
      • Net I/O: 65.6 KB/s
    • rate 40 req/s, 1000 requests per same session
      • Total: connections 1000 requests 1000 replies 1000 test-duration 51.456 s
      • Connection time ms: min 36.3 avg 6694.1 max 45082.6 median 3408.5 stddev 7158.8
      • Request rate: 19.4 req/s (51.5 ms/req)
      • Reply rate replies/s: min 0.0 avg 19.9 max 27.8 stddev 10.5 (10 samples)
      • Net I/O: 76.8 KB/s
    • rate 100 req/s, 1000 requests per same session
      • Total: connections 1000 requests 1000 replies 1000 test-duration 55.236 s
      • Connection time ms: min 34.0 avg 16248.8 max 46095.5 median 11825.5 stddev 13749.8
      • Request rate: 18.1 req/s (55.2 ms/req)
      • Reply rate replies/s: min 0.0 avg 18.1 max 26.8 stddev 11.4 (11 samples)
      • Net I/O: 71.5 KB/s