Load Testing Outlook 2010 on XenApp – Hosted vs. Streamed to Server

After this initial test comparing the performance of Word 2010 Hosted against Streamed to Server on XenApp 6 showed some considerable advantages to using locally installed Office 2010 instead of Streaming it, I stepped up my testing and included Outlook 2010 instead.  The test was more intensive, moving between the different screens of Outlook (Inbox, Tasks, Contacts, etc) and creating and deleting items. 

Again, I set up EdgeSight for Load Testing to run 50 users on a single server – logging in at the rate of one every 20 seconds, staying logged in and “working” for 10 minutes and then all logging out in 3 minutes.  They once again looked nice when they were all logged in…

Outlook 2010 Hosted

Outlook 2010 performed very well on XenApp 6 when installed locally.  CPU time was quite acceptable throughout the test, being quite close to the performance of Word 2010 at the same load.  Memory usage was slightly higher than Word 2010, being about 7gb. 

Outlook 2010 Streamed to Server

Streaming added a couple of issues to the performance of Outlook 2010.  Firstly, the logon phase involved far more CPU activity, being flat at 100% for about the last three minutes.  CPU usage then remained very high through the phase of the test where Outlook was looping through simulated user activity, before dropping off nicely at the end of the test as the users logged off.  Memory usage was more worrying.  Its not that it was much higher than hosted – it was higher, though not substantially so.  The worrying part was that it increased through the 10 minutes after logons stopped as each OUTLOOK.EXE process slowly ate more RAM.  The Hosted Outlook processes did not behave in this way.  We tried this again over a much longer period with more users and this continued, Outlook using ever more memory until being logged off

Conclusion

This test was more conclusive than the Word 2010 test, with far more gap between Outlook 2010 when Hosted and when Streamed to Server.  Performance of the application when open did appear to be roughly the same, but with logon time more than double for the streamed application, this is not enough to justify streaming.  The higher CPU usage after logons stopped would indicate using Streaming to Server would reduce the number of active user sessions the server could support quite substantially and on this hardware could not be recommended.

Load Testing Word 2010 on XenApp – Hosted vs. Streamed to Server

As part of a decision on how to present Office 2010 to our XenApp 6 farm we benchmarked Office applications as Streamed and Hosted applications.  When we tried this on XenApp 5 with Office 2007 the difference between the two did not seem very great – this time around the Streaming version seems to put on some considerable load.

With each test we logged on 50 users in 16 minutes (one logon every 20 seconds) and ran them for 10 minutes, typing random text Word.  They then all logged out over a 3 minute period.  They look very pretty when they’re all running:

Word 2010 Hosted

The locally installed on the server version of Word 2010 performed very well.  It easily handled logins every 20 seconds with the CPU (on a medium spec dual processor server) at about 30% in the logon phase, 15% when all the users were logged in and a peak of over 50% during the rather rapid logoff. Memory usage peaked at about 6gb

Word 2010 Streamed To Server

The second test was using the same hardware, users and script as the first – but this time Word was streamed to the server.  The difference was apparent from the first login, which took a lot longer – about 25-30 seconds, almost double the Hosted logon time.  CPU usage was consistently 10-15% higher in the initial logon phase and then built alarmingly as the user count reached 50.  It then dropped off very nicely once the users were logged in.  Memory usage was higher, peaking at about 7gb.

Conclusions

Overall the Hosted version performed very much better than the Streamed to Server version with Word 2010.  As you might expect, most of the difference was in the logon phase, where CPU activity was much lower on the hosted version.  Logon times were also longer using Streaming, by at least 15 seconds.  This makes sense as the Streamed to Server version has a bit more work to do creating the Sandbox environment.  The lower memory footprint of the Hosted install overall might also mean that greater user density would be possible using Word 2010 installed directly on the servers.

However, the Streamed to Server version did have lower CPU usage in the later phases, so this test might be considered more even, depending on the pattern of your logons and logoffs.  It must be said that the test did not make an effort to push Word in this middle phase, it was just some light typing.  A more intensive test might show more difference at this point.

This test was conducted using v6 of the streaming client and profiler and XenApp 6 running on Windows Server 2008 R2 on a single dual core server with 12gb RAM.  The test was performed with EdgeSight for Load Testing v3.7.

Installing EdgeSight For Load Testing for XenApp 6

The version of most software appears to have been incremented with the release of XenApp 6, and this includes EdgeSight For Load Testing (ESLT) which is now Build 3.6.1.24.

Oddly previous versions of ESLT appear to be a higher build number – the version I was testing XenApp 5 with was build 5.2 and it shipped with build 5.0. I assume they have reordered their build numbers! Please note that ESLT is a different program from EdgeSight – they can be installed on the same machine, but there is no reason as such to do so, and neither application requires the other to work.

• Get a Windows Server 2008 or 2008 R2 server (I’ve tested XenApp 6 with an R1 server just fine).  You will need a XenApp License server set up too before you can use ESLT properly.  Make sure its got a lot of memory if its going to run a lot of sessions – I have found to launch more than 100 sessions meant more than 4gb of RAM and of course 64-bit OS if using Windows 2008.
• Be aware that ESLT actually consists of 2 parts – the Launcher and the Controller.  You would only have one Controller usually to kick off the tests but you could have the Launcher software installed on several machines to actually launch sessions from lots of places.  This might be better for you if you don’t have a powerful server to run everything – personally I’ve always used one server for everything which is fine – if its up to it.
• Uninstall any previous version of ESLT.
• Get the folder “Load Testing Services” off the XenApp 6 DVD and copy it to your new Load Testing server.  Or download it from My Citrix.
• Run EdgeSight for Load Testing.msi

• Click next, agree to the EULA and click next again.
• Click Custom (always click custom!)

• Set all three components to install.  The “Web Interface XML Service” allows you to connect to applications more easily without messing about creating ICA files so is good to have.  It doesn’t actually use the Web Interface.
• Enter a really good password

• ESLT WI Support.msi is only needed if you are installing ESLT on a Web Interface server
• After installation, you can make sure the essential service is started – its the Citrix EdgeSight Launcher Service and replaces the Launcher application from previous versions of ESLT

• You can now run the LT Controller from the Start Menu

• You will get a handy message about the changes you should make to your Citrix servers in order to be able to do Load Testing. These are important so don’t ignore them. You might well not want these settings on your live production servers!

• To implement these settings, log on to your XenApp 6 servers and click Start, All Programs, Administrative Tools > Remote Desktop Services and click “Remote Desktop Session Host Configuration”
• Right click ICA-TCP in the middle of the screen and click Properties. You should be able to follow the instructions above on the Sessions and Log on Settings tabs. So this on any server you will use for hosting Load Tested applications.
• After the handy message about server settings, enter your password to get to the main interface.
• The last step is to license it, or you will only have a 30 day 15 user license (which is not of much use). Click the Licenses menu and select License Server Configuration.

You are now ready to do load testing!  The next high level next steps are

• Create usernames to perform the load testing – I would create as many users as you will want to create sessions rather than reuse the same username lots of times.  This also gives you the option to give them mailboxes etc later.  Make your life easy though and make their usernames the same except for an incrementing number on the end and keep the passwords the same too.
• Create a new EdgeSight test and connect it to an application on your farm (using an ICA file is one easy way to do this)
• Enter your usernames and passwords
• Record a script for load testing an application.  Best practice is to have folders for the steps – a load section and a log off section with a folder in between set to “Iterate”.  This section can then repeat infinitely until ESLT starts logging users off.  If you don’t do this a long test will see constant logon/logoff activity which can cripple the test.
• Enter figures for how to load the test.  Usually, this will consist of a “log on” period where all your users log in at a realistic rate, a middle period where load is at maximum (this is a good time to test logon times and application performance of an extra session by logging on manually) and a short log off period where the tests complete and log off again.
• If you’re testing Office 2007, you can download very good sample scripts from here: http://support.citrix.com/article/CTX122568.  They’re worth looking at even if you don’t use them as they’re well done scripts.  They can be adapted to work in Office 2003 and Office 2010 as well.
• Under Display > Counters set up connections to useful perfmon counters on the servers being monitored – such as
  \\servername\Processor(_Total)\% Processor Time
• Run tests – this will spawn many windows on your desktop (only have one user logged on to the launcher server or they can appear on the wrong session).  Usually its best to right click the stack of windows and click “Show Windows Stacked” to display them nicely.
• Record the results – screenshots of the “Monitor” section of Display are best though you can save reports.