ABSTRACT
The following is a detailed summary of why we have seen performance issues in IVR Version 6.0.X.X and how we resolved them in Version 7.0.X.X.
For the technical details around Microsoft Workflow 4, I found a great document that illustrates the improvements from version 3 to 4: http://msdn.microsoft.com/en-us/library/gg281645(v=vs.110).aspx
BACKGROUND INFORMATION
It is important to understand the mechanics of our IVR system before understanding the hard limitations. When a call enters our IVR, the call flow must be compiled for every caller on a port. This will allow the port to obtain the directives on how to handle the call as the port itself only understand basic functionality. It is important to understand that the call flows are always compiled at the completion of the last call, or when the service starts. The port will also not answer the call until the call flow compilation is complete. That is the reason why larger call flows require extended compilation time and in high call volume scenarios with small port sizing, the caller may hear a ringing delay before the IVR answers the call. As the compilation time directly affects the call per hour limit of the IVR port, this is why part of our best practices are to make use of our Subroutine activity along with keeping the call flow size under a manageable level as a preventative measure.
Each possible call flow step or branch is considered a “Path”. Higher path count increases the call flow compilation time. Here’s an example of a simple call flow with an Answer, Menu (with 1 option), Transfer and Hang up activity;
- Answer activity will count as 1 path
- Menu activity will count as 8 paths
- 1 path for the failure branch
- 3 paths for the invalid branch (as it contains the branch, the Play message and a Go To by default)
- 3 paths for the timeout branch (as it contains the branch, the Play message and a Go To by default)
- 1 path for the digit option
- Transfer activity will count as 2 paths (7 paths in version 7.0 with the inclusion of hang up activities in the abandoned/failure branches)
- 1 path for the success branch
- 1 path for the failure branch
- Hang up activity will count as 1 path
Total path count = 12 paths (18 in version 7.0)
As each port can handle 3600 seconds of time per hour, the higher path and port count decreases the amount of call time available per hour. The following chart outlines the number of paths and ports versus the amount of call per hour handle capacity:
The outcome of having high compilations time will reduce the IVR port’s capacity to handle more calls per hour before experiencing delays in the port answering incoming calls. While the usual symptom is the IVR Service restarting, causing an outage, the impact of the performance issues ties to high compilations can be traced in logs and alarms.
The relevant alarms tied to this problem are as follows;
VWM0022 - CallReceivedEvent Queue is filling up
http://micc.mitel.com/kb/KnowledgebaseArticle51648.aspx
When the CallReceivedEvent queue starts backing up, this is an indication that the workflow is taking a long time to be created. This is a result of the XOML/XAML file being too large, normally associated with too many branches within a single call flow. As a consequence, the time to answer will start to increase since the CallReceived events are not processed fast enough.
VWM0024 - Call flow creation exceeded 4 seconds
http://micc.mitel.com/kb/KnowledgebaseArticle51684.aspx
Now that the mechanics have been explained, there are multiple other factors that could affect and improve the IVR performance;
Performance factors affecting IVR Performance
- Number of ports per IVR server
- More ports require increased resources for MiAUDIO, MiTAI and logic processing
- Co-location of other services
- Resource starvation from CCM Enterprise Services, third party applications (Microsoft SQL Server, Microsoft Exchange, etc.)
- Telephone system performance
- Audio quality, MiTAI processing, routing times
- Limit paths per call flow through subroutines
- Limit ports per IVR Server to 60 (MiContact Center Engineering guidelines)
- Off-board IVR services wherever possible, refrain from co-locating with resource heavy services such as Microsoft SQL Server
- Limit IVR ports per 3300 to 120 (Mitel Engineering guidelines)
- Ensure that MiTAI on 3300 is only enabled on required Class of Services
- Load balance calls between IVR Servers by interleaving IVR ports in hunt groups
VERSION 7.0 IMPROVEMENTS
1. Workflow caching
Due to MS Workflow limitations in 6.0, we could not cache the compiled call flows in memory. In version 7.0, the first call will be compiled and cached in memory, therefore, unless changes are made to the call flow, the compilation time will be dramatically reduced.
2. Microsoft Workflow 4
While the Workflow 3 framework was in a single thread environment, compiling/building the XOML (call flow) would be executed on a single thread. Assuming that it takes 1 second to build a call flow, getting 10 calls within the same second would result in the first build to take 1 second to execute, the second will take approximately 2 seconds, third one will take 3-4 seconds, and so on. This has a considerable performance hit as all the work is queued within that single thread. Microsoft re-designed Workflow 4 to offer a multi-threaded environment and offers massive improvements in performance to our version 7.0 release.
3. 64-bit Application
While 6.0 was running as a 32-bit application, we were limited in terms of memory usage. In larger environments/configurations, we are now capable of taking advantage of 64-bit services.
APPLIES TO
7.0, 7.1
Keywords: 7.0 IVR performance improvements best practices