Contact Center Enterprise (CCE) End to End Voice Call Flow Troubleshooting and Configuration - LTRCCT-2051 - Cisco Live
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Contact Center Enterprise (CCE)
End to End Voice Call Flow
Troubleshooting and Configuration
LTRCCT-2051
Taylan Kucuk – Technical Leader [CCBU]
Carles Duz Palau – Technical Leader [CCBU]
Ramiro Amaya – Technical Leader [CX]
1
Abstract
This lab session is an intermediate level session intended for engineers with prior Cisco Contact
Center Enterprise (CCE) experience.
A deep dive interactive explanation of the CCE Voice call flow and component interaction will be
provided. Students will then be taught how to implement and troubleshoot the solution through
hands-on lab exercises. Cisco Contact Center experts will share powerful tools to enhance your
troubleshooting skills.
All the content and labs will be based on the latest 12.5 Release.
2
Abstract .................................................................................................................................................... 2
Lab Topology & Access .............................................................................................................................. 4
Self-Assessment ........................................................................................................................................ 5
Exercise 1 - Troubleshooting and Configuring Ingress Leg (Part 1) ............................................................. 6
Exercise 2 - Troubleshooting and Configuring VRU Leg (Part 2) ............................................................... 23
Exercise 3 – Troubleshooting & Configuring IVR Treatment (Part 3) ........................................................ 44
Exercise 4 - Troubleshooting & Configuring Agent Leg (Part 4) ................................................................ 54
BONUS Exercise 1: Troubleshooting with DB & Useful Queries ............................................................... 70
BONUS Exercise 2: CVPParserG3 ............................................................................................................. 77
BONUS Exercise 3: VVB Cache Management and IIS Content Expiry........................................................ 84
Challenge Labs – Break / Fix .................................................................................................................... 91
3
Lab Topology & Access
Pod Addressing and Credentials
Component Hostname IP Address OS Login Password Web Login Password
DC/DNS/hMail DC 10.10.10.3 CC\Administrator Train1ng!
AgentDesktop1 AGENT1 10.10.10.211 CC\JDoe Train1ng! Jabber: jdoe Train1ng!
AgentDesktop2 AGENT2 10.10.10.212 CC\DMarino Train1ng! Jabber: dmarino Train1ng!
CUCM CUCM 10.10.10.40 administrator Train1ng! Admin Train1ng!
I&MP IMP 10.10.10.41 administrator Train1ng! Admin Train1ng!
PCCE All in 1 PCCEAllin1 10.10.10.10 CC\Administrator Train1ng! CC\Administrator Train1ng!
Finesse A FINESSEA 10.10.10.38 administrator Train1ng! admin Train1ng!
CUIC+IDS+ LD CUIC 10.10.10.35 administrator Train1ng! admin Train1ng!
VVB CVVB 10.10.10.30 administrator Train1ng! admin Train1ng!
CVP CVP 10.10.10.20 CC\Administrator Train1ng! adminstrator Train1ng!123
Note: The lab pod is not configured and sized according to SRND production requirements. It was created using VM clones of a
master pod with the goal of hosting multiple distinct but identical pods. These labs are for instructional purposes only; please do
not consider the deployment model and allocated hardware resources as a reference for a production system.
Agent Information
Agent Name Username Password Agent ID Extension
John Doe jdoe@cc.lab Train1ng! 2001 2001
Dan Marino dmarino Train1ng! 2002 2002
Ray Lewis rlewis@cc.lab Train1ng! 2003 2005
4
Self-Assessment
Objective: Self-assessment of your troubleshooting skills prior to the session
In this lab exercise, you will analyze the pre-collected log files for a failing call scenario.
Consider this a role-based session where you are the TAC engineer and the presenters are
the end customer. For this exercise, the debugging is already set, the issue is reproduced, and
all log files are collected.
Steps
1. Remote Desktop to the CVP Server.
2. Please locate the 66666666 folder on the desktop of CVP Server.
3. You have 5 minutes to find the root cause of the issue.
5
Exercise 1 - Troubleshooting and Configuring Ingress Leg (Part 1)
Objectives
In this lab exercise you will learn how to troubleshoot issues on the ingress leg and also will
learn the related configuration to get the ingress leg up and running.
You will also familiarize yourself with Notepad++ and will learn efficient methods to filter and
search within the logs and easily identify and follow your call in the log files.
Furthermore, you will also use Diagnostic Portico to collect the Router Logs.
Call Flow Details
Contact Center number: 40400
ICM Script: Troubleshoot
Expected behaviour: Welcome Prompt Queue Music Agent
Agent: Jdoe@cc.lab – 2001
Call Flow: Jabber CUCM GW CVP ICM
Steps
1. Log in to the PCCEALLIN1 remote desktop session.
2. Place a new call to 40400 from Jabber. If Jabber is not running, start it from the
Desktop or task bar shortcut .
3. You should hear the prompt “I am sorry we are currently experiencing system problems
and are unable to process your call. Please try again later” and the call drops.
4. As you can see, you did not hear any IVR treatment. So, you can say the call failed in
the early stages of the call flow.
5. As has been mentioned, before enabling any trace or collecting any logs, the best way
to start troubleshooting is to check the ICM script to see where the call fails.
6
6. To do so, open the Script Editor by clicking on the icon on the task bar.
7. Open the Troubleshoot script by clicking File Open and then double-clicking on the
Troubleshoot script.
8. Click on the Monitor icon to start monitoring the script.
9. In monitoring mode, you can see the number of calls hitting each node. This way you
can see which path your call follows and where it fails if there is an issue.
10.Place one more call from Jabber to 40400 and check the script.
11.At this point, you see numbers still showing 0. So, this means this call did not hit the
script. This indicates the call failed in Part1 – Ingress leg.
7
12.Now, let’s collect the logs and try to identify where exactly the call failed. We will not set
any tracing. Instead, we will try to understand how far we can go with default traces. We
will start with CVP and Router logs only.
13.Open the Unified CCE Tools folder on the desktop.
14.Double-click the Diagnostic Framework Portico shortcut.
15.Continue past the certificate warning.
16.Log in details will be prepopulated. Click ok to login.
8
17.In the list of commands on the left-hand side, click on ListTraceFiles.
18.For the Component, select the Router A/rtr. For the ToDate, make sure it is matching
the current time on the PCCEALLIN1 Windows time and then click Submit.
19.You will see that the trace file will be ready in a few seconds.
20.Click on that link. You will see an empty pop-up window open.
21.Wait a few seconds (may take up to a minute) and you will see a notification at the
bottom of the page.
22.Click on the button and select Save As.
9
23.On the left-hand side, under Quick Access, select CVP Desktop shortcut, and click on
Save. This will save the collected router log file in the CVP server’s desktop.
24.Remote desktop to CVP Server. On the desktop, locate the router log zip file you just
saved in the previous step. Right-click and first select 7-Zip and then Extract Here.
25.This will unzip the log files. You will see a new folder called EMSLogFile created on the
desktop.
1026.Enter that folder by double-clicking. Right-click on the Router log file, then select Edit
with Notepad++.
27.This will open the router log file with Notepad++.
28.Next, we will also open the corresponding CVP log file with Notepad++. To do so, click
on the Open button in Notepad++.
1129.CVP log files are stored in C:\Cisco\CVP\logs folder. Navigate to that location. Then
click on the Date Modified column to order the files from newest to oldest. Then select
the latest CVP file on top and click on the Open button.
30.Now, you will have both CVP and Router log files open in Notepad++.
Recap so far:
We have placed a call to 40400 and heard the error prompt. We have collected Router logs
through Diagnostic Portico and saved them on the CVP server desktop. We have opened both
the Router logs and the CVP logs (the latter located in C:\Cisco\CVP\logs folder) with
Notepad++. We are now ready to start analysing the root cause of the problem.
31.Before we start analysing the logs, let’s remember the call flow and the message
exchange for the Ingress Leg.
1232.In summary, GW/CUBE sends SIP INVITE to CVP. CVP then sends a New Call message
to the VRU PG. The message is then forwarded as Route Request to the Router.
33.Now go back to Notepad++ and select the CVP log file.
34.The easiest way to find calls in CVP logs is to search for the string “new call with guid”.
To do this, click on the Search menu and then select Find.
35. Type new call with guid and click on Find All in Current Document.
36.You will see in the bottom part of the Notepad++ that all lines having this string will be
listed. So, you can simply see all calls hitting the CVP server. As this is a lab system and
there are no other calls, you can simply go to the last line and find your call.
1337.Double-click on the last line, and Notepad++ will take you back to the CVP logs and
focus on that line. If you receive a notification that the file has been modified by another
program and if you would like to reload it click on NO.
38.On that line, you can see the message NEW CALL with guid followed by 32
hexadecimal characters. This is the CALLGUID. As explained by the presenter,
CALLGUID is the unique internal ID for this call used by CVP to correlate all messages
related to the call.
CALLGUID = 00B777000001000000000002280A0A0A
39.Double-click on that ID (you will see notepad++ already highlight CALLGUID in all lines).
Next, right-click and select Style token. Click on Using 1st Style.
40.CVP Logs can have many lines if you have a busy Contact Center and receive many
calls. Instead of reading line by line, we will filter the file by searching a special string.
Double-click on the highlighted CallGUID again and click on the menu option Search
Find. Notepad++ will already bring up callguid in the search window.
1441.Type “CALLGUID=“ in front of the ID. Check the Match Case checkbox and click on Find
All in Current Document.
42.Again, you will see that all lines with this string will be listed in the bottom part of the
Notepad++. We will use only these lines to continue troubleshooting.
43.The most important part of troubleshooting is to know which messages we would
normally see if the call had been working correctly. As you recall from the presentation,
once CVP receives the SIP Invite from ingress GW, it sends a NEW_CALL request to the
Router. In return, the Router sends a Temporary Connect message. Scroll right a little bit
and review the selected lines to locate the [NEW_CALL] message.
44.What is the next message you see? As you can see, CVP is receiving a
DIALOG_FAILURE_EVENT from the Router instead of a Temporary Connect message.
45.We can see that the CVP is sending the expected message. However, it does not
receive the expected response. It looks like the issue is not on the CVP side. Instead,
the issue seems to be coming from the Router side.
1546.Next, we will go to the Router logs and see what happens there. Now, how do we
locate our call in the Router logs? We do this by using the Dialog IDs. Check the line
containing NEW_CALL message [Publishing to UCCE]. You will find the DialogID there:
DialogID = 2
47.Open the Router (rtr) logs with Notepad++ and find the call. To find the call in the rtr
logs, you can simply search for the string “( x”, which in the example above is
(2 x. Press CTRL+F, then type the dialogue ID you found in your lab in the previous step
(which is (2 x in our example). Uncheck the Match case checkbox and click on Find All
in Current Document.
48.This again will show, on the bottom part of Notepad++, all the lines including this string.
You can see that the Router receives the NewCall and creates the RouterCallKey and
RouterCallKeyDay.
49.Check the next lines which indicate the root cause.
Conclusion of Troubleshooting Performed
We have identified that the root cause of the issue is that the Router cannot find a dialed
number configured for 40400 and no script is scheduled for the called number 40400.
Remember the required configuration that allows the Ingress Leg to work successfully.
1650.For a call to hit the ICM script, we need to make sure that:
a. The Call Type is configured
b. The Dialed number is configured and associated with the Call Type
c. The Script is configured and Scheduled for the Call Type.
Configuration Steps
51.Remote Desktop to AGENT1. Open Chrome and navigate to the CCE Web
Administration page using the bookmark in the Chrome browser.
52.If prompted, click on Advanced.
1753.Then click on the Proceed to pcceallin1.cc.lab (unsafe).
54.Log in with the Web Admin credentials [administrator@cc.lab – Train1ng!].
55.If you are familiar with earlier versions of PCCE, you will notice the changes in the User
eXperience (UX) on the PCCE Web Admin page. This is a brand-new look and feel
starting with PCCAE 12.0(1).
56.We will first configure Call Type. To do so, click on Call Settings.
1857.Then click on Route Settings.
58.This will open the Route Settings window. Click on the Call Type tab.
59.Click on the New button.
60.Create a new call type with the details shown in the image below and click on the Save
button.
1961.Next, we will configure the Dialed Number. Click on the Dialed Number tab.
62.Click on the New button.
63.Create the Dialed Number with details shown in the image below and click on the Save
button.
64.As a last step, we need to configure the Script and schedule it. As we saw at the
beginning of the exercise, the script has already been created for this session.
65.Log in to PCCEAllin1 remote desktop session and maximize the Script Editor. (This
should already be open with the Troubleshoot script. Reopen it if you have closed it.)
66.Next, we need to schedule this script for a CallType so that when the Router receives a
new call for Dialed number 40400, it can match that call to this script.
2067.In the Script Editor, click on Script Call Type Manager.
68.On the Schedules tab, click on the Call Type dropdown. Select troubleshoot.ct and click
on the button.
69. This will open a new window. Select the Troubleshoot script and click OK.
70.This will schedule the script for the troubleshoot.ct Call Type, which is associated with
Dialed number 40400. Click first on the button for the configuration to take
effect, and then click to close Call Type Manager.
21Summary of Configuration Steps
• We have configured a new Call Type troubleshoot.ct.
• We have configured a new Dialed number 40400 and associated it with
troubleshoot.ct CallType.
• Troubleshoot Script was already created as pre-work.
• We scheduled the Troubleshoot Script for troubleshoot.ct Call Type.
71.According to the information provided by the proctors during this session, Part1 (Ingress
Leg) is completed when the call hits the Start node in the script. Our script should
already be in monitoring mode. Please put it in monitoring mode if this is not the case.
72.On PCCEALLIN1, open Jabber and place a call to 40400.
73.Congratulations!! Now you can hear the Welcome IVR prompt. You can drop this call.
74.Check your script. You will notice that the number on the start node increases to 1 after
a couple of seconds. This indicates our Ingress leg has been successfully completed.
Congratulations: This completes Exercise 1.
Now your proctor will provide more details about the Ingress leg through the
presentation.
22Exercise 2 - Troubleshooting and Configuring VRU Leg (Part 2)
Objectives
In this lab exercise you will learn how to troubleshoot issues on the VRU leg and will learn the
related configuration to get the VRU leg up and running.
You will familiarize yourself with Cisco Log Analysis Visualization Tool (CLAV) which is a
troubleshooting tool comes with CVP installation
Call Flow Details
Contact Center number: 40400
ICM Script: Troubleshoot
Expected behaviour: Welcome Prompt Queue Music Agent
Agent: Jdoe@cc.lab – 2001
Call Flow: Jabber CUCM GW CVP ICM
Steps:
1. Log in to AGENT1 remote desktop.
2. Open the LabExerciseFiles Folder on the desktop.
3. Double-click on the Exercise2-Break.bat file.
234. This will open a command prompt window and execute some commands. The window
will automatically close after execution.
5. Log in to the PCCEALLIN1 remote desktop session.
6. Maximize the Script Editor. Troubleshoot script should be open from previous exercise.
(open it if you have closed). Click on Script Monitor Options Reset Current Script.
This will reset counters of the visited notes. (you need to wait couple of seconds).
7. Place a new call to 40400 from Jabber.
8. You will hear the prompt “I am sorry we are currently experiencing system problems
and are unable to process your call. Please try again later” and the call drops.
249. Wait a few seconds and you will see that the counters in the script increment. Try to find
where the call fails.
10.You can see that the call successfully has hit the script (Part 1 Ingress Leg completed
successfully.) You can also see that it goes through several Set variable nodes. Then, it
reaches Send to VRU node, but does not continue from there. So, this call fails in Send
to VRU node. This indicates that this call failed in Part 2 (VRU Leg). Let’s troubleshoot
and see what went wrong.
11.As explained by the proctor, CVP logs are always the best place to start. Again, we will
not enable any tracing just to see how far we can go with the default tracing and with
CVP Logs only. In addition, instead of Notepad++, this time we will utilize the Cisco Log
Analysis Visualization Tool (CLAV) to analyze the CVP logs.
12.Remote Desktop to CVP and minimize the open programs. (If not closed in previous
exercise, you should still have the notepad++ open.
13.On the CVP Server, click on windows start button.
14. Type the Cisco log and click on the Cisco Log Analysis Visualization Tool icon shown
under the search field in the top right-hand corner.
2515.Once the program opens, click on Add File(s) button.
16.Navigate to the C:\Cisco\CVP\logs folder. Click on the rightmost top icon and select
Details.
17.Order the files by date by double-clicking on the button. Select the latest
CVP log file on top and double-click on it.
2618.Selected log file is now added to the Call Flow Tool. Click on the
button.
19.You will see that a graph with message flows for different calls is created on the right-
hand side.
20.Default CVP traces don’t include any SIP tracing. Since we haven’t enabled tracing, we
will not see any SIP messages in the logs. Therefore, the first message we can see for
this call will be the NEW_CALL message from SIP Subsystem to the ICM Subsystem,
and then from ICM subsystem to UCCE.
21.As we are only interested in the latest call, scroll down and find that call. You can also
use the Filter Results left panel. Select only the latest GUCID in the list, then click the
Filter Results. Note that each call will be presented with a different color, which makes it
easier for you to find messages for the latest call. (Maximize the Tool to have better
visibility on all messages.) Please note that the order of the components and colors
might be different for your call. Locate the NEW_CALL messages from SIP_SS to
ICM_SS and from ICM_SS to UCCE.
2722.Now let’s check what happens after the ingress leg. Recall the presentation and the
messages for the VRU Leg.
23.We expect the following message exchanges to happen. Find these messages for your
call in the Call Flow Tool.
2824.Router Sends Temporary_Connect message to CVP (From UCCE to ICM Sub System).
25.Next, ICM_SS will send a Connect message to SIP_SS. This should be followed by an
INVITE from SIP_SS to VVB. (As explained above, we will not see the SIP invite with
default CVP tracing.)
26.If the SIP communication between CVP and VVB is successful, we should see a
NEW_CALL message from VVB to CVP. However, checking the messages, after the
CONNECT message we see an EVENT_REPORT message (EVENT_ID:
CONNET_FAILURE) followed by DIALOG_FAILURE_EVENT message.
2927.At this stage, it looks like there is an issue on the CVP side, as we do not see the
expected message in CVP. Can we continue our troubleshooting without enabling SIP
tracing or is it time to enable more tracing?
28.Let’s check the content of the DIALOG_FAILURE_EVENT message. Hover the mouse
over the DIALOG_FAILURE_EVENT message. This will show the content of the
message.
29.There is not much information here either. The next step, we need to see log files to
understand the root cause. We are interested in the period from the last good message
(TEMPORARY_CONNECT and CONNECT) to the first bad message we can identify
(DIALOG_FAILURE_EVENT).
3030.Hover the mouse over the CONNECT message and click on the See message in the log
file link.
31.This will open a new tab with the log file and will highlight the lines including this
CONNECT message.
32.Scroll right and locate these messages in the log.
33.Something must have gone wrong between these messages. Check those lines and try
to identify the issue.
34.We clearly see the root cause of the issue. CVP does not have a route to direct the call
for the number 88888888881001. (From the presentation, we know that the
8888888888 is the Network VRU label and 1001 is the correlation ID.)
35.Remember the required configuration from the presentation.
3136.Looks like we are missing the route for the VRU label (8888888888).
37.As you can see here, without enabling any tracing, and only with CVP Logs, we were
able to narrow down the issue and find the potential root cause.
38.In UCCE Releases and in the PCCE releases prior to 12.0, CVP Routes are configured
under the OAMP through System Dialed number Pattern configuration. Starting with
PCCE 12.0, now all CVP configuration is done through the PCCE Web Admin page.
Let’s check the configuration to see what the problem is with this route
(88888888881001).
39.Remote Desktop to AGENT1 and maximize Chrome. CCE Web Administration page
should still be open from the previous exercise. (If you have closed it, open again
through the CCE bookmark and log in with the Web login credentials provided
[Administrator@cc.lab, Train1ng!].
40.Click on the Overview button on the left-hand side. This will take to the home page.
3241.Click on Call Settings.
42.Then click on Route Settings.
43.On the top right-hand side, click on the Routing Pattern tab.
44.We can see that different routes are configured. But we cannot find any routing pattern
matching 88888888881001 in order to route calls.
45.To add this route, click on the button.
3346.As shown below, fill the Routing Pattern as 888*. For Pattern Type select VRU and for
the Destination select cvvb.cc.lab SIP server group. (This Server group has our VVB
configured inside). Once completed, click on the Save button.
47.So far, we have identified that the route for the Network VRU label was missing in the
CVP configuration. Therefore, CVP did not know where to send the invite when it
received the Network VRU Label. We have now added this route. Let’s make a new test
call and see if it works now.
48.Log in to PCCEALLIN1 and place a new call to 40400 from Jabber.
49.You will hear the prompt “I am sorry we are currently experiencing system problems
and are unable to process your call. Please try again later” and the call drops.
50.Looks like there are still some other issues in the system. Go back to the CVP Server.
CLAV tool should still be open.
51.In the New Search Tab, select the already added log file and click on Remove File(s)
button.
3452.Now let’s add the latest log file. To do so, click on Add File(s) button.
53.Select the latest CVP log file and double-click on it.
54.Selected log file is now added to the Call Flow Tool. Click on the
button and you will see message flows of the calls on the right-hand side.
55.As we are only interested in the latest call, scroll down and find that call. (Maximize the
Tool to have better visibility on all messages.) Please note that the order of the
components and colors might be different for your call. Locate the NEW_CALL
messages from SIP_SS to ICM_SS and from ICM_SS to UCCE.
3556.We expect the following message exchanges to happen. Find these messages for your
call in the Call Flow Tool.
I. Router Sends Temporary_Connect message to CVP (to ICM Sub System).
II. Next, ICM_SS will send a Connect message to SIP_SS and this should be followed
by an INVITE from SIP_SS to VVB. (But bear in mind we haven’t enabled yet SIP
tracing. Therefore, we will not see the SIP invite.)
36III. If the SIP communication between CVP and VVB is successful, we should see a
NEW_CALL message from VVB to CVP. Although we added the route for VRU Label,
we still see a DIALOG_FAILURE_EVENT instead of a NEW_CALL message.
57.Let’s check the content of the DIALOG_FAILURE_EVENT message to see if there is any
difference. Hover the mouse over the DIALOG_FAILURE_EVENT message. This will
show the content of the message.
3758.There is not much information here either. As before, we need to see the log files to
understand the root cause. Again, we are interested in the period from the last good
message (TEMPORARY_CONNECT and CONNECT) to the first bad message we can
identify (DIALOG_FAILURE_EVENT).
59.Hover the mouse over the CONNECT message and click on the See message in the log
file link.
60.This will open a new tab with a log file and will highlight the lines including this
CONNECT message.
61.Scroll right and locate these messages in the log, as we did before.
3862.Without any detailed analysis or deep dive, if you compare these logs with the ones
from step 32, we can see that adding the route for the Network VRU Label 888*
changed something in the logs.
63.Let’s have a closer look and see what happens after the Connect message. As you can
see, after adding the route for the Network VRU Label, now CVP knows where to route
this call and we can see this clearly in the logs.
64.So far everything looks good; continue checking the logs. Next you will see an outgoing
INVITE message with Network VRU Label + Correlation ID with destination cvvb.cc.lab
SIP Server group (VVB). As mentioned before, with default tracing you will not see the
content of the SIP messages, but you can still see the information about them and can
follow the call flow.
65.The LEGID highlighted in the above screenshot is the SIP CALLID for the VRU leg.
66.By following, the messages with this LEGID, you can also have an idea of what
happened on the SIP layer to the VRU leg. Below are the messages seen with this
LEGID. Find them for your call.
67.When you check the next message with this LEGID, you will see REJECTED WITH 404 -
Not Found Reason: Q.850; cause=1 message. [you need to scroll right to see the
complete message]
68.At this stage, we know that ICM sends the Temporary_Connect to CVP, CVP sends SIP
invite to VVB (after we add the route for Network VRU Label 888*). But Looks like VVB
does not know what to do with this and returns 404 Not Found message.
69.Next, we need to check VVB and find what is root cause for this. But before this, let’s
remember the required configuration on the VVB.
3970.In UCCE Releases and in the PCCE releases prior to 12.0, VVB SIP triggers are
configured under the VVB Administration page through SubSystems SIP Telephony
SIP Triggers configuration. However, starting with PCCE 12.0, now all VVB
configuration is done through PCCE Web Admin page.
71.Remote Desktop to AGENT1. You should still have the CCE Web Administration page
open in chrome. (reopen if you have closed it.)
72.Click on the Overview button on the left-hand side. This will take to the home page.
73.Click on Infrastructure Settings.
4074.Then click on Device Configuration.
75.This will open the Device Configuration window. From the left-hand side, select
Virtualized Voice Browser. Then click on the Applications & Triggers tab to see the
configured triggers for the applications.
76.For VVB to establish the VRU leg, it needs to run the Comprehensive application (This
corresponds to Bootstrap.tcl in VXML GW). In our case, we see that an invite with
Network VRU label (888*) reaches the VVB. However, VVB does not trigger the
41Comprehensive application and does not even have a target specified for this number.
Therefore, VVB returns 404 back to CVP.
77.Once the Comprehensive application is selected, we can only see one trigger
configured and this is 77777777*.
78.Let’s configure a new trigger with our Network VRU label (888*) for the Comprehensive
application. To do so, click on the button next to Configured Triggers. This will open
a small window. Type 888* and click on the Add button.
79.You will see that the new trigger has been added. Click on the Save button to save this
configuration.
80.Now that we have created our SIP trigger as well, let’s place another call and see if it
works.
4281.Log in to PCCEALLIN1 and place a new call to 40400 from Jabber. Ypu can drop the
call once you hear the Welcome Prompt.
82.Congratulations! You have just fixed the VRU Leg of your call flow.
Summary of the Exercise
1. As our setup is PCCE, it already comes with a preconfigured Network VRU label for
CVP. This label is 7777777777. We also had CVP Dialed Number 777* pointing to VVB,
and SIP Trigger 7777777777* configured in VVB. So, our setup was configured
correctly.
2. What we did in Steps 3 was actually to utilize PCCE REST APIs in order to change the
Network VRU Label from 7777777777 to 8888888888 by executing a batch file.
3. Because of this change, for the next call, the Router sent 8888888888 as Network VRU
Label in the Temporary_Connect message.
4. However, CVP did not have a route for 8888888888 and we could see that CVP did not
know what to do with this call. CVP then replied with DIALOG_FAILURE message to the
Router and the call dropped.
5. Next, we added Routing Pattern 888* in PCCE configuration and pointed it to VVB. So
now CVP knows that when it gets this label, it should route the call to VVB.
6. After this step, our call still failed, but this time, we saw that CVP, after receiving
Temporary_Connect message, sent the SIP invite to VVB. However, VVB returned 404
Not Found.
7. For the VRU Leg to work, VVB needs to have a SIP Trigger configured which is
associated with the Comprehensive application.
8. After we configured the SIP trigger 8888888888 and associated it with the
Comprehensive application, our call worked.
Congratulations: This completes Exercise 2.
Now your proctor will provide more details about VRU leg through the presentation.
43Exercise 3 – Troubleshooting & Configuring IVR Treatment (Part 3)
Objectives
In this lab exercise you will learn how to troubleshoot issues on the IVR Treatment and will
learn the related configuration to fix the IVR related issues.
You will familiarize yourself with Wiresharkk tool and learn which traffic and protocol is
important in the IVR treatment.
Furthermore, you will also get details on VVB cache management and IIS Management.
Call Flow Details
Contact Center number: 40400
ICM Script: Troubleshoot
Expected behaviour: Welcome Prompt Queue Music Agent
Agent: Jdoe@cc.lab – 2001
Call Flow: Jabber CUCM GW CVP ICM
Steps:
1. Switch to the PCCEALLIN1 remote desktop session.
2. Maximize the Script editor. You should still have the Troubleshoot script open. Click on
Script Monitor Options Reset Current Script. This will reset counters of the visited
nodes.
443. Place a new call to 40400 from Jabber.
4. You will hear the awesome welcome prompt and then the menu options. Press 1 on
your keyboard.
5. You will hear a prompt saying “one” followed by another short prompt, but then you will
hear the error prompt “I am sorry we are currently experiencing system problems and
are unable to process your call. Please try again later.”
6. Let’s have a look at the script and identify where the call fails. We see that the script has
followed the upper path (option 1 in menu) as no agent is available. Then, continue to
the Run External Script to play queue music, but we see it has followed the failure path
from there. So it looks like our call failed at Run External Script.
7. We have seen in the presentation what happens during a Run External Script request.
Let’s look at the slides again.
458. As you can see ICM sends the details about the script to CVP and the rest is HTTP
communication between VVB and CVP.
9. We will place the call again, but this time open Wireshark and sniff network traffic in the
background.
10. Log in to the CVP server and close all open applications from the previous exercise.
Open Wireshark by double-clicking the wireshark icon on the desktop.
11.If you receive any notification on Wireshark update. Just click on Remind me later
Button. Wireshark has already been upgraded the week before CiscoLive.
4612.Click on Button to start the capture.
13.Switch back to the PCCEALLIN1 and place a call to 40400. Select option 1 and drop the
call once you begin to hear the error prompt.
14.Go back to the CVP server and stop the sniffer by clicking on the button .
15.Wireshark can filter the messages and show you only the ones you are interested in. As
in this case, we are only interested in the HTTP traffic, so you can filter the HTTP
messages only.
16.To do this, type http on the Display filter text box hit Enter.
17.As you can see, Wireshark now shows you only the HTTP messages.
18.You can see that VVB (10.10.10.30) is sending an HTTP GET request to CVP
(10.10.10.20) for /en-us/app/QueuMusic.wav and in return receives a 404 No Found
Response.
Conclusion of Troubleshooting Performed
We identified that the root cause of the issue is that CVP cannot find the requested file.
Side Question: Why do we see the HTTP GET request for QueuMusic.wav but not for the other
prompts played?
Starting with CVP 11.6 ES84 (and CVP 12.0). VVB will not send anymore GET request for the
files there are already cached, and Max-age is not reached. We created a bonus exercise for
you to see HTTP requests and learn more about VVB and IIS content / Cache creation. After
47you complete all core Exercises in this lab, you can go through that Exercise or keep it for
reference for future.
Required Configuration
19.We know that VVB is requesting the QueuMusic.wav file located under the /en-us/app
folder and CVP replies with 404 Not Found. Let’s go to the CVP server and check if this
file really exists and is located there, but where is this /en-us/app folder located? Where
do we define this?
20.To identify the prompt location, we need to know:
a. The Media Server
b. Root folder location
21.You can identify the Media Server in two ways. The first way is to configure a
user.microapp.media_server set variable in the script. See the example below: [just for
reference, no configuration needed.]
22.This node would be seen in the script.
23.Now log in to PCCEALLIN1 and within the Script Editor, check if you have any set
Variable node within the Troubleshoot Script to set the Media Server.
24.We do not have this node in our script. If we do not have this node, how do we know
which is the Media Server in this case? In our lab, this is done by the feature Default
Media Server. This is a global setting and can be specified in the PCCE Web Admin
page under Device Configuration CVP IVR (in earlier versions or also with UCCE
4812.0 this is under OAMP Media Server configuration). So, the second way is to
configure a default Media Server.
25.Remote Desktop to AGENT1. You should still have the Chrome open with CCE Web
Administration page (open again if you closed or session timed out.)
26.Click on the Overview button on the Left-hand side. This will take to the home page.
27.Click on Infrastructure Settings.
28.Then click on Device Configuration.
29.Select the CVP Server. Then click on the IVR Tab and at the bottom you will see the
Default Media Server configuration.
4930.The default media server is used by the micro-applications if the ECC variable
user.microapp.media_server is missing or empty in the Unified ICM script.
31.Now we know the server holding the Media files is the same CVP Server. Our next task
is to identify the root folder for the media files.
32.What we call the Media Server is nothing more than a web server running Microsoft
Internet Information Services (IIS) and the default root folder for IIS is
C:\inetpub\wwwroot folder.
33.Log in to the CVP Server, which in our case is also acting as a Media Server. Open
Windows Explorer and navigate to C:\inetpub\wwwroot.
34.Now remember the HTTP Request our VVB sent.
35.Let’s navigate to en-us/app folder and check if you have the QueuMusic.wav file there.
Click on the Name tab to order files by name. Then scroll down to locate filenames
starting with Q.
5036.We see several files with similar names but note that none of the files match the exact
name we are looking for. [The exact name is QueuMusic.wav.]
37.It looks like somebody typed the wrong name when saving the prompt on the Media
Server. You have two options to fix this configuration mistake:
• Rename the file Rename QueuMusicTroubleshoot.wav to QueuMusic.wav.
• Rename the Network VRU Script to match the prompt name.
38.Here we will go with the first of these two options. Rename the prompt
QueuMusicTroubleshoot.wav as QueuMusic.wav.
39.Log back in to PCCEALLIN1 and place a call to 40400 again. When you hear the menu
prompt, press 1.
40.Congratulations! Now you can hear the Queue Music. This completes this exercise. You
can now drop this call.
Additional Information [just for reference, no configuration needed]
This part explains how you can add/remove/modify the Network VRU Scripts.
• On the Agent1, on the CCE Web Administration page, click on Overview.
• Click on Call Settings.
51• Then click on IVR Settings.
• Here you can see all Network VRU Scripts. The relevant one for this exercise is
highlighted.
Summary of Exercise 3
• We have monitored the script and noticed that the call is failing to play queue music at
Run External Script node.
• We know that ICM sends the details about the script to CVP and CVP passes this,
together with the right template, on to VVB. The rest is HTTP communication between
VVB and CVP.
52• We started a Wireshark trace, captured the call, and then filtered the sniffer to see only
HTTP messages.
• We noticed that VVB was requesting a QueuMusic.wav file and CVP replied with 404
Not Found. This was the root cause of the issue.
• Next, we identified that we are using a default Media Server which is 10.10.10.20.
• Because our Media Server is an IIS server, the root folder for prompts is
C:\inetpub\wwwroot.
• We then tried to locate QueuMusic.wav file under C:\inetpub\wwwroot\en-us\app\
folder, but we noticed there was no QueuMusic.wav file. Instead we found
QueuMusicTroubleshoot.wav.
• At this step, there were two possible solutions:
o Rename the prompt
o Modify the Network VRU script
• We modified the prompt name, and this fixed the issue.
Congratulations!! This completes Exercise 3.
Please now continue with the presentation. Your proctor will provide you with further details
and tips about the IVR treatment (Part 3).
53Exercise 4 - Troubleshooting & Configuring Agent Leg (Part 4)
Objectives
In this lab exercise you will learn how to troubleshoot issues on the Agent Leg and will learn
the related configuration to fix the Agent Leg.
You will again utilize the Wireshark tool. However, totally different feature of Wireshark to
troubleshoot the SIP signaling.
Call Flow Details
Contact Center number: 40400
ICM Script: Troubleshoot
Expected behaviour: Welcome Prompt Queue Music Agent
Agent: Jdoe@cc.lab – 2001
Call Flow: Jabber CUCM GW CVP ICM
Steps:
1. Log in to AGENT1 remote desktop session and close all open applications from previous
exercises.
2. Open the LabExerciseFiles folder on the desktop.
3. Double-click on the Exercise4-Break.bat file.
544. This opens a command prompt window and executes some commands. The window
automatically closes after execution.
5. Log in to AGENT2 remote desktop session. Make sure Jabber is registered.
6. Open Firefox browser and navigate to Finesse Desktop through bookmarks. If you
receive certificate warning, click on Advanced and then click on Accept the Risk and
Continue.
7. Log in agent dmarino with the following credentials and click on Remember Password
when asked [dmarino, Train1ng!, 2002].
558. Keep the status of the agent as Not Ready.
9. Log in to the PCCEALLIN1 remote desktop session and place a call to 40400.
10.You will hear the Welcome prompt and then the Menu. When prompted, press 2. As
the agent is in Not Ready state, you will be placed in queue and you will begin to hear
“special” Queue Music.
11.On Agent2, Finesse Desktop makes the agent Ready by clicking on Not Ready button
and selecting Ready.
12.You will see the agent status changed to Reserved.
13.Then you will hear the error prompt “I am sorry we are currently experiencing system
problems and are unable to process your call. Please try again later.” You can now drop
the call.
14.The agent is shown as Reserved but does not receive the call. Our call is failing on the
Agent Leg. Let’s remember what happens on this leg of the call:
5615.Once the agent becomes available, the Router sends the agent extension as a label to
CVP with the CONNECT message. At the same time, it sends a DEVICE TARGET
PRECALL INDICATION message to the Agent PG to reserve the agent.
16.Our agent has been reserved, so this means the communication between the Router,
the Agent PG and Finesse is working as it should. So, we do not need to check this part.
17.Now, let’s focus on the other part to see what is happening there. First CVP will
disconnect the VRU leg and send an invitation to VVB to play the RingBack Tone.
5718.Remember from your call, when the agent goes READY, the agent gets Reserved.
19.However, in the previous call we did not hear the agent’s phone ring on the Agent
Desktop. Although the agent got reserved, Jabber phone did not ring. This indicates,
that there must be an issue with the call signaling. Let’s go back to the next steps on
the Agent Leg.
5820.At this stage, CVP sends an invite to CUCM in order to establish the Agent Leg. Then
CUCM sends SIP messages to connect this call to the agent phone. In our case, the
agent phone is not ringing. There are two possible reasons for this:
a. CUCM has not received an invite from CVP.
b. CUCM has received the invite but cannot connect the call to the agent phone.
21.Let’s check first if the CUCM receives the SIP invite. For this step, you are free to use
the troubleshooting tool you like best. (Notepad++, Wireshark, CLAV tool). Below you
will see an analysis of this troubleshooting step using Wireshark.
Analysis with WireShark
22.Let the agent Dmarino in Ready state. (we know there is no issue with IVR treatment or
the agent reservation, so we do not need to troubleshoot those parts).
23.Log in to CVP Server. Maximize the Wireshark and click on the button. (Wireshark
should still be running from the previous exercise.)
24.Click on Continue Without Saving.
Note: If the Wireshark was closed in the previous example, reopen it and double click on the
25.Remove the filters by deleting it and press Enter (http filter from previous exercise
should still be there.)
26.Log in to PCCEALLIN1 and place a call to 40400 again. You will again hear the welcome
prompt and then Menu. Press 2 on the keyboard again. You will hear the Error prompt.
You can now drop this call.
5927.Log back in to CVP Server and stop Wireshark by clicking the stop button.
28.In the previous exercise we used http filters to see the http traffic. This time we are
interested in the SIP communication and will utilize another feature of Wireshark. From
the Wireshark User Interface, click on Telephony VOIP Calls.
29.You will see 4 Call Legs.
30.Let’s remember the SIP call legs in a Comprehensive Call Flow.
• Ingress Leg
• VRU Leg
• RingBack Leg
• Agent Leg
31.Pay attention to State Column for those legs. Do you notice the REJECTED message for
the Agent Leg?
32.Select only that leg and click on the Flow Sequence button in order to generate the flow
diagram. (IF you would like to see all SIP messages for this call, you can select all legs
and click on the Flow Sequence button.)
33.As you can see here, CVP (10.10.10.20) sends an INVITE to CUCM (10.10.10.40). but
receives a 404 Not Found Message
60Conclusion of Troubleshooting Performed
We can see that CVP (10.10.10.20) sent the invite to CUCM (10.10.10.40), but CUCM replies
with a 404 Not Found message back. This indicates that CUCM cannot route this call to the
extension 2002.
Required Configuration
1. Let’s remember the required configuration for Agent Leg to work.
2. As we isolated the issue on CUCM side, we can check the Agent DN, SIP Trunk and
Calling Search Space (CSS)/ Partitions. Since our Agent can log in to Jabber phone with
DN 2002 and can log in to the Finesse, we do not expect any issues on the phone
configurations.
3. Next Let’s check the Partitions and the Calling Search Spaces.
4. On AGENT1 open the chrome and navigate to CUCM CUCM Administration.
5. If you receive certificate warning, Click on Advanced and then click on the Proceed to
cucm.cc.lab (unsafe) link.
6. Log in with [admin, Train1ng!] credentials.
7. Navigate to Device Phone.
618. Select Directory Number in the search criteria and type 2002. Then, click on the Find
button.
9. Now you see the phone for the agent Dmarino listed.
10.Click on it to see the configuration of the phone. On the right-hand side, you see the
phone configuration and on the left-hand side you can see the line configuration.
11.We can see that 2002 DN is in troubleshoot partition. Click on the Line configuration to
open the Line configuration page.
6212.Here we can confirm the partition is troubleshoot, and by clicking on it we can see other
available partitions.
13.For this call to be connected to the Agent, the SIP trunk between CVP and CUCM
should be able to call this phone. And for this to happen, the Calling Search Space of
the SIP Trunk should include the partition of this DN.
When a calling search space is assigned to a device, the list of partitions in the calling search
space comprises only the partitions that the device can reach. All other DNs that are in
partitions that are not in the device calling search space receive a busy signal.
14.Navigate to Device Trunk and click on the Find button.
15.Here we see several trunks configured. The one we are interested in is the
sipTrunkCVP.
16.Click on it to see the configuration. Here we are interested in the Calling Search Space
of this trunk. The Calling Search of this trunk should include the troubleshoot partition so
that this trunk can place a call to the Agent extension 2002.
17.Scroll down to locate the Inbound Calls section and check the Calling Search Space of
the trunk.
6318.As you can see here, the trunk has None Calling Search Space. This is basically the
default calling search space and does not include any partition. This explains why the
call from CVP to Agent could not connect.
19.To fix this problem, we will change the partition of 2002 and set it to None.
(Alternatively, you can create a new CSS and add the troubleshoot partition in it and
associate this new CSS to trunk.)
20.Navigate back to Device phone.
21.Click on the phone for the agent Dmarino.
22.Click on the Line configuration to open the Line configuration page.
.
23.Change the Route Partition to < None >.
6424.Click anywhere on the page for the new configuration to load. And you will see at the
top of the page a notification that Directory Number configuration has been refreshed.
25.Click on the button and then on the button.
26.Click on the OK button in the pop-up window.
27.Now go back to Agent2 Finesse Desktop. You will see Dmarino is signed out. Sign in
again with agent dmarino [dmarino, Train1ng!, 2002] and set it to Ready.
28.Now go back to the PCCEALLIN1 and place a call to 40400. You will again hear the
welcome prompt and then the Menu. Press 2 to select option 2. Navigate back to
Agent2 and you will see your agent being reserved.
6529.At the same time, you will also see that the call is delivered to the agent and the phone
is ringing.
30.Click on the button.
31. You will see the call is connected between the agent and the customer. The agent is
now in Talking State.
32.Congratulations!! With all the troubleshooting steps and configurations, you have
successfully deployed the CVP comprehensive call flow.
Summary of the Exercise 4
• We notice that the agent is being reserved but has not received the call.
• This tells us that Part1, Part2 and Part3 were successful. Therefore, we only need to
focus on Part 4 (Agent Leg).
• In the agent leg, since the agent has been reserved, we do not need to check anything
on Agent PG. if something were wrong on Agent PG, the agent would not have been
reserved.
• So, we conclude that either CUCM has not receive the invite from CVP or CUCM has
received the Invite but cannot connect the call to the agent’s phone.
66• Upon checking the Wireshark traces, we notice that invite has been sent to CUCM, but
CUCM has replied with 404 Not Found.
• We then check the partition configuration of the agent’s phone and determine that the
partition is troubleshoot.
• Next, we check the SIP trunk which is used to route calls between CVP and CUCM and
we determine that the Calling Search Space (CSS) of this trunk is set to None.
• The issue here is that the None CSS does not include the troubleshoot partition and
therefore the call does not connect.
• Once we set the partition of 2002 to the issue is solved.
Congratulations!! This completes Exercise 4.
Please follow the presentation now. Your proctor will provide more details and tips on the
Agent Leg.
67Complete!
Congratulations! You have successfully completed the LTRCCT-2051
We hope you have enjoyed the session.
Please let us know if anything could be improved, or if you would just like to discuss something
in more detail.
We are also available Through Webex Teams after the event!
We take great pride in these labs and appreciate your feedback in the session survey!!
Taylan, Carles, Ramiro
We also created additional Bonus Exercises For you!
Feel free to go through it if you still have time now or use it as a reference.
6869
BONUS Exercise 1: Troubleshooting with DB & Useful Queries
Objectives
In this lab exercise you will learn how to troubleshoot issues with the Database.
- Initially you will learn the two most important tables and understand how to identify
your call.
- Subsequently, you will be provided 2 custom Queries written by us to get very
detailed information from the Database.
Basics:
Steps:
1. Remote Desktop to PCCEALLIN1.
2. Open Microsoft SQL Server through the shortcut in the taskbar.
3. Click on Connect Button.
4. Click on the New Query Button.
5. Select the AWDB as target DB for our query.
706. This will open a new tab blank tab where we can prepare our queries.
7. As a first step we will query the Route_Call Detail Table. We have been placing several
calls during this lab session and let’s have a look to Database what has been recorded
for those calls. Copy and paste the below query to SQL Management Studio This will
return all calls you have placed today during the lab session.
select * from Route_Call_Detail where ANI='1001' and DateTime >= DATEADD(day, -1,
GETDATE())
8. Click on Execute Button.
9. This will bring all the calls made today from the ANI 1001 and list them in the bottom
part of the SQL Management Studio.
10.As we learned today in the session, each call on the CCE side has a unique identifier
which is RouterCallKeyDay & RouterCallKey Combination.
11.Once we identify the RouterCallKeyDay & RouterCallKey then we can simply query our
Database with these values instead of DateTime and ANI information. Let’s pick up the
Last call in the list and use the information.
7112.Please modify your query now with the RouterCallKeyDay and RouterCallKey values and
run again. (Copy paste may not work as your values might be different than the Lab
guide).
select * from Route_Call_Detail where RouterCallKeyDay = 153054 and RouterCallKey =
312
13.This will return only the data for our call
14.Go through the results and see what you can gather from this query result. Definitions of
all the fields for all tables are published in the Database Schema Handbook (At the time
of this lab guide prepared Database Schema Handbook for v12.5 was not yet available.
Hence, we are referring to 12.0 guide. However, concerning this exercise, there won’t
be any changes and same document can be referred to.)
15.Now let’s use the same RouterCallKeyDay and RouterCallKey details and let’s query the
Termination_Call_Detail Table.
select * from Termination_Call_Detail where RouterCallKeyDay = 153054 and
RouterCallKey = 312
16.You can simply type the new query under the existing one and Execute them together.
17.You will see that the results are listed again in the bottom.
7218.As you can see here, for this call, we had only one entry in Route_Call_Detail table,
However, we have 3 entries in the Termination_Call_Detail table. How do you explain
this? Review the content of the Termination_Call_Detail and try to answer question.
Expert Queries:
Steps:
1. On the SQL Management Studio Click on Open Button.
2. Navigate to Desktop LabExerciseFiles Folder and open the SQLQuery-RCD.sql file.
3. This will open a new tab and load our custom SQL query.
734. Only part you need to modify is the RouterCallKeyDay and RouterCallKey values. These
values are defined as local variable within the Query.
5. Locate the ---- Setting Values section on the top of the query and modify the
@RCKDay and @RCK values.
Before:
After: (your values might be different than lab guide.)
6. Click on Execute Button to run the query.
747. Now have a look to the results. With this query you can get many information about your
call. Here are some examples:
- Time of the call
- Calling Number
- Called number
- From Which CVP the call arrived (Routing Client)
- BeginCallType and Begin Script
- RouteName (skill group)
- Label (Agent DN in our case)
And Many more…
8. Now let’s open the 2nd custom query. Click on Open and this time select the SQLQuery-
TCD.sql file.
9. modify the @RCKDay and @RCK values like we did before.
10.Click on Execute Button to run the query.
7511. With this query now, we can see that for each leg of this call we have a new entry in
the database. Checking the DNIS value, we can see that the first row is for ingress leg,
2nd row is for the VRU leg and finally the 3rd row is the Agent leg.
12.Review the outputs of both queries to see what all information you can gather from the
database without even checking the logs.
13.The proctors will share the Queries with you.
76BONUS Exercise 2: CVPParserG3 Exclusive
This tool was developed by Ricardo Mancera (Cisco CX) and is used internally by
Cisco TAC Teams to analyze UCCE Calls. It is not a Cisco supported tool, but with this
exercise you will have the chance to become familiar with it. You can take it with you today
as an exclusive for this Cisco Live.
Disclaimer:
License
1. Ricardo Mancera grants you a revocable, nonexclusive, nontransferable, limited license to
download, install and use the Application solely for your personal, noncommercial purposes
strictly in accordance with the terms of this Agreement.
2. Title, copyright, intellectual property rights and distribution rights of the Software remain
exclusively with the Vendor. Intellectual property rights include the look and feel of the
Software.
3. This Agreement constitutes a license for use only and is not in any way a transfer of
ownership rights to the Software.
4. Failure to comply with any of the terms under the License section will be considered a
material breach of this Agreement.
Restrictions
You agree not to, and you will not permit others to:
a) License, sell, rent, lease, assign, distribute, transmit, host, outsource, disclose or otherwise
commercially exploit the Application or make the Application available to any third party.
b) The Software may not be modified, reverse-engineered, or de-compiled in any manner
through current or future available technologies.
Modifications to Application
Ricardo Mancera reserves the right to modify, suspend or discontinue, temporarily or
permanently, the Application or any service to which it connects, with or without notice and
without liability to you.
Additional Clause
This software is not supported in any form or fashion by Cisco Systems.
77You can also read
