RGS routes calls to Busy agents (alredy In a Call)...

Hi guys,

our Call Centre agents are reporting they get 2nd incoming call transferred via the same workflow as the 1st call.
Their status is In a Call and obviously call should not be routed to them.

Screenshot
http://img855.imageshack.us/img855/8094/rgsbug.jpg

Apparently this happens when there are a whole bunch of incoming calls.

Do you know what is the coded logic for RGS quering agents status?
Does it query RTC db for agents status?

My guess is there's some OCS server component underperforming and RGS application queries something (db?) to find this particular agent status is Available (not updated yet to In A Call).

OCS 2007 R2 Front-End is running on 3.5.6907.221 updates (RTCC/3.5.0.0 Response_Group_Service)

Mediation Server
RTCC/3.5.0.0 MediationServer/3.5.6907.196

thanks for any hint

 

November 24th, 2011 1:39pm

Hi plug,

A workflow defines the behavior of a call from the time the phone rings to the time somebody answers the call. The workflow can include queue and routing information, as well as interactive voice response.

For detail, see this document Creating Workflows. Hope it’s useful.

Free Windows Admin Tool Kit Click here and download it now
November 28th, 2011 6:09am

thx, but I already got this workflow configured.

Let me reword the scenario:
- the Agent gets an incoming call notification transferred via the workflow
- the Agent answers the call
- Agent see his/her presence updated to "In a Call"
- there is ANOTHER incoming call notification transferred via the very same workflow from other Caller

This should never happen as RGS doesn't route call to Busy agents.
Unless it doesn't know Agent status is not "Avaliable" anymore..

 

November 28th, 2011 4:16pm

 

plug n pray,

 

"Match Making" component of RGS queries database for Agent's presence State.

It is possible that it may not be getting the updated State because of performance issues.

You can use  RGS performace Counters to monitor the activity.

"LC:RGS - 01 -Response Group Service Match Making"\"Total Number of presence notifications received"

"LC:RGS - 01 -Response Group Service Match Making"\"Total Number of presence subscriptions terminated"

 

Also please review the capacity of your server

http://technet.microsoft.com/en-us/library/dd425159(office.13).aspx 

 

Table 6. Response Group Service User Model

Component Per Enterprise deployment Per Standard Edition server

Active agents (formal and informal)

1,200

1,200

Number of standard Response Groups

450

150

Number of queues used

One unique queue for each hunt group, two for the One-Level Interactive response group

One unique queue for each hunt group, two for the One-Level Interactive response group

Distribution of routing methods on groups

Parallel routing: 40%

Longest idle: 40%

Serial: 10%

Round robin: 10%

Parallel routing: 40%

Longest idle: 40%

Serial: 10%

Round robin: 10%

Percentage of workflows that use speech recognition in their interactive voice response (IVR) versus workflows that use only dual tone multi-frequency (DTMF) in their IVR

Speech recognition/Text-to-speech (SR/TTS) + DTMF: 50% DTMF: 50%

SR/TTS + DTMF:50% DTMF: 50%

Number of hunt groups (mix of 50% simple and 50% complex hunt groups)

600

300

Average number of agents per group

10 agents

10 agents

Average number of groups an agent is a member of

Two groups

Two groups

Number of groups per queue (average)

90%: One group 10%: Two groups

90%: One group 10%: Two groups

Number of simultaneous response group calls

480

60

Average call duration (IVR portion + music on hold)

30 seconds

30 seconds

Average call duration with the agent

3 minutes

3 minutes

Number of sign-in/sign-out cycles for formal agents in a day (based on an 8-hour day)

4

4

 

 

 

Table 7. Maximum Supported Users for Each Topology

Topology Servers required Maximum users supported

Standard Edition server

One Standard Edition server

5,000

Enterprise pool, consolidated configuration

Eight Enterprise Edition Front-End Servers running all server roles

One Back-end SQL Server

100,000

Dd425159.note(en-us,office.13).gifNote: If deploying only IM and presence, Office Communications Server 2007 R2 supports 200,000 client end points, where each end point is a client program, such as Communicator, based on eight Front End Servers and a 16-core computer running the Microsoft SQL Server database software. The back-end database must run on a 4-way processor, quad core, or 8-way, dual core, 2.0 GHz+ computer.

 

-Santosh

 

  • Marked as answer by plug n pray Monday, December 05, 2011 2:25 PM
  • Unmarked as answer by plug n pray Saturday, May 05, 2012 3:28 PM
Free Windows Admin Tool Kit Click here and download it now
December 3rd, 2011 8:37am

 

plug n pray,

 

"Match Making" component of RGS queries database for Agent's presence State.

It is possible that it may not be getting the updated State because of performance issues.

You can use  RGS performace Counters to monitor the activity.

"LC:RGS - 01 -Response Group Service Match Making"\"Total Number of presence notifications received"

"LC:RGS - 01 -Response Group Service Match Making"\"Total Number of presence subscriptions terminated"

 

Also please review the capacity of your server

http://technet.microsoft.com/en-us/library/dd425159(office.13).aspx 

 

Table 6. Response Group Service User Model

Component Per Enterprise deployment Per Standard Edition server

Active agents (formal and informal)

1,200

1,200

Number of standard Response Groups

450

150

Number of queues used

One unique queue for each hunt group, two for the One-Level Interactive response group

One unique queue for each hunt group, two for the One-Level Interactive response group

Distribution of routing methods on groups

Parallel routing: 40%

Longest idle: 40%

Serial: 10%

Round robin: 10%

Parallel routing: 40%

Longest idle: 40%

Serial: 10%

Round robin: 10%

Percentage of workflows that use speech recognition in their interactive voice response (IVR) versus workflows that use only dual tone multi-frequency (DTMF) in their IVR

Speech recognition/Text-to-speech (SR/TTS) + DTMF: 50% DTMF: 50%

SR/TTS + DTMF:50% DTMF: 50%

Number of hunt groups (mix of 50% simple and 50% complex hunt groups)

600

300

Average number of agents per group

10 agents

10 agents

Average number of groups an agent is a member of

Two groups

Two groups

Number of groups per queue (average)

90%: One group 10%: Two groups

90%: One group 10%: Two groups

Number of simultaneous response group calls

480

60

Average call duration (IVR portion + music on hold)

30 seconds

30 seconds

Average call duration with the agent

3 minutes

3 minutes

Number of sign-in/sign-out cycles for formal agents in a day (based on an 8-hour day)

4

4

 

 

 

Table 7. Maximum Supported Users for Each Topology

Topology Servers required Maximum users supported

Standard Edition server

One Standard Edition server

5,000

Enterprise pool, consolidated configuration

Eight Enterprise Edition Front-End Servers running all server roles

One Back-end SQL Server

100,000

Dd425159.note(en-us,office.13).gifNote: If deploying only IM and presence, Office Communications Server 2007 R2 supports 200,000 client end points, where each end point is a client program, such as Communicator, based on eight Front End Servers and a 16-core computer running the Microsoft SQL Server database software. The back-end database must run on a 4-way processor, quad core, or 8-way, dual core, 2.0 GHz+ computer.

 

-Santosh

 

  • Marked as answer by plug n pray Monday, December 05, 2011 2:25 PM
  • Unmarked as answer by plug n pray Saturday, May 05, 2012 3:28 PM
December 3rd, 2011 8:37am

 

plug n pray,

 

"Match Making" component of RGS queries database for Agent's presence State.

It is possible that it may not be getting the updated State because of performance issues.

You can use  RGS performace Counters to monitor the activity.

"LC:RGS - 01 -Response Group Service Match Making"\"Total Number of presence notifications received"

"LC:RGS - 01 -Response Group Service Match Making"\"Total Number of presence subscriptions terminated"

 

Also please review the capacity of your server

http://technet.microsoft.com/en-us/library/dd425159(office.13).aspx 

 

Table 6. Response Group Service User Model

Component Per Enterprise deployment Per Standard Edition server

Active agents (formal and informal)

1,200

1,200

Number of standard Response Groups

450

150

Number of queues used

One unique queue for each hunt group, two for the One-Level Interactive response group

One unique queue for each hunt group, two for the One-Level Interactive response group

Distribution of routing methods on groups

Parallel routing: 40%

Longest idle: 40%

Serial: 10%

Round robin: 10%

Parallel routing: 40%

Longest idle: 40%

Serial: 10%

Round robin: 10%

Percentage of workflows that use speech recognition in their interactive voice response (IVR) versus workflows that use only dual tone multi-frequency (DTMF) in their IVR

Speech recognition/Text-to-speech (SR/TTS) + DTMF: 50% DTMF: 50%

SR/TTS + DTMF:50% DTMF: 50%

Number of hunt groups (mix of 50% simple and 50% complex hunt groups)

600

300

Average number of agents per group

10 agents

10 agents

Average number of groups an agent is a member of

Two groups

Two groups

Number of groups per queue (average)

90%: One group 10%: Two groups

90%: One group 10%: Two groups

Number of simultaneous response group calls

480

60

Average call duration (IVR portion + music on hold)

30 seconds

30 seconds

Average call duration with the agent

3 minutes

3 minutes

Number of sign-in/sign-out cycles for formal agents in a day (based on an 8-hour day)

4

4

 

 

 

Table 7. Maximum Supported Users for Each Topology

Topology Servers required Maximum users supported

Standard Edition server

One Standard Edition server

5,000

Enterprise pool, consolidated configuration

Eight Enterprise Edition Front-End Servers running all server roles

One Back-end SQL Server

100,000

Dd425159.note(en-us,office.13).gifNote: If deploying only IM and presence, Office Communications Server 2007 R2 supports 200,000 client end points, where each end point is a client program, such as Communicator, based on eight Front End Servers and a 16-core computer running the Microsoft SQL Server database software. The back-end database must run on a 4-way processor, quad core, or 8-way, dual core, 2.0 GHz+ computer.

 

-Santosh

 

  • Marked as answer by plug n pray Monday, December 05, 2011 2:25 PM
  • Unmarked as answer by plug n pray Saturday, May 05, 2012 3:28 PM
Free Windows Admin Tool Kit Click here and download it now
December 3rd, 2011 8:37am

 

plug n pray,

 

"Match Making" component of RGS queries database for Agent's presence State.

It is possible that it may not be getting the updated State because of performance issues.

You can use  RGS performace Counters to monitor the activity.

"LC:RGS - 01 -Response Group Service Match Making"\"Total Number of presence notifications received"

"LC:RGS - 01 -Response Group Service Match Making"\"Total Number of presence subscriptions terminated"

 

Also please review the capacity of your server

http://technet.microsoft.com/en-us/library/dd425159(office.13).aspx 

 

Table 6. Response Group Service User Model

Component Per Enterprise deployment Per Standard Edition server

Active agents (formal and informal)

1,200

1,200

Number of standard Response Groups

450

150

Number of queues used

One unique queue for each hunt group, two for the One-Level Interactive response group

One unique queue for each hunt group, two for the One-Level Interactive response group

Distribution of routing methods on groups

Parallel routing: 40%

Longest idle: 40%

Serial: 10%

Round robin: 10%

Parallel routing: 40%

Longest idle: 40%

Serial: 10%

Round robin: 10%

Percentage of workflows that use speech recognition in their interactive voice response (IVR) versus workflows that use only dual tone multi-frequency (DTMF) in their IVR

Speech recognition/Text-to-speech (SR/TTS) + DTMF: 50% DTMF: 50%

SR/TTS + DTMF:50% DTMF: 50%

Number of hunt groups (mix of 50% simple and 50% complex hunt groups)

600

300

Average number of agents per group

10 agents

10 agents

Average number of groups an agent is a member of

Two groups

Two groups

Number of groups per queue (average)

90%: One group 10%: Two groups

90%: One group 10%: Two groups

Number of simultaneous response group calls

480

60

Average call duration (IVR portion + music on hold)

30 seconds

30 seconds

Average call duration with the agent

3 minutes

3 minutes

Number of sign-in/sign-out cycles for formal agents in a day (based on an 8-hour day)

4

4

 

 

 

Table 7. Maximum Supported Users for Each Topology

Topology Servers required Maximum users supported

Standard Edition server

One Standard Edition server

5,000

Enterprise pool, consolidated configuration

Eight Enterprise Edition Front-End Servers running all server roles

One Back-end SQL Server

100,000

Dd425159.note(en-us,office.13).gifNote: If deploying only IM and presence, Office Communications Server 2007 R2 supports 200,000 client end points, where each end point is a client program, such as Communicator, based on eight Front End Servers and a 16-core computer running the Microsoft SQL Server database software. The back-end database must run on a 4-way processor, quad core, or 8-way, dual core, 2.0 GHz+ computer.

 

-Santosh

 

  • Marked as answer by plug n pray Monday, December 05, 2011 2:25 PM
  • Unmarked as answer by plug n pray Saturday, May 05, 2012 3:28 PM
December 3rd, 2011 11:37am


I am having this same issue in Lync server. I see that your answer was related to performance but specifically can you tell me what you did to correct the issue? Where did you fix the performance issue?

Thanks for your help

Free Windows Admin Tool Kit Click here and download it now
March 1st, 2012 12:31am

No clean resolution found.

There were no hits for this counter
"LC:RGS - 01 -Response Group Service Match Making"\"Total Number of presence subscriptions terminated"

I've asked operators to just close incoming call toast - it bounces the caller back to queue without the caller noticing.
Eventually RGS hunts truly Available agent..
March 1st, 2012 4:31pm

Just to follow up on this. I have opened a SEV-A case with Microsoft on this issue. They were able to reproduce the behavior in their lab. Possible bug in the code. I will keep you up to date on the resolution.
Free Windows Admin Tool Kit Click here and download it now
March 1st, 2012 5:54pm

Just to follow up. We opened the SEV-A case with the Lync Product group. They were able to reproduce the exact error described in their lab environment. Once they attempted to fix the issue, they made the product unstable to the point that it was unusable. The Lync product team has confirmed that it is an issue as that it will not be fixed in this version of Lync. It is being considered as a possible fix for wave 15.

Sorry for the long wait, but we have been waiting for the Lync product team to complete their testing. We are looking to 3rd parties for our call center application.

April 25th, 2012 6:06pm

Hi Bob,

    Do you happen to know if this is specific to a specific routing method?  We are having the same problem and are using the Longest Idle method.  I'm wondering if this could be solved by using another routing method such as Round Robin or Parallel.  I'm assuming if an Agent is on a call they won't be present with a call and in the case of Parallel, calls would be placed on hold if all Agents are busy.

Thanks,

Ricardo

Free Windows Admin Tool Kit Click here and download it now
May 4th, 2012 5:28pm

 

Pap3rkut,

Sad to say but we tried EVERY routing method out there. All routing methods presented the same behavior. We tried a host of different scenarios and all failed. It appears there is an issue with the call handling and the match-making service that is causing this. This is NOT, I repeat, NOT a performance issue as described earlier in Santosh_More's reply.

Just in case your are interested.... Here is the MS Case Number:  SR 112022755545407

And here is what the letter from Microsoft stated:

Dear Bob Ausmus,

I am writing you concerning your support case 112022755545407 (RGS agents receive a toast for second waiting call in RGS once they accept the first waiting call from RGS) with Microsoft Customer Service and Support.

<You> reported that there is an issue using Response Group Service with Lync 2010. Under certain circumstances, an RGS agent would receive a second toast on the screen once the first call is accepted from the queue.

As a result the positive user experience for the agents hosted on the Lync 2010 Response Group Service is hampered.
Lync 2010 Product Feature Team investigated the issue, collaborated with internal groups and understood the root cause of the issue. They have confirmed that unfortunately, developing an architecturally sound solution would require rigorous testing and the current functionality cannot be easily changed without risking destabilizing the product if delivered in the form of an interim patch.

They have advised several workarounds for this issue.
1. Declining the second toast once it is alerted on the agent desktop <This is not a solution>
2. Reducing the alert time for the agent <This did not work either>

Given the very high risk of a potential fix and that there are two workarounds available, we are unable to accept the request for the current major release of Lync Server 2010. It will be considered for the next major release of Lync.

We hope you understand the challenges explained, and respect the decision that was reached. I personally thank you for continuing to use Microsoft products and services, your valuable feedback, and your understanding as we partner to look for the best possible solution.

There You have it. Broken and they aren't going to fix it. Yeah..... #FAIL

May 4th, 2012 5:57pm

Bob,

    I very much appreciate the effort you've gone through working this problem and at least providing an answer to me to report back to the supervisors and managers expiriencing this issue.  In short, that really sucks.  Best of luck to you and I hope Microsoft resolves this sooner than later.

Thanks,

Ricardo

Free Windows Admin Tool Kit Click here and download it now
May 4th, 2012 6:40pm

Anybody knows is there even a hope to get this fixed on the Lync 2010? I can see "plug n pray" had this issue on the O2007R2. And this sounds very much similar with this case:

http://social.technet.microsoft.com/Forums/en-US/ocsclients/thread/188fd6cd-cbd8-40e1-b969-a62609cca25a/

Where you cannot avoid second phone call if you are already having one.

Or has anybody tested this on the Lync 2013?

November 5th, 2012 5:53pm

Hi Bob,

    I've recently been revisiting this issue.  Have you happened to have heard from Microsoft or tested yourself if this issue is resolved by Lync 2013?  I've budgeted to have this installed this summer and was hoping it could solve our problems with the response groups.

Thanks,

Ricardo

Free Windows Admin Tool Kit Click here and download it now
March 25th, 2013 7:59pm

Bob any update?  Do you know if this was ever corrected by MS?
August 30th, 2013 6:31pm

I am experiencing this exact issue in Lync 2013. I haven't yet investigated whether this might be performance related, but before I do wondered if anyone knew if MS attempted to fix this in Lync 2013?
Free Windows Admin Tool Kit Click here and download it now
January 21st, 2014 7:08pm

Sorry guys, I have been in dream world and slacking on my posts. I have heard that this is NOT corrected in 2013. It didn't make the cut from what I have been told.

We bailed on using Lync for our Call Center. We use it for the company phone system, but for the call center we use Interactive Intelligence (I3) which integrates with Lync.

Hope this helps...
January 24th, 2014 2:50pm

This is a known issue in Lync Server 2013.

If you want to Vote to have this issue addressed, please click below:
http://lync.ideascale.com/a/dtd/response-group-routes-calls-to-busy-agent-bug/559917-16285

Free Windows Admin Tool Kit Click here and download it now
January 24th, 2014 3:04pm

Looks like Skype for Business Server June 2015 update may have fixed this issue.
https://support.microsoft.com/en-us/kb/3068197

anyone that is experiencing it, can you verify it has fixed your scenario?
Thanks!

June 22nd, 2015 9:06am

This topic is archived. No further replies will be accepted.

Other recent topics Other recent topics