occasional failure on sending callrecording

[dkratt]

We are seeing occasional failures in the system sending us the binary callrecording.
Logs show the posting attempt:

Posting binary content "callrecording" of size 819200 type "audio/basic"
Posted form data is multipart encoded
Attempting to fetch https://**************/save

We have attempted with data, with submit (changing our far end to produce voiceXML)

<data srcexpr="storedRecordingURL" method="post" enctype="multipart/form-data" namelist="callrecording n c x p" fetchtimeout="120s"/>

We seem to have this issue specifically on certain hang up conditions.
Everything else functions correctly. Any other posts we do are fine. There are no logged errors, no conditions of the post failing. We are setting <property name="recordcallappend" value="true"/> at the beginning of the call, and turn recording on in each form.

At the far end, we end up with only 1 form variable coming through when this fails, and request.files is not populated .
The firewall is not the issue, we thought maybe some IPS protection was causing some stripping out (why would it occur on only some calls) but concluded it is already not present in the post on the way in the door.

Any lower level way to see what is occurring here, why the post shows it occurred, but at the far end we see nothing was "attached"?

[support]

Hi Donald,

We have had a chance to look over the code and information you sent us and we have an idea of what may be the problem. The most common cause for this situation is a server configuration issue where the maximum post, memory or file size is too small and the file upload is lost. We recommend checking these settings and increasing them as necessary.The more specific details of how to check/alter these settings will depend on your environment. We hope this information helps you to solve your problem!

Regards,
Plum Support

[dkratt]

Hmmmm.
I have a 30 mb (30000000 bytes) max allowed content length.
My application pool on this, on all farm servers, has no CPU limit virtual memory limit, no private memory limit, is configured for always on (for responsiveness, we do not let the process sleep or shut down from inactivity).

So nothing looks off.
Again, I have had whole recordings larger than those that have failed posted with no issue.
The server is also not responding with a HTTP Error 404.13.

[dkratt]

OK, we can never make it fail when the script goes to completion without the dialer hanging up prior to the exit.

If the hangup happens, it is a crap shoot whether the recording is attached when it comes into our service.
Discussed whether it might be specific IVRs, so we logged that. It is on different IVRs.

All complete call recordings are longer than any incompletes, so has nothing to do with some size limit on our side.
It is always a hang up prior to exit.


recording,hangup,call end,IVR
TRUE,intro 1,11/20/15 7:56:12,69.25.74.78
TRUE,intro 1,11/20/15 7:57:40,69.25.74.73
TRUE,intro 1,11/20/15 7:58:45,69.25.74.78
TRUE,intro 2,11/20/15 8:00:19,69.25.74.73
FALSE,intro 3,11/20/15 8:02:11,69.25.74.73
FALSE,intro 3,11/20/15 8:04:31,69.25.74.73
FALSE,intro 1,11/20/15 8:05:38,69.25.74.78
TRUE,intro 1,11/20/15 8:22:38,69.25.74.73
FALSE,intro 1,11/20/15 8:23:32,69.25.74.82
TRUE,intro 1,11/20/15 8:25:28,69.25.74.82
FALSE,intro 1,11/20/15 8:26:51,69.25.74.88
TRUE,intro 2,11/20/15 8:28:12,69.25.74.73
TRUE,intro 4,11/20/15 8:31:40,69.25.74.121
TRUE,NO HANGUP,11/20/15 8:35:28,69.25.74.92
TRUE,NO HANGUP,11/20/15 8:37:40,69.25.74.121
TRUE,NO HANGUP,11/20/15 8:38:11,69.25.74.92
TRUE,NO HANGUP,11/20/15 8:40:42,69.25.74.121
TRUE,NO HANGUP,11/20/15 8:41:21,69.25.74.92
TRUE,NO HANGUP,11/20/15 8:43:13,69.25.74.73
TRUE,NO HANGUP,11/20/15 8:44:13,69.25.74.82


It is always conditions where logs look like this:

impl->dxi->waitForPlayEOD() detected a disconnect. Abandoning queued data.
Can not queue audio -- line disconnected
impl->dxi->waitForPlayEOD() detected a disconnect. Abandoning queued data.
Can not queue audio -- line disconnected
received event: connection.disconnect.hangup:
VXI::assign_element(name="Disconnected" expr = "<private>")
VXI::assign_element(name="StatusCode" expr = "<private>")
VXI::assign_element(name="NeedCustomerSession" expr = "<private>")
Entering form = 'main' form item = '$_internalName_419049'
VXI::throw_element - throwing (threepv.session.exit, )
received event: threepv.session.exit:
VXI::assign_element(name="TimeEnd" expr = "<private>")
Posted form data is URL encoded
Attempting to fetch https://*********/LogCall.aspx
Posting binary content "callrecording" of size 94208 type "audio/basic"
Posted form data is multipart encoded
Attempting to fetch https://*********/save

[support]

Hi Donald,

It seems as if the issue you're having may be much more involved than we initially thought. I have currently escalated your issue to the platform engineering team, who will be evaluating this soon. We will get back to you with more information, or reach out to you if we have any more questions.

Regards,
Plum Support

[dkratt]

OK, we are able to make the failure happen by manipulating the namelist on the post, again, only on early hangups.

If we put the callrecording variable last in the namelist, on these hangups we get the recording in the post.
If we do not put it last, it is unpredictable.

So this consistently works

Code: Select all

<data srcexpr="storedRecordingURL" method="post" enctype="multipart/form-data" namelist="n c x p callrecording" fetchtimeout="120s"/>

But, this works consistently only when we the caller does not hang up

Code: Select all

<data srcexpr="storedRecordingURL" method="post" enctype="multipart/form-data" namelist="callrecording n c x p" fetchtimeout="120s"/>

So, we can work with that ... but it seems like some issue with some asynchronous processing on your side that depends on timing.

[support]

Thanks for that additional information, Donald. we have passed it along to the platform engineers currently investigating your issue. More input is always helpful, and we will be getting back to you as soon as we have an update.

Regards, Plum Support

[dkratt]

This issue is still occurring with our live traffic that we have started to cut over.

Again, this always has to do with an early disconnect, which is otherwise handled, rather than the FIA going through to a graceful exit without a disconnect occurring.

Can we escalate? Reaching out to our support reps.