Every now and then any Siebel implementation hits one of the most problematic issues to resolve: memory leaks! Working with this financial services customer for years, they adopted in 2012 Siebel eMail Response. You would think that as many modules in Siebel – eMail Response should be pretty mature. Hence when you hit a memory leak – it would point towards any sort of customization.
Not in this case. It appeared that the Communications Outbound Manager kept crashing a couple of times a day – when reaching the 1.7 GB memory limit. If you take a step back – the Communications Outbound Manager should be pretty standard in what it does. It really is a very basic process is executes when it either is invoked using the CreateRequest or SendMessage method.
Luckily the customer is using Oracle ATS for Load testing. Very helpful because it enabled us to quickly create a simple script to ran a number of Replies to emails and have them sent. Sending invokes the SendMessage method on the Communications Outbound Manager business service. We noticed that the CommOutboundMgr component steadily grew while emails got sent out.
The initial idea was to monitor the SQL cursor usage which turned out to be a lucky pick:
SUM (A.VALUE) TOTAL_CUR,
A.STATISTIC# = B.STATISTIC#
AND B.NAME = ‘openened cursors current’
AND S.USERNAME = ‘SBLCMP_COMMOUTMGR’
Every email sent increased the number of open cursors by about 5 without freeing any cursor.
A good start.
Next with any reproduction case, would be to reproduce against a Siebel Vanilla SRF. So we changed the SRF used by the CommOutboundMgr.
Repeating the same scenario, guess what, no SQL cursor leak to be seen. Is it then caused by customization after all?
Well, not in this case. Siebel provided starting with 18.104.22.168 a new eMail Response UI. For that a number of .sif archives were required to be imported. Could these changes be the culprit? One would not expect this – would we be the first customer to hit this issue? Being on Siebel 22.214.171.124 – it’s a pretty descent release giving that the latest shipped fixpack is 126.96.36.199.
I decided to compare the trace files of the CommOutboundMgr – the Vanilla one with the Custom one. Many differences – well actually not. Besides this business service “Set Parentemail Status”…
Doing a quick search on My Oracle Support (well – as Oracle employee having full visibility) I almost instantaneously found an SR pointing towards a know Bug
The BUG 14203363 – MEMORY LEAK IN SET PARENT EMAIL BS
And to find out it will only be fixed in the upcoming 188.8.131.52 fix pack. While reading the Bug details it became clear this is our issue. Since the fix is known – I requested a Quickfix. Hopefully to be turned around in the next 4-6 weeks…