Researching a Siebel memory leak…

Image

Every now and then any Siebel implementation hits one of the most problematic issues to resolve: memory leaks! Working with this financial services customer for years, they adopted in 2012 Siebel eMail Response. You would think that as many modules in Siebel – eMail Response should be pretty mature. Hence when you hit a memory leak – it would point towards any sort of customization.

Not in this case. It appeared that the Communications Outbound Manager kept crashing a couple of times a day – when reaching the 1.7 GB memory limit. If you take a step back – the Communications Outbound Manager should be pretty standard in what it does. It really is a very basic process is executes when it either is invoked using the CreateRequest or SendMessage method.

Luckily the customer is using Oracle ATS for Load testing. Very helpful because it enabled us to quickly create a simple script to ran a number of Replies to emails and have them sent. Sending invokes the SendMessage method on the Communications Outbound Manager business service. We noticed that the CommOutboundMgr component steadily grew while emails got sent out. 

The initial idea was to monitor the SQL cursor usage which turned out to be a lucky pick:

SELECT
  SUM (A.VALUE) TOTAL_CUR,
  CEIL(AVG(A.VALUE)),
  MAX(A.VALUE) MAX_CUR
FROM
  V$SESSSTAT A,

  V$STATNAME B,
  V$SESSION S
WHERE
  A.STATISTIC# = B.STATISTIC#
  AND S.SID=A.SID
  AND B.NAME = ‘openened cursors current’
  AND S.USERNAME = ‘SBLCMP_COMMOUTMGR’
GROUP BY
  S.USERNAME,
  S.MACHINE

Every email sent increased the number of open cursors by about 5 without freeing any cursor.

A good start.

Next with any reproduction case, would be to reproduce against a Siebel Vanilla SRF. So we changed the SRF used by the CommOutboundMgr.

Repeating the same scenario, guess what, no SQL cursor leak to be seen. Is it then caused by customization after all?

Well, not in this case. Siebel provided starting with 8.1.1.4 a new eMail Response UI. For that a number of .sif archives were required to be imported. Could these changes be the culprit? One would not expect this – would we be the first customer to hit this issue? Being on Siebel 8.1.1.7 – it’s a pretty descent release giving that the latest shipped fixpack is 8.1.1.9.

I decided to compare the trace files of the CommOutboundMgr – the Vanilla one with the Custom one. Many differences – well actually not. Besides this business service “Set Parentemail Status”…

Doing a quick search on My Oracle Support (well – as Oracle employee having full visibility) I almost instantaneously found an SR pointing towards a know Bug

The BUG 14203363 – MEMORY LEAK IN SET PARENT EMAIL BS

And to find out it will only be fixed in the upcoming 8.1.1.10 fix pack. While reading the Bug details it became clear this is our issue. Since the fix is known – I requested a Quickfix. Hopefully to be turned around in the next 4-6 weeks… 

Advertisements

One thought on “Researching a Siebel memory leak…

  1. Hey, that is an interesting article. We have been facing the same for past few months and have raised a Oracle SR’s but to no avail. For now we have implemented the auto recycle parameter for 1GB. We are on 8119 to fix some of our outbound email issues. Now that you mention of this BS.. will check our logs, do some compare like you mentioned above and see if we have similar issues.

    Thanks a ton for documenting this. Any idea if the quickfix is available now.
    Regards.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s