Siebel IP15.8/14.14 Has Arrived


Siebel IP15.8/14.14 arrived, per scheduled date. The traditional document to keep a close eye on “Oracle Support Document 1614310.1 (Siebel Patchset Installation Guides for Siebel Innovation Packs)” has been updated as well. Unfortunately, the awaited “Desktop Integration Siebel Agent (DISA)” framework solution has been pushed into the February patchset IP15.9/14.15.

There are a couple of bugs to mention here:


– Jeroen



Simplifying Simple Siebel Things: Logout

Not gracefully logging out of Siebel is one of those user habits which is hard to kill. Especially because in the high-interactivity days, closing the browser would gracefully end the Siebel session on the server. Well… that changed with the OpenUI client because there is no way in which the browser ‘X’ button to either close a tab or the browser all together can be intercepted (similarly for the back button).

End result: having potentially many orphan sessions, waiting until the session times-out. Well, that can be considered still a positive scenario: the session will time-out. But might you have message broadcasting enabled, it won’t. Unless, again, you take preventive measures explained in this post about white-listing certain events.

But another, yet worse scenario to happen is that an user has uncommitted data, pending to be committed. What will happen using the ‘X’ approach? You can guess what, you will loose whatever is pending. Likely that it happens? Sure! And this will hurt your user adoption and creates frustration among the user base, guaranteed. The same obviously goes for that so-often used back button, that set aside.

In a previous post I discussed one possible approach, indicating users they have unsaved data. I’m about to write a post, with an alternative solution for the same issue. Keep a close eye on the blog 🙂

But back to logout. I firmly believe the standard’s theme ‘logout option’ hidden behind the ‘user’ button, is, well, too much hidden. The same goes for the logout option in the application menu. So I decided (and others will have done similar, for sure) to add the logout button where everybody would expect such a button: top-right corner or the browser’s window.

Added a tiny bit of additional functionality:

a) Greying-out the background
b) Enabling the spinning Siebel wheel ‘busy’ cursor

I decided to keep it purely an image (or better: glyph), to prevent any localization. If I’d have opted for a ‘logout’ text (either with our without the glyph), I would rather have created the logout button through Siebel Tools, leveraging translation through a symbolic string. Changing the behavior (e.g. moving it to the far-right and adding the logout code, would then still be done client-side by javascript plus CSS).

Here I decided to generate the button during postload, with some basic CSS. You can simply add this code to your existing postload. Otherwise, you need to create one.



I created the CSS for Aurora. Technically it does work for the Synergy theme. You just need to tweak the CSS a bit more.


– Jeroen

Siebel OpenUI Boilerplate on Github!

Chances are that you have used or even are a frequent user of Duncan Ford‘s boilerplate code generator for Siebel OpenUI. His boilerplate code generator surely is the safest and most effective way to start any undertaking comprising a custom presentation model, physical renderer or control plug-in wrapper. It prevents you from those silly typo’s which can easily waste your time. All together, a huge efficiency tool.

Well, Duncan decided to move his code from JSFiddle to Github. For a very good reason of course. This move permits those with ideas for improvements to add them directly to Github’s managed issues log, and, for those more skilled, to allow to directly submit pull-requests with changes made. Which Duncan then will be able to test and decide upon

Although the existing URLs which you might or might not have bookmarked, have been changed by Duncan as well to reflect the new location. Below the direct Github URLs.

– Jeroen


Siebel Patch Set 15.7 / 14.13 Has Been Released

The December Patchset slipped a couple of days into 2016, likely due to the holiday season. But here it it. Patch numbers are respectively 22392806 (IP15) and 22387952 (IP14).

Wading through the release notes I stumbled into this bug, drawing my immediate interest:


Checking on the reproduction steps, it becomes apparent that it’s a rather simple scenario resulting in a very undesirable result:

(1) Login to Open UI client.
(2) Go to [Accounts]->[Accounts List]
(3) Create records or run a query so the list applet has 11 records.
(4) Place the focus on Name field on any record.
(5) Click 'Show More' icon to see the all 11 records.
(6) Click 'Show Less' icon to see the 1st 10 records.
(7) Click 'Next record set' icon to show the 11th record.
(8) Now you see that the 10th record show the 11th record's data.
(9) Click 'Previous record' icon 10 times to navigate the records upward one by one.
(10) You will see that Name field data on several records are modified and saved to the database. The original data is gone. The issue occurs when the list applet has more than 10 records and less than 20 records.

Happy this one got identified and fixed! If one thing can be more UX-damaging, it would be data loss.

One other Bug to mention:


This is related with the in IP15.4 / IP14.10 implemented ST Script engine performance improvements, primarily focused on more efficient memory use. This fix can be optionally enabled using the USE_NEW_RM = 1 / USE_NEW_MM = 1 environment variables.

– Jeroen

Desktop Integration Siebel Agent (DISA)

The 2016 SOD already mentioned a totally new Siebel acronym. Because running out of three-letter-acronyms, resorting to four-letter acronyms 😉


The SOD also mentioned backporting the DISA framework to IP14/15. And that seems to materialize. Although it definitely slipped the December 2015 Patchset (14.13/15.7) for which it was initially targeted – the odds are that it will quite likely make it into the January 2016 Patchset (14.14/15.8).

So again, what’s this Desktop Integration Siebel Agent all about? Well, first of all it’s the must-have feature closing the important Siebel client-side integration gap. Especially because Google started deprecating the NPAPI in Chrome, Mozilla will follow deprecating NPAPI in Firefox later in 2016 and Microsoft’s Edge never had and never will have the capability.

DISA will be all about bringing Siebel OpenUI on-par with the old ‘high interactivity’ days where the Web Client Automation Server was a strong asset, enabling relatively easy client-side integrations with desktop applications and a Siebel user session.

DISA will be a local client-side running executable, which means it will only be available for Windows clients. The DISA will act as a Websocket-server, and because today all of the relevant browsers will support the Websocket protocol,  browsers running a Siebel OpenUI session can act as a Websocket-client. The framework will enable bi-directional, real-time communications between browsers.

Unsure whether at the time DISA will be released as backport from the under-development IP16 codeline is which of the features will be enabled alongside? I personally expect that initially the DISA framework as-is will be released, and that out-of-the-box enhancements such as listed below will be released gradually in subsequent patchsets.

DISA targeted features:

» Email (F9) Integration with Microsoft Outlook and IBM Lotus Notes
» Invoke external email clients with F9 on Siebel Open UI
» Populate email fields (To, CC, BCC) in external email clients
» Use HTML templates to provide pre-defined, formatted email content
» Send file attachments
» Read & write files in agent’s local machines
» CTI Hoteling feature
» Ability to retrieve client machine IP address in Siebel Open UI Client
» Outlook to Siebel Attachment Drag-and-Drop. Enables agents to drag an email from a Microsoft Outlook email client and drop it as a Siebel CRM file attachment
» Inline attachment editing. Provides the capability for a user to open a file stored in the Siebel File System, make changes to the document using a location application (such as Microsoft Word), and then save and close the application with the new version automatically replacing the previous version in the Siebel File System.
» Batch Fulfillment Printing
» Allows Fulfillment Center personnel to select a set of Correspondence Requests and send them to a printer.

– Jeroen


Server busy, server busy, the server is very busy…

No need to explain that these infamous two words hide worlds of potential causes, if you happen to run a Siebel implementation. I got involved in a situation at one of my customers, where end-users were reporting end-less occurrences on the above mentioned. User’s reported to be kicked-out for no reason, often loosing in-flight data while having a customer on-call. The volume of reports was more than substantial, though initially real, objective measured figures were not available.

First and foremost focus in such situation should always be: are there any frequent object manager crashes? Unfortunately customer was not running the Server Task Persistence component, part of the Siebel Failure Diagnostics Framework which got introduced as part of Innovation Pack 2012. This out-of-the-box feature helps administrators analyzing crashes, and viewing key characteristics ‘comfortable’ from within the Siebel UI. So without going through the laborious process grabbing manually the call stack from the /bin directory and decoding the FDR files. Added ‘bonus’ is that you immediately know which users got affected by a specific crash.

Luckily, the background utility ‘siebprocdiag’ used by the Server Task Persistence component to gather the key details, can be also be run manually on a specific server using. Siebprocdiag will essentially cycle through all available .fdr files, decode them and relate them back to call stacks reported in the crash.txt file. The way in which siebprocdiag reports the results of its FDR analysis can be solid gold. It reports in the form of “user navigated to view ‘x’, invoked method ‘y’ on applet ‘z’ which in turn invoked business service ‘abc’ leading to a crash”.

Detailed output will be:

  • Siebel Server and server component names
  • Time of failure
  • Process, thread, and task IDs
  • Number of affected tasks
  • Last set of meaningful business processes that occurred at the time of the failure
  • Whether a new process was created, and if so, the list of current users impacted by the failure
  • A list of users whose sessions were lost following the failure
  • Content of the actual crash.txt file that was logged, which provides the call stack, register contents, and memory information

Well all said and done, we figured out there were crashes and a good number of them across all four Windows servers hosting Call center object managers. The use of siebprocdiag proved vital in quick root-cause analysis, it turned out a buggy business component definition leading to incomplete SQL statements being generated, that business component got introduced in the most recent SRF release.

But guess what, after all the object manager crashes ceased to exists after an emergency release – users kept reporting ‘server busy’. What was going on?

We started to increase ‘EventContext’ tracing, to get a better grip and to hopefully see some type of pattern. Unfortunately, that lead nowhere.

One step further we added ObjMgrSessionLog=5 and ProcessRequest=5 in order to trail the flow of events. At the same time we cranked-up SWSE monitoring per three usual system environment variables (SIEBEL_LOG_EVENTS = 4, SIEBEL_SESSMGR_TRACE = 1, SIEBEL_SISNAPI_TRACE = 1). Although SWSE tracing is flooded with irrelevant (no, what I mean is ‘benign’) traces, it typically can be well filtered based on SARM ID which can always correlate an Object manager task with SWSE tracing.

The SARM ID is printed in every line of a component log. The fact it correlated with SWSE tracing.

TaskConfig    TaskCfgParamInit    3    0000003c56710e30:0    2015-12-30 15:28:14    The Parameters for the current task (8405474) are :

Since the customer is running a high-available configuration including a set of hardware load-balanced web servers, we decided to temporarily by-pass the load-balancer. Luckily the vast majority of end-users are on Citrix, hence we able to provision a changed short-cut instantaneously. By-passing the load balancer we both could cancel-out the load-balancer  being potential culprit as well as analysing is simply easier on a single web server. Load-wise it should create no issue to run on a single leg temporarily.

Run a couple of hours, the incidents kept flowing in and pretty quickly we noticed this pattern:

ObjMgrSessionLog    ObjMgrLogin    3    00005710566f096c:0    2015-12-30 15:43:38    Username: XYZ, Login Status: Attempt,  Session Id: bP6I.5OZwhK.IxZSBDlRU3dYBTdk2UeLUkP8fk83cyQcXmEKcPnGWD4mUAYLDnQCtsMEDoCp8E6aDkhoXWRCgX-4QnqEWCECWJHyMZKAcVqMF6MI1szESLb78y05d7O-SIO8bmbJ2W5SFgJvqj8kD-V1Gx.zFc4hjxWshoKQSlg_, IP Address: xx.xx.xx.xx

Usually, if you spot “Login Status: Attempt” in the middle of a web session, it simply means a time-out. But under normal circumstances, the message in the log should be preceded by “SBL-SMI-00126: The session has timed out”. The “Login Status: Attempt” message, would in case of WebSSO ensure the user is logged-in automatically, after a session timout. Moreover, in case of a traditional time-out, you should be easily able to spot a gap in the log corresponding the the SessionTimeout configured on the web server (eapps.cfg).

After collecting during a good deal of a day we confirmed a couple of things:

  1. The message “Login Status: Attempt” (if not preceded by SBL-SMI-00126) clearly indicated a an ‘affected user’.
  2. Extrapolating on the occurrences we collected over the period of one hour on one single server, we found the issue to hit approximately every users two times their working day. Wow.

But the most intriguing fact: at the same time, groups of users got affected. Sometimes a little as 2, sometimes as many as 9. And these groups of users were always related to one specific object manger process Id.

This latter piece of information led us to believe a relationship with SISNAPI session multiplexing between SWSE and Siebel Servers. The SWSE multiplexes Siebel web client session over a pool of SISNAPI – which of course are just TCP/IP – connections. SWSE maintains a ‘pool’ of SISNAPI connections for each target object manager process. Multiplexing is defined by the SessPerSisn value on the object manager, and defaults to 20. The size of the pool is calculated by:

Max Tasks of the component / Max number of Object Manager multi-threads processes / SessPerSisn. This figure should lead to an integer value. So take Max Tasks = 200, MaxMTServers = 5 and the default SessPerSisn = 20 would lead to 2 SISNAPI connections for each multi-threaded process. I have to admit, in the past 15+ years I have never been required to tweak this value.

We verified these connections running “netstat -ano | find “2321”. The output contains a trailing column a PID, this PID related to the SiebService process on the target Siebel server. So basically you separate the output, and confirm the number of TCP/IP connections between SWSE and a particular Siebel Server.

netstat -ano | find “2131”
  TCP    ESTABLISHED     3464
  TCP    ESTABLISHED     3464
  TCP    ESTABLISHED     3464
  TCP    ESTABLISHED     3464

Anyways, working with the network team we ran a short while Wireshark between the SWSE and one specific Siebel Server. And since we could indicate second-close when an incident produced itself, Wireshark data could be filtered appropriately without being overwhelmed. And guess what? Spot-on, we found a match. It appeared that the Web server closed a TCP/IP connection with details: [RST, ACK, CWR]. The network team immediately noticed the uncommon “CWR” flag which was set on the Reset (RST) packet. CWR means ‘congestion window reduced’, which essentially is a mechanism for a sender to re-negotiate the window in which packets are being exchanged with the receiving party, to make it smaller because of perceived packet-loss or congestion.

A bit more details here, is that CWR has a related feature named ‘Explicit Congestion Notification’ or ECN. And interestingly, ECN is a feature which on Windows releases prior to Windows 2012 has been available, but default it has been always disabled. But with Windows 2012 the feature has been enabled by default. Since the network and infrastructure teams were not prone, disabling ECN because they felt the applications should be resilient we considered changing SessPerSisn.

Theoretically: lowering SessPerSisn would increase the number of SISNAPI connections, and likely lowering the possibility of congestion on one specific TCP/IP connections. At the same time, we would be reducing the number of users impacted. Side-effect of lowering SessPerSisn would be increased memory foot-print for the SWSE plugin plus additional OS memory overhead for both Web servers as well as Siebel Servers. Additional memory for the SWSE plugin should be approximately 1 Mb / SISNAPI connection. Additional OS memory was estimate to be 100 KB / connection. In the customer’s setup that would not accumulate to huge figures.

Decision was made to lower SessPerSisn from default ’20’ to ’10’ – so basically doubling the number of TCP/IP connections. The change got implemented, and seems to have the looked-for effect. The ‘Server Busy’ messages have not been seen anymore!

Now – the story is not over and done for me. Some questions need still to be answered from Oracle’s side:

  1. Should we recommend other values for SessPerSisn for Windows 2012+ implementations?
  2. How to effectively ‘measure’ what an appropriate value for SessPerSisn should be. It’s appreciated that doubling the number of SISNAPI connections seems to work, but it would be good if it could be pro-actively monitored when congestion lingers.

– Jeroen