… how could I have been more wrong 🙂
One of my customers took the challenge to upgrade from 220.127.116.11 to Innovation Pack 2014 almost the day it became available for download. They did run a successful sandbox upgrade to Innovation Pack 2013 a couple of months back. The killer feature for them was the Java-enabled “on-the-fly” inline editing solution for attachments in Open UI. Well, essentially bringing back pre-existing like-for-like functionality from the High Interactivity client.
So a couple of weeks back the the AIX-centric Siebel Enterrprise got upgraded and patched with PS1 and we ran the Incremental Repository Merge. So far so good. The Aurora themed tab-style upgraded application does look decently enough. And functionality-wise no big issues at first sight.
Next step was to upgrade the Windows-based Document server. And well, that turned out to be a painstaking exercise. Ough! I expected this to go down smoothly and simple, after the AIX upgrade and repository merge went fine. But in the end this took quite a bit of effort and Tech support involvement before the server was up and functional.
What happened? Well the Siebel Server did start-up fine after the IP14 PS1 upgrade. The only Component which was initially not getting online was the…. Document Server. All other system components were running just fine. From the DocServer_xxx logs little supporting evidence of what could be the culprit. The enterprise log did seem to point towards a crash of the component seconds after it got started.
This was all in the DocServer log:
ObjMgrSRFLog Open 5 0004dff754850044:0 2014-12-11 15:35:23 Begin: OpenSRF
ObjMgrSRFLog Open 5 0004dff754850044:0 2014-12-11 15:35:25 End: OpenSRF
ObjMgrLicense Load 5 0004dff754850044:0 2014-12-11 15:35:25 Begin: LoadLicenseManager
ObjMgrLicense Load 5 0004dff754850044:0 2014-12-11 15:35:25 Encoding successful
And this was seen in the Enterprise log:
ServerLog ProcessExit 1 00000b3654890b14:0 2014-12-11 16:12:20 DocServer 4676 0xc000037408 Process 4676 exited with error - Exception code 0xc000037408
GenericLog GenericError 1 00000b3654890b14:0 2014-12-11 16:12:20 (schedule.cpp (928) err=1376352 sys=6) SBL-SMI-00096: The maximum number of restarts (1) for MinUpTime has been reached. The component DocServer will not be restarted again.
But there was no crash.txt nor a .fdr file in the /siebsrvr/bin directory.
Initially Tech support hinted that we should not override the Password at the component level, but should rather use the Enterprise password. After deleting the parameter override to revert back to an Enterprise level set password for the DocServer, the situation became worse:
ServerLog LstnObjInherit 3 000bb1fb548b00dc:0 2014-12-22 11:31:34 Inherited listening object for port 49156
ServerLog LstnObjPrivCreate 3 000bb1fc548b00dc:0 2014-12-22 11:31:34 Created port 49166 for Document Server
GenericLog GenericError 1 000bb1fc548b00dc:0 2014-12-22 11:31:58 (ccfom.cpp (492) err=4522127 sys=0) SBL-SCL-00143: Internal: input disassembly failed.
SmiLayerLog Error 1 000bb1fc548b00dc:0 2014-12-22 11:31:58 Terminate process due to unrecoverable error: 4522127. (Main Thread)
So, reverted back to a Component-level set Password.
Ok, back to the Enterprise log. It hinted on a crash. And yes, after disabling “Windows Error Reporting” on the box and installing the Windows Debug Diagnostic Tool we were able to move forward. We created a simple rule to take effect on a crash of any “siebmtshmw” (a Siebel multi-threaded) process and enabled it.
Ah! Now we got a crash.txt and the coredump was saved as well.
Immediately the root-cause became apparent:
- CALL STACK -
ntdll +0x332b0 = RtlImageNtHeader() +0x11c
ntdll +0x335b7 = RtlImageNtHeader() +0x423
ntdll +0x334a2 = RtlImageNtHeader() +0x30e
kernel32 +0x114ad = HeapFree() +0x14
MSVCR110 +0xdac2 = free() +0x1a
sslcosd +0xb197 = UTLOsdRealloc() +0x67
sslcrsa128 +0x62eb = AllocateRC2CrypterAdpt() +0x333b
sslcrsa128 +0x603b = AllocateRC2CrypterAdpt() +0x308b
sslcrsa128 +0xc5bd = AllocateRC2CrypterAdpt() +0x960d
sslcrsa128 +0x4da4 = AllocateRC2CrypterAdpt() +0x1df4
sslcrsa128 +0x5783 = AllocateRC2CrypterAdpt() +0x27d3
sslcrsa128 +0x3a1b = AllocateRC2CrypterAdpt() +0xa6b
Mmm… related with encryption…
Looking further down the call stack…
10000000-10043000: O:\sba81\siebsrvr\BIN\sslcrsa128.DLL, 18.104.22.168
Huh? 22.214.171.124! For those unfamiliar with the “Siebel Strong Encryption Pack”, well, this file is used to increase the encryption key-strength used by Siebel. The standard key-strength is 56 bits, by copying this file from the \siebsrvr\bin\ssep directory the key-strength can be increased to 128 bits.
So what happened during installation?
The Siebel IP14 feature which copies “Customer Modified Files” during a software migration, also copied this library… Instead it should have copied the same file from the \siebsrvr\bin\ssep. Overwriting the 126.96.36.199 file with the IP14-seeded \siebsrvr\bin\ssep\ version did the trick. But at the expense of numerous hours of analysis.
So – a clear bug in the IP14 installer for which a Bug will be raised. This might affect your customer too, if they are using SSEP on their current release. Be forewarned!