IBM Business Monitor 8.0.1.2, Cognos BI and Unix - Dependencies Again

This follows on from an earlier post: -

I saw this exception: -

10.99.131.190:9081      28605   2014-08-27 06:40:23.824 +0      na      na                      Thread-125      DISP    5003    1       Audit.Other.dispatcher.DISP.pogo        pogo    com.cognos.pogo.reportservice.ProcessManager            Failure <messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>Process BIBusTKServerMain failed to start properly.</messageString></message></messages>  External Report Server process BIBusTKServerMain cannot be startedProcess BIBusTKServerMain failed to start properly.java.io.IOException: Process BIBusTKServerMain failed to start properly.   at com.cognos.pogo.reportservice.ReportServerProcess.getProcessOutput(ReportServerProcess.java:182)     at com.cognos.pogo.reportservice.ReportServerProcess.start(ReportServerProcess.java:125)        at com.cognos.pogo.reportservice.ProcessFacade.createServerProcess(ProcessFacade.java:275)      at com.cognos.pogo.reportservice.ProcessFacade.<init>(ProcessFacade.java:158)   at com.cognos.pogo.reportservice.ProcessFacade.<init>(ProcessFacade.java:117)   at com.cognos.pogo.reportservice.RSComponentFactory.newProcessFacade(RSComponentFactory.java:76)        at com.cognos.pogo.reportservice.ProcessManager.createProcessFacade(ProcessManager.java:516)    at com.cognos.pogo.reportservice.ProcessManager.

in cogserver.log during the startup of the Cognos Dispatcher, included as part of IBM Business Monitor 8.0.1.2.

I also saw this: -

2014-08-28 15:09:04.276 FATAL [m.cognos.pogo.reportservice.ProcessManager] metadataServiceHandler-MDSRVProcessManager-ProcessMgrThread: External Report Server process BmtMDProviderMain cannot be started java.io.IOException: Process BmtMDProviderMain failed to start properly.

in the pogo_2014_08_28.log file.

I started digging into the two exceptions.

Both relate to Cognos binaries, and the errors imply that there's something missing e.g. a dependency.

First of all, I wanted to locate the offending binary: -

cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1
find . -name BIBusTKServerMain

which showed one copy.

When I try and execute this: -

I get this: -

./BIBusTKServerMain: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory

However, there is a 64-bit version: -

cd /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin64
./BIBusTKServerMain

port=39164 pid=29992

I then dug further using the ldd command: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin64
ldd BIBusTKServerMain

linux-vdso.so.1 =>  (0x00007fff58759000)
/lib64/libfreebl3.so (0x0000003b25a00000)
libtcmalloc_minimal.so.0 => ./libtcmalloc_minimal.so.0 (0x00007f71de1d6000)
libBIBusTK.so => ./libBIBusTK.so (0x00007f71ddf91000)
libBIBusTKServer.so => ./libBIBusTKServer.so (0x00007f71ddd5a000)
libCCLCore.so => ./libCCLCore.so (0x00007f71dd9f8000)
libX11.so.6 => /usr/lib64/libX11.so.6 (0x0000003b17800000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000003b14c00000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003b25000000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003b1bc00000)
libm.so.6 => /lib64/libm.so.6 (0x0000003b15800000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003b1b000000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003b15400000)
libc.so.6 => /lib64/libc.so.6 (0x0000003b15000000)
libxerces-c.so.27 => ./libxerces-c.so.27 (0x00007f71dd3d1000)
librt.so.1 => /lib64/librt.so.1 (0x0000003b16000000)
libCCLCryptoShared.so => ./libCCLCryptoShared.so (0x00007f71dd1c7000)
libCCLIDOM.so => ./libCCLIDOM.so (0x00007f71dcf78000)
libCCLHttptools.so => ./libCCLHttptools.so (0x00007f71dcd54000)
libIBJStreamsDLL.so => ./libIBJStreamsDLL.so (0x00007f71dcb30000)
libCCLCFGAPI.so => ./libCCLCFGAPI.so (0x00007f71dc903000)
libcogipf2.so => ./libcogipf2.so (0x00007f71dc6c3000)
coglog4ccl.so => ./coglog4ccl.so (0x00007f71dc420000)
libz.so.1 => ./libz.so.1 (0x00007f71dc20a000)
libicui18ncognos.so.48 => ./libicui18ncognos.so.48 (0x00007f71dbdfc000)
libicuuccognos.so.48 => ./libicuuccognos.so.48 (0x00007f71dba7c000)
libicudatacognos.so.48 => ./libicudatacognos.so.48 (0x00007f71da5f0000)
libxcb.so.1 => /usr/lib64/libxcb.so.1 (0x0000003b18000000)
/lib64/ld-linux-x86-64.so.2 (0x0000003b14800000)
libXau.so.6 => /usr/lib64/libXau.so.6 (0x0000003b17c00000)

I did the same with the other binary - BmtMDProviderMain: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/SupClusterMember1/bin
ldd BmtMDProviderMain

ERROR: ld.so: object '/lib64/libfreebl3.so' from LD_PRELOAD cannot be preloaded: ignored.
linux-gate.so.1 =>  (0x006fa000)
libCCLIDOM.so => ./libCCLIDOM.so (0x0026a000)
libxerces-c.so.27 => ./libxerces-c.so.27 (0x002b5000)
libCCLCore.so => ./libCCLCore.so (0x00c59000)
libBIBusTKServer.so => ./libBIBusTKServer.so (0x001ac000)
libBIBusTK.so => ./libBIBusTK.so (0x001e1000)
libBMTUtil.so => ./libBMTUtil.so (0x00bcc000)
libX11.so.6 => /usr/X11R6/lib/libX11.so.6 (0x006fb000)
libdl.so.2 => /lib/libdl.so.2 (0x00221000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00ea7000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0079f000)
libm.so.6 => /lib/libm.so.6 (0x00226000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00695000)
libpthread.so.0 => /lib/libpthread.so.0 (0x006b3000)
libc.so.6 => /lib/libc.so.6 (0x0088a000)
libIBJStreamsDLL.so => ./libIBJStreamsDLL.so (0x006ce000)
libicuuccognos.so.48 => ./libicuuccognos.so.48 (0x00a21000)
libicudatacognos.so.48 => ./libicudatacognos.so.48 (0xf64e7000)
libz.so.1 => ./libz.so.1 (0x00250000)
libicui18ncognos.so.48 => ./libicui18ncognos.so.48 (0x00ec2000)
librt.so.1 => /lib/librt.so.1 (0x006ef000)
libCCLCFGAPI.so => ./libCCLCFGAPI.so (0x00b8f000)
libCCLHttptools.so => ./libCCLHttptools.so (0x00dc8000)
libcogipf2.so => ./libcogipf2.so (0x00dea000)
coglog4ccl.so => ./coglog4ccl.so (0x0677d000)
libCCLCryptoShared.so => ./libCCLCryptoShared.so (0x00bbc000)
libIBJHelpers.so => ./libIBJHelpers.so (0x00264000)
libBmtResources.so => ./libBmtResources.so (0x00bc5000)
libCST.so => ./libCST.so (0xf53a5000)
/lib/ld-linux.so.2 (0x00161000)
libCCLVirtualMemMgmt.so => ./libCCLVirtualMemMgmt.so (0x00e29000)
libtcmalloc_virtualmm.so.0 => ./libtcmalloc_virtualmm.so.0 (0x00e8d000)
libCAM_Crypto_Interface.so => ./libCAM_Crypto_Interface.so (0x03a48000)
libcrypto.so.0.9.8 => ./libcrypto.so.0.9.8 (0x071d2000)

In both cases, I was looking for Not Found exceptions ( ironically I took the above listings AFTER I'd solved the problem - see below ).

The Not Found exception indicates a missing library.

Initially, I thought that the solution was to add: -

export LD_PRELOAD=/lib64/libfreebl3.so

into the shell ( ~/.bashrc ) for the user that actually runs WAS - wasadmin in my case.

However, that was a herring rouge.

The actual solution was to add the missing dependent libraries. The main missing library, as per the exception: -

was libX11.so.6.

I did have the 64-bit version of this library: -

/usr/lib64/libX11.so.6
/usr/lib64/libX11.so.6.3.0

but NOT the 32-bit version: -

/usr/X11R6/lib/libX11.so.6
/usr/X11R6/lib/libX11.so.6.1

I installed XFree86-libs-3.3.6-20.i386.rpm to give me the 32-bit libraries; mainly because I didn't have access to the source Red Hat Enterprise Linux 6.4 ISO repository.

I'd also installed libstdc++, again 32-bit as well as 64-bit.

I am going to go back through the entire setup again, using a vanilla RHEL image, in order that I can document the end-to-end process.

Interestingly, I'm seeing the same Cognos Dispatcher ( Report Studio ) issue on AIX 7.1.

The solution appears to be the same.

On a PoC environment, where the Cognos Report Studio works OK, I do have libX11.a installed, as below: -

I've asked our AIX admin to install the X11.base.lib licensed program, and will see where we get to tomorrow.