All users are not created equally. Authorization is the process of ensuring that a user (or other computer system) has permission to perform a given act. IBM Business Process Manager defines a very fine-grained authorization model. Getting this model right – ensuring that only the right people have access to certain resources – is key to securing your Business Process Management environment. Using excerpts from J Keith Wood and Jens Engelke's new IBM Redbooks publication IBM Business Process Manager Security: Concepts and Guidance, here's the top 5 authorization security concerns we are seeing in Business Process Management today.
↧
Top 5 authorization security considerations in Business Process Management
↧
Slightly strange problem with the Integrated Solutions Console of WebSphere Application Server Network Deployment 8.0.0.5
My colleague, Simmo, hit a problem earlier today with the Integrated Solutions Console (ISC) in a newly built WAS 8.0.0.5 Deployment Manager profile.
When he logged in, he saw the normal navigation tree on the left-hand side, but nothing much on the right-hand side, apart from the Welcome pane - "Integrated Solutions Console provides a common administrative console for multiple products…".
When he clicked on the WebSphere Application Server link within the Welcome pane, he saw: -
SRVE0190E: File not found: /secure/layouts.
Also, in the Deployment Manager's SystemOut.log, he saw exceptions such as: -
[2/28/13 4:35:03:769 PST] 00000019 SystemOut O In DefinitionsXmlParser parse Exception occurred org.xml.sax.SAXParseException: The value of attribute "value" associated with an element type "put" must not contain the '<' character.
[2/28/13 4:35:03:823 PST] 00000019 servlet E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause action: java.io.FileNotFoundException: SRVE0190E: File not found: /JDBCProvider.content.main[2/28/13 4:35:06:651 PST] 00000019 SystemOut O In DefinitionsXmlParser parse Exception occurred org.xml.sax.SAXParseException: The value of attribute "name" associated with an element type "putList" must not contain the '<' character.
[2/28/13 4:35:06:692 PST] 00000019 servlet E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause action: java.io.FileNotFoundException: SRVE0190E: File not found: /DataSource.content.main
[2/28/13 4:35:06:692 PST] 00000019 servlet E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause action: java.io.FileNotFoundException: SRVE0190E: File not found: /DataSource.content.main
etc.
He found this old Technote from 2005: -
which provides an iFix ( for WAS 7.0 ! ), and also provided a local fix: -
Relink the console-defs.idx in the config root and install root.
Note: these commands should be entered on one line
$ rm <config_root>/systemApps/isclite.ear/isclite.war/WEB-INF/console-defs.idx
$ ln -s <install_root>/systemApps/isclite.ear/isclite.war/WEB-INF/console-defs.idx <config_root>/systemApps/isclite.ear/isclite.war/WEB-INF/console-defs.idx
He did this, and restarted the DM, but to no avail.
However, the Technote also included a temporary fix: -
run iscdeploy -restore
which, thankfully, did the job.
To achieve this, he ran: -
$ <install_root>/bin/iscdeploy.sh -restore
restarted the DM again, and … job done :-)
Well done, Simmo.
↧
↧
Fun with IBM HTTP Server
I was getting HTTP 404 for: -
https://10.222.36.21:8443/teamserver/faces/login.jsp
Looking at the IHS error.log, I saw: -
[Thu Feb 28 11:39:16 2013] [error] [client 28.103.96.87] File does not exist: /opt/IBM/HTTPServer/htdocsext/teamserver
and the Plugin log ( with LogLevel="Error" in plugin-cfg.xml ): -
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - DETAIL: ws_common: websphereShouldHandleRequest: trying to match a route for: vhost='10.222.36.21'; uri='/teamserver/faces/login.jsp'
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: webspherePortNumberForMatching: Using logical.
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:5065' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:9445' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:5063' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:9444' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:5061' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:9443' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Failed to match: 10.222.36.21:8443
This led me to check the WAS Virtual Hosts ( via the WAS ISC -> Environment -> Virtual hosts -> default_host ) and realised that 9443 was there but 8443 was NOT :-)
I changed it, regenerated the plugin, restarted the cluster and …. job done :-)
https://10.222.36.21:8443/teamserver/faces/login.jsp
Looking at the IHS error.log, I saw: -
[Thu Feb 28 11:39:16 2013] [error] [client 28.103.96.87] File does not exist: /opt/IBM/HTTPServer/htdocsext/teamserver
and the Plugin log ( with LogLevel="Error" in plugin-cfg.xml ): -
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - DETAIL: ws_common: websphereShouldHandleRequest: trying to match a route for: vhost='10.222.36.21'; uri='/teamserver/faces/login.jsp'
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: webspherePortNumberForMatching: Using logical.
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:5065' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:9445' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:5063' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:9444' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:5061' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Comparing '*:9443' to '10.222.36.21:8443' in VhostGroup: default_host
[Thu Feb 28 11:39:16 2013] 00003c42 04cfe950 - TRACE: ws_common: websphereVhostMatch: Failed to match: 10.222.36.21:8443
This led me to check the WAS Virtual Hosts ( via the WAS ISC -> Environment -> Virtual hosts -> default_host ) and realised that 9443 was there but 8443 was NOT :-)
I changed it, regenerated the plugin, restarted the cluster and …. job done :-)
Can you say "Doh!" ?
↧
Resolving WAS -> DB2 problems - create the darned database
If you get: -
[2/28/13 12:22:51:686 GMT] 00000067 DataSourceCon E DSRA8040I: Failed to connect to the DataSource. Encountered "": java.sql.SQLException: null DSRA0010E: SQL State = 58031, Error Code = -1,031
Check that the database has been created: -
$ db2 create database RTSDB automatic storage yes using codeset UTF-8 territory US pagesize 32768
DB20000I The CREATE DATABASE command completed successfully.
[2/28/13 12:22:51:686 GMT] 00000067 DataSourceCon E DSRA8040I: Failed to connect to the DataSource. Encountered "": java.sql.SQLException: null DSRA0010E: SQL State = 58031, Error Code = -1,031
Check that the database has been created: -
$ db2 create database RTSDB automatic storage yes using codeset UTF-8 territory US pagesize 32768
DB20000I The CREATE DATABASE command completed successfully.
↧
Importing SSL certificates into the WebSphere Application Server CACerts certificate store
So I saw this in my WAS logs ( for the IBM Business Monitor instance that hosts Cogos ): -
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/PCSR011.Support/logs/cogserver.log
CM-SYS-5157 Content Manager failed to notify the dispatcher "https://rhel6.uk.ibm.com:8443/p2pd/servlet/dispatch" of a running status change. CM-REQ-4128 An input/output error occurred executing an external request to the connector "CM". java.security.cert.CertificateException: CAM-CRP-1072 The certificate with the DN 'CN=rhel6.uk.ibm.com, O=ibm' issued by the Certificate Authority with the DN 'CN=rhel6.uk.ibm.com, O=ibm' is not trusted. Cause: java.security.cert.CertificateException: CAM-CRP-1072 The certificate with the DN 'CN=rhel6.uk.ibm.com, O=ibm' issued by the Certificate Authority with the DN 'CN=rhel6.uk.ibm.com, O=ibm' is not trusted. Runtime Exception stack trace: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: CAM-CRP-1072 The certificate with the DN 'CN=rhel6.uk.ibm.com, O=ibm' issued by the Certificate Authority with the DN 'CN=rhel6.uk.ibm.com, O=ibm' is not trusted. at com.ibm.jsse2.o.a(o.java:33) at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:649) at com.ibm.jsse2.kb.a(kb.java:56) at com.ibm.jsse2.kb.a(kb.java:502) at com.ibm.jsse2.lb.a(lb.java:107) at com.ibm.jsse2.lb.a(lb.java:570) at com.ibm.jsse2.kb.s(kb.java:327) at com.ibm.jsse2.kb.a(kb.java:529) at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:300) at com.ibm.jsse2.SSLSocketImpl.h(SSLSocketImpl.java:403) at com.ibm.jsse2.SSLSocketImpl.a(SSLSocketImpl.java:774) at com.ibm.jsse2.k.write(k.java:7) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:76) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:115) at org.apache.commons.httpclient.HttpConnection.write(HttpConnection.java:975) at org.apache.commons.httpclient.HttpConnection.write(HttpConnection.java:943) at com.cognos.cm.connectors.BusPostMethod.writeRequestBody(BusPostMethod.java:117) at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114) at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at com.cognos.cm.connectors.BusConnectionPool$BusConnection.execute(BusConnectionPool.java:85) at com.cognos.cm.connectors.CMConnector.CMConnector$CMConnectorRunnable.run(CMConnector.java:469) at java.lang.Thread.run(Thread.java:772) Caused by: java.security.cert.CertificateException: CAM-CRP-1072 The certificate with the DN 'CN=rhel6.uk.ibm.com, O=ibm' issued by the Certificate Authority with the DN 'CN=rhel6.uk.ibm.com, O=ibm' is not trusted. at com.cognos.accman.jcam.crypto.misc.CAMX509TrustManager14.checkServerTrusted(CAMX509TrustManager14.java:354) at com.ibm.jsse2.lb.a(lb.java:323) ... 21 more
This led me to the JVM's CACerts store: -
Dumped out the current list of CAs in CACerts
$ /opt/IBM/HTTPServer/java/jre/bin/ikeycmd -cert -list -db /opt/IBM/WebSphere/AppServer/java/jre/lib/security/cacerts -pw changeit -type jks > /tmp/cacerts.out
Used a simple shell script to remove these 77 CAs
for c in `cat /tmp/cacerts.out`
do
/opt/IBM/HTTPServer/java/jre/bin/ikeycmd -cert -delete -db /opt/IBM/WebSphere/AppServer/java/jre/lib/security/cacerts -pw changeit -type jks -label $c
done
do
/opt/IBM/HTTPServer/java/jre/bin/ikeycmd -cert -delete -db /opt/IBM/WebSphere/AppServer/java/jre/lib/security/cacerts -pw changeit -type jks -label $c
done
none of which are required for MY specific environment.
Used OpenSSL to retrieve the certificate from the IHS box to a file
$ openssl s_client -connect https://rhel6.uk.ibm.com:8443> ihs.cer
Edited the file down to the certificate itself
$ vi ihs.cer
<snip>
-----BEGIN CERTIFICATE-----
MIICzjCCAbagAwIBAgIEURUitDANBgkqhkiG9w0BAQUFADApMQwwCgYDVQQKEwNp
Ym0xGTAXBgNVBAMTEHJoZWw2LnVrLmlibS5jb20wHhcNMTMwMjA4MTYwNzE2WhcN
MTQwMjA4MTYwNzE2WjApMQwwCgYDVQQKEwNpYm0xGTAXBgNVBAMTEHJoZWw2LnVr
LmlibS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCDsSbw27hU
M+1xIMapH4KJD3maj046A6k2GyUzsvKzH25OA/cmSo4LQByAHXc6O9wlFAbsRf4H
NWZemrGPtk2DqJQkLUMnTckLL5hb3Vt0RocnoBfbPJgNB8jwkUocF22aLEJ0Btqi
ppCbX2Gmkg+vFNMPSkGL89W5QGcnxeAwWASJRcITIEsxJ8JNRqZtrCz2IiUrC7li
ef1eIObt7eW/tr6xLpup8K0DFtI1FK27wMTCSb9B84yWYyIWKm30cgPlUHZqJJfQ
Rt5veMdmVfWN0vA4t6ctux5bD/HmcjMHRvXrM98vU2juq8l4JBd2HXVRZzblCRom
baAz5oqT4D6XAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBAEAyzeQPW5Y88qUHg8sb
OpzzDE7pQMAudsBUlZG7hyz2hFXrao2P2h/JkOoOlvjmYm+e2A1GxO2DRVEj6tvw
V5AzjVw2XuVL0nRmaDDVOFgMsV+6LurXFj4TWG7nbFNvFNBHKGDzfbl9j4Zp8fsK
CGU1k8xQVQIykP6PptazjD5nAny4C1bWVDNNPiev7pJSR4Ki5tMuCNK5EHqeZ1PX
gO4kiBLDdS7RNgszxgkCdPF4pPvqis0tcl9zoXenzNOyEvJU9L1/Q7dhTq81uBMU
e0kh5b3OXPT92UD7xB87o8oUTlNrPfX4fKR6zTDEYLhgPTwYBGDpQJj6mD3wbnJV
a3I=
-----END CERTIFICATE-----
MIICzjCCAbagAwIBAgIEURUitDANBgkqhkiG9w0BAQUFADApMQwwCgYDVQQKEwNp
Ym0xGTAXBgNVBAMTEHJoZWw2LnVrLmlibS5jb20wHhcNMTMwMjA4MTYwNzE2WhcN
MTQwMjA4MTYwNzE2WjApMQwwCgYDVQQKEwNpYm0xGTAXBgNVBAMTEHJoZWw2LnVr
LmlibS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCDsSbw27hU
M+1xIMapH4KJD3maj046A6k2GyUzsvKzH25OA/cmSo4LQByAHXc6O9wlFAbsRf4H
NWZemrGPtk2DqJQkLUMnTckLL5hb3Vt0RocnoBfbPJgNB8jwkUocF22aLEJ0Btqi
ppCbX2Gmkg+vFNMPSkGL89W5QGcnxeAwWASJRcITIEsxJ8JNRqZtrCz2IiUrC7li
ef1eIObt7eW/tr6xLpup8K0DFtI1FK27wMTCSb9B84yWYyIWKm30cgPlUHZqJJfQ
Rt5veMdmVfWN0vA4t6ctux5bD/HmcjMHRvXrM98vU2juq8l4JBd2HXVRZzblCRom
baAz5oqT4D6XAgMBAAEwDQYJKoZIhvcNAQEFBQADggEBAEAyzeQPW5Y88qUHg8sb
OpzzDE7pQMAudsBUlZG7hyz2hFXrao2P2h/JkOoOlvjmYm+e2A1GxO2DRVEj6tvw
V5AzjVw2XuVL0nRmaDDVOFgMsV+6LurXFj4TWG7nbFNvFNBHKGDzfbl9j4Zp8fsK
CGU1k8xQVQIykP6PptazjD5nAny4C1bWVDNNPiev7pJSR4Ki5tMuCNK5EHqeZ1PX
gO4kiBLDdS7RNgszxgkCdPF4pPvqis0tcl9zoXenzNOyEvJU9L1/Q7dhTq81uBMU
e0kh5b3OXPT92UD7xB87o8oUTlNrPfX4fKR6zTDEYLhgPTwYBGDpQJj6mD3wbnJV
a3I=
-----END CERTIFICATE-----
</snip>
Imported the certificate into CACerts
$ /opt/IBM/HTTPServer/java/jre/bin/ikeycmd -cert -add -db /opt/IBM/WebSphere/AppServer/java/jre/lib/security/cacerts -pw changeit -type jks -file ihs.cer
Restarted my JVM ….
↧
↧
IBM Business Monitor 8.0.1 - DB2 or not DB2
This post is most definitely a Work-in-Progress, as I get to grips with IBM Business Monitor ( aka Business Activity Monitoring or BAM ).
INFO, "[main]", "Silent Execution Mode (start)"
EXEC, "[main]", "Loading configuration file"
SUCCESS, "[main]", "Completed successfully."
EXEC, "[Validation]", "Checking for errors and configuration integrity"
SUCCESS, "[Validation]", "Completed successfully."
EXEC, "[Cryptography]", "Generating cryptographic information"
SUCCESS, "[Cryptography]", "Completed successfully."
EXEC, "[Data Encryption]", "Checking integrity of encrypted data"
SUCCESS, "[Data Encryption]", "Completed successfully."
EXEC, "[Backup]", "Backing up configuration files"
SUCCESS, "[Backup]", "Completed successfully."
EXEC, "[Save Configuration]", "Saving configuration parameters"
SUCCESS, "[Save Configuration]", "Completed successfully."
EXEC, "[Checking upgrade status]", "Checking upgrade status"
SUCCESS, "[Checking upgrade status]", "Completed successfully."
EXEC, "[Cryptography]", "Generating cryptographic information"
SUCCESS, "[Cryptography]", "Completed successfully."
EXEC, "[''VMMAuth'']", "Testing ''WBM_Provider'' namespace."
SUCCESS, "[''VMMAuth'']", "Completed successfully."
EXEC, "[Trust Root Test]", "Checking for same Trust Root."
SUCCESS, "[Trust Root Test]", "Completed successfully."
EXEC, "[CSK Request Test]", "Checking CSK availability."
SUCCESS, "[CSK Request Test]", "Completed successfully."
EXEC, "[Content Manager database connection]", "Testing Content Manager database connection."
ERROR, "[Content Manager database connection]", "The database connection failed."
ERROR, "[Content Manager database connection]", "Content Manager failed to start because it could not load driver "com.ibm.db2.jcc.DB2Driver"."
FAILURE, "[Content Manager database connection]", "Failed."
INFO, "[main]", "Silent Execution Mode (end)"
127.0.0.1:9081639022013-02-08 16:29:02.408+0server.startup : 1LOGSV62351server.Audit.IPFStartServiceLogServiceSuccess<parameters><item name="Port"><![CDATA[9381]]></item><item name="Mode"><![CDATA[UDP]]></item><item name="Secure"><![CDATA[FALSE]]></item></parameters>
127.0.0.1:9081639022013-02-08 16:29:22.361+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2057 Creating content store tables (schema version 6.0022).
127.0.0.1:9081639022013-02-08 16:31:56.905+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal PRECACHEINITACTIONS task started.
127.0.0.1:9081639022013-02-08 16:31:56.967+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2216 A Content Manager internal PRECACHEINITACTIONS task is complete.
127.0.0.1:9081639022013-02-08 16:31:57.103+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal POSTINITACTIONS task started.
127.0.0.1:9081639022013-02-08 16:31:57.389+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2216 A Content Manager internal POSTINITACTIONS task is complete.
127.0.0.1:9081639022013-02-08 16:32:16.391+0Initialization_SESSInitialization_REQThread-54CM62351Audit.RTUsage.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal CMUPGRADESCRIPT task started.
127.0.0.1:9081639022013-02-08 16:32:17.257+0 Thread-54CM62351Audit.RTUsage.cms.CMStartServiceContentManagerServiceSuccessCM-SYS-5090 Content Manager build 10.1.6235.601 started (None;20110713180135, schema version 6.0022, implementation: CMDbStore - Java CMCache).
127.0.0.1:9081639022013-02-08 16:32:17.257+0 Thread-54CM62351Audit.RTUsage.cms.CMStartServiceContentManagerServiceInfCM-SYS-5159 Content Manager is running in active mode.
127.0.0.1:9081639022013-02-08 16:32:19.456+0pogoStartupnana0Thread-54DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServicerelationalMetadataServiceSuccess
127.0.0.1:9081639022013-02-08 16:32:19.588+0pogoStartupnana0Thread-54DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.contentmanager.coordinator.CMBootstrapgetActiveContentManagerContentManagerFailure<messages><message><messageString>DPR-CMI-4006 Unable to determine the active Content Manager. Will retry periodically.</messageString></message></messages>
127.0.0.1:9081639022013-02-08 16:32:19.605+0pogoStartupnana0Thread-54DISP62351Audit.Other.dispatcher.DISP.pogopogocom.cognos.pogo.contentmanager.coordinator.ActiveCMControlFailure<messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>DPR-DPR-1004 Expecting a BI Bus XML response but got: </messageString></message></messages>DPR-CMI-4007 Unable to perform an active Content Manager election on the local IP node. For more information, see the dispatcher and Content Manager detailed logs. Ensure that the local Content Manager service is started.DPR-DPR-1004 Expecting a BI Bus XML response but got: com.cognos.pogo.bibus.CommandExecutionException: DPR-DPR-1004 Expecting a BI Bus XML response but got: at com.cognos.pogo.bibus.BIBusCommand.handleDefaultException(BIBusCommand.java:294)at com.cognos.pogo.bibus.BIBusCommand.execute(BIBusCommand.java:194at com.cognos.pogo.contentmanager.StandbyContentManagerElectionCommand.executeIfLocal(StandbyContentManagerElectionCommand.java:53)at com.cognos.pogo.contentmanager.coordinator.ActiveCMControl.sendElectCommand(ActiveCMControl.java:230)at com.cognos.pogo.contentmanager.coordinator.ActiveCMControl.startTimer(ActiveCMControl.java:175)at com.cognos.pogo.contentmanager.coordinator.ActiveCMControl.startupFindCM(ActiveCMControl.java:142)at com.cognos.pogo.contentmanager.coordinator.RuntimeInfoPublishHandler.start(RuntimeInfoPublishHandler.java:113)at com.cognos.pogo.services.DefaultHandlerService.start(DefaultHandlerService.java:94)at com.cognos.pogo.services.DispatcherServices.startInititalServices(DispatcherServices.java:388)at com.cognos.pogo.transport.PogoServlet$PogoStartup.run(PogoServlet.java:690)at com.cognos.pogo.util.threads.SafeThread.safeRun(SafeThread.java:70)at com.cognos.pogo.util.threads.SafeThread.run(SafeThread.java:61)Caused by: com.cognos.pogo.bibus.CommandExecutionException: DPR-DPR-1004 Expecting a BI Bus XML response but got: at com.cognos.pogo.bibus.BIBusCommand.checkResponseContentType(BIBusCommand.java:326)at com.cognos.pogo.bibus.BIBusCommand.getSinglePartInputStream(BIBusCommand.java:273)at com.cognos.pogo.bibus.BIBusCommand.getInputStream(BIBusCommand.java:259)at com.cognos.pogo.bibus.BIBusCommand.parseResponse(BIBusCommand.java:248)at com.cognos.pogo.bibus.BIBusCommand.processResponse(BIBusCommand.java:242)at com.cognos.pogo.bibus.BIBusCommand.executeCommand(BIBusCommand.java:204)at com.cognos.pogo.bibus.BIBusCommand.execute(BIBusCommand.java:190)... 10 more
127.0.0.1:9081639022013-02-08 16:49:51.894+0nanastopper-runTimeInfoPublisherDISP62351Audit.Other.dispatcher.DISP.pogopogocom.cognos.pogo.services.DefaultHandlerServiceFailure<messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>null</messageString></message></messages>problem stopping handler runTimeInfoPublishernulljava.lang.NullPointerExceptionat com.cognos.pogo.contentmanager.coordinator.RuntimeInfoPublishHandler.stopAciveCMChecking(RuntimeInfoPublishHandler.java:273)at com.cognos.pogo.contentmanager.coordinator.RuntimeInfoPublishHandler.stop(RuntimeInfoPublishHandler.java:124)at com.cognos.pogo.services.DefaultHandlerService.stop(DefaultHandlerService.java:168)at com.cognos.pogo.services.DispatcherServices$ServiceStopper.safeRun(DispatcherServices.java:335)at com.cognos.pogo.util.threads.SafeThread.run(SafeThread.java:61)
127.0.0.1:9081639022013-02-08 16:49:51.894+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceCPS Producer Registration ServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.101+0stopper-eventManagementServiceEMS62351Audit.ems.EMSStopServiceEventServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.103+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceMetadataServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.111+0stopper-jobServiceJOBS62351Audit..JOBSStopServiceJobServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.115+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceDimensionManagementServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.165+0stopper-monitorServiceMS62351Audit.ms.MSStopServiceMonitorServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.169+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceReportServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.173+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceContentManagerCacheServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.177+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceCacheServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.177+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServicecamAsyncAASuccess
127.0.0.1:9081639022013-02-08 16:49:52.178+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceIBM Cognos Enhanced Search ServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.179+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceReportDataServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.187+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceGraphicsServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.193+0nanastopper-graphicsServiceDISP62351Audit.Other.dispatcher.DISP.pogopogocom.cognos.pogo.services.DefaultHandlerServiceFailure<messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>null</messageString></message></messages>problem stopping handler graphicsServiceHandlernulljava.lang.NullPointerExceptionat com.ibm.cgsBus.service.GraphicsServiceHandler.stop(Unknown Source)at com.cognos.pogo.services.DefaultHandlerService.stop(DefaultHandlerService.java:168)at com.cognos.pogo.services.DispatcherServices$ServiceStopper.safeRun(DispatcherServices.java:335)at com.cognos.pogo.util.threads.SafeThread.run(SafeThread.java:61)
127.0.0.1:9081639022013-02-08 16:49:52.196+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceSystemServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.197+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServicebuxServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.223+0stopper-logServiceLOGSV62351server.Audit.IPFStopServiceLogServiceSuccess<parameters><item name="Port"><![CDATA[9381]]></item><item name="Mode"><![CDATA[UDP]]></item><item name="Secure"><![CDATA[FALSE]]></item></parameters>
127.0.0.1:9081550872013-03-02 17:07:19.386+0server.startup : 0LOGSV62351server.Audit.IPFStartServiceLogServiceSuccess<parameters><item name="Port"><![CDATA[9381]]></item><item name="Mode"><![CDATA[UDP]]></item><item name="Secure"><![CDATA[FALSE]]></item></parameters>
127.0.0.1:9081550872013-03-02 17:07:27.722+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal PRECACHEINITACTIONS task started.
127.0.0.1:9081550872013-03-02 17:07:27.976+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal POSTINITACTIONS task started.
127.0.0.1:9081550872013-03-02 17:07:27.725+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2216 A Content Manager internal PRECACHEINITACTIONS task is complete.
127.0.0.1:9081550872013-03-02 17:07:28.072+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2216 A Content Manager internal POSTINITACTIONS task is complete.
127.0.0.1:9081550872013-03-02 17:07:28.238+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal CMUPGRADESCRIPT task started.
127.0.0.1:9081550872013-03-02 17:07:28.911+0 Thread-54CM62351Audit.RTUsage.cms.CMStartServiceContentManagerServiceSuccessCM-SYS-5090 Content Manager build 10.1.6235.601 started (None;20110713180135, schema version 6.0022, implementation: CMDbStore - Java CMCache).
127.0.0.1:9081550872013-03-02 17:07:28.913+0 Thread-54CM62351Audit.RTUsage.cms.CMStartServiceContentManagerServiceInfCM-SYS-5159 Content Manager is running in active mode.
127.0.0.1:9081550872013-03-02 17:07:29.357+0pogoStartupnana0Thread-54DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServicerelationalMetadataServiceSuccess
127.0.0.1:9081550872013-03-02 17:07:29.587+0BootstrapConfigurePublishna0Thread-54DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.contentmanager.coordinator.CMBootstrapStartServicedispatcherBootstrapInfoDPR-DPR-1002 Successfully registered the dispatcher http://rhel6.uk.ibm.com:9081/p2pd in Content Manager.
127.0.0.1:9081550872013-03-02 17:08:03.321+0Thread-76AAA62351Audit.cms.CAM.AAAStartServicecamAsyncServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:14.125+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceFragment Server ServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:14.129+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceBatchReportServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:14.131+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.async.service.connection.bibustkserver.BIBusTKServerManagerStartcom.cognos.pogo.reportserviceInfoStart a External Process because the start count 1 is larger than 0.
127.0.0.1:9081550872013-03-02 17:08:39.399+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceQueryServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:39.903+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceSystemServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:39.902+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServicebuxServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:46.171+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceGraphicsServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:46.281+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceReportDataServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:51.733+0Thread-76xts62351Audit.ps.xtsUpgradePresentationServiceSuccessPRS-ACF-0413 Successfully wrote the upgraded cogadmin/system.xml file.
127.0.0.1:9081550872013-03-02 17:09:00.803+0Thread-76xts62351Audit.ps.xtsStartServicePresentationServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:00.804+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceIBM Cognos Enhanced Search ServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:00.804+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServicecamAsyncAASuccess
127.0.0.1:9081550872013-03-02 17:09:00.808+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceCacheServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:00.808+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceContentManagerCacheServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:00.985+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceReportServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:01.008+0Thread-76MS62351Audit.ms.MSStartServiceMonitorServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:01.008+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceDimensionManagementServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:01.009+0Thread-76JOBS62351Audit..JOBSStartServiceJobServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:01.289+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceMetadataServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:01.434+0Thread-76EMS62351Audit.ems.EMSStartServiceEventServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:13.223+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceCPS Producer Registration ServiceSuccess
BAM includes, and makes use, of Cognos Business Intelligence 10.1 ( aka Cognos BI ), which is installed with the product.
Cognos BI runs, in this particular instance, on one of the WebSphere Application Server (WAS) instances that forms the BAM infrastructure.
In my case, I'm following good practice for scalability and resilience by deploying BAM in the so-called "Gold Standard' deployment pattern, similar to IBM Business Process Manager.
This involves BAM ( or BPM ) being clustered across four discrete WAS instances: -
Messaging engine cluster
Business Space application
- Messaging engine for the IBM Business Monitor bus
- Messaging engine for the common event infrastructure (CEI) bus
Support cluster
- CEI event service
- Event emitter services
- Action services
- Monitor scheduled services
- IBM Cognos Business Intelligence service
Application cluster
- Monitor model applications Web cluster
Business Space application
- Business Space widgets
- Representational State Transfer (REST) services application
and is known as the "Remote Messaging, Remote Support, and Web topology pattern".
The IBM documentation: -
explains this, and other topologies, further.
So, as is evident from above, Cognos is deployed onto the Support cluster.
I had assumed that Cognos would be a well-behaved client of WAS, and make use of JNDI and JDBC to connect, via WAS, to it's underlying database - COGNOSCS.
However, now I'm not quite so sure.
I have already blogged about the need to configure WAS to make use of the 32-bit native DB2 drivers: -
and thought that was all I had to do.
However, I spent the best part of a day struggling to work out why Cognos wouldn't fully start.
In terms of logging, apart from the WAS SystemOut.log file: -
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/PCSR011.Support/SystemOut.log
Cognos also has its own set of logs: -
$ ls -al /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/PCSR011.Support/logs
total 44
drwxr-xr-x 3 wasadmin mqbrkrs 4096 Mar 2 17:08 .
drwxr-xr-x 30 wasadmin mqbrkrs 4096 Feb 8 14:04 ..
-rw-r--r-- 1 wasadmin wasadmins 1401 Mar 3 09:40 cmMetrics_2013-03-02.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:06 cogconfigipf.log
-rw-r--r-- 1 wasadmin wasadmins 1729 Mar 2 17:06 cogconfig_response.csv
-rw-r--r-- 1 wasadmin wasadmins 311 Mar 2 17:07 cogserver_default_55087.log
-rw-r--r-- 1 wasadmin wasadmins 155 Mar 2 17:08 cogserver_default_55441.log
-rw-r--r-- 1 wasadmin wasadmins 6504 Mar 2 17:09 cogserver.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:06 ipfInternal_cpp.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:06 ipfInternal_java.log
-rw-r--r-- 1 wasadmin wasadmins 7870 Mar 3 07:48 pogo_2013-03-02.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:07 pogoMetrics_2013-03-02.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:08 qs_cmMetrics_2013-03-02.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:08 qs_pogo_2013-03-02.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:08 qs_pogoMetrics_2013-03-02.log
drwxr-xr-x 4 wasadmin wasadmins 4096 Mar 2 17:08 XQE
drwxr-xr-x 30 wasadmin mqbrkrs 4096 Feb 8 14:04 ..
-rw-r--r-- 1 wasadmin wasadmins 1401 Mar 3 09:40 cmMetrics_2013-03-02.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:06 cogconfigipf.log
-rw-r--r-- 1 wasadmin wasadmins 1729 Mar 2 17:06 cogconfig_response.csv
-rw-r--r-- 1 wasadmin wasadmins 311 Mar 2 17:07 cogserver_default_55087.log
-rw-r--r-- 1 wasadmin wasadmins 155 Mar 2 17:08 cogserver_default_55441.log
-rw-r--r-- 1 wasadmin wasadmins 6504 Mar 2 17:09 cogserver.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:06 ipfInternal_cpp.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:06 ipfInternal_java.log
-rw-r--r-- 1 wasadmin wasadmins 7870 Mar 3 07:48 pogo_2013-03-02.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:07 pogoMetrics_2013-03-02.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:08 qs_cmMetrics_2013-03-02.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:08 qs_pogo_2013-03-02.log
-rw-r--r-- 1 wasadmin wasadmins 0 Mar 2 17:08 qs_pogoMetrics_2013-03-02.log
drwxr-xr-x 4 wasadmin wasadmins 4096 Mar 2 17:08 XQE
More specifically: -
$ cat cogconfig_response.csv
INFO, "[main]", "Silent Execution Mode (start)"
EXEC, "[main]", "Loading configuration file"
SUCCESS, "[main]", "Completed successfully."
EXEC, "[Validation]", "Checking for errors and configuration integrity"
SUCCESS, "[Validation]", "Completed successfully."
EXEC, "[Cryptography]", "Generating cryptographic information"
SUCCESS, "[Cryptography]", "Completed successfully."
EXEC, "[Data Encryption]", "Checking integrity of encrypted data"
SUCCESS, "[Data Encryption]", "Completed successfully."
EXEC, "[Backup]", "Backing up configuration files"
SUCCESS, "[Backup]", "Completed successfully."
EXEC, "[Save Configuration]", "Saving configuration parameters"
SUCCESS, "[Save Configuration]", "Completed successfully."
EXEC, "[Checking upgrade status]", "Checking upgrade status"
SUCCESS, "[Checking upgrade status]", "Completed successfully."
EXEC, "[Cryptography]", "Generating cryptographic information"
SUCCESS, "[Cryptography]", "Completed successfully."
EXEC, "[''VMMAuth'']", "Testing ''WBM_Provider'' namespace."
SUCCESS, "[''VMMAuth'']", "Completed successfully."
EXEC, "[Trust Root Test]", "Checking for same Trust Root."
SUCCESS, "[Trust Root Test]", "Completed successfully."
EXEC, "[CSK Request Test]", "Checking CSK availability."
SUCCESS, "[CSK Request Test]", "Completed successfully."
EXEC, "[Content Manager database connection]", "Testing Content Manager database connection."
ERROR, "[Content Manager database connection]", "The database connection failed."
ERROR, "[Content Manager database connection]", "Content Manager failed to start because it could not load driver "com.ibm.db2.jcc.DB2Driver"."
FAILURE, "[Content Manager database connection]", "Failed."
INFO, "[main]", "Silent Execution Mode (end)"
and: -
$ cat cogserver.log
127.0.0.1:9081639022013-02-08 16:29:02.408+0server.startup : 1LOGSV62351server.Audit.IPFStartServiceLogServiceSuccess<parameters><item name="Port"><![CDATA[9381]]></item><item name="Mode"><![CDATA[UDP]]></item><item name="Secure"><![CDATA[FALSE]]></item></parameters>
127.0.0.1:9081639022013-02-08 16:29:22.361+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2057 Creating content store tables (schema version 6.0022).
127.0.0.1:9081639022013-02-08 16:31:56.905+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal PRECACHEINITACTIONS task started.
127.0.0.1:9081639022013-02-08 16:31:56.967+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2216 A Content Manager internal PRECACHEINITACTIONS task is complete.
127.0.0.1:9081639022013-02-08 16:31:57.103+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal POSTINITACTIONS task started.
127.0.0.1:9081639022013-02-08 16:31:57.389+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2216 A Content Manager internal POSTINITACTIONS task is complete.
127.0.0.1:9081639022013-02-08 16:32:16.391+0Initialization_SESSInitialization_REQThread-54CM62351Audit.RTUsage.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal CMUPGRADESCRIPT task started.
127.0.0.1:9081639022013-02-08 16:32:17.257+0 Thread-54CM62351Audit.RTUsage.cms.CMStartServiceContentManagerServiceSuccessCM-SYS-5090 Content Manager build 10.1.6235.601 started (None;20110713180135, schema version 6.0022, implementation: CMDbStore - Java CMCache).
127.0.0.1:9081639022013-02-08 16:32:17.257+0 Thread-54CM62351Audit.RTUsage.cms.CMStartServiceContentManagerServiceInfCM-SYS-5159 Content Manager is running in active mode.
127.0.0.1:9081639022013-02-08 16:32:19.456+0pogoStartupnana0Thread-54DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServicerelationalMetadataServiceSuccess
127.0.0.1:9081639022013-02-08 16:32:19.588+0pogoStartupnana0Thread-54DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.contentmanager.coordinator.CMBootstrapgetActiveContentManagerContentManagerFailure<messages><message><messageString>DPR-CMI-4006 Unable to determine the active Content Manager. Will retry periodically.</messageString></message></messages>
127.0.0.1:9081639022013-02-08 16:32:19.605+0pogoStartupnana0Thread-54DISP62351Audit.Other.dispatcher.DISP.pogopogocom.cognos.pogo.contentmanager.coordinator.ActiveCMControlFailure<messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>DPR-DPR-1004 Expecting a BI Bus XML response but got: </messageString></message></messages>DPR-CMI-4007 Unable to perform an active Content Manager election on the local IP node. For more information, see the dispatcher and Content Manager detailed logs. Ensure that the local Content Manager service is started.DPR-DPR-1004 Expecting a BI Bus XML response but got: com.cognos.pogo.bibus.CommandExecutionException: DPR-DPR-1004 Expecting a BI Bus XML response but got: at com.cognos.pogo.bibus.BIBusCommand.handleDefaultException(BIBusCommand.java:294)at com.cognos.pogo.bibus.BIBusCommand.execute(BIBusCommand.java:194at com.cognos.pogo.contentmanager.StandbyContentManagerElectionCommand.executeIfLocal(StandbyContentManagerElectionCommand.java:53)at com.cognos.pogo.contentmanager.coordinator.ActiveCMControl.sendElectCommand(ActiveCMControl.java:230)at com.cognos.pogo.contentmanager.coordinator.ActiveCMControl.startTimer(ActiveCMControl.java:175)at com.cognos.pogo.contentmanager.coordinator.ActiveCMControl.startupFindCM(ActiveCMControl.java:142)at com.cognos.pogo.contentmanager.coordinator.RuntimeInfoPublishHandler.start(RuntimeInfoPublishHandler.java:113)at com.cognos.pogo.services.DefaultHandlerService.start(DefaultHandlerService.java:94)at com.cognos.pogo.services.DispatcherServices.startInititalServices(DispatcherServices.java:388)at com.cognos.pogo.transport.PogoServlet$PogoStartup.run(PogoServlet.java:690)at com.cognos.pogo.util.threads.SafeThread.safeRun(SafeThread.java:70)at com.cognos.pogo.util.threads.SafeThread.run(SafeThread.java:61)Caused by: com.cognos.pogo.bibus.CommandExecutionException: DPR-DPR-1004 Expecting a BI Bus XML response but got: at com.cognos.pogo.bibus.BIBusCommand.checkResponseContentType(BIBusCommand.java:326)at com.cognos.pogo.bibus.BIBusCommand.getSinglePartInputStream(BIBusCommand.java:273)at com.cognos.pogo.bibus.BIBusCommand.getInputStream(BIBusCommand.java:259)at com.cognos.pogo.bibus.BIBusCommand.parseResponse(BIBusCommand.java:248)at com.cognos.pogo.bibus.BIBusCommand.processResponse(BIBusCommand.java:242)at com.cognos.pogo.bibus.BIBusCommand.executeCommand(BIBusCommand.java:204)at com.cognos.pogo.bibus.BIBusCommand.execute(BIBusCommand.java:190)... 10 more
127.0.0.1:9081639022013-02-08 16:49:51.894+0nanastopper-runTimeInfoPublisherDISP62351Audit.Other.dispatcher.DISP.pogopogocom.cognos.pogo.services.DefaultHandlerServiceFailure<messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>null</messageString></message></messages>problem stopping handler runTimeInfoPublishernulljava.lang.NullPointerExceptionat com.cognos.pogo.contentmanager.coordinator.RuntimeInfoPublishHandler.stopAciveCMChecking(RuntimeInfoPublishHandler.java:273)at com.cognos.pogo.contentmanager.coordinator.RuntimeInfoPublishHandler.stop(RuntimeInfoPublishHandler.java:124)at com.cognos.pogo.services.DefaultHandlerService.stop(DefaultHandlerService.java:168)at com.cognos.pogo.services.DispatcherServices$ServiceStopper.safeRun(DispatcherServices.java:335)at com.cognos.pogo.util.threads.SafeThread.run(SafeThread.java:61)
127.0.0.1:9081639022013-02-08 16:49:51.894+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceCPS Producer Registration ServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.101+0stopper-eventManagementServiceEMS62351Audit.ems.EMSStopServiceEventServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.103+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceMetadataServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.111+0stopper-jobServiceJOBS62351Audit..JOBSStopServiceJobServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.115+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceDimensionManagementServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.165+0stopper-monitorServiceMS62351Audit.ms.MSStopServiceMonitorServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.169+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceReportServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.173+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceContentManagerCacheServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.177+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceCacheServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.177+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServicecamAsyncAASuccess
127.0.0.1:9081639022013-02-08 16:49:52.178+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceIBM Cognos Enhanced Search ServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.179+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceReportDataServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.187+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceGraphicsServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.193+0nanastopper-graphicsServiceDISP62351Audit.Other.dispatcher.DISP.pogopogocom.cognos.pogo.services.DefaultHandlerServiceFailure<messages><message><messageString>DPR-DPR-1035 Dispatcher detected an error.</messageString></message><message><messageString>null</messageString></message></messages>problem stopping handler graphicsServiceHandlernulljava.lang.NullPointerExceptionat com.ibm.cgsBus.service.GraphicsServiceHandler.stop(Unknown Source)at com.cognos.pogo.services.DefaultHandlerService.stop(DefaultHandlerService.java:168)at com.cognos.pogo.services.DispatcherServices$ServiceStopper.safeRun(DispatcherServices.java:335)at com.cognos.pogo.util.threads.SafeThread.run(SafeThread.java:61)
127.0.0.1:9081639022013-02-08 16:49:52.196+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServiceSystemServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.197+0shutdownnana0Non-deferrable Alarm : 2DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStopServicebuxServiceSuccess
127.0.0.1:9081639022013-02-08 16:49:52.223+0stopper-logServiceLOGSV62351server.Audit.IPFStopServiceLogServiceSuccess<parameters><item name="Port"><![CDATA[9381]]></item><item name="Mode"><![CDATA[UDP]]></item><item name="Secure"><![CDATA[FALSE]]></item></parameters>
So, thus far, I've not managed to get to the bottom of why Cognos appears to have an issue with DB2, given that WAS is connecting quite happily using JNDI/JDBC.
As per the aforementioned blog post: -
I thought I'd fixed the core issue of allowing Cognos access to the native 32-bit JDBC driver, by setting LD_LIBRARY_PATH to: -
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/PCSR011.Support/bin64:/opt/ibm/db2/V9.7/lib32/
but it's still not happy, as can be seen above.
In addition, I was seeing the following exception: -
DPR-ERR-2109 The dispatcher cannot service the request at this time. The dispatcher is still initializing. Contact administrator if this problem persists.
when attempting to access the Cognos Dispatcher URL: -
Despite reading scores of IBM documentation: -
In the end, I looked back through a design document that I'd recently written for an IBM Business Monitor client, who's actually using Oracle rather than DB2, where I'd recommended that they use Dynamic Query Mode, which makes full use of WAS' own JDBC driver, rather than needing a native, client-side driver.
This Technote describes this more fully: -
but, in essence, it's a matter of setting a generic JVM argument ( for the Support cluster member ): -
-Dinit.dqm.enabled=true
and restarting the cluster member.
Having done this, I still see: -
<snip>
EXEC, "[Content Manager database connection]", "Testing Content Manager database connection."
ERROR, "[Content Manager database connection]", "The database connection failed."
ERROR, "[Content Manager database connection]", "Content Manager failed to start because it could not load driver "com.ibm.db2.jcc.DB2Driver"."
FAILURE, "[Content Manager database connection]", "Failed."
</snip>
ERROR, "[Content Manager database connection]", "The database connection failed."
ERROR, "[Content Manager database connection]", "Content Manager failed to start because it could not load driver "com.ibm.db2.jcc.DB2Driver"."
FAILURE, "[Content Manager database connection]", "Failed."
</snip>
in cogconfig_response.csv, but Cognos now behaves itself perfectly, as per conserver.log: -
$ cat cogserver.log
127.0.0.1:9081550872013-03-02 17:07:19.386+0server.startup : 0LOGSV62351server.Audit.IPFStartServiceLogServiceSuccess<parameters><item name="Port"><![CDATA[9381]]></item><item name="Mode"><![CDATA[UDP]]></item><item name="Secure"><![CDATA[FALSE]]></item></parameters>
127.0.0.1:9081550872013-03-02 17:07:27.722+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal PRECACHEINITACTIONS task started.
127.0.0.1:9081550872013-03-02 17:07:27.976+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal POSTINITACTIONS task started.
127.0.0.1:9081550872013-03-02 17:07:27.725+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2216 A Content Manager internal PRECACHEINITACTIONS task is complete.
127.0.0.1:9081550872013-03-02 17:07:28.072+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2216 A Content Manager internal POSTINITACTIONS task is complete.
127.0.0.1:9081550872013-03-02 17:07:28.238+0Initialization_SESSInitialization_REQThread-54CM62351Audit.cms.CMStartServiceContentManagerServiceInfoCM-SYS-2215 A Content Manager internal CMUPGRADESCRIPT task started.
127.0.0.1:9081550872013-03-02 17:07:28.911+0 Thread-54CM62351Audit.RTUsage.cms.CMStartServiceContentManagerServiceSuccessCM-SYS-5090 Content Manager build 10.1.6235.601 started (None;20110713180135, schema version 6.0022, implementation: CMDbStore - Java CMCache).
127.0.0.1:9081550872013-03-02 17:07:28.913+0 Thread-54CM62351Audit.RTUsage.cms.CMStartServiceContentManagerServiceInfCM-SYS-5159 Content Manager is running in active mode.
127.0.0.1:9081550872013-03-02 17:07:29.357+0pogoStartupnana0Thread-54DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServicerelationalMetadataServiceSuccess
127.0.0.1:9081550872013-03-02 17:07:29.587+0BootstrapConfigurePublishna0Thread-54DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.contentmanager.coordinator.CMBootstrapStartServicedispatcherBootstrapInfoDPR-DPR-1002 Successfully registered the dispatcher http://rhel6.uk.ibm.com:9081/p2pd in Content Manager.
127.0.0.1:9081550872013-03-02 17:08:03.321+0Thread-76AAA62351Audit.cms.CAM.AAAStartServicecamAsyncServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:14.125+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceFragment Server ServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:14.129+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceBatchReportServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:14.131+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.async.service.connection.bibustkserver.BIBusTKServerManagerStartcom.cognos.pogo.reportserviceInfoStart a External Process because the start count 1 is larger than 0.
127.0.0.1:9081550872013-03-02 17:08:39.399+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceQueryServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:39.903+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceSystemServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:39.902+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServicebuxServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:46.171+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceGraphicsServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:46.281+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceReportDataServiceSuccess
127.0.0.1:9081550872013-03-02 17:08:51.733+0Thread-76xts62351Audit.ps.xtsUpgradePresentationServiceSuccessPRS-ACF-0413 Successfully wrote the upgraded cogadmin/system.xml file.
127.0.0.1:9081550872013-03-02 17:09:00.803+0Thread-76xts62351Audit.ps.xtsStartServicePresentationServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:00.804+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceIBM Cognos Enhanced Search ServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:00.804+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServicecamAsyncAASuccess
127.0.0.1:9081550872013-03-02 17:09:00.808+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceCacheServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:00.808+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceContentManagerCacheServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:00.985+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceReportServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:01.008+0Thread-76MS62351Audit.ms.MSStartServiceMonitorServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:01.008+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceDimensionManagementServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:01.009+0Thread-76JOBS62351Audit..JOBSStartServiceJobServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:01.289+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceMetadataServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:01.434+0Thread-76EMS62351Audit.ems.EMSStartServiceEventServiceSuccess
127.0.0.1:9081550872013-03-02 17:09:13.223+0nanaThread-76DISP62351Audit.Other.dispatcher.DISP.com.cognos.pogo.services.DispatcherServicesStartServiceCPS Producer Registration ServiceSuccess
and I can access the Cognos URLs: -
http://rhel6.uk.ibm.com:9081/p2pd/servlet/dispatch/ext?b_action=xts.run&m=portal/welcome/welcome.xts
and also add/remove/edit Cognos iWidgets within Business Space: -
One final point - it's sometimes necessary to modify the Cognos configuration outside of the WAS/BAM environment.
This can be achieved in one of two ways: -
(1) Use the cogconfig.sh GUI -
$ /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/PCSR011.Support/bin64/cogconfig.sh
This requires a GUI environment ( X11 or VNC on Linux ), which may be a problem for some Unix boxes.
For me, I accessed my RHEL VM as follows: -
$ ssh -X wasadmin@rhel6
and then started the tool: -
$ source /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/setupCmdLine.sh
$ /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/PCSR011.Support/bin64/cogconfig.shwhich loads the GUI up via my Mac's X11 client Xquartz.
(2) Use the wsadmin scripts that come with BAM: -
wbmDeployCEIEventService to create and configure the CEI event service and its required resources for receiving and sending events
wbmConfigureEventEmitterFactory to configure the outbound event emitter factory for sending events
wbmDeployMessagingEngine to install and configure the messaging engine and service integration bus
wbmDeployActionServices to install and configure the action services for invoking actions
wbmDeployScheduledServices to install and configure the Monitor scheduled services for scheduling recurring services
wbmDeployEventEmitterServices to install and configure the JMS and REST event emitter services
wbmDeployBPMEmitterServices to install and configure the BPM emitter service for use by IBM BPM
wbmDeployCognosService to install the IBM Cognos® Business Intelligence service for multidimensional analysis, or wbmSetCognosDispatcher to point to an IBM Cognos BI service that is already installed
wbmSetCognosDatabaseUser to set the IBM Cognos BI content store database password, and wbmSetCognosAdminUser to set the IBM Cognos BI administrator password.
as per the following link: -
Of course, there is a third option - one that I would not recommend - to hand-edit the Cognos configuration file: -
/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/cognos/PCSR011.Support/configuration/cogstartup.xml
This is, I believe, modified ( and, more importantly, validated ) by options (1) and (2) above.
As I say, this is a W-I-P. Next on my list is to find out whether I can configure the Cognos dispatcher to work with SSL.
I did try changing the dispatcher endpoints to HTTPS URLs, but then got exceptions such as: -
CM-SYS-5157 Content Manager failed to notify the dispatcher "https://rhel6.uk.ibm.com:8443/p2pd/servlet/dispatch" of a running status change. CM-REQ-4128 An input/output error occurred executing an external request to the connector "CM". java.security.cert.CertificateException: CAM-CRP-1072 The certificate with the DN 'CN=rhel6.uk.ibm.com, O=ibm' issued by the Certificate Authority with the DN 'CN=rhel6.uk.ibm.com, O=ibm' is not trusted. Cause: java.security.cert.CertificateException: CAM-CRP-1072 The certificate with the DN 'CN=rhel6.uk.ibm.com, O=ibm' issued by the Certificate Authority with the DN 'CN=rhel6.uk.ibm.com, O=ibm' is not trusted. Runtime Exception stack trace: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: CAM-CRP-1072 The certificate with the DN 'CN=rhel6.uk.ibm.com, O=ibm' issued by the Certificate Authority with the DN 'CN=rhel6.uk.ibm.com, O=ibm' is not trusted.
but that's a problem for another day.
↧
Configuring IBM Business Process Management Human Task Management widgets for use in WebSphere Portal
Saw this on Twitter earlier: -
The Human Task Management widgets in IBM®Business Process Manager V7.5 are designed to run in a Business Space environment, but they can also be run in IBM WebSphere®Portal V7. This article describes how to install, configure, and customize these widgets in a Portal environment. This enables you to build a unified environment of portlets and widgets within a single web portal page, so that you can run BPM processes and tasks in your portal pages. This content is part of the IBM Business Process Management Journal.
↧
Performance issues while accessing LDAP in Federated repository environment
Also saw this on Twitter today: -
Problem(Abstract)
Slow response time is experienced while VMM makes calls to LDAP during login, membership resolution, search, and so on.
Symptom
Slow response time of VMM when configured with LDAP.
Cause
There could be several cause for slow response time: amount of data in LDAP, how data is stored in LDAP, LDAP response time, how the LDAP repository is configured in VMM, and so on. Review the Resolving the problem section and find the cause that might be causing the issue in your environment.
Slow response time is experienced while VMM makes calls to LDAP during login, membership resolution, search, and so on.
Symptom
Slow response time of VMM when configured with LDAP.
Cause
There could be several cause for slow response time: amount of data in LDAP, how data is stored in LDAP, LDAP response time, how the LDAP repository is configured in VMM, and so on. Review the Resolving the problem section and find the cause that might be causing the issue in your environment.
↧
Fun with Linux disks
So this morning, when I landed into work, I saw an unexpected set of symptoms from my partially built IBM Business Monitor environment.
First of all, I saw this when I attempted to list my WAS profiles: -
$ /opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/bin/manageprofiles.sh -list
JVMSHRC226E Error opening shared class cache file
JVMSHRC336E Port layer error code = -300
JVMSHRC337E Platform error message: Read-only file system
JVMJ9VM015W Initialization error for library j9shr26(11): JVMJ9VM009E J9VMDllMain failed
Could not create the Java virtual machine.
Then I saw these errors: -
JVMSHRC336E Port layer error code = -300
JVMSHRC337E Platform error message: Read-only file system
JVMJ9VM015W Initialization error for library j9shr26(11): JVMJ9VM009E J9VMDllMain failed
Could not create the Java virtual machine.
Then I saw these errors: -
<snip>
[3/1/13 15:30:05:202 GMT] 0000001c MMRoutingConf E com.ibm.wbimonitor.lifecycle.routing.MMRoutingConfigFlowDaemon checkForStateChanges CWMLC0274E: An error occurred while trying to determine the state of an MM. This will be retried shortly. The condition was caused by: com.ibm.wbimonitor.persistence.metamodel.spi.MetaModelPersistenceException: com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=MONITOR.META_MODEL_UNVERSIONED_T, DRIVER=4.11.69.
[3/1/13 15:30:05:740 GMT] 0000001b LifecycleStop E com.ibm.wbimonitor.lifecycle.LifecycleStopRequestScanTask run() CWMLC0012E: Unexpected exception [com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=MONITOR.META_MODEL_T, DRIVER=4.11.69].
</snip>
[3/1/13 15:30:05:740 GMT] 0000001b LifecycleStop E com.ibm.wbimonitor.lifecycle.LifecycleStopRequestScanTask run() CWMLC0012E: Unexpected exception [com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=MONITOR.META_MODEL_T, DRIVER=4.11.69].
</snip>
in the SystemOut.log file for the Deployment Manager, even though I knew that (a) it was fine on Friday and (b) that DB2 was up-and-running ( validated via the command db2 connect to MONITOR ).
When I went to check the file systems on the box: -
$ mount
/dev/mapper/vglinux-rootlv on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/mapper/vglinux-tmplv on /tmp type ext3 (rw)
/dev/mapper/vglinux-varlv on /var type ext3 (rw)
/dev/dasda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/mapper/vgapp-optlv on /opt type ext3 (rw)
/dev/mapper/vgapp-homelv on /home type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
LBPM002L:/store/ on /store type nfs (rw,addr=10.222.36.21)
mount: warning: /etc/mtab is not writable (e.g. read-only filesystem).
It's possible that information reported by mount(8) is not
up to date. For actual information about system mount points
check the /proc/mounts file.
When I attempted to change the RWX permissions for /tmp: -
$ chmod 777 /tmp
chmod: changing permissions of `/tmp': Read-only file system
When I attempted to update the locate database ( as root ): -
$ updatedb
When I attempted to update the locate database ( as root ): -
$ updatedb
updatedb: can not open a temporary file for `/var/lib/mlocate/mlocate.db'
At this point, I figured that something serious had occurred; this was validated by messages such as: -
<snip>
At this point, I figured that something serious had occurred; this was validated by messages such as: -
<snip>
EXT3-fs error (device dm-1): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only
EXT3-fs error (device dm-1): ext3_lookup: unlinked inode 229390 in dir #229378
EXT3-fs error (device dm-1): ext3_lookup: unlinked inode 229384 in dir #229378
EXT3-fs error (device dm-1): ext3_lookup: unlinked inode 229390 in dir #229378
EXT3-fs error (device dm-1): ext3_lookup: unlinked inode 229384 in dir #229378
attempt to access beyond end of device
</snip>
Remounting filesystem read-only
EXT3-fs error (device dm-1): ext3_lookup: unlinked inode 229390 in dir #229378
EXT3-fs error (device dm-1): ext3_lookup: unlinked inode 229384 in dir #229378
EXT3-fs error (device dm-1): ext3_lookup: unlinked inode 229390 in dir #229378
EXT3-fs error (device dm-1): ext3_lookup: unlinked inode 229384 in dir #229378
attempt to access beyond end of device
</snip>
in the kernel ring buffer ( via the dmesg command ).
Then I called the Unix sysadm who realised that there was a much bigger problem with the disks, leading to a rebuild ( of / as /opt and /home appeared to be OK ).
Ah well, one lives and one learns ….
↧
↧
Fun with DB2 UDB following loss of / file system
As per my earlier post: -
The test connection operation failed for data source Monitor_Admin_Database on server dmgr at node BAMDMNODENode with the following exception: java.sql.SQLNonTransientException: DB2 SQL Error: SQLCODE=-30082, SQLSTATE=08001, SQLERRMC=42;ROOT CAPABILITY REQUIRED;, DRIVER=4.11.69 DSRA0010E: SQL State = 08001, Error Code = -30,082. View JVM logs for further details.
having seen exceptions such as: -
com.ibm.wbimonitor.lifecycle.LifecycleStopRequestScanTask run() CWMLC0012E: Unexpected exception [com.ibm.wbimonitor.persistence.spi.MonitorPersistenceException: com.ibm.websphere.ce.cm.StaleConnectionException: DB2 SQL Error: SQLCODE=-30082, SQLSTATE=08001, SQLERRMC=42;ROOT CAPABILITY REQUIRED;, DRIVER=4.11.69 DSRA0010E: SQL State = 08001, Error Code = -30,082].
[3/4/13 13:24:10:878 GMT] 00000014 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/logs/ffdc/dmgr_d363f0f1_13.03.04_13.24.10.8779174053313322880198.txt com.ibm.wbimonitor.lifecycle.LifecycleUpdateScanTask 106
[3/4/13 13:24:10:879 GMT] 00000014 LifecycleUpda E com.ibm.wbimonitor.lifecycle.LifecycleUpdateScanTask run() CWMLC0012E: Unexpected exception [com.ibm.wbimonitor.persistence.spi.MonitorPersistenceException: com.ibm.websphere.ce.cm.StaleConnectionException: DB2 SQL Error: SQLCODE=-30082, SQLSTATE=08001, SQLERRMC=42;ROOT CAPABILITY REQUIRED;, DRIVER=4.11.69 DSRA0010E: SQL State = 08001, Error Code = -30,082].
I validated the password ( su - db2inst1 ), and also updated the J2C alias for the db2inst1 user, but to no avail.
There were a few Google hits relating to the "ROOT CAPABILITY REQUIRED" part of the error message, including that made reference to the db2ckpwd process ( which should be running as root ).
This led me to the realization that the DAS ( dasusr1 ) and instance ( db2inst1 ) were "orphaned", as validated by these two commands: -
$ /opt/ibm/db2/V9.7/instance/db2ilist
and
$ /opt/ibm/db2/V9.7/instance/daslist
both of which returned no data.
DB20000I The UPDATE DATABASE MANAGER CONFIGURATION command completed successfully.
$ db2 get dbm config | grep SVCE
TCP/IP Service name (SVCENAME) = db2c_db2inst1
SSL service name (SSL_SVCENAME) =
$ db2stop
03/04/2013 13:53:25 0 0 SQL1064N DB2STOP processing was successful.
SQL1064N DB2STOP processing was successful.
$ db2start
03/04/2013 13:53:29 0 0 SQL1063N DB2START processing was successful.
SQL1063N DB2START processing was successful.
I saw this when testing JDBC connections: -
The test connection operation failed for data source Monitor_Admin_Database on server dmgr at node BAMDMNODENode with the following exception: java.sql.SQLNonTransientException: DB2 SQL Error: SQLCODE=-30082, SQLSTATE=08001, SQLERRMC=42;ROOT CAPABILITY REQUIRED;, DRIVER=4.11.69 DSRA0010E: SQL State = 08001, Error Code = -30,082. View JVM logs for further details.
having seen exceptions such as: -
com.ibm.wbimonitor.lifecycle.LifecycleStopRequestScanTask run() CWMLC0012E: Unexpected exception [com.ibm.wbimonitor.persistence.spi.MonitorPersistenceException: com.ibm.websphere.ce.cm.StaleConnectionException: DB2 SQL Error: SQLCODE=-30082, SQLSTATE=08001, SQLERRMC=42;ROOT CAPABILITY REQUIRED;, DRIVER=4.11.69 DSRA0010E: SQL State = 08001, Error Code = -30,082].
[3/4/13 13:24:10:878 GMT] 00000014 FfdcProvider W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/logs/ffdc/dmgr_d363f0f1_13.03.04_13.24.10.8779174053313322880198.txt com.ibm.wbimonitor.lifecycle.LifecycleUpdateScanTask 106
[3/4/13 13:24:10:879 GMT] 00000014 LifecycleUpda E com.ibm.wbimonitor.lifecycle.LifecycleUpdateScanTask run() CWMLC0012E: Unexpected exception [com.ibm.wbimonitor.persistence.spi.MonitorPersistenceException: com.ibm.websphere.ce.cm.StaleConnectionException: DB2 SQL Error: SQLCODE=-30082, SQLSTATE=08001, SQLERRMC=42;ROOT CAPABILITY REQUIRED;, DRIVER=4.11.69 DSRA0010E: SQL State = 08001, Error Code = -30,082].
I validated the password ( su - db2inst1 ), and also updated the J2C alias for the db2inst1 user, but to no avail.
There were a few Google hits relating to the "ROOT CAPABILITY REQUIRED" part of the error message, including that made reference to the db2ckpwd process ( which should be running as root ).
This led me to the realization that the DAS ( dasusr1 ) and instance ( db2inst1 ) were "orphaned", as validated by these two commands: -
$ /opt/ibm/db2/V9.7/instance/db2ilist
and
$ /opt/ibm/db2/V9.7/instance/daslist
both of which returned no data.
In the end, I had to remove and recreate the db2inst1, db2fenc1 and dasusr1 users, and recreate the instances: -
$ userdel -r dasusr1
$ userdel -r db2fenc1
$ userdel -r db2inst1
$ useradd -u 501 -g db2iadm1 -G db2iadm1,dasadm1 db2inst1
$ useradd -u 502 -g db2fadm1 db2fenc1
$ useradd -u 500 -g dasadm1 dasusr1
$ passwd db2inst1
$ passwd db2fenc1
$ /opt/ibm/db2/V9.7/instance/dascrt -u dasusr1
$ /opt/ibm/db2/V9.7/instance/db2icrt -a SERVER -u db2fenc1 db2inst1
and set up autostart etc.
$ su - db2inst1
$ db2iauto -on db2inst1
$ db2set DB2AUTOSTART=YES
$ db2set DB2COMM=tcpip
$ userdel -r dasusr1
$ userdel -r db2fenc1
$ userdel -r db2inst1
$ useradd -u 501 -g db2iadm1 -G db2iadm1,dasadm1 db2inst1
$ useradd -u 502 -g db2fadm1 db2fenc1
$ useradd -u 500 -g dasadm1 dasusr1
$ passwd db2inst1
$ passwd db2fenc1
$ /opt/ibm/db2/V9.7/instance/dascrt -u dasusr1
$ /opt/ibm/db2/V9.7/instance/db2icrt -a SERVER -u db2fenc1 db2inst1
and set up autostart etc.
$ su - db2inst1
$ db2iauto -on db2inst1
$ db2set DB2AUTOSTART=YES
$ db2set DB2COMM=tcpip
$ db2 update dbm config using SVCENAME db2c_db2inst1
DB20000I The UPDATE DATABASE MANAGER CONFIGURATION command completed successfully.
$ db2 get dbm config | grep SVCE
TCP/IP Service name (SVCENAME) = db2c_db2inst1
SSL service name (SSL_SVCENAME) =
$ db2stop
03/04/2013 13:53:25 0 0 SQL1064N DB2STOP processing was successful.
SQL1064N DB2STOP processing was successful.
$ db2start
03/04/2013 13:53:29 0 0 SQL1063N DB2START processing was successful.
SQL1063N DB2START processing was successful.
and we are back in the game.
↧
More on IBM Business Monitor and Cognos BI - Patience is the key
If you get this: -
IBM Cognos
An error has occurred.
DPR-ERR-2109 The dispatcher cannot service the request at this time. The dispatcher is still initializing. Contact your administrator if this problem persists.
Please try again or contact your administrator.
IBM Cognos
An error has occurred.
DPR-ERR-2109 The dispatcher cannot service the request at this time. The dispatcher is still initializing. Contact your administrator if this problem persists.
Please try again or contact your administrator.
from the Cognos Dispatcher: -
just wait and watch: -
$ tail -f /opt/IBM/WebSphere/AppServer/profiles/BAMN1Profile/logs/BAMSR011.Support/SystemOut.log
for: -
[3/4/13 16:10:13:968 GMT] 00000000 WsServerImpl A WSVR0001I: Server BAMSR011.Support open for e-business
….
[3/4/13 16:13:19:678 GMT] 000000a5 SystemOut O The dispatcher is ready to process requests.
Note the six minute gap between the two ….
$ tail -f /opt/IBM/WebSphere/AppServer/profiles/BAMN1Profile/logs/BAMSR011.Support/SystemOut.log
for: -
[3/4/13 16:10:13:968 GMT] 00000000 WsServerImpl A WSVR0001I: Server BAMSR011.Support open for e-business
….
[3/4/13 16:13:19:678 GMT] 000000a5 SystemOut O The dispatcher is ready to process requests.
Note the six minute gap between the two ….
↧
"Array index out of range: 0" seen when updating AJAX Proxy Config in IBM Business Monitor 8.0.1
If you get: -
WASX7015E: Exception running command: "AdminTask.updateBlobConfig('[-clusterName BAMR01.WebApp -propertyFileName "/opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/BusinessSpace/BAMSR01.WebApp/mm.runtime.prof/config/proxy-config.xml" -prefix "Mashups_"]')"; exception information:
java.lang.Exception: java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 0
when running: -
wsadmin> AdminTask.updateBlobConfig('[-clusterName BAMR01.WebApp -propertyFileName "/opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/BusinessSpace/BAMSR01.WebApp/mm.runtime.prof/config/proxy-config.xml" -prefix "Mashups_"]')
Check the value of the -clusterName variable.
It was BAMR01.WebApp. It should have been BAMSR01.WebApp.
wsadmin> AdminTask.updateBlobConfig('[-clusterName BAMSR01.WebApp -propertyFileName "/opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/BusinessSpace/BAMSR01.WebApp/mm.runtime.prof/config/proxy-config.xml" -prefix "Mashups_"]')
'updateBlobConfig is executed succesfully'
wsadmin> AdminConfig.save()
AdminNodeManagement.syncActiveNodes()
quit
''
wsadmin>---------------------------------------------------------------
AdminNodeManagement: Synchronize the active nodes
Usage: AdminNodeManagement.syncActiveNodes()
Return: If the command is successfully invoked, a value of 1 is returned.
---------------------------------------------------------------
BAMNODE1
1
WASX7015E: Exception running command: "AdminTask.updateBlobConfig('[-clusterName BAMR01.WebApp -propertyFileName "/opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/BusinessSpace/BAMSR01.WebApp/mm.runtime.prof/config/proxy-config.xml" -prefix "Mashups_"]')"; exception information:
java.lang.Exception: java.lang.Exception: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: 0
when running: -
wsadmin> AdminTask.updateBlobConfig('[-clusterName BAMR01.WebApp -propertyFileName "/opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/BusinessSpace/BAMSR01.WebApp/mm.runtime.prof/config/proxy-config.xml" -prefix "Mashups_"]')
Check the value of the -clusterName variable.
It was BAMR01.WebApp. It should have been BAMSR01.WebApp.
wsadmin> AdminTask.updateBlobConfig('[-clusterName BAMSR01.WebApp -propertyFileName "/opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/BusinessSpace/BAMSR01.WebApp/mm.runtime.prof/config/proxy-config.xml" -prefix "Mashups_"]')
'updateBlobConfig is executed succesfully'
wsadmin> AdminConfig.save()
AdminNodeManagement.syncActiveNodes()
quit
''
wsadmin>---------------------------------------------------------------
AdminNodeManagement: Synchronize the active nodes
Usage: AdminNodeManagement.syncActiveNodes()
Return: If the command is successfully invoked, a value of 1 is returned.
---------------------------------------------------------------
BAMNODE1
1
In other words, I was at home to Mr. Typo :-)
It took me two hours to realise this, do I feel stupid or what ?
At least, this post may save someone else from making the same daft mistake :-)
↧
More on the AJAX Proxy
Following on from my earlier posts, I've now discovered, thanks to @WAS_John, how to determine whether the AJAX Proxy config ( proxy-config.xml ) actually made it into my cluster member's configuration.
He explained that the update process: -
wsadmin> AdminTask.updateBlobConfig('[-clusterName BAMSR01.WebApp -propertyFileName "/opt/IBM/WebSphere/AppServer/profiles/BAMDMProfile/BusinessSpace/BAMSR01.WebApp/mm.runtime.prof/config/proxy-config.xml" -prefix "Mashups_"]')
actually updates resources.xml.
However, I wasn't 100% clear which specific file got updated.
This is how I checked: -
$ cd /opt/IBM/WebSphere/AppServer/profiles/BAMN1Profile/config/cells
$ ls -al `find . -name resources.xml`
-rw-r--r-- 1 wasadmin wasadmins 53144 Mar 4 14:18 ./BAMCELL/applications/commsvc.ear/deployments/commsvc/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 51832 Mar 4 14:18 ./BAMCELL/clusters/BAMSR01.AppTarget/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 135131 Mar 4 14:18 ./BAMCELL/clusters/BAMSR01.Messaging/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 120465 Mar 4 14:18 ./BAMCELL/clusters/BAMSR01.Support/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 855651 Mar 6 11:16 ./BAMCELL/clusters/BAMSR01.WebApp/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 51382 Mar 4 14:04 ./BAMCELL/nodes/BAMNODE1/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 53783 Mar 4 14:18 ./BAMCELL/nodes/BAMNODE1/servers/BAMSR011.AppTarget/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 53783 Mar 4 14:18 ./BAMCELL/nodes/BAMNODE1/servers/BAMSR011.Messaging/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 53783 Mar 4 14:18 ./BAMCELL/nodes/BAMNODE1/servers/BAMSR011.Support/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 53783 Mar 4 14:18 ./BAMCELL/nodes/BAMNODE1/servers/BAMSR011.WebApp/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 200966 Mar 4 14:14 ./BAMCELL/resources.xml
Note that I have highlighted the file that changed most recently.
$ view ./BAMCELL/clusters/BAMSR01.WebApp/resources.xml
…
<resourceProperties xmi:id="J2EEResourceProperty_1362406193149" name="proxy-config.xml" value="<?xml version="1.0" encoding="UTF-8"?><!-- ***************************************************************** --><!-- Licensed Materials - Property of IBM --><!-- 5724-L01, 5655-N53, 5724-I82, 5655-R15 --><!-- (C) Copyright IBM Corporation 2006, 2012. All rights reserved. --><!-- US Government Users Restricted Rights - Use, duplication or --><!-- disclosure restricted by GSA ADP Schedule Contract with --><!-- IBM Corp. --><!-- ***************************************************************** --><proxy:proxy-rules xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:proxy="http://www.ibm.com/xmlns/prod/sw/ajax/proxy-config/1.1">	<proxy:mapping contextpath="/proxy/*"/> 	<proxy:policy url="endpoint://*" acf="none" basic-auth-support="true">		<proxy:actions>			<proxy:method>GET</proxy:method>			<proxy:method>POST</proxy:method>			<proxy:method>PUT</proxy:method>			<proxy:method>DELETE</proxy:method>		</proxy:actions>		<proxy:headers>			<proxy:header>Cache-Control</proxy:header>			<proxy:header>Pragma</proxy:header>			<proxy:header>User-Agent</proxy:header>			<proxy:header>Accept*</proxy:header>			<proxy:header>Content*</proxy:header>			<proxy:header>X-Method-Override</proxy:header>			<proxy:header>X-HTTP-Method-Override</proxy:header>			<proxy:header>If-Match</proxy:header>			<proxy:header>If-None-Match</proxy:header>			<proxy:header>If-Modified-Since</proxy:header>			<proxy:header>If-Unmodified-Since</proxy:header>			<proxy:header>Slug</proxy:header>			<proxy:header>SOAPAction</proxy:header>								</proxy:headers>		<proxy:cookies>			<proxy:cookie>LtpaToken</proxy:cookie>			<proxy:cookie>LtpaToken2</proxy:cookie> 			<proxy:cookie>JSESSIONID</proxy:cookie>					</proxy:cookies>	</proxy:policy>	<proxy:policy url="http://www.ibm.com/*" acf="none" basic-auth-support="true">		<proxy:actions>			<proxy:method>GET</proxy:method>		</proxy:actions>	</proxy:policy>	<proxy:policy url="http://www-03.ibm.com/*" acf="none" basic-auth-support="true">		<proxy:actions>			<proxy:method>GET</proxy:method>		</proxy:actions>	</proxy:policy>	<proxy:policy url="http://www.redbooks.ibm.com/*" acf="none" basic-auth-support="true">		<proxy:actions>			<proxy:method>GET</proxy:method>		</proxy:actions>	</proxy:policy>		<proxy:policy url="http://www.google.com/ig/*" acf="none" basic-auth-support="true">		<proxy:actions>			<proxy:method>GET</proxy:method>		</proxy:actions>	</proxy:policy>	 <proxy:mapping contextpath="/cognosProxy/*">	 <proxy:policy url="endpoint://*" acf="none" basic-auth-support="true">			<proxy:actions>				<proxy:method>GET</proxy:method>				<proxy:method>POST</proxy:method>				<proxy:method>PUT</proxy:method>				<proxy:method>DELETE</proxy:method>			</proxy:actions>			<proxy:headers>				<proxy:header>Cache-Control</proxy:header>				<proxy:header>Pragma</proxy:header>				<proxy:header>User-Agent</proxy:header>				<proxy:header>Accept*</proxy:header>				<proxy:header>Content*</proxy:header>				<proxy:header>X-Method-Override</proxy:header>				<proxy:header>X-HTTP-Method-Override</proxy:header>				<proxy:header>If-Match</proxy:header>				<proxy:header>If-None-Match</proxy:header>				<proxy:header>If-Modified-Since</proxy:header>				<proxy:header>If-Unmodified-Since</proxy:header>				<proxy:header>Slug</proxy:header>				<proxy:header>SOAPAction</proxy:header>						</proxy:headers>			<proxy:cookies>				<proxy:cookie>LtpaToken</proxy:cookie>				<proxy:cookie>LtpaToken2</proxy:cookie> 				<proxy:cookie>JSESSIONID</proxy:cookie>				<proxy:cookie>CRN</proxy:cookie>				<proxy:cookie>caf</proxy:cookie>				<proxy:cookie>cam_passport</proxy:cookie>				<proxy:cookie>cc_session</proxy:cookie>				<proxy:cookie>userCapabilities</proxy:cookie>				<proxy:cookie>usersessionid</proxy:cookie>								</proxy:cookies>	 </proxy:policy> <proxy:meta-data> <proxy:name>use-context-path-for-cookies</proxy:name> <proxy:value>true</proxy:value> </proxy:meta-data> </proxy:mapping>		<proxy:meta-data>		<proxy:name>socket-timeout</proxy:name>		<proxy:value>30000</proxy:value>	</proxy:meta-data>	<proxy:meta-data>		<proxy:name>connection-timeout</proxy:name>		<proxy:value>30000</proxy:value>	</proxy:meta-data>		<proxy:meta-data>		<proxy:name>retries</proxy:name>		<proxy:value>2</proxy:value>	</proxy:meta-data>	<proxy:meta-data>		<proxy:name>max-connections-per-host</proxy:name>		<proxy:value>10</proxy:value>	</proxy:meta-data>	<proxy:meta-data>		<proxy:name>max-total-connections</proxy:name>		<proxy:value>200</proxy:value>	</proxy:meta-data>	<proxy:meta-data>		<proxy:name>unsigned_ssl_certificate_support</proxy:name>		<proxy:value>false</proxy:value>	</proxy:meta-data>	<proxy:meta-data>		<proxy:name>forward-http-errors</proxy:name>		<proxy:value>true</proxy:value>	</proxy:meta-data></proxy:proxy-rules>" description="Automatically generated! Do not change the value here! Please use the according file in the config directory and rerun the update task!" required="false"/>
…
$ ls -al `find . -name resources.xml`
-rw-r--r-- 1 wasadmin wasadmins 53144 Mar 4 14:18 ./BAMCELL/applications/commsvc.ear/deployments/commsvc/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 51832 Mar 4 14:18 ./BAMCELL/clusters/BAMSR01.AppTarget/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 135131 Mar 4 14:18 ./BAMCELL/clusters/BAMSR01.Messaging/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 120465 Mar 4 14:18 ./BAMCELL/clusters/BAMSR01.Support/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 855651 Mar 6 11:16 ./BAMCELL/clusters/BAMSR01.WebApp/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 51382 Mar 4 14:04 ./BAMCELL/nodes/BAMNODE1/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 53783 Mar 4 14:18 ./BAMCELL/nodes/BAMNODE1/servers/BAMSR011.AppTarget/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 53783 Mar 4 14:18 ./BAMCELL/nodes/BAMNODE1/servers/BAMSR011.Messaging/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 53783 Mar 4 14:18 ./BAMCELL/nodes/BAMNODE1/servers/BAMSR011.Support/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 53783 Mar 4 14:18 ./BAMCELL/nodes/BAMNODE1/servers/BAMSR011.WebApp/resources.xml
-rw-r--r-- 1 wasadmin wasadmins 200966 Mar 4 14:14 ./BAMCELL/resources.xml
Note that I have highlighted the file that changed most recently.
$ view ./BAMCELL/clusters/BAMSR01.WebApp/resources.xml
…
<resourceProperties xmi:id="J2EEResourceProperty_1362406193149" name="proxy-config.xml" value="<?xml version="1.0" encoding="UTF-8"?><!-- ***************************************************************** --><!-- Licensed Materials - Property of IBM --><!-- 5724-L01, 5655-N53, 5724-I82, 5655-R15 --><!-- (C) Copyright IBM Corporation 2006, 2012. All rights reserved. --><!-- US Government Users Restricted Rights - Use, duplication or --><!-- disclosure restricted by GSA ADP Schedule Contract with --><!-- IBM Corp. --><!-- ***************************************************************** --><proxy:proxy-rules xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xmlns:proxy="http://www.ibm.com/xmlns/prod/sw/ajax/proxy-config/1.1">	<proxy:mapping contextpath="/proxy/*"/> 	<proxy:policy url="endpoint://*" acf="none" basic-auth-support="true">		<proxy:actions>			<proxy:method>GET</proxy:method>			<proxy:method>POST</proxy:method>			<proxy:method>PUT</proxy:method>			<proxy:method>DELETE</proxy:method>		</proxy:actions>		<proxy:headers>			<proxy:header>Cache-Control</proxy:header>			<proxy:header>Pragma</proxy:header>			<proxy:header>User-Agent</proxy:header>			<proxy:header>Accept*</proxy:header>			<proxy:header>Content*</proxy:header>			<proxy:header>X-Method-Override</proxy:header>			<proxy:header>X-HTTP-Method-Override</proxy:header>			<proxy:header>If-Match</proxy:header>			<proxy:header>If-None-Match</proxy:header>			<proxy:header>If-Modified-Since</proxy:header>			<proxy:header>If-Unmodified-Since</proxy:header>			<proxy:header>Slug</proxy:header>			<proxy:header>SOAPAction</proxy:header>								</proxy:headers>		<proxy:cookies>			<proxy:cookie>LtpaToken</proxy:cookie>			<proxy:cookie>LtpaToken2</proxy:cookie> 			<proxy:cookie>JSESSIONID</proxy:cookie>					</proxy:cookies>	</proxy:policy>	<proxy:policy url="http://www.ibm.com/*" acf="none" basic-auth-support="true">		<proxy:actions>			<proxy:method>GET</proxy:method>		</proxy:actions>	</proxy:policy>	<proxy:policy url="http://www-03.ibm.com/*" acf="none" basic-auth-support="true">		<proxy:actions>			<proxy:method>GET</proxy:method>		</proxy:actions>	</proxy:policy>	<proxy:policy url="http://www.redbooks.ibm.com/*" acf="none" basic-auth-support="true">		<proxy:actions>			<proxy:method>GET</proxy:method>		</proxy:actions>	</proxy:policy>		<proxy:policy url="http://www.google.com/ig/*" acf="none" basic-auth-support="true">		<proxy:actions>			<proxy:method>GET</proxy:method>		</proxy:actions>	</proxy:policy>	 <proxy:mapping contextpath="/cognosProxy/*">	 <proxy:policy url="endpoint://*" acf="none" basic-auth-support="true">			<proxy:actions>				<proxy:method>GET</proxy:method>				<proxy:method>POST</proxy:method>				<proxy:method>PUT</proxy:method>				<proxy:method>DELETE</proxy:method>			</proxy:actions>			<proxy:headers>				<proxy:header>Cache-Control</proxy:header>				<proxy:header>Pragma</proxy:header>				<proxy:header>User-Agent</proxy:header>				<proxy:header>Accept*</proxy:header>				<proxy:header>Content*</proxy:header>				<proxy:header>X-Method-Override</proxy:header>				<proxy:header>X-HTTP-Method-Override</proxy:header>				<proxy:header>If-Match</proxy:header>				<proxy:header>If-None-Match</proxy:header>				<proxy:header>If-Modified-Since</proxy:header>				<proxy:header>If-Unmodified-Since</proxy:header>				<proxy:header>Slug</proxy:header>				<proxy:header>SOAPAction</proxy:header>						</proxy:headers>			<proxy:cookies>				<proxy:cookie>LtpaToken</proxy:cookie>				<proxy:cookie>LtpaToken2</proxy:cookie> 				<proxy:cookie>JSESSIONID</proxy:cookie>				<proxy:cookie>CRN</proxy:cookie>				<proxy:cookie>caf</proxy:cookie>				<proxy:cookie>cam_passport</proxy:cookie>				<proxy:cookie>cc_session</proxy:cookie>				<proxy:cookie>userCapabilities</proxy:cookie>				<proxy:cookie>usersessionid</proxy:cookie>								</proxy:cookies>	 </proxy:policy> <proxy:meta-data> <proxy:name>use-context-path-for-cookies</proxy:name> <proxy:value>true</proxy:value> </proxy:meta-data> </proxy:mapping>		<proxy:meta-data>		<proxy:name>socket-timeout</proxy:name>		<proxy:value>30000</proxy:value>	</proxy:meta-data>	<proxy:meta-data>		<proxy:name>connection-timeout</proxy:name>		<proxy:value>30000</proxy:value>	</proxy:meta-data>		<proxy:meta-data>		<proxy:name>retries</proxy:name>		<proxy:value>2</proxy:value>	</proxy:meta-data>	<proxy:meta-data>		<proxy:name>max-connections-per-host</proxy:name>		<proxy:value>10</proxy:value>	</proxy:meta-data>	<proxy:meta-data>		<proxy:name>max-total-connections</proxy:name>		<proxy:value>200</proxy:value>	</proxy:meta-data>	<proxy:meta-data>		<proxy:name>unsigned_ssl_certificate_support</proxy:name>		<proxy:value>false</proxy:value>	</proxy:meta-data>	<proxy:meta-data>		<proxy:name>forward-http-errors</proxy:name>		<proxy:value>true</proxy:value>	</proxy:meta-data></proxy:proxy-rules>" description="Automatically generated! Do not change the value here! Please use the according file in the config directory and rerun the update task!" required="false"/>
…
Note that I've highlighted the property that I had previously changed - unsigned_ssl_certificate_support=false.
Nice one, John
↧
↧
IBM Business Monitor - Fixing REST Endpoints
If you see this: -
![]()
![]()
then you probably forgot to set your REST endpoints when you imported your Deployment Environment into your IBM Business Process Manager or IBM Business Monitor environment.
Note the highlighted portion of the URL - this points at the WRONG host :-)
Silly you :-)
Good news, it can be resolved simply using the WAS Integrated Systems Console ( ISC ) and, I imagine, wsadmin.
Here's the ISC route: -
↧
Twitter, Facebook, and Blog information for WebSphere and CICS Support
Following an earlier post: -
here's an updated list: -
All well worth following ….
↧
What's new in IBM Business Process Manager V8
Saw this on Twitter, thanks to @IBM_BPM
Summary: This article describes the highlights of the newly announced IBM Business Process Manager V8, including a newly redesigned Process Portal, integration with Enterprise Content Management systems, searching and sharing of content between Process Centers, enhanced governance capabilities, and other new features. This content is part of the IBM Business Process Management Journal.
↧
Webcast replay: WebSphere Application Server Top Five Performance Topics
Saw this from @IBM_AppServer on Twitter: -
Delivered straight from the source, this presentation looks into the most common performance topics addressed by WebSphere Application Server Level 2 Support. Empowered with this information, you'll be ble to either avoid these particular issues altogether or at the very least expedite resolution.
↧
↧
IBM Redbooks: Building and Implementing a Social Portal
Saw this in a newsletter from my friends at Portal and noted that three of the authors are friends of mine: -
Brian Farbrother is head of the WebSphere technical team within IBM's Premier UK Business Parter, Portal. He is a Solution Architect, specializing in Portals and enterprise collaboration with over 15 years experience in IT. Highly passionate and practically experienced in architecture, strategy, design, development and governance areas, Brian is particularly interested in architecting and implementing enterprise Portal solutions. Since joining Portal, Brian has dedicated his time to both architecting solutions for clients and actively leading teams of architects and developers on client collaboration programmes.
Peter Hood is an experienced IBM Certified IT Specialist originally from Australia. He holds a Masters in Internet and Web Computing from Royal Melbourne Institute of Technology, Melbourne (RMIT). For the past 5 years Peter has been working for IBM Software Group (SWG) in Ireland and the UK, providing technical leadership on collaboration and social technologies within the development and services organisation. With over 15 years of experience in the IT industry, Peter has advised organisations on large integration programs in various industry sectors. He has provided leadership and strategy on general technology architecture, technical consultancy, and led many web based application design and development projects. He has co-authored several IBM Rebooks, IBM WebSphere and Microsoft .NET Interoperability, ISBN/ISSN: 0738495573, and Patterns: Implementing Self-Service in an SOA Environment, ISBN/ISSN: 073849626X. You can reach him at pete.hood@uk.ibm.com
David Strachan is a Solution Architect in the pan-European IBM Software Services for Collaboration team, based in Edinburgh, UK. He has worked in the software industry for 14 years and held positions at IBM as well as at IBM business partners. He holds a degree from the University of Cambridge. David focusses on Web Experience architecture, bringing together content management, social and portal to deliver exceptional web experiences, and has led large Portal, WCM and Connections deployments at a range of customers. David has been working in this field for more than 10 years and regularly presents on Web Experience topics at IBM conferences.
• Introduction to social portal
• Architecture and technical integration setup
• Pattern 1: Adding social alongside an existing intranet
• Pattern 2: Putting social into context
• Partern 3: Making WCM authoring social
↧
Seeing "mount clntudp_create: RPC: Program not registered" when starting NFS on Red Hat Enterprise Linux
This was a classic Doh! moment.
and then started the service: -
I kept seeing: -
mount clntudp_create: RPC: Program not registered
when I was trying to check the status of my NFS exports: -
$ cat /etc/exports
/media/Software*(rw,no_root_squash)
using the command: -
$ showmount -e
It took me a few minutes to realise why.
NFS wasn't running, as evidenced by: -
$ chkconfig --list | grep -i nfs
nfs 0:off1:off2:off3:off4:off5:off6:off
nfslock 0:off1:off2:off3:on4:on5:on6:off
nfslock 0:off1:off2:off3:on4:on5:on6:off
I set the service to start at boot-up ( whenever run levels 3, 4 or 5 are started ): -
$ chkconfig --levels 345 nfs on
and validated this: -
$ chkconfig --list | grep -i nfs
nfs 0:off1:off2:off3:on4:on5:on6:off
nfslock 0:off1:off2:off3:on4:on5:on6:off
nfslock 0:off1:off2:off3:on4:on5:on6:off
and then started the service: -
$ service nfs start
Starting NFS services: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
Starting NFS quotas: [ OK ]
Starting NFS daemon: [ OK ]
Starting NFS mountd: [ OK ]
Now I can see my exports: -
$ showmount -e
Export list for vhost4288:
/media/Software *
/media/Software *
↧
More fun with NFS on Red Hat Enterprise Linux
Hot on the heels of my last post: -
Seeing "mount clntudp_create: RPC: Program not registered" when starting NFS on Red Hat Enterprise Linux
mount.nfs: mounting 122.55.228.172:/media/Software failed, reason given by server:
No such file or directory
Seeing "mount clntudp_create: RPC: Program not registered" when starting NFS on Red Hat Enterprise Linux
I hit another apparent show-stopper with NFS.
In fact, I hit two problems.
In the first instance, I was seeing: -
$ mount 122.55.228.172:/media/Software /mnt
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified
I spent nearly an hour trying to debug this, until inspiration ( and, perhaps, a spot of Google helped ).
Guess what ?
There were bloomin' firewalls running on the two boxes - client and server.
I quickly killed them off: -
$ service iptables stop
iptables: Flushing firewall rules: [ OK ]
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Unloading modules: [ OK ]
iptables: Setting chains to policy ACCEPT: filter [ OK ]
iptables: Unloading modules: [ OK ]
and I then hit my second problem: -
$ mount 122.55.228.172:/media/Software /mnt
mount.nfs: mounting 122.55.228.172:/media/Software failed, reason given by server:
No such file or directory
I re-checked /etc/exports on the NFS server, which read: -
/media/Software*(rw,no_root_squash)
but I was also seeing: -
Mar 12 18:07:47 vhost4288 mountd[26662]: can't stat exported dir /media/Software: No such file or directory
in it's own logs - /var/log/messages.
I then looked again at the actual directory itself: -
$ ls -al /media
only to find that the directory had been miskeyed as: -
total 16
drwxr-xr-x 2 root root 4096 Mar 12 12:26 .
drwxr-xr-x 28 root root 4096 Mar 12 12:26 ..
-rw-r--r-- 1 root root 0 Mar 12 12:26 .hal-mtab
lrwxrwxrwx 1 root root 15 Dec 10 08:41 Sotfware -> /data/Software/
drwxr-xr-x 2 root root 4096 Mar 12 12:26 .
drwxr-xr-x 28 root root 4096 Mar 12 12:26 ..
-rw-r--r-- 1 root root 0 Mar 12 12:26 .hal-mtab
lrwxrwxrwx 1 root root 15 Dec 10 08:41 Sotfware -> /data/Software/
Can you spot the obvious error ?
Yep, someone had misspelt Software as Sotfware.
In the end, I just changed /etc/exports to: -
/data/Software*(rw,no_root_squash)
and re-ran my mount command: -
$ mount 122.55.228.172:/data/Software /mnt
Can you say "Doh!" ? I bet you can ….
↧