So, whilst running the bootstrap command: -
LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 11-JUL-2018 12:45:09
Copyright (c) 1991, 2016, Oracle. All rights reserved.
Starting /home/oracle/app/oracle/product/12.2.0/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 12.2.0.1.0 - Production
System parameter file is /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Log messages written to /home/oracle/app/oracle/diag/tnslsnr/bpm/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bpm.uk.ibm.com)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=bpm.uk.ibm.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date 11-JUL-2018 12:45:09
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Listener Log File /home/oracle/app/oracle/diag/tnslsnr/bpm/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bpm.uk.ibm.com)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
The listener supports no services
The command completed successfully
So now we have this: -
/opt/ibm/WebSphereProfiles/Dmgr01/bin/bootstrapProcessServerData.sh -clusterName AppCluster
I saw this: -
…
java.lang.Exception: java.lang.reflect.InvocationTargetException
…
…
Caused by: java.lang.reflect.InvocationTargetException
…
…
Caused by: java.lang.IllegalStateException: Failed to initialize registry
…
…
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'handlersMap': Cannot create inner bean 'com.lombardisoftware.server.ejb.persistence.PSDefaultHandler#f4e9076' of type [com.lombardisoftware.server.ejb.persistence.PSDefaultHandler] while setting bean property 'sourceMap' with key [TypedStringValue: value [Branch], target type [null]]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.lombardisoftware.server.ejb.persistence.PSDefaultHandler#f4e9076' defined in class path resource [registry.persistence.xml]: Cannot resolve reference to bean 'dao.branch' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dao.branch' defined in class path resource [registry.persistence.xml]: Instantiation of bean failed; nested exception is java.lang.ExceptionInInitializerError
…
…
etc.
So I checked the log: -
view /opt/ibm/WebSphereProfiles/Dmgr01/logs/bootstrapProcesServerData.AppCluster.log
and saw this: -
…
[11/07/18 12:20:27:801 BST] 00000001 ProviderTrack I com.ibm.ffdc.osgi.ProviderTracker AddingService FFDC1007I: FFDC Provider Installed: com.ibm.ffdc.util.provider.FfdcOnDirProvider@cfa421b1
[11/07/18 12:20:28:470 BST] 00000001 LocalCache I CWLLG2155I: Cache settings read have been from file file:////opt/ibm/WebSphere/AppServer/BPM/Lombardi/process-server/twinit/lib/basic_resources.jar!/LombardiTeamWorksCache.xml.
[11/07/18 12:20:28:768 BST] 00000001 XmlBeanDefini I org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
[11/07/18 12:20:28:784 BST] 00000001 SQLErrorCodes I org.springframework.jdbc.support.SQLErrorCodesFactory <init> SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]
[11/07/18 12:20:28:787 BST] 00000001 SQLErrorCodes W org.springframework.jdbc.support.SQLErrorCodesFactory getErrorCodes Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory, cause: IO Error: The Network Adapter could not establish the connection
…
[11/07/18 12:20:28:470 BST] 00000001 LocalCache I CWLLG2155I: Cache settings read have been from file file:////opt/ibm/WebSphere/AppServer/BPM/Lombardi/process-server/twinit/lib/basic_resources.jar!/LombardiTeamWorksCache.xml.
[11/07/18 12:20:28:768 BST] 00000001 XmlBeanDefini I org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions Loading XML bean definitions from class path resource [org/springframework/jdbc/support/sql-error-codes.xml]
[11/07/18 12:20:28:784 BST] 00000001 SQLErrorCodes I org.springframework.jdbc.support.SQLErrorCodesFactory <init> SQLErrorCodes loaded: [DB2, Derby, H2, HSQL, Informix, MS-SQL, MySQL, Oracle, PostgreSQL, Sybase]
[11/07/18 12:20:28:787 BST] 00000001 SQLErrorCodes W org.springframework.jdbc.support.SQLErrorCodesFactory getErrorCodes Error while extracting database product name - falling back to empty error codes
org.springframework.jdbc.support.MetaDataAccessException: Error while extracting DatabaseMetaData; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory, cause: IO Error: The Network Adapter could not establish the connection
…
I checked that the Oracle 12c listener appeared to be running: -
netstat -aon|grep LISTEN
tcp 0 0 127.0.0.1:1521 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN off (0.00/0/0)
tcp6 0 0 :::36875 :::* LISTEN off (0.00/0/0)
tcp6 0 0 :::22 :::* LISTEN off (0.00/0/0)
tcp6 0 0 ::1:25 :::* LISTEN off (0.00/0/0)
tcp6 0 0 :::5500 :::* LISTEN off (0.00/0/0)
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN off (0.00/0/0)
tcp6 0 0 :::36875 :::* LISTEN off (0.00/0/0)
tcp6 0 0 :::22 :::* LISTEN off (0.00/0/0)
tcp6 0 0 ::1:25 :::* LISTEN off (0.00/0/0)
tcp6 0 0 :::5500 :::* LISTEN off (0.00/0/0)
lsnrctl status listener
LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 11-JUL-2018 12:42:50
Copyright (c) 1991, 2016, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date 10-JUL-2018 20:22:19
Uptime 0 days 16 hr. 20 min. 30 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Listener Log File /home/oracle/app/oracle/diag/tnslsnr/bpm/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=bpm.uk.ibm.com)(PORT=5500))(Security=(my_wallet_directory=/home/oracle/app/oracle/admin/orcl/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "orcl.uk.ibm.com" has 1 instance(s).
Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclXDB.uk.ibm.com" has 1 instance(s).
Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully
LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 11-JUL-2018 12:42:50
Copyright (c) 1991, 2016, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date 10-JUL-2018 20:22:19
Uptime 0 days 16 hr. 20 min. 30 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Listener Log File /home/oracle/app/oracle/diag/tnslsnr/bpm/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=bpm.uk.ibm.com)(PORT=5500))(Security=(my_wallet_directory=/home/oracle/app/oracle/admin/orcl/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "orcl.uk.ibm.com" has 1 instance(s).
Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclXDB.uk.ibm.com" has 1 instance(s).
Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully
Notice the problem ?
Both the netstat and the lsnrctl commands show that the listener is "bound" to 127.0.0.1 rather than to the server's hostname.
lsnrctl stop LISTENER
LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 11-JUL-2018 12:44:58
Copyright (c) 1991, 2016, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=bpm.uk.ibm.com)(PORT=1521)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
The command completed successfully
LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 11-JUL-2018 12:44:58
Copyright (c) 1991, 2016, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=bpm.uk.ibm.com)(PORT=1521)))
TNS-12541: TNS:no listener
TNS-12560: TNS:protocol adapter error
TNS-00511: No listener
Linux Error: 111: Connection refused
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1521)))
The command completed successfully
lsnrctl start LISTENER
LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 11-JUL-2018 12:45:09
Copyright (c) 1991, 2016, Oracle. All rights reserved.
Starting /home/oracle/app/oracle/product/12.2.0/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 12.2.0.1.0 - Production
System parameter file is /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Log messages written to /home/oracle/app/oracle/diag/tnslsnr/bpm/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bpm.uk.ibm.com)(PORT=1521)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=bpm.uk.ibm.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date 11-JUL-2018 12:45:09
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Listener Log File /home/oracle/app/oracle/diag/tnslsnr/bpm/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bpm.uk.ibm.com)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
The listener supports no services
The command completed successfully
I then saw this: -
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor
which I solved via a previous blog post: -
So, the conclusion is that the Listener needs to know / care about the host name of the box: -
…
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bpm.uk.ibm.com)(PORT=1521)))
…
whereas the BPM -> Oracle connectivity needs to know / care about the Oracle Service Name: -
sqlplus / as sysdba
SQL*Plus: Release 12.2.0.1.0 Production on Wed Jul 11 15:06:23 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> select value from v$parameter where name='service_names';
VALUE
--------------------------------------------------------------------------------
orcl.uk.ibm.com
SQL> exit
SQL*Plus: Release 12.2.0.1.0 Production on Wed Jul 11 15:06:23 2018
Copyright (c) 1982, 2016, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> select value from v$parameter where name='service_names';
VALUE
--------------------------------------------------------------------------------
orcl.uk.ibm.com
SQL> exit
so the JDBC data sources need to look like this: -
jdbc:oracle:thin:@//bpm.uk.ibm.com:1521/orcl.uk.ibm.com
This needs to be reflected in the BPM Deployment Environment properties file: -
…
bpm.de.db.1.databaseName=orcl.uk.ibm.com
…
bpm.de.db.2.databaseName=orcl.uk.ibm.com
…
…
bpm.de.db.3.databaseName=orcl.uk.ibm.com
…
…
So now we have this: -
/opt/ibm/WebSphereProfiles/Dmgr01/bin/bootstrapProcessServerData.sh -clusterName AppCluster
Bootstraping data into cluster AppCluster and logging into /opt/ibm/WebSphereProfiles/Dmgr01/logs/bootstrapProcesServerData.AppCluster.log
WASX7357I: By request, this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode.
'BootstrapProcessServerData admin command completed successfully.....'
Bootstraping data into cluster AppCluster and logging into /opt/ibm/WebSphereProfiles/Dmgr01/logs/bootstrapProcesServerData.AppCluster.log
WASX7357I: By request, this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode.
'BootstrapProcessServerData admin command completed successfully.....'