[java.lang.ClassNotFoundException][oracle.bpel.services.workflow.task.impl.TaskService.startup][soa-infra]

It’s not very often I have to deal with administration of WebLogic and SOA. However I do administer small installation for development purposes in one company. 

One of the servers crushed few dayes ago and from that moment we were unable to correctly start SOA infrastructure.

Main log showed very strange error:

<feb 26, 2016 9:51:40 AM CET> <emergency> <oracle.dfw.incident> <BEA-000000> <incident 6 created with problem key "DFW-99998 [java.lang.ClassNotFoundException][oracle.bpel.services.workflow.task.impl.TaskService.startup][soa-infra]">

I tried to search the web with very little success. Later I’ve got a "great" idea “Hey, let’s check diagnostic logs!” …. Next time I’ll do it right at the start. The error in the log was as follows:

####<feb 26, 2016 11:12:09 AM CET> <error> <deployer> <hp02.elso-systems.com> <wls_soa1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<wls Kernel>> <> <0000lct3knd3j4wvlyvh8a1mo2gy000002> <1456481529274> <BEA-149231> <unable to set the activation state to true for the application 'soa-infra'. weblogic.application.ModuleException: [HTTP:101216]Servlet: "FabricInit" failed to preload on startup in Web application: "/soa-infra". oracle.fabric.common.FabricException: Error in getting XML input stream: oramds:/deployed-composites/Name_rev1.0/composite.xml: oracle.mds.exception.MDSException: MDS-00054: The file to be loaded oramds:/deployed-composites/Name_rev1.0/composite.xml does not exist. ...

Well that's much more precise! Now if you do search for that, you’ll probably get to Oracle Support Doc ID 1380835.1 - SOA/BPM: Solving MDS-00054 error preventing soa-infra to start correctly. Basically what you are dealing with here is incorrectly deployed component and until the problem is fixed; your soa infra will not come up. And the only way  to deal with it when your soa infra is not up is by removing it. There is a nice guide on how to remove component which is causing the problem:

Check the SOA logs and determine which composite is causing the problem and then follow the below process to undeploy the composite by editing deployed-composites.xml:

1. Download and copy the ShareSoaInfraPartition.ear file to $MIDDLEWARE_HOME/oracle_common/common/bin

2. cd to $MIDDLEWARE_HOME/oracle_common/common/bin and run wlst.sh

3.  Connect to a SOA server:

wls:/offline> connect()
Please enter your username :weblogic
Please enter your password :
Please enter your server URL [t3://localhost:7001] :server
Connecting to t3://server:7014 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'name'.

4. run the below command to deploy ShareSoaInfraPartition.ear to the server:

wls:/vzpsoa/serverConfig> deploy('ShareSoaInfraPartition','ShareSoaInfraPartition.ear',upload='true')
Deploying application from /opt/oracle/middleware/soa111/oracle_common/common/bin/ShareSoaInfraPartition.ear to targets  (upload=true) ...
<Feb 26, 2016 12:28:50 PM CET> <Info> <J2EE Deployment SPI> <BEA-260121> <Initiating deploy operation for application, ShareSoaInfraPartition [archive: /opt/oracle/middleware/soa111/oracle_common/common/bin/ShareSoaInfraPartition.ear], to AdminServer .>
.......Completed the deployment of Application with status completed
Current Status of your Deployment:
Deployment command type: deploy
Deployment State       : completed
Deployment Message     : [Deployer:149194]Operation 'deploy' on application 'ShareSoaInfraPartition' has succeeded on 'AdminServer'

5. Now run the below command by changing the "toLocation" ('/fmw11g/fmw1115/Middleware' is some location path on SOA machine)

wls:/vzpsoa/serverConfig> exportMetadata(application='ShareSoaInfraPartition',server='AdminServer',toLocation='/opt/oracle/middleware/soa111',docs='/deployed-composites/deployed-composites.xml')
Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainRuntime)

Executing operation: exportMetadata.

Operation "exportMetadata" completed. Summary of "exportMetadata" operation is:
1 documents successfully transferred.
List of documents successfully transferred:

/deployed-composites/deployed-composites.xml

6. A deployed-composites folder will be created at "toLocation" path with deployed-composites.xml in it

7. Delete the composite which is causing the problem and save the file

For example, the MediatorTest composite:

<composite-series name="default/MediatorTest" default="default/MediatorTest!1.0">
<composite-revision dn="default/MediatorTest!1.0" state="on" mode="active" location="dc/soa_58b98be8-9ec8-41af-bb83-590f6004d1aa">
<composite dn="default/MediatorTest!1.0*soa_58b98be8-9ec8-41af-bb83-590f6004d1aa" deployedTime="2011-11-17T09:01:54.750+05:30"/>

8. Now run the below command by changing the "fromLocation" (this should be the same location as previous)

wls:/vzpsoa/serverConfig> importMetadata(application='ShareSoaInfraPartition',server='AdminServer',fromLocation='/opt/oracle/middleware/soa111',docs='/deployed-composites/deployed-composites.xml')

Executing operation: importMetadata.

Operation "importMetadata" completed. Summary of "importMetadata" operation is:
1 documents successfully transferred.
List of documents successfully transferred:

/deployed-composites/deployed-composites.xml

9. Now bounce your server and the composite will not be deployed by SOA when it comes up and hence that should bring your soa-infra up.

Note:  When you remove a composite that contains task definitions manually, the Task definition references to the composite are still in WFTASKMETADATA table. Even though the composite is not loaded during startup there will be an  exception when loading the task definition,

<Error> <oracle.soa.services.workflow.task>
<BEA-000000> <<.> Could not locate composite.

The references to WFTASKMETADATA for that specific composite/version can only be removed when a composite containing task definitions is cleanly undeployed using em/wlst.