BEA-000362 due to "Unable to obtain lock" on AdminServer lock file

Problem:

When starting up the Oracle WebLogic 11g AdminServer, you may get the following error:

<Dec 3, 2010 8:57:32 AM EST> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

weblogic.management.ManagementException: Unable to obtain lock on /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/tmp/AdminServer.lok. Server may already be running
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:159)
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:58)
        at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:74)
        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
        at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:802)
        at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:489)
        at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:446)
        at weblogic.Server.main(Server.java:67)

>
<Dec 3, 2010 8:57:32 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
<Dec 3, 2010 8:57:32 AM EST> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Dec 3, 2010 8:57:32 AM EST> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN>

Solution:

1. Shutdown the AdminServer gracefully as follows:

export MW_HOME=/u01/app/oracle/middleware

export DOMAIN_HOME=$MW_HOME/user_projects/domains/soa_domain

$DOMAIN_HOME/bin/stopWebLogic.sh

2. Remove the lock file:

rm /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/tmp/AdminServer.lok

3. Restart the AdminServer as follows, and you should be okay:

export MW_HOME=/u01/app/oracle/middlware

export DOMAIN_HOME=$MW_HOME/user_projects/domains/soa_domain

cd $DOMAIN_HOME/bin

nohup $DOMAIN_HOME/bin/startWebLogic.sh &

Applicable Versions:

• Oracle WebLogic 11g (11.1.1.x)

BEA-171520 and "Could not obtain an exclusive lock for directory" when starting up WebLogic 11g AdminServer

Problem:

If you have configured your Oracle WebLogic 11g server to perform LDAP authenticaion, when starting up the WebLogic 11g AdminServer, you may receive the following error:

<Dec 3, 2010 8:58:34 AM EST> <Warning> <EmbeddedLDAP> <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>

Investigation:

Upon investigating the AdminServer log file, located here:

/u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/logs/AdminServer.log

You may notice the following, which pretty much indicates the same thing:

####<Dec 3, 2010 8:58:54 AM EST> <Warning> <EmbeddedLDAP> <oradev> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1291384734208> <BEA-171520> <Could not obtain an exclusive lock for directory: /u01/app/oracle/middleware/user_projects/domains/soa_domain/servers/AdminServer/data/ldap/ldapfiles. Waiting for 10 seconds and then retrying in case existing WebLogic Server is still shutting down.>

After shutting down the AdminServer, do a 'ps' for the remaining running processes, and you may find lingering processes which may have to manually be killed.

oracle@oradev:/home/oracle> ps -ef | grep soa | grep -v grep
oracle   29085 24648  0 09:32 pts/3    00:00:00 /bin/sh /u01/app/oracle/middleware/user_projects/domains/soa_domain/bin/startWebLogic.sh
oracle   29133 29085 45 09:32 pts/3    00:00:37 /u01/app/oracle/jdk1.6.0_21/bin/java -server -Xms768m -Xmx768m -XX:PermSize=256m -XX:MaxPermSize=512m -Dweblogic.Name=AdminServer -Djava.security.policy=/u01/app/oracle/middleware/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=/u01/app/oracle/middleware/wlserver_10.3 -Dwls.home=/u01/app/oracle/middleware/wlserver_10.3/server -Dweblogic.home=/u01/app/oracle/middleware/wlserver_10.3/server -Ddomain.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain -Dcommon.components.home=/u01/app/oracle/middleware/oracle_common -Djrf.version=11.1.1 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Djrockit.optfile=/u01/app/oracle/middleware/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt -Doracle.domain.config.dir=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig -Doracle.server.config.dir=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/servers/AdminServer -Doracle.security.jps.config=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/jps-config.xml -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Digf.arisidbeans.carmlloc=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/carml -Digf.arisidstack.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain/config/fmwconfig/arisidprovider -Dweblogic.alternateTypesDirectory=/u01/app/oracle/middleware/Oracle_SOA1/modules/oracle.ossoiap_11.1.1,/u01/app/oracle/middleware/Oracle_SOA1/modules/oracle.oamprovider_11.1.1 -Dweblogic.jdbc.remoteEnabled=true -Doracle.security.jps.policy.migration.validate.principal=false -da:org.apache.xmlbeans... -Dsoa.archives.dir=/u01/app/oracle/middleware/Oracle_SOA1/soa -Dsoa.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dsoa.instance.home=/u01/app/oracle/middleware/user_projects/domains/soa_domain -Dtangosol.coherence.clusteraddress=227.7.7.8 -Dtangosol.coherence.clusterport=9778 -Dtangosol.coherence.log=jdk -Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl -Dcom.sun.management.jmxremote -Djava.protocol.handler.pkgs=oracle.mds.net.protocol|oracle.fabric.common.classloaderurl.handler|oracle.fabric.common.uddiurl.handler|oracle.bpm.io.fs.protocol -Dweblogic.transaction.blocking.commit=true -Dweblogic.transaction.blocking.rollback=true -Djavax.net.ssl.trustStore=/u01/app/oracle/middleware/wlserver_10.3/server/lib/DemoTrust.jks -Dem.oracle.home=/u01/app/oracle/middleware/oracle_common -Djava.awt.headless=true -Dbam.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dums.oracle.home=/u01/app/oracle/middleware/Oracle_SOA1 -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/u01/app/oracle/middleware/patch_wls1033/profiles/default/sysext_manifest_classpath weblogic.Server

Solution:

1. Shutdown the AdminServer

2. Do a "ps -ef | grep oracle | grep -v grep" and observe and see if you have any lingering processes

3. If so, then do a "kill <processid>"

4. If that doesn't kill the process gracefully, then do a "kill -9 <processid>"

5. Restart the AdminServer and you should be okay

Applicable Versions:

• Oracle WebLogic 11g (11.1.1.x)

Incompatible state when starting up a WebLogic managed server

Problem:

After starting up the Oracle WebLogic 11g AdminServer server, you may get the following error when trying to start up any of your managed servers:

For server soa_server1, the Node Manager associated with machine LocalMachine is not reachable.
All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.

Solution:

This is simply due to the Node Manager not running.

1. Start up the Node Manager as follows:

export MW_HOME=/u01/app/oracle/middlware
cd $MW_HOME/wlserver_10.3/server/bin
nohup $MW_HOME/wlserver_10.3/server/bin/startNodeManager.sh &

Applicable Versions:

• Oracle WebLogic 11g (11.1.1.x)