OWSM God spotted in Virginia!

As I was parking at the gym last night, I came across this. Apparently the OWSM God lives in Virginia!

For those who don't know, OWSM means only one thing... Oracle Web Services Manager.

OWSM provides security, policy management, and monitoring of your services. It can provide authentication, authorization, and encryption of your web services. 

I've used it on a few projects. One global customer required the messages between their U.S. and European data centers to be encrypted. Another customer required it to secure the services between their internal systems and external partners. A third customer implemented it, and later on removed it, because it resulted in a 15% performance hit which they could not afford at the time.

For more information, check out the following...

OWSM Wiki on Oracle Wiki
http://wiki.oracle.com/page/OWSM+Links

Oracle Web Services Manager Product Overview
http://www.oracle.com/appserver/web-services-manager.html

OWSM OBE's (Oracle By Example)
http://www.oracle.com/technology/obe/fusion_middleware/owsm/index.html

com.oracle.bpel.client.delivery.ReceiveTimeOutException in 50% of instances in a BPEL Process Manager cluster

An issue with a recent customer starting occurring in some BPEL processes when we migrated off of a single-node Oracle BPEL Process Manager 10g environment to a 2-node cluster.

The Problem

1. BPEL process SubscriberActivation is instantiated (instance 4961221)

2. It synchronously calls GetDeviceState (instance 4961222)

3. GetDeviceState completes successfully within 6.106 seconds

4. The response is never received by the first process around 50% of the time and returns the following BPEL fault:

<summary>
  when invoking locally the endpoint 'http://oradev1.thisisahmed.com:7777/orabpel/default/GetDeviceState/1.0', ; nested exception is
  com.oracle.bpel.client.delivery.ReceiveTimeOutException: Waiting for response has timed out
</summary>

With debugging enabled, the domain.log shows the following:

<2010-06-25 07:55:49,226> <DEBUG> <default.collaxa.cube.engine.delivery> <DeliveryHandler::initialRequestAnyType>
com.oracle.bpel.client.delivery.ReceiveTimeOutException: Waiting for response has timed out. The conversation id is bpel://localhost/default/SubscriberActivation~1.0/5040805-BpInv1-BpSeq4.7-2. Please check the process instance for detail.
at com.collaxa.cube.engine.delivery.DeliveryHandler.initialRequestAnyType(DeliveryHandler.java:543)
at com.collaxa.cube.engine.delivery.DeliveryHandler.initialRequest(DeliveryHandler.java:457)

5. The parent process times out in 1000 seconds which is the value of our syncMaxWaitTime (as shown in the Tree Finder figure above).

This only happens when both nodes of the cluster are up and running. If only a single node is running, this issue does not occur.

Troubleshooting Efforts

The BPEL Process Manager Developer's Guide 10g (10.1.3.1.0) asks you to try the following, which is not applicable to our situation.

(a) Increasing transaction-timeout="7200" in $ORACLE_HOME/j2ee/oc4j_soa/config/transaction-manager.xml 

(b) Increasing transaction-timeout="3600" to a lower value for CubeEngineBean, DispatcherBean, CubeDeliveryBean, DeliveryBean, DomainManagerBean, and ProcessManagerBean in
$ORACLE_HOME/j2ee/oc4j_soa/application-deployments/orabpel/ejb_ob_engine/orion-ejb-jar.xml 

(c) Increasing syncMaxWaitTime to 1000 in $ORACLE_HOME/bpel/domains/default/config/domain.xml

Using TCP instead of UDP for the BPEL PM cluster (in $ORACLE_HOME/bpel/system/config/jgroups-protocol.xml) has no bearing either.

Adding the transaction participate property to the partnerlink won't help either:  <property name="transaction">participate</property>

Cause of Problem & Analysis

This problem is caused by the flawed design of the flow in that it doesn't support operating in a BPEL Process Manager cluster.

Both BPEL1 and BPEL2 are designed as synchronous processes.

Success scenario:

1. Client makes sync request to BPEL1.
2. BPEL1 makes sync request to BPEL2.
3. BPEL2 makes sync request to an external service (and BPEL 2 receives the sync response back).
4. BPEL2 has an async “receive” activity.
5. BPEL2 responds to BPEL1 synchronously
6. BPEL1 responds to client synchronously.

Timeout scenario:

1. Client makes sync request to BPEL1.
2. BPEL1 makes sync request to BPEL2.
3. BPEL2 makes sync request to external service (and BPEL 2 receives the sync response back).
4. BPEL2 has an async “receive” activity, but receives response on Node 2.
5. BPEL2 tries to reply to BPEL1, but no link back to BPEL1 (BPEL2 completes successfully though).
6. BPEL1 times out.

Even though BPEL2 is designed as a synchronous process, the onWait forces it to become asynchronous, thus dehydrating, and rehydrating when it receives the callback. The problem is that the callback could be received on the other node, which is the exact behavior we are seeing here 50% of the time.

In fact, this 2 year old blog post of mine discusses the same issue:
http://blog.thisisahmed.com/2008/10/behavior-of-bpel-processes-in-bpel.html

Setting up email notifications for AIA 2.0

Here, I quickly describe how to set up the system to allow for email notifications upon AIA errors. This is not an exhaustive overview of AIA Error Handling.

1. Configure outgoing email
Edit $ORACLE_HOME/bpel/system/services/config/ns_emails.xml and configure the following:

<EmailAccount>
  <Name>AIA Error Handler</Name>
  <GeneralSettings>
    <FromName>AIA Error Handler</FromName>
    <FromAddress>do-not-reply@thisisahmed.com</FromAddress>
  </GeneralSettings>
  <OutgoingServerSettings>
    <SMTPHost>yourmailserver</SMTPHost>
    <SMTPPort>25</SMTPPort>
  </OutgoingServerSettings>
</EmailAccount>

Also ensure that NotificationMode="EMAIL" is set in the same file.

2. Set the AIA Error Handler role
Ensure that the AIA Error Handler defaults to the ‘AIAIntegrationAdmin’ role.
Edit $AIA_HOME/config/AIAConfigurationProperties.xml and ensure the following is set:

<ModuleConfiguration moduleName="ErrorHandler">
  .
  .
  .
  <Property name="EH.DEFAULT.ACTOR.ROLE">AIAIntegrationAdmin</Property>
  .
  .
  .
</ModuleConfiguration>

3. Create user account
a. Log in to Enterprise Manager (http://host:7777/em)
b. Click on “oc4j_soa”
c. Click on “Administration”
d. Click on “Security Providers”
e. Click on “Instance Level Security”
f. Click on “Realms”
g. Click on the number below ‘Users’
h. Click on “Create”
i. Enter username, password, and select ‘AIAIntegrationAdmin’ as the role
j. Click on “OK”

* The JAZN user information created above is stored in $ORACLE_HOME/j2ee/oc4j_soa/config/system-jazn-data.xml.

4. Set property for BPM user
Edit $ORACLE_HOME/bpel/system/services/config/users-properties.xml and add a new UserObject as follows:

<userObject>
  <name>Ahmed Aboulnaga</name>
  <description>Ahmed Aboulnaga</description>
  <email>ahmed.aboulnaga@somerandomsite.info</email>
  <title>AIAIntegrationAdmin</title>
  <firstName>Ahmed</firstName>
  <lastName>Aboulnaga</lastName>
  <timeZone>America/New_York</timeZone>
  <languagePreference>en-US</languagePreference>
  <notificationPreferences>Mail</notificationPreferences>
</userObject>

Logging in to the BPM Worklist
a. Navigate to BPM Worklist (http://host:7777/integration/worklistapp/Login)
b. Enter username and password

Searching for assigned tasks
a. Click on “My Tasks”
b. Search for ‘My & Group’, ‘Any’, and ‘Assigned’

Claiming, reviewing, and closing tasks
a. Select the action “Claim” and click on “Go”
b. Review the error. Note the following at minimum:

Reporting Date Time         <-- Timestamp
Fault Message Text           <-- Error
Faulting Service ID            <-- BPEL process name
Faulting Servicenstance ID <-- BPEL instance ID

You may optionally log in to the BPEL Console to do further troubleshooting/analysis of the issue.

c. Select the task action “COMPLETED” and click on “Go”    <-- if you want to close the task

Querying the AIA_ERROR_TOPIC from the database
The SQL query is as follows:

SELECT q_name, enq_uid, msgid, expiration, enq_time, a.user_data.text_len, a.user_data.text_vc
FROM aia_error_topic_table a

SYS.AQ$_JMS_TEXT_MESSAGE has the following for data types:

• header        AQ$_JMS_HEADER
• text_len      INT
• text_vc        VARCHAR2(4000)
• text_lob      CLOB

 This has been tested on a rather older version of AIA 2.0.

ORA-06531 "Reference to uninitialized collection" when executing a BPEL process in SOA Suite 10g

It was somewhat frustrating getting to the bottom of this issue.

Problem:

1. We set up a new Oracle SOA Suite 10g (10.1.3.3.1) 64-bit environment.

2. We deployed the same codebase that was deployed to all other environments.

3. Executing a simple BPEL process that invokes a DB Adapter returned the following error:

10/05/28 01:03:27 1 - org.collaxa.thirdparty.apache.wsif.WSIFException: file:/aiaapp/oracle/product/10.1.3/OracleAS_1/bpel/domains/coxwirelessdomain/tmp/.bpel_InventoryLookupSCMProvABCSImpl_1.0_e2ce717b0cd798581b48968c2ab6c2c0.tmp/CallERP.wsdl [ CallERP_ptt::CallERP(InputParameters,OutputParameters) ] - WSIF JCA Execute of operation 'CallERP' failed due to: Error while trying to prepare and execute an API.
An error occurred while preparing and executing the APPS.CCI_INV_O_21_ITEM_AVAIL_PKG.ITEM_CHECK API. Cause: java.sql.SQLException: ORA-06531: Reference to uninitialized collection
ORA-06512: at "APPS.CCI_INV_O_21_ITEM_AVAIL_PKG", line 90
ORA-06512: at line 1
[Caused by: ORA-06531: Reference to uninitialized collection
ORA-06512: at "APPS.CCI_INV_O_21_ITEM_AVAIL_PKG", line 90
ORA-06512: at line 1
]
; nested exception is:
ORABPEL-11811
Error while trying to prepare and execute an API.

Investigation:

1. The error occurs when using a JNDI (error shows up in the log, and no BPEL instance is created).

2. The error occurs when using MCF properties (instance is created, same error appears).

3. The connection pool and datasources are configured and working correctly.

4. The procedure APPS.CCI_INV_O_21_ITEM_AVAIL_PKG_N.ITEM_CHECK works when invoked from sqlplus using the same values used in our BPEL process.

The ORA-06531 error obviously seems to indicate an error with the package, as it is clearly an error returned from the database. But we just confirmed that it is neither a connectivity issue nor is there anything wrong with the package itself.

Solution:

The DB Adapter deployed was an older version. Get a copy of the newer ‘DBAdapter.rar’ and redeploy it as follows:

opmnctl stopall

cd $ORACLE_HOME/j2ee/oc4j_soa

cp -Rp connectors connectors.old

cd $ORACLE_HOME/j2ee/oc4j_soa/connectors

rm -rf DbAdapter

mkdir DbAdapter

chmod 700 DbAdapter

cp /tmp/DBAdapter.rar $ORACLE_HOME/j2ee/oc4j_soa/connectors/DbAdapter

opmnctl startall

Explanation:
It appears that we have hit Oracle Bug 5954475 (which is related to the DB Adapter) and supposedly fixed by the 10.1.3.3 patchset, as documented in Oracle Metalink Note ID 437825.1.

The ‘DBAdapter.rar’ deployed on this environment was 356,884 bytes in size, while on our other environments was 374,183 bytes.

It seems that when we applied the 10.1.3.3 patchset, it did not fully patch everything. Fortunately, patchsets can be re-applied if a component failed to patch (it is smart enough to determine if a specific component has been patched and ignore it).

In our case, the patchset did update the DB Adapter, but failed during the redeploy phase. When patchsets update BPEL libraries, they will typically do it in the $ORACLE_HOME/bpel/lib directory or $ORACLE_HOME/bpel/system/services/lib directory. Adapters are updated in the /bpel/services/lib directory and are stored there for re-deployment.

Since the ‘DBAdapter.rar’ file in the /bpel/services/lib directory is larger than the one in the /connectors directory, this is an indication that the patchset did not re-deploy it after it patched it.

I did not dig deeper into this to investigate why, but at least our issue was resolved.

Migrating BPEL instances from one Oracle BPEL Process Manager 10g environment to another

We were cutting over to a new Oracle SOA Suite infrastructure (i.e., retiring the first instance, and cutting over to a new one).

Even though the codebase deployed on both environments were identical, the development team wanted all instances migrated as well. (The primary reason was that they were already overruling the BPEL instance title and inserting order and customer numbers there, so they would like to use that as a reference when troubleshooting.)

Migrating BPEL instances:

1. Purge all instances via the BPEL Console on the new instance (via the “Purge all instances” button).

2. Shutdown both SOA Suite midtiers.

3. Export the following tables from the old dehydration store and import it into the new one (using standard ‘exp’ and ‘imp’ database utilities):

ORABPEL.CUBE_INSTANCE
ORABPEL.CUBE_SCOPE
ORABPEL.AUDIT_DETAILS
ORABPEL.AUDIT_TRAIL

4. Run the following query on both dehydration store databases, and note the next_range value:

SELECT next_range FROM orabpel.id_range;

5. Select the largest of the two values, we’ll denote it as XYZ.

6. Run the following update statements on the new dehydration store database:

UPDATE orabpel.id_range SET next_range = XYZ + 10000 WHERE range_name = 'cikey';

UPDATE orabpel.cube_instance SET state = 9;

COMMIT;

That’s it. This effectively migrates all instances from the old environment to the new.

Things to note:

• These instructions are applicable to Oracle SOA Suite 10g (10.1.3.x).
• Updating the next_range value requires a restart of the midtier.
• The steps above intentionally force all instances to be “stale”.
• Backup your tables in the new instance before executing the instructions above just in case.

How to become an Oracle SOA Architect Certified Expert

I just took and passed the Oracle SOA Foundations, Adoption and Implementation exam (1Z0-114). The exam is actually acquired from BEA, and is a combination of both the BEA SOA Foundations and BEA SOA Adoption and Implementation exams. Passing the exam grants you an OCE (Oracle Certified Expert) as an Oracle SOA Architect.

Certification Details
    Certification:              Oracle SOA Architect Certified Expert (OCE)
    Exam:                       Oracle SOA Foundations, Adoption and Implementation
    Time:                        120 minutes
    Cost:                         $195
    Passing score:            66%
    Number of questions:  78 

Study Material
    BEA White Paper Domain Model For SOA

Preparation Time
You will need 20 hours (or more, depending on your current knowledge).

Sample Questions
    BEA SOA Adoption & Implementation Mock Exam
    Oracle SOA Foundations, Adoption and Implementation Sample Questions 
    BEA SOA Adoption and Implementation Sample Questions
    BEA SOA Foundations Sample Questions 
    Sample questions for IBM SOA Test 664/665
    IBM SOA Fundamentals (000-664 Exam) 

Degree of Difficulty
Oracle certification exams are not usually easy, but this provides recognition and credibility to the certification.

Based on the scale below, if you have extensive architecture, design, and implementation experience in SOA and related technologies, the exam is slightly above average in difficulty (straight black line). If you are a SOA developer with little architecting experience, the exam may be on the difficult end (dotted black line).

Around a third of the exam focuses on explicit technical questions (e.g., a Service Directory provides the ability to discover existing services). The second third involves concepts surrounding the BEA Domain Model (see Study Material). The last third focuses on story-board scenarios, and asks questions in areas such as canonical design, SOA benefits, cost justifications, architectural decisions, process approaches, roadmap prioritization, SOA adoption challenges, and more. 

Preparing for the Exam

1. Read the study material and know it by heart.
 
2. You should understand and know how to explain each of the following:

• BPEL, service composition, orchestration, composite applications, service reuse.
• ESB, VETO, transformation, routing.
• Fine-grained vs. coarse-grained services.
• XA protocol, compensating transactions, idempotent operations.
• Calling/exchange paradigms: asynchronous, synchronous, callback, polling, fan-out, callout, uni-directional, bi-directional, fork/join.
• Terminology: Web services, portlets, WSDL, XML, UDDI, WSRP, JSR, AJAX, HTTP, SSL, SOAP, EDA, JMS, SAML, SLA, canonicals, adapters.
• WS-* specifications: WS-ReliableMessaging, WS-Policy, WS-Coordination, WS-Security, WS-Trust, WS-Addressing.
• Products: Service Registry, Service Bus, Policy Directory, BAM, Web Services Manager. 

3. Go through the BEA SOA Adoption & Implementation Mock Exam and Oracle SOA Foundations, Adoption and Implementation Sample Questions.

4. Leverage your SOA architecture and implementation experience. 

Oracle Links 
For more information about the certification and the exam, check out:
    http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=284
    http://education.oracle.com/pls/web_prod-plq-dad/db_pages.getpage?page_id=41&p_exam_id=1Z0_114

My Opinions
The exam assumes architecture level experience within the SOA space. 

For example, what role does BAM (Business Activity Monitoring) play in your enterprise? What is the best way to push data to BAM? How does this, if at all, impact your main operational process? Unless you have architected the usage of and/or implemented BAM, you may not know. If you can answer these questions, then you should be able to answer one of the BAM related questions in the exam.

Being a perfectionist, I noticed a few grammatical errors in the exam, missing periods, double quotes not closed, etc., but nothing that would impact your understanding of the question.

Final thoughts:

• Follow what I mentioned in "Preparing for the Exam". This will give you what you need to cover 75% of the exam.
• Questions are not technical in nature. For example, you will not find questions that ask "What is the request security token structure in WS-Trust?" You will most likely encounter questions such as "Company X currently has services on a legacy system and want to migrate them off. What should they do?"
• Go through the mock exams.
• Know the BEA White Paper Domain Model For SOA by heart. If you would like the summary and study notes I made from this white paper, then contact me and I'll be happy to share it.
• The IBM mock exams I listed above have a few questions that are similar to those that appear in the exam, so going through them will probably firm up your concepts. 

Good luck!

Prerequisites for installing Oracle SOA Suite on Red Hat 64-bit

Oracle SOA Suite 10g (10.1.3.x) is certified to install and run on 64-bit Linux. Ignore the Oracle install notes and follow these instructions instead, and you can guarantee it to work flawlessly.

These instructions are specific to Oracle SOA Suite 10g (10.1.3.1) and Red Hat ES 5.4 x86_64, installed as user "oracle". You can then proceed normally with your patching/upgrades.

1. Modify /etc/redhat-release.

This is to fake out the installer to recognize this as a certified OS.

cp /etc/redhat-release /etc/redhat-release.original
export ORAREL="Red Hat Enterprise Linux ES release 3 (Nahant Update 8)"
echo $ORAREL > /etc/redhat-release

2. Create user and group.

groupadd dba
groupadd oinstall
useradd -c "Oracle software owner" -g oinstall -G dba oracle
passwd oracle

3. Confirm that the following RPMs are installed.

This can be done by using the following command. Ensure that the appropriate 32-bit and 64-bit packages are installed accordingly. All of the required RPMs can be found on the install media.

Command:

rpm -qa --queryformat "%{NAME}-%{VERSION}.%{RELEASE} (%{ARCH})\n" | grep <package>

Packages:

binutils-2.17.50.0.6-12.el5.x86_64.rpm
compat-db-4.2.52-5.1.i386.rpm
compat-db-4.2.52-5.1.x86_64.rpm
compat-glibc-2.3.4-2.26.i386.rpm
compat-glibc-2.3.4-2.26.x86_64.rpm
compat-libstdc++-296-2.96.138.i386.rpm
compat-libstdc++-33-3.2.3.61.i386.rpm
compat-libstdc++-33-3.2.3.61.x86_64.rpm
control-center-2.16.0.16.el5.i386.rpm
control-center-2.16.0.16.el5.x86_64.rpm
gcc-4.1.2.46.el5.x86_64.rpm
gcc-c++-4.1.2.46.el5.x86_64.rpm
glibc-2.5.42.i686.rpm
glibc-2.5.42.x86_64.rpm
glibc-common-2.5.42.x86_64.rpm
glibc-devel-2.5.42.i386.rpm
glibc-devel-2.5.42.x86_64.rpm
libgcc-4.1.2.46.el5.i386.rpm
libgcc-4.1.2.46.el5.x86_64.rpm
libpng-1.2.10-7.1.el5_3.2.i386.rpm (optional)
libpng-1.2.10-7.1.el5_3.2.x86_64.rpm (optional)
libstdc++-4.1.2.46.el5.i386.rpm
libstdc++-4.1.2.46.el5.x86_64.rpm
libstdc++-devel-4.1.2.46.el5.i386.rpm
libstdc++-devel-4.1.2.46.el5.x86_64.rpm
libXpm-3.5.5.3.i386.rpm
make-3.81.3.el5.x86_64.rpm
openmotif-2.3.1-2.el5.i386.rpm
openmotif-2.3.1-2.el5.x86_64.rpm
ORBit2-2.14.3-5.el5.i386.rpm (optional)
ORBit2-2.14.3-5.el5.x86_64.rpm (optional)
pdksh-5.2.14.36.el5.x86_64.rpm
setarch-2.0.1.1.x86_64.rpm
sysstat-7.0.2.3.el5.x86_64.rpm

4. Apply Oracle patch 6078836.

This is done by simply downloading the patch and executing the following commands.

unzip p6078836_101330_LINUX.zip
cp 6078836/libdb.so.2 /usr/lib
rm -rf 6078836

5. Add the following to /etc/security/limits.conf.

Use tab as the delimiter.

* soft nproc 2047
* hard nproc 16384
* soft nofile 2048
* hard nofile 65536

6. Add the following to /etc/pam.d/login.

session required /lib/security/pam_limits.so

7. Add the following to /etc/profile.

if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

8. Ensure that you have at least 512 MB of SWAP space.

Simply run the following command to confirm.

grep SwapTotal /proc/meminfo

9. Confirm that 'perl' and 'fuser' are installed.

Simply run the following command to confirm.

which perl ; which fuser

10. If using a RAC database, manually set the SYS password on all RAC nodes to same password.

The SYS password must be set to the same value on all RAC nodes. For example, the following command would have to be executed on all RAC nodes:

ALTER USER sys IDENTIFIED BY manager;

11. Force the OS to link to 32-bit (you must undo this post-installation).

mv /usr/bin/gcc /usr/bin/gcc.oracle.original
echo "/usr/bin/gcc.oracle.original -m32 \$*" > /usr/bin/gcc
chmod 755 /usr/bin/gcc

12. Immediately prior to installation, set the shell to 32-bit (as ‘oracle’).

linux32 bash
export LDEMULATION=elf_i386

13. Start the installer (as ‘oracle’)!

Advanced find/replace commands in Linux

Working on a recent SOA/AIA project, I had to write numerous recursive find/replace commands to update hostnames, ports, values, etc.

Below are some random examples with a brief explanation.

Example 1 - Recursively find a string

Recursively search through all files in $ORACLE_HOME/bpel/domains/default/tmp and list out the filenames that have the hostname string "dev78.net" in them.

find $ORACLE_HOME/bpel/domains/default/tmp -type f | xargs grep "dev78.net"

Example 2 - Recursively replace a string

Recursively search from the current directory, and replace all references of "orabpel" with "orabpel2" in all files.

find . -type f -exec sed -i "s%orabpel%orabpel2%" {} \;

Example 3 - Recursively replace a string in a specific location within the line

In the command below, everything between the first % and the second % represents the original search string. Everything between the second % and the third % is the new string to be replaced.

find . -type f -exec sed -i "s%\(<soapEndpointURI>\)\(.*\)?wsdl\(.*\)%\1\2\.wsdl\?wsdl\3%" {} \;

The original search string consists of 4 parts:

(1) <soapEndpointURI>
(2) *
(the string "?wsdl" in between 2 and 3)
(3) *

The new string search string keeps parts 1, 2, and 3 intact, but replaces the text in between with ".wsdl?wsdl".

So if the original string looked like this:

<soapEndpointURI>http://thisisahmed/hello?wsdl</soapEndpointURI>

It would now look like this:

<soapEndpointURI>http://thisisahmed/hello.wsdl?wsdl</soapEndpointURI>

Example 4 - Exclude files

Same as Example 3, but will not search/replace inside of .class, .jar, or .zip files.

find . -type f \( -iname "*.class" ! -iname "*.jar" ! -iname "*.zip" \) -exec sed -i "s%\(\)\(.*\)?wsdl\(.*\)%\1\2\.wsdl\?wsdl\3%" {} \;

Example 5 - Recursively replace the hostname in URLs

For every string that takes the form "http://*:7777" (the * is just a wildcard, but can be any value), replace it with "http://${HOSTNAME}:7777" where ${HOSTNAME} is an environment variable.

find -type f -exec sed -i "s%\(ocation=\"http://\)\(.*\):7777\(.*\)%\1${HOSTNAME}:7777\3%" {} \;

Oracle SOA Suite 10g prerequisites for Red Hat Linux ES/AS 4

This is just a quick cheat sheet to make sure you have all the appropriate prerequisites for Oracle SOA Suite 10g (10.1.3.x) on Red Hat Linux ES/AS 4 (32-bit).

1. Ensure that you have at least 512 MB SWAP space.

grep SwapTotal /proc/meminfo

2. Ensure that you have enough disk space.

Confirm that you have at least 10 GB free for your software.
Confirm that you have at least 5 GB free for your temporary scripts directory.
Confirm that you have at least 400 MB free in /tmp.

df -m

3. Confirm that the following Red Hat 4 32-bit RPMs or later are installed:

binutils-2.15.92.0.2-13
compat-db-4.1.25-9
compat-glibc-7.x-2.2.4.32.6
compat-libstdc++-296-2.96-132.7.2
compat-libstdc++-7.3-2.96.128
compat-libstdc++-devel-7.3-2.96.128
control-center-2.8.0-12
gcc-3.4.3-22.1
gcc-c++-3.4.3-22.1
glibc-2.3.4-2.9
glibc-common-2.3.4-2.9
gnome-libs-1.4.1.2.90-44.1
libstdc++-3.4.3-22.1
libstdc++-devel-3.4.3-22.1
make-3.80-5
openmotif21-2.1.30-11.RHEL4.4
pdksh-5.2.14-30
setarch-1.6-1
sysstat-5.0.5-1
xscreensaver-4.18-5.rhel4.2

4. Confirm that 'perl' and 'fuser' are installed.

which perl
which fuser

5. Confirm that /etc/security/limits.conf has the following (must use tabs between fields):

* soft nproc 2047
* hard nproc 16384
* soft nofile 2048
* hard nofile 65536

6. Confirm that /etc/pam.d/login has the following line (must use tab between 2nd and 3rd field):

session required /lib/security/pam_limits.so

7. Confirm that /etc/profile has the following lines:

if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

You should now expect your Oracle SOA Suite installation to go smoothly.

ORA-27102: out of memory

Recently tried starting up a newly create 9i (9.2.0.5) database instance and encountered the following:

oracle@ebs:/u01/oracle/devdb/9.2.0> sqlplus "/ as sysdba"

SQL*Plus: Release 9.2.0.5.0 – Production on Wed Feb 17 16:53:32 2010

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Connected to an idle instance.

SQL> startup
ORA-27102: out of memory
Linux Error: 22: Invalid argument

This was installed on Red Hat Linux AS 4 U4 x86 with 8GB of RAM.

The solution was simply to adjust the SHMMAX kernel parameters. Basically, edit /etc/sysctl.conf and modify the following:

OLD: kernel.shmmax = 4294967296
NEW: kernel.shmmax = 2147483648

SHMMAX defines the maximum size (in bytes) of a single shared memory segment that a Linux process can allocate in its virtual address space. 32-bit Red Hat Enterprise Linux supports shared memory segments up to 4GB.

Per the Red Hat Knowledgebase:

If SHMMAX is set to a value larger then or equal to 4294967296 bytes (4 GB) on a 32-bit system, SHMMAX will be set to 0 bytes since it wraps around the 4 GB value. This means that SHMMAX should not exceed 4294967295 (one byte below 4 GB) on a 32-bit system.

Redeploying the OJMS Adapter due to “Unable to find/read file META-INF/ra.xml”

Upon startup of Oracle SOA Suite, the OC4J_SOA OPMN logs were showing errors related to my OJMS adapter:

10/01/28 19:17:50 WARNING: ApplicationStateRunning.loadConnectors Exception loading connector OjmsAdapter : Unable to find/read file META-INF/ra.xml in /u01/app/oracle/product/10.1.3/soa_1/j2ee/oc4j_soa/connectors/OjmsAdapter/OjmsAdapter archive (IO Error: No such file or directory)

Basically, the OJMS adapter is not found, which was strange since I was pretty sure it was deployed.

The oc4j-connectors.xml file was configured correctly:

<connector name="OjmsAdapter" path="OjmsAdapter.rar">
<config-property name="lookupMethod" value="resourceProvider"/>
<config-property name="esourceProviderName" value="MY_AQJMS_POOLED_DS"/>
</connector>

I am not sure why the adapter was missing, but fortunately I had a backup copy of both the adapter JAR and oc4j-ra.xml, and used the following steps to redeploy the adapter manually:

mkdir -p $ORACLE_HOME/j2ee/oc4j_soa/connectors/OjmsAdapter/OjmsAdapter

cp OjmsAdapter.rar $ORACLE_HOME/j2ee/oc4j_soa/connectors/OjmsAdapter/OjmsAdapter

cd $ORACLE_HOME/j2ee/oc4j_soa/connectors/OjmsAdapter/OjmsAdapter

unzip OjmsAdapter.rar

mv $ORACLE_HOME/j2ee/oc4j_soa/connectors/OjmsAdapter/OjmsAdapter/OjmsAdapter.rar $ORACLE_HOME/j2ee/oc4j_soa/connectors/OjmsAdapter

mkdir $ORACLE_HOME/j2ee/oc4j_soa/application-deployments/default/OjmsAdapter

cp /tmp/oc4j-ra.xml $ORACLE_HOME/j2ee/oc4j_soa/application-deployments/default/OjmsAdapter

I restarted the OC4J_SOA container, and was good to go.

Naming host and domain names for Oracle software

Let me start with the conclusion first.

Conclusion:

If you are an Oracle expert, when installing products, configuring software, or setting up servers, for hostnames, domain names, and subdomain names...

• Do not use underscores ("_")
• Do not use dashes ("-")
• In fact, don't use any special characters at all if you can control it
• Don't even think of uppercasing or camelcasing

Example #1 - OEM Grid Control issues due to underscore in hostname

In 2007, we had a BEA WebLogic Server 9.1 running on a hostname called SERVER_A and wanted to use OEM 10g Grid Control R3 to monitor this WebLogic instance. This Oracle web page indicated that the "Plug-in for BEA WebLogic is built into Oracle Enterprise Manager 10g Grid Control Release 2, Release 3, and Release 4."

BEA WebLogic Server 9.1 was already installed on Windows 2003 Enterprise and we successfully installed the OEM Agent 10.2.0.3 on the same server. However, we were not able to add the Weblogic server as a target.

Cause of problem

Underscore in the hostname.

Explanation

In our case here, as a product, OEM 10g Grid Control was unable to register targets with underscore in their hostname.

Don't get creative. Avoid underscores.

Example #2 - E-Business Suite installation failure due to dash in hostname

In 2010, we were installing Oracle E-Business Suite 11i (11.5.10) for Linux on a server hostname taking the format ORADEV.MY-DOMAIN.ORG. Now I had ran into a few snags installing it the first time due to missing prerequisites, but those were quickly resolved and installation proceeded fine.

During the installation, the adrundb.sh script reported "Cannot create control file for..." and threw "RW-50010" and "RW-50004" error codes.

Cause of problem

Dash in the fully qualified hostname.

Explanation

The log showed:

ALTER DATABASE RENAME GLOBAL_NAME to oradev.my-domain.org
*
ERROR at line 1:
ORA-00933: SQL command not properly ended

Basically, the Oracle database cannot do an ALTER DATABASE with a fully qualified hostname that includes dashes (if not surrounded by double quotes). I was pretty much unable to work around this as the command was automatically created and executed by the installer. We had no choice but to rename the server and restart the installation.

Don't get creative. Avoid dashes.

(I believe this issue was corrected in the 11.5.10.2 installer.)

Scenario #3 - Configuring software using uppercase or camelcase hostnames

In 2009, I had a colleague who would use all uppercase ("WWW.GOOGLE.COM") or camelcase ("www.Google.com") when updating configuration files of various Oracle software. I tried explaining that although this was technically and syntactically correct, "why the $#%! would you do that?!"

He came from a Windows background, and I tried explaining that although it was indeed valid and case insensitive, "you do understand that what comes AFTER the domain name could be case sensitive?" in an attempt to convince him not to be creative.

The point is, yes, it's valid. But from the two scenarios above, some software either will strictly adhere to the standard (rightly so) or is just badly written that it doesn't factor in the "creative" people who want to name their servers "This_Server!.is-Awesome.com".

Final word:

In the DNS world, there is some industry confusion surrounding the ability to use underscore in subdomains, but it it is not a valid character in a domain name. There are cases where RFC-compliant mail servers will refuse to deliver mail for Windows computers with domain names containing underscores. In another example, Microsoft SQL Server maps underscores to backslashes. We have already shown above how 2 Oracle products behave when using underscores or dashes in hostnames.

The point here being that the more creative you get with your hostnames and domain names, the higher chance that the software you install or configure will have issues.

Compiling Apache 2.0 to support mod_rewrite

For a recent project, I downloaded the Apache 2.2.14 Unix source (http://apache.mirrors.tds.net/httpd) to assist in our automated BPEL/ANT compilations.

We added the following rewrite rules:

RewriteEngine On
RewriteCond %{QUERY_STRING} ^wsdl$
RewriteRule ^(.*)$ $1.wsdl [PT]

But when we started Apache, we received the following error:

oracle@oradev:/u01/httpd-2.2.14/bin> ./apachectl start

Syntax error on line 411 of /u01/httpd-2.2.14/conf/httpd.conf:
Invalid command 'RewriteEngine', perhaps misspelled or defined by a module not included in the server configuration

This is because when Apache was compiled using make, we forgot to include the appropriate flag.

This is the correct way to compile Apache to include the mod_rewrite module:

cd /u01
gtar -xzvf httpd-2.2.14.tar.gz
cd httpd-2.2.14
./configure --prefix=${PWD} --enable-rewrite=shared
make
make install

Afterwards, you will notice the mod_rewrite module being automatically configured to load in httpd.conf.

Now adding your rewrite rules will work.

Troubleshooting ORABPEL-05215 errors

The ORABPEL-05215 is annoying to troubleshoot. Advice online is pretty generic, and when I ran into this, I decided to once and for all get to the bottom of it.

My Scenario

1. I used ANT to compile the BPEL process.

ant --noconfig -buildfile build.xml compile

Compilation successful. The JAR file was successfully created in the ~/output directory.

HelloWorld/output/bpel_HelloWorld_1.0.jar

2. I used ANT to deploy the BPEL process.

ant --noconfig -buildfile build.xml deployProcess

Deployment failed. The output was as follows:

Buildfile: build.xml

deployProcess:
[echo]
[echo] --------------------------------------------------------------
[echo] | Deploying bpel process HelloWorld on sawftdev, port 7777
[echo] --------------------------------------------------------------
[echo]
[deployProcess] Deploying process /u01/deploy/code/SVN/HelloWorld/output/bpel_HelloWorld_1.0.jar

BUILD FAILED

/u01/deploy/code/SVN/HelloWorld/build.xml:124: A problem occured while connecting to server "sawftdev" using port "7777": bpel_HelloWorld_1.0.jar failed to deploy. Exception message is: ORABPEL-05215
Error while loading process.
The process domain encountered the following errors while loading the process "HelloWorld" (revision "1.0"): Failed to compile classes.
Failed to compile the generated BPEL classes for "HelloWorld".
.
If you have installed a patch to the server, please check that the bpelcClasspath domain property includes the patch classes.

Not enough information is provided unfortunately.

Deploying the JAR file manually via the BPEL Console produced the same result:

* bpel_AccSusResPortalProvABCSImpl_1.0.jar failed to deploy.
Error while loading process. The process domain encountered the following errors while loading the process "HelloWorld" (revision "1.0"): Failed to compile classes. Failed to compile the generated BPEL classes for "HelloWorld". . If you have installed a patch to the server, please check that the bpelcClasspath domain property includes the patch classes.

3. I restarted the OC4J_SOA container. No luck. I changed some of my JVM memory settings. No luck.

The Discovery

1. My server already had output and error logs separated for the OC4J_SOA container.

opmn.xml had the following configuration for the log files for the OC4J_SOA container:

<data id="oc4j-options" value="-out /u01/app/oracle/product/10.1.3/soa_1/opmn/logs/soa_group_1~oc4j_soa~soa_group_1~1.out -err /u01/app/oracle/product/10.1.3/soa_1/opmn/logs/soa_group_1~oc4j_soa~soa_group_1~1.err">

So basically, output and errors are contained in separate logs.

2. The soa_group_1~oc4j_soa~soa_group_1~1_2009_12_30_16_04_34.out log file showed:

<2009-12-30> Error while invoking bean "domain manager": [com.collaxa.cube.engine.deployment.ProcessLoaderException: Error while loading process.
The process domain encountered the following errors while loading the process "HelloWorld" (revision "1.0"): Failed to compile classes.
Failed to compile the generated BPEL classes for "HelloWorld".
.
If you have installed a patch to the server, please check that the bpelcClasspath domain property includes the patch classes.
]
ORABPEL-05215

Error while loading process.
The process domain encountered the following errors while loading the process "HelloWorld" (revision "1.0"): Failed to compile classes.
Failed to compile the generated BPEL classes for "HelloWrld".
.
If you have installed a patch to the server, please check that the bpelcClasspath domain property includes the patch classes.

This is similar to the output I received during deployment.

3. But the soa_group_1~oc4j_soa~soa_group_1~1_2009_12_30_16_04_34.err showed something different:

09/12/30 16:06:13 /u01/app/oracle/product/10.1.3/soa_1/bpel/domains/default/tmp/.bpel_HelloWorld_1.0_fc0bd0106b3c9d6a8abf672e863c8d72.tmp/BPEL-INF/src/bpel/helloworld/Execute6.java:3: Package com.sawft.util not found in import.
09/12/30 16:06:13 import org.w3c.dom.Element;import java.util.Properties;import java.io.FileInputStream;import com.sawft.util.*;import com.sawft.*;
09/12/30 16:06:13 ^
09/12/30 16:06:13 1 errors

Now we're getting somewhere.

This BPEL process apparently has embedded Java code and was importing packages that where not included in the bpelcClasspath on the server.

The .bpel file contained import statements such as:

<bpelx:exec import="com.sawft.*"></bpelx:exec&gt

My Solution

1. I added the necessary JARs to the classpath in $ORACLE_HOME/bpel/domains/default/config/domain.xml (the bpelcClasspath parameter).

What you should do

Leverage the OPMN logs when troubleshooting this issue. As a developer, you may need to reach out t the application administrator.

Interesting bug related to ANT deployments and importing JARs

This finding is related to the same issue I was troubleshooting in my next post.

Basically, if you use ANT to deploy a BPEL process to a domain that is not the default domain, and this BPEL process uses embedded Java and imports custom Java packages, unless those JARs are in the bpelcClasspath of the default domain as well, deployment will fail.

This has been tested on Oracle BPEL Process Manager 10g (10.1.3.3.1).

Scripted install for an Oracle SOA Suite 10g (10.1.3) cluster

Around a year ago, I wrote scripts that would allow you to silently install Oracle SOA Suite 10g as a 2-node cluster. Manual intervention is reduced by 95% (I would have reduced it by 100% if not for bugs in the silent installer).

For those who have installed Oracle SOA Suite in High Availability per the Oracle® Application Server
Enterprise Deployment Guide 10g Release 3 knows that the installation and configuration takes time (often a full day or more) and is loaded with endless manual steps.

My scripts takes away all the manual effort.

Completing the installation of a functioning 2-node SOA Suite cluster can be completed in 60 minutes.

Here is a small portion of the property file you must set up.

#----------------------------------------
# Parent directory of scripts
#----------------------------------------
SAWFT_INSTALL_DIRECTORY=/u01/temp2/soa_suite_cluster_install_scripts

#----------------------------------------
# Location of install software
#----------------------------------------
SAWFT_SOFTWARE_DIRECTORY=/u01/temp/software

#----------------------------------------
# User/Group owner
#----------------------------------------
SAWFT_INSTALL_USER=oracle
SAWFT_INSTALL_GROUP=oinstall

#----------------------------------------
# Hostname of this server
#----------------------------------------
SAWFT_HOSTNAME=oradev1.sawft.com
SAWFT_HOSTNAME_NODE1=oradev1.sawft.com
SAWFT_HOSTNAME_NODE2=oradev2.sawft.com

#----------------------------------------
# Used to create staticports file for installer
#----------------------------------------
SAWFT_J2EE_OPMN_REQUEST_PORT=6101
SAWFT_J2EE_OPMN_LOCAL_PORT=6201
SAWFT_J2EE_OPMN_REMOTE_PORT=6004

SAWFT_GTWY_OPMN_REQUEST_PORT=6102
SAWFT_GTWY_OPMN_LOCAL_PORT=6202
SAWFT_GTWY_OPMN_REMOTE_PORT=6007

SAWFT_OHS_OPMN_REQUEST_PORT=6100
SAWFT_OHS_OPMN_LOCAL_PORT=6200
SAWFT_OHS_OPMN_REMOTE_PORT=6001
.
.
.

If you would like a copy of these scripts, please contact me via the 'Contact me' link on http://thisisahmed.com.

A few things worth noting:

• The scripts work perfect, but there is redundant and/or messy code in some areas.
• Currently it installs a 2-node Oracle SOA Suite 10g 10.1.3.3.1 cluster on Red Hat Linux, but can be modified to support later versions or additional nodes.
• There are bugs in the Oracle Application Server and Oracle Web Services Manager silent installers that require the GUI installer to be used in a few steps (see http://blog.thisisahmed.com/2008/11/numerous-bugs-in-oracle-soa-suite-10g.html).

I am still doing some minor cleanup to them, so my response may be slow.

---

Update on 2009-12-01:
The scripts can be downloaded from here.

Bug in the SOA Suite 10g 10.1.3.5 patchset for Windows affects BPEL redeployments

My recommendation at this time is not to proceed with applying the Oracle SOA Suite 10g Patch Set 5 (10.1.3.5.0) for Windows. There is a bug that causes redeployment of BPEL processes to fail.

When redeploying an updated version of a BPEL process that is already deployed to the server, your ant or JDeveloper will return the following error:

* Error when deploying BUILD FAILED C:\data\HelloWorld\build.xml:79: A problem occured while connecting to server "localhost" using port "80": bpel_HelloWorld_v2009_06_01__55238.jar failed to deploy. Exception message is: Error deploying BPEL suitcase. An error occurred while attempting to deploy the BPEL suitcase file "C:\oracle\product\10.1.3\soa_1\bpel\domains\default\tmp\bpel_11747128.tmp"; the exception reported is: archive cannot rename C:\oracle\product\10.1.3\soa_1\bpel\domains\default\tmp\.bpel_HelloWorld_v2009_06_01__55238_a4ef0434bf12f22a3374aab6a0a942a1.tmp

Reviewing the domain.log reveals an ORABPEL-05250 error, essentially reporting the same thing:

<2009-10-12 09:39:30,281> <WARN> <default.collaxa.cube.engine.deployment> <DeploymentManager::deploySuitecase> Process bpel://localhost/default/HelloWorld~v2009_06_01__55238/ is being re-deployed (same version number) <2009-10-12 09:39:30,281> <ERROR> <default.collaxa.cube.engine.deployment> <DeploymentManager::deploySuitecase> archive cannot rename C:\oracle\product\10.1.3\soa_1\bpel\domains\default\tmp\.bpel_HelloWorld_v2009_06_01__55238_a4ef0434bf12f22a3374aab6a0a942a1.tmp <2009-10-12 09:39:30,312> <ERROR> <default.collaxa.cube> <BaseCubeSessionBean::logError> Error while invoking bean "domain manager": Error deploying BPEL suitcase. An error occurred while attempting to deploy the BPEL suitcase file "C:\oracle\product\10.1.3\soa_1\bpel\domains\default\tmp\bpel_11747128.tmp"; the exception reported is: archive cannot rename C:\oracle\product\10.1.3\soa_1\bpel\domains\default\tmp\.bpel_HelloWorld_v2009_06_01__55238_a4ef0434bf12f22a3374aab6a0a942a1.tmp ORABPEL-05250 Error deploying BPEL suitcase. An error occurred while attempting to deploy the BPEL suitcase file "C:\oracle\product\10.1.3\soa_1\bpel\domains\default\tmp\bpel_11747128.tmp"; the exception reported is: archive cannot rename C:\oracle\product\10.1.3\soa_1\bpel\domains\default\tmp\.bpel_HelloWorld_v2009_06_01__55238_a4ef0434bf12f22a3374aab6a0a942a1.tmp at com.collaxa.cube.engine.deployment.DeploymentManager.deploySuitcase(DeploymentManager.java:871) at com.collaxa.cube.ejb.impl.BPELDomainManagerBean.deploySuitcase(BPELDomainManagerBean.java:465) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Even if you remove the read-only attributes or set the Windows permissions and owner attributes to the Administrators group on the /domains folder, the behavior is the same.

The server deploys the project to a temporary directory in $ORACLE_HOME/bpel/domains/tmp and attempts to rename the existing project before replacing it with the new one. This is when the error occurs. Some unknown Java process within the BPEL server locks both the bpelclasses.jar file and one of your project WSDLs (e.g., _HelloWorld.wsdl), preventing the renaming of the parent directory.

To workaround this, you have 3 options:

1. Undeploy the process, shutdown SOA Suite, manually delete the project directory from the /tmp directory, start up SOA Suite, then redeploy your process.
   
2. Use a process unlocker utility to force remove the process lock on the JAR and WSDL files mentioned above, then redeploy your process.
3. Don't upgrade to 10.1.3.5 for Windows for now.

Unless there is a particular bug fix that you desperately need to push, avoid upgrading to 10.1.3.5 on Windows for the time being. As mentioned above, there are workarounds, but they are annoying and may disrupt operations.