A Real-World Experience using AIA Productivity Tools

One of the joys of working in product development, especially in my earlier customer success advocate role, is the satisfaction received when helping resolve customers needs with your products!

A few months back, a key telecom customer needed to describe the semantic Enterprise Business Object (EBO) structure in a hierarchical/indented model to a flat spreadsheet so that designers can perform application mapping for the Process Integration Packs (a.k.a. PIP, basically pre-packaged integration accelerators for specific business process flows). I pointed them to a not-well publicised (and highly underestimated tool) called the XSL Mapping Analyzer (a.k.a XMAN).

Below is an email from the customer describing how useful they found the tool, in fact just one of the utilities! Thanks to the OCS Team (Balaji S G and Kurt) and KT (Sungwoo Yu) for this valuable feedback!

Side note: XMAN is actually a set of utilities my team built to parse XSL and EBO XSDs into various formats for semantic analysis of attribute level mapping; including HTML, CSV, flattened CSV and XML. It’s real power is in providing tactical SOA governance if you are using AIA canonicals, as it can help flag functional interoperability issues across projects using the same EBO, thereby encouraging service reuse and providing the ROI on your SOA. To learn more about XMAN usage, see my earlier blog post and OOW 2009 session.

— Email below published verbatim with prior owner’s permission —

Hi Balaji,

I’m glad to share my experience with Rajesh’ blog

Specially, I use to visit and learn a best practices using aia from that popular blog.

I’d like to briefly introduce my situation.

I’m working at telecom Project aims to build next generation BSS/OSS for 35 mil subscribers utilized by Oracle RODOD(Siebel,BRM,OSM, OPH and etc) .

I’m in charge of Integration team.

As we try to fit OOTB PIPs with our business, we need to analysis OOTB PIPs and design extension of PIPs.

These days our team cooperation with OCS for Communication PIP extension design on AIA Technical Document Design.

Although we cannot totally understand what existing input/output message and mapping in complex process integrations, we have to provide explain how prebuilt aia integration build interoperability with Siebel, BRM.

Basically, we can refer PIP implementation documents. However, if we closely looked at message level input/output and mapping details for reviewing service detail communication with edge application. It had been challenge.

At that time, I concerned that below so that I requested the OCS guidance

For we describe edge ABM structure or complex EBMs on spreadsheet, how can we describe different level elements on spreadsheet automatically?

Why we need to describe on spreadsheet, most of edge application function designer is not familiar with complex related xsd and different level of structure specially ebo.

So, when designed for extension of pips, Edge application function designer is unable to consider EBO structure.

If AIA team share used each operation EBM by spreadsheet, they will be easy to approach understanding EBO

Do you have any idea for that?

1.       How to automatically convert message structure xsd to table. (It could be different structure distinguished by indentation)

2.       If we describe different level elements distinguish using XPath, How we automatically extract XPath of all nodes from EBM.

Thanks to OCS(Balaji, Kurt) helps

We can utilize AIA Development Tools such as XMAN, XSD Flatter and generated XMAN reports RV11.1.

Particularly, XSD Flatter is right to solve above two problems.

For instance, I’m going to explain how I can extract element tree on spreadsheet from SalesOrderEBM.xsd

source $AIA_INSTANCE/bin/aiaenv.sh.

cd $AIA_Installed Directory/DeveloperTools/XSDFlattener/bin/

 

xsd2csv.sh -inputSchemaURL $AIA_Metadata_Directory /AIAComponents/EnterpriseObjectLibrary/Industry/Communications/EBO/SalesOrder/V2/SalesOrderEBO.xsd -rootElement ProcessSalesOrderEBM

As converted, I’ve got a csv file with x-path, cardinality, annotation and element tree with display hierarchy using indentation, and then reformatted below.

Even, I can convert ABM of Siebel Integration Object Message as well.

xsd2csv.sh -inputSchemaURL $AIA_Metadata_Directory /AIAComponents/ApplicationObjectLibrary/Siebel/V1/schemas/SWIOrderIO.xsd -rootElement ListOfSWIOrderIO

 

In addition, I aware generated RV11.1 XMAN report already provided whole EBM csv information by support.oracle.com

Also I can generate csv report from all of EBM using eol2csv.

eol2csv.sh -inputDir $AIA_Metadata_Directory /AIAComponents/ApplicationObjectLibrary/Siebel/V1/schemas -outputDir ~/eoloutput/AIAComponents/ApplicationObjectLibrary/Siebel/V1/schemas

Thanks

Sungwoo

Book Review: Oracle Application Integration Architecture (AIA) Foundation Pack 11gR1: Essentials

AIA Essentials Book

I was recently asked by Packt Publishing to review the above mentioned book. I have worked in AIA development for three years (and still work closely with the team) driving many cross-functional Process Integration Packs (PIPs) architecture initiatives, developing Foundation Pack developer productivity tools, managing SAP PIP development and leading pre- and post sales solution architecture support for strategic accounts to enable successful implementations. Having seen issues faced with complex SOA based integration projects first-hand, it was an interesting opportunity to review a book that attempts to explain the “essentials” of my product from an implementors perspective.

Disclaimer: All opinions are my own and not that of my employer. I have received no compensation for this review and was sent an e-book by the publisher. My review should be considered biased due to my proximity to the product, having reviewed a lot of the product documentation itself and having trained many SI partners on the very same topic!

The book targets SOA practitioners who need to understand the AIA Foundation Pack. There are chapters dedicated for Overview, Enterprise Business Objects (EBO), Enterprise Business Messages (EBM), Enterprise Business Services (EBS), Application Business Connector Services (ABCS), Enterprise Business Flows (EBF), Security, Versioning, Design Patterns, Error Handling and Logging, Service Management using Oracle Enterprise Repository (OER), Composite Application Validation System (CAVS) and finally ends with a case study.

My first concern when flipping through the table of contents was that it felt like a rehash of the product documentation, specifically the Concepts and Technologies Guide. So let’s get that out-of-the-way. I know for a fact that the Oracle documentation is the definitive source of truth on Oracle products (refer to the disclaimer about my bias). However, the AIA documentation targets multiple audiences (including internal Oracle PIP development teams) and hence has some information not immediately relevant for Foundation Pack implementors. That is an opportunity this book exploits and it offers the following benefits:

First, it brings together the Concepts and Technologies Guide as well as the Developer’s Guide and presents it in an informal, almost conversational tone. In addition, it omits some advanced concepts (e.g. ABCS customizations) which mostly make sense for PIP teams or ISV. This makes the book a bit easier to consume for those new to the concept.

Second, the chapters have nuggets of field knowledge from the implementation experiences of the author, for example ABCS best practices and stressing on not equating web services with SOA.

Third, there are a lot of screen shots that hand holds newcomers to the concepts, and a case study towards the end that brings the concepts together.

Fourth, the book does not attempt to teach SOA concepts and that is a GOOD thing. Working experience with SOA is a mandatory pre-requisite. I have seen many consultants who attend the AIA training after just having completed a SOA class the week before, and who are expected to be on-site the week after to tie together decades old legacy systems with heterogeneous data models. Does. Not. Work.

I did find quite a few issues with the book that I would have liked to see addressed.

First, the formatting. I am not sure if this only affects the ePub e-book, but the headings are difficult to keep track of since they are only differentiated by color and subtle differences in size. A simple heading numbering would have made it a lot easier to follow, especially when a lot of concepts (such as EBO) are repeated in different context throughout the book.

Second, the editing. Some sections stop abruptly as though the author meant to follow-up but did not get to it. For example, a chapter 1 section ends with “Let’s discuss a little more about Oracle AIA PIP before getting into the AIA Foundation Pack”. The next section then starts with the Foundation Pack, instead of the PIP. In another example, EBO customizations refer liberally to EBMs, which have not yet been covered at that point. The EBO chapter contains an installation note, which looks out-of-place and would have been better served in an appendix.

Third, the conversational tone is great for being able to “talk through” complex concepts, but it can be misconstrued easily. For example, the Foundation Pack chapter states that canonicals are the best practice for service virtualization (not always true). In another example, the author states that in a typical integration scenario the source and target “should be” mapped directly from object to object. What I believe the author meant is that it “usually is” or “typically”, rather than “should be”.

Fourth, there are a lot of code snippets, but the book lacks concrete examples that could have been used throughout the book to highlight the concepts. Particularly, PIPs are a best practice implementation of the Foundation Pack and I would have liked to see a lot more PIP examples. Also, I could not find reference to any accompanying sample code, JDeveloper projects and/or VMs that readers can download and follow through.

Fifth, some concepts have limited or no coverage. For example, there is passing mention of the Project Lifecycle Workbench (PLW) and no mention  of developer tools and utilities such as XSL Mapping Analyzer (XMAN) and PIP Auditor. Information on the latter can be found in my earlier blog posts. Also, one of the key tenets of AIA is SOA governance and I would have preferred more in-depth coverage on its practical implementation.

So, is this book better than the product documentation? No. It’s different. It does mimic the product documentation structure and brings together selective content from multiple guides, but it is still far from a one-stop resource on AIA. Having said that, lack of skilled SOA and AIA resources increases implementation risk, so any book that spreads practical AIA knowledge is always worth a read. The ideal audience for the book would be project managers and architects. Integration developers would be served better with detailed product documentation, hands-on training and blogs to stay current with implementation best practices.

5 Performance Testing Considerations for Application Integrations

Image via Flickr

Enterprise integrations are complex, both functionally, due to implementation of a business process; and technically, due to introduction of one or more runtime layers between applications. Since these integrations typically represent end-to-end business flows, developers need to ensure that the performance meets the business need.

Here are some considerations when planning for performance testing of service oriented architecture (SOA) projects that integrate enterprise applications, such as Oracle’s Application Integration Architecture (AIA).

Update April 21, 2011: AIA specific tuning details can be found in Chapter 28 of the Developer’s Guide for AIA 11gR1 (E17364-02).

1. Define the End Goal. Clearly.

It may sound obvious, but it is the main cause of performance testing efforts going awry – lack of a clear end goal.

Note: “make it run faster” does not count as a clear goal!

Quantify desired metrics in an objective manner by setting Key Performance Indicators (KPI). Here are some KPIs you may want to check for:

• Throughput of the end-to-end business flow by users, payload size, volume
• Response Time for the end-to-end business flow by users, payload size, volume
• Throughput of integration layer only (legacy application interactions stubbed out)
• Response of integration layer only (legacy application interactions stubbed out)

2. Use Metrics Relevant to the Business

System performance KPI should be derived from business metrics so that it involves both business and IT. This results in a more realistic goal than arbitrary benchmarks set by developers or vendors. For example, the throughput KPI could be derived based on a formula that uses software cost and peak order volume to result in a “minimum orders per CPU core per minute” indicator that satisfies the business needs.

When looking at transactions, always consider “peak” spikes vs the average. For example, orders coming in usually have peak periods (e.g. holiday season sales), wherein the system will be subject to transaction load that is a magnitude higher than on non-peak times. Defining KPIs based on peak transaction volumes will not only help in setting realistic goals, but ensures true success of the project when it actually handles the load when it is most needed by the business.

Finally, don’t try to boil the ocean – identify a subset of the integration use cases which are prone to performance bottlenecks and meet all the KPIs before attempting other ones.

3. Do you REALLY Need Production Grade Hardware for Testing?

Using dedicated hardware is always better than sharing existing development or QA environments. However, every business has different needs with their enterprise applications and even this changes by business process. For example, an order-to-cash process may have a need for consistently high target performance metrics with medium-high load; as compared to the financial close process, which may need it once every quarter with high load.

Instead of buying or configuring hardware that necessarily matches every possible target scenario, consider the use of commodity hardware with matching “normalized” KPIs that are downsized from the target business scenario. For example, say the production hardware uses a given compute unit (CPU/memory/cache specification); and the commodity hardware is determined to be one-fourth the compute unit. If the business KPI target is 40 orders/CPU core/minute on the production grade hardware, then the internal, normalized KPI would be one-fourth of that i.e. performance testing would need to achieve 10 orders/CPU core/minute on the commodity hardware to be considered successful.

Of course, the benchmark may not scale as linearly, but this can be easily factored into the equation, providing a good educated estimate of the integration performance. Compared to the alternative of not testing due to hardware unavailability and discovering issues in production, use of commodity hardware and normalized KPIs can be a very viable performance testing approach.

4. Choose a Consistent Testing Strategy

For integration scenarios, a bottoms up testing strategy may be useful to consider, i.e. optimize a single use case fully (to reach desired KPIs) before introducing additional artifacts or flows.

Plan on the sequencing of the use cases appropriately, which can save some cycles e.g. between a Query and an Insert use case, the Query may look simpler, but it needs data which can anyway be seeded by the Insert use case, so it may make sense to proceed with Insert first. Also, identify the “data profiles” for the use cases and create representative sample data e.g. B2B orders may have 50-100 lines per order whereas B2C orders may only have 4-5 lines/order.

For each use case, once KPIs are met with for a particular number of users, payload size etc., run longevity tests for at least 24 hours to ensure that the flow does not have memory leaks or other issues. Check the desired metrics e.g. JVM garbage collection, database AWR reports etc. and purge data after each run to ensure consistency between tests.

When the above passes, gradually increase number of users and increase payload on the same use case to identify system limitations when under load. Once the specific use case is optimized to KPI for concurrent users / payload, add new flows to the mix and tune.

While the above may again seem obvious, the temptation to “switch gears” when one use case is not fully working can cause a lot of overhead in switching context by the project teams and setting up data for the new use case etc. It is better to complete one full use case successfully before targeting others.

5. What about Standalone Testing for Integrations?

Standalone testing – stubbing out enterprise applications – is useful strategy to identify integration hotspots and remove the unknowns of the enterprise application performance from the integration scenario. However, be aware that it will not identify all performance issues. Developing stubs requires substantial investment to emulate the edge applications and may be non-trivial for enterprise applications that typically have complex setups. Furthermore, some integration settings on the SOA server will typically change when the applications are introduced, so avoid over-tuning the solution when performing standalone integration testing.

Performance testing and tuning is still somewhat of an art that requires a good understanding of the technologies, its limitations, and all the available tuning “knobs” in each technology to achieve the KPI requirements of the integration flow. At the same time, the non-technical, project related aspects of the testing exercise is also essential to the success of the initiative as a whole.

Oracle OpenWorld 2010: Practical Insight on Using AIA Session

Session S316877 – Practical Insight on using Application Integration Architecture
at Moscone West L3, Room 3011
on Thursday, September 23 at 1:30pm

The session was quite well attended for a Thursday afternoon (and final day of OpenWorld) with almost 100 attendees. I shared some of the best practices we have learned from our customer implementation experiences, focusing on three areas:

1. REUSE – if you are not reusing services, you might as well not use SOA
2. BUILD NEW – using a holistic approach to SOA with governance utilities
3. PLAN ROLLOUT – to ensure success in your SOA implementation

Here is the webinar of that session.

You can consider this session as a follow-up to the Oracle OpenWorld 2009 AIA Best Practices session.

Oracle OpenWorld 2010: Must-see AIA Sessions

This year’s OpenWorld should be one of the biggest, so to ensure you don’t miss your favorite AIA sessions, here is the Focus on AIA session guide (pdf).

Also, check out the 5 must-see AIA sessions, which includes my session “S316877 – Practical Insight on using Application Integration Architecture” (yes, we do have to spell out AIA in full, but that’s another story). That’s at Moscone West L3, Room 3011 on Thursday, September 23 at 1:30pm. I will be presenting on the usage of various AIA features and tools in a typical implementation, so if there is anything specific you would like to hear about, drop me a comment below.

See you at OpenWorld!

Correction Sep 14, 2010: There are six sessions, not five listed on the page.