Category: Technical

Configuring Weblogic For Restart After Reboot on Windows

I have been working on a customer project involving new installations of Oracle Database (a two-node failover cluster using Oracle Clusterware–good stuff) and two load-balanced BEA Oracle Weblogic servers for the middle tier. The middle tier environment runs on Windows Server 2003 Enterprise x64 Edition and is managed by an outsourced hosting facility. I had … Continue reading “Configuring Weblogic For Restart After Reboot on Windows”

I have been working on a customer project involving new installations of Oracle Database (a two-node failover cluster using Oracle Clusterware–good stuff) and two load-balanced BEA Oracle Weblogic servers for the middle tier. The middle tier environment runs on Windows Server 2003 Enterprise x64 Edition and is managed by an outsourced hosting facility.

I had worked with Weblogic before this project, but not on Windows and I was a little surprised by the difficulty finding relatively simple information on BEA’s support site as well as in their documentation.

<rant>I have to say that I’m pleased that Oracle closed this deal so that (I never thought I’d say this) they can make BEA’s support as good as Oracle Support and Metalink. I know it will take quite a while–probably 6-12 months–but it absolutely needs to happen. It’s little reminders like these that make me glad I work with Oracle (or the companies they ultimately buy) and not other software. I’m as guilty as anyone of not knowing how good we Oracle professionals have it when it comes to good, well-organized documentation, a solid support site with a killer knowledge repository (with an awesome search), and a warm, active community of users (too many “thanks to” to mention at this point).</rant>

Back to the reason you’re reading: this post will summarize what turns out to be the relatively easy changes required to ensure that your Windows-based Weblogic managed servers will start up at boot time.

When creating a new Weblogic Domain on Windows, you’ll sometimes have the opportunity to choose the Sun JDK or BEA JRockit JDK to use for your domain’s Java engine. Be conscious of which one you choose (for WL 10.0, it’s on the screen where you choose a Development or Production configuration). I chose BEA JRockit for my environment’s servers because on 64-bit Windows, it’s the only one supported. For 32-bit Windows, you have the option to use Sun’s JDK, but then we’d have a mix and more potential for new bugs to pop up due to the JDK differences. There’s a small change in this process when using Sun’s JDK versus BEA JRockit and I’ll note that as well.

To ensure that your Windows-based BEA Weblogic Managed Server(s) start at boot time, follow these steps:

  1. After creating the new domain and the new Managed Server (Managed Server is a Weblogic term to identify the differences between an Admin Server and the application server where applications should be deployed), modify the Managed Server settings.
  2. Lock & Edit the configuration and proceed to Environment > Servers > (name of your Managed Server) > Configuration tab > Server Start subtab. Then scroll down to the Arguments box and enter -Xnohup in the box. If you’re using Sun’s JDK instead of BEA JRockit, enter -Xrs in that box instead. This is documented at http://edocs.bea.com/wls/docs100/server_start/nodemgr.html#wp1101004.
  3. Once that’s in place, you can Activate Changes.
  4. The admin server says that no restarts are needed, but I would restart it just to make sure. I’m not sure how the admin server can change a JDK flag without restarting the application server. Maybe I’m just not knowledgeable enough to know how it works, but I think it’s just not smart enough to know that you *do* have to restart.
  5. The last change was the one that I missed initially. Briefly, the reason I missed it was that it has to do with crash recovery and in my opinion, a server reboot shouldn’t cause an application server to crash, so I ignored this part of the documentation. One has to ask why you wouldn’t want crash recovery enabled by default anyway, but that’s probably for another rant some other day. Anyway, the final change is to modify a property for the node manager process. Edit BEA_HOME\wlserver_10.0\common\nodemanager\nodemanager.properties and set CrashRecoveryEnabled=true (it is in the file set to false by default). Save and exit the editor.
  6. This is Windows, so go to the Services control panel and restart the BEA Products NodeManager service to put the change into effect.
  7. Ensure that your managed server is up and running. Then, test the changes you’ve made by rebooting the node and see that your managed server restarts after the reboot is complete (and you’ve given time for node manager to start the managed server).

If you have problems, check the node manager logfile (BEA_HOME\wlserver_10.0\common\nodemanager\nodemanager.log) as it will be most useful in determining what happened. If you don’t see any hint that it even tried to restart the server after the reboot, then it’s probably because the crash recovery setting is not enabled–make sure you changed the right thing in the right file.

I didn’t test to see if this process will restart the admin server as well, but I think it probably will or at least should. With a production configuration, you have to enter the username/password for the admin server when starting it, so you may have to store that in the admin server configuration, but that should be a relatively easy fix. In my case, we didn’t want our admin server running all the time and only start it when needed, so having it start after a reboot wasn’t necessary or desired.

PIPELINED PL/SQL function performance

I recently had an opportunity to do a deeper dive into a performance issue related to the performance of a PIPELINED function. The results weren’t quite what I had expected, so I thought they may be of general interest. As with many performance issues, the issues encountered here are likely data- and database-specific. This was … Continue reading “PIPELINED PL/SQL function performance”

I recently had an opportunity to do a deeper dive into a performance issue related to the performance of a PIPELINED function. The results weren’t quite what I had expected, so I thought they may be of general interest. As with many performance issues, the issues encountered here are likely data- and database-specific. This was testing performed on 10.2.0.3 64-bit non-RAC on Solaris 9.

The PIPELINED function was called like this:

select * from TABLE(pkg1.f1(arg1,arg2,arg3));

And pkg1 is defined like this:

CREATE OR REPLACE PACKAGE pkg1 AS
TYPE coll_table IS TABLE OF custom_persistent_obj_type;
FUNCTION f1 (
p1        IN   NUMBER,
p2             IN   NUMBER,
p3        IN   NUMBER
) RETURN coll_table PIPELINED;
END;
/

The function pkg1.f1 returns a collection of object types, is a little more than 1400 lines long, and contains more than 10 distinct, separate queries, each of them significantly complex involving at least 2 tables, many of them with multiple levels of subqueries. Anyhoo, my chore was to open the idea bin and see if there were some possible enhancements that hadn’t been considered yet.

While reviewing overall performance metrics for the instance, I noted that the DBMS_PICKLER package appeared in the top 10 resource-intensive SQL statements on a regular basis. More digging identified this package as being the primary workhorse behind the TABLE function. So, then the test was clear. Let’s see what happens if we remove PIPELINED, remove TABLE and call the function in a more conventional way. So, my version of the pkg.f would create a collection, add records to it throughout the function’s execution, then return the whole collection at the end with a RETURN statement. The calling function would then need to process this collection into some form to make it more usable by the calling application (similar to the functionality that TABLE provides). For this task, I wrote a stand alone function called DAN.

The new package, pkg2, is defined like this:

CREATE OR REPLACE PACKAGE pkg2 AS
  TYPE coll_table IS TABLE OF custom_persistent_obj_type;
  FUNCTION f2 (
    p1        IN   NUMBER,
    p2             IN   NUMBER,
    p3        IN   NUMBER
  )  RETURN coll_table;
END;
/

DAN is defined as:

CREATE OR REPLACE FUNCTION dan (p_b_tab  IN  pkg2.coll_table,
    p_delimiter     IN  VARCHAR2 DEFAULT ',')
  RETURN VARCHAR2 IS
    l_string     VARCHAR2(32767);
  BEGIN
    for i in p_b_tab.FIRST .. p_b_tab.LAST LOOP
      l_string := l_string || to_char(p_b_tab(i).id1)  ||p_delimiter
                           ||         p_b_tab(i).id2   ||p_delimiter
                           || to_char(p_b_tab(i).array_id)   ||p_delimiter
                           ||         p_b_tab(i).value
      ;
    END LOOP;
  RETURN l_string;
END dan;
/

So, my new units would be called like this:

DECLARE
  l_string      VARCHAR2(32767);
BEGIN
  l_string := dan(pkg2.f2(arg1,arg2,arg3));
  DBMS_OUTPUT.PUT_LINE(l_string);
END;
/

Of course, performance is our primary objective in this case, so timing these two examples side-by-side is the primary chore. I was satisfied that they performed equivalent work and provided equivalent results using different methods. The calling program (custom java, in this case) would have to be modified to handle the changes required, but that may be worth it depending on the time difference.

So, I devised a script to do some very rough testing. The script looked like this:

set timing on echo on termout on pages 0 lines 450 feedb on serveroutput on size 100000
select * from table(pkg1.f1(1095482041,60983,1));
DECLARE
  l_string      VARCHAR2(32767);
BEGIN
  l_string := dan(pkg2.f2(1095482041,60983,1));
  DBMS_OUTPUT.PUT_LINE(l_string);
END;
/

The results were somewhat surprising. Keep in mind that all I’m changing is the way in which data is returned and processed from the function. The data returned, though in slightly different formats, is exactly the same and is not filtered further after it is returned (so there’s no benefit in having it available to run SQL against it).

SELECT (sec) PL/SQL (sec) Difference (sec) Percent Improvement
0.15 0.09 0.06 40.00%
0.14 0.09 0.05 35.71%
0.14 0.10 0.04 28.57%
0.16 0.09 0.07 43.75%
0.15 0.09 0.06 40.00%
0.14 0.12 0.02 14.29%
0.18 0.12 0.06 33.33%
0.18 0.10 0.08 44.44%
0.25 0.12 0.13 52.00%
0.14 0.10 0.04 28.57%
0.18 0.10 0.08 44.44%
0.15 0.10 0.05 33.33%
0.17 0.10 0.07 41.18%
0.18 0.10 0.08 44.44%
0.14 0.10 0.04 28.57%
0.17 0.11 0.06 35.29%
0.17 0.10 0.07 41.18%
0.16 0.10 0.06 37.01% Average
0.14 0.09 0.02 14.29% Min
0.25 0.12 0.13 52.00% Max
8.84% Standard Deviation

As you can see, the results are pretty clear that there is a performance improvement in this case when we avoid using PIPELINED functions. I will also say that the result set from the function body is only about 30 rows. From my review of common uses for PIPELINED functions, it appears that larger collections (result sets) may benefit more from PIPELINED functions as they can likely take more advantage of the parallelism allowed with PIPELINED functions.

2:30am will never happen on Sunday morning

For those that didn’t bother to install the OS, Oracle or other DST patches (to handle the adjusted DST change schedule) last year and elected to just change the time on your servers by hand twice a year, get ready to do that again this weekend. Personally, I’d rather burn two whole regular work days … Continue reading “2:30am will never happen on Sunday morning”

For those that didn’t bother to install the OS, Oracle or other DST patches (to handle the adjusted DST change schedule) last year and elected to just change the time on your servers by hand twice a year, get ready to do that again this weekend. Personally, I’d rather burn two whole regular work days figuring out and installing the patches than get up to modify the time on a server at 2am on Sunday morning, but that’s just me I guess.

For those that forgot the crontab rule about not scheduling things to occur between 1am and 3am (because they either get skipped or run twice once a year), prepare to be confused as 2:30am doesn’t ever happen on clocks in most US states/regions (here are the exceptions).

I’ll look forward to a little more daylight during the hours that count and one hour less sleep on Saturday night.

On a side note, let’s hope that this little posting will get me started on regular posting here…at least for a while!

Cool thing happened on Twitter today…

A neat thing happened today on Twitter. While I admit that I don’t necessarily “get it” as fast as some of my “web 2.0” friends do, I haven’t seen this happen too much on Twitter since I’ve been following it in the past several months. I’m sure it probably happens all the time to cool … Continue reading “Cool thing happened on Twitter today…”

A neat thing happened today on Twitter. While I admit that I don’t necessarily “get it” as fast as some of my “web 2.0” friends do, I haven’t seen this happen too much on Twitter since I’ve been following it in the past several months. I’m sure it probably happens all the time to cool people, but I was lucky enough to cross over for a few minutes and that’s notable.

Basically, the “thing” was that someone needed help understanding how to get started with an OID installation for managing TNS connect descriptors. He wanted (and needed) to use an existing database since he was resource-constrained and wasn’t sure what the installation process looks like for such an installation.

Here’s the combined thread between @fuadar, @topperge and me (@dannorris) just a few minutes ago:

fuadar: looking for someone or some document to install oid in an existing 10.2 database need only names service resolution
dannorris: @fudar It’s much easier to just have it install its own DB. If you use existing DB, you must run metadata repos creation asst first.
fuadar: @dnanorris out of space already have a database out there for other functions. trying to setup oid to solve our tnsnames issues
dannorris: @fudar Issues? Honestly, OID usually introduces more issues than it solves when it comes to TNS. It’s a lot more complex than a text file.
fuadar: @dannorris true but i’m trying to come up with some way to manage acouple of hundred servers and a couple of thousand clients
dannorris: @fudar It’s definitely the right direction to head–just need realistic expectations about complexity and manageability–not easier!
fuadar: @dannorris agree just looking for better documentation
topperge: @fuadar fudar, all you need is RepCA and install the identity repos, http://tinyurl.com/yweyr8
dannorris: @fuadar Better free up some space first–you’ll need a gig or two I’d expect. (ps sorry for misspelling your handle)
fuadar: @topperge so what you are saying is just go thru the oid software install process and then so the repca manually
fuadar: @topperge i am using the Oracle Identity management dvd’s 10.1.4.0.1
dannorris: @fuadar Install RepCA first, run it, then install OID from IdM and tell it to use the repos you created.
dannorris: @fuadar be sure to check DB prereqs (version, pkgs, options, etc.). Follow section here http://snurl.com/1xzda
fuadar: @dannorris thanks reinstalling the software now
topperge: @fuadar There is a 10.1.4 MRCA with the DVDs, install from that first , then install from the OIM Infrastructure CD second
topperge: @fuadar then make sure you patch to 10.1.4.2 which is patch 5983637 on metalink (doing the same install right now)

Even patch numbers! Posting that same question to a forum would likely have taken several hours to get responses–and precise responses as well. Now, I don’t want everyone to believe that @topperge (Matt Topper) and I sit around all day looking for questions we can answer on Twitter. However, I am on Twitter most of the time (even though I don’t tweet that often) and occasionally will throw a response or post in when I think of it. Matt is usually there and seems to behave similarly most of the time.

The bottom line: today, Twitter helped someone solve a real technical problem much faster than they were likely to solve it via other means (web 2 dot oh or otherwise). I don’t know that it happens every day, but we can only save one life at a time :).

You can follow me (@dannorris) on twitter, but as I don’t say much, you won’t likely be impressed. After all, I’m no Jake Kuramoto.

On handling logging in a script

This tip isn’t really new, but it is one of those things that I used to know, forgot, remembered, and am now blogging so that it will be easier to find (for me) when I finally forget it again. Besides, I was hanging out with Bex Huff earlier tonight and he told me I need … Continue reading “On handling logging in a script”

This tip isn’t really new, but it is one of those things that I used to know, forgot, remembered, and am now blogging so that it will be easier to find (for me) when I finally forget it again. Besides, I was hanging out with Bex Huff earlier tonight and he told me I need to blog more and this was what popped into my head. On a side note, I would encourage everyone to seek out someone that you “know” from this internet thang when you’re traveling. Meeting in person certainly beats any “8 things” list :). I happen to be in Minneapolis this month, so I looked up Bex and also hope to find a bar or similar establishment where Billy can don the kilt 2.0 for another picture (that may be asking a lot in 31-degree temps, though).

Enough soapbox…on with the tech talk. Continue reading “On handling logging in a script”

Interesting Metalink findings

I generally don’t spend a lot of time surfing around Metalink. Normally, I get in, find the bug or patch or whatever thing I need, and get out. However, my current project involves a database upgrade for a very performance-sensitive application (they have an SLA that they actually have to honor–or honour for my UK … Continue reading “Interesting Metalink findings”

I generally don’t spend a lot of time surfing around Metalink. Normally, I get in, find the bug or patch or whatever thing I need, and get out. However, my current project involves a database upgrade for a very performance-sensitive application (they have an SLA that they actually have to honor–or honour for my UK friends :), so I’ve been doing a bit of research. Coincidentally, a posting to Oracle-L recently allowed me to mention one of my research findings and several subscribers (one publicly) there responded that they had never seen the document and that it was great. Well, it really is great and Oracle Support or whomever it is that supplied the content deserves a great round of applause for putting it together.

The document of which I speak is the Oracle 10g Upgrade Companion. This document contains more than just the upgrade steps, but starts with a list of recommended patches, then goes on to include sections on Behavior Changes (this is especially valuable and absent from most other upgrade plans), Best Practices for the upgrade process, and Documentation references. While I am reporting that this is a great resource, I have two general suggestions for improvement Continue reading “Interesting Metalink findings”

PortableApps.com

One of my peers introduced me to the website www.portableapps.com a week or two ago. This site takes popular applications, modifies them to avoid requiring a typical installation so that you don’t need to have administrative privileges on the system to run them. This allows those of us (especially consultants) that work on others’ systems … Continue reading “PortableApps.com”

One of my peers introduced me to the website www.portableapps.com a week or two ago. This site takes popular applications, modifies them to avoid requiring a typical installation so that you don’t need to have administrative privileges on the system to run them. This allows those of us (especially consultants) that work on others’ systems to be able to use some of our favorite tools without requesting special privileges.

I’d encourage you to check them out if you’d like to be able to put your favorite utility or web browser on a USB drive, plug it in to a computer you’ve never used before and start using the utility without delay. It can provide a great deal of privacy protection as well since you don’t leave behind your browser cache, bookmarks, or site visit history.

If you’re looking for a tool to access the internet, archive some files, transfer files, read MS Word documents or many others, grab the Portable Apps utility of your choice and avoid the Windows-isms. They’ve got a great inventory of products already and they seem to be actively updating the existing tools as well as adding new ones.

Using mod_rewrite to rewrite OC4J-served URLs – a complete review

We recently ran into an issue in a customer configuration where rewriting URLs using pass-through didn’t function as expected with OC4J-deployed applications. As it turned out, there’s a bug in the OC4J container and a relatively easy workaround for some. The situation was this (names changed to protect the innocent): An existing Java application deployment … Continue reading “Using mod_rewrite to rewrite OC4J-served URLs – a complete review”

We recently ran into an issue in a customer configuration where rewriting URLs using pass-through didn’t function as expected with OC4J-deployed applications. As it turned out, there’s a bug in the OC4J container and a relatively easy workaround for some.

The situation was this (names changed to protect the innocent):

  • An existing Java application deployment existed using JRun on Solaris. In that deployment, an application called “abc” would be called as “http://abcapp.corp.com/servlet/login”
  • Applications were to be migrated to Oracle Application Server 10.1.3.1.0.
  • Deployments on OAS were required to prefix the application with something and they used the application name. So, on the new site, the application would need to be called as “http://abcapp.corp.com/abc/servlet/login”. This was undesirable since bookmarks would have to be updated. While it could easily be handled with redirections, the desired behavior was to have all URLs match what they were on the old deployment.

On the surface, this seems like a relatively simple problem to solve using a RewriteRule with the [PT] option and few RewriteConds in the Apache configuration. That is, until you find the bug in OC4J that makes it impossible. First, let’s review the configuration parameters. Continue reading “Using mod_rewrite to rewrite OC4J-served URLs – a complete review”

Another “special” circumstance when running OAS on Windows

We encountered an “interesting” challenge recently where some, not all, OC4J containers in an Oracle Application Server 10.1.3.1.0 installation would “crash” (they would stop running). There was no apparent pattern to the “crazy” crashing containers. The system administrator was actively doing application (re)deployments at the rate of 3-4 per week. The containers seemed to be … Continue reading “Another “special” circumstance when running OAS on Windows”

We encountered an “interesting” challenge recently where some, not all, OC4J containers in an Oracle Application Server 10.1.3.1.0 installation would “crash” (they would stop running). There was no apparent pattern to the “crazy” crashing containers. The system administrator was actively doing application (re)deployments at the rate of 3-4 per week. The containers seemed to be “crashing” randomly, sometimes throughout the day, sometimes just after a deployment.

We increased many timeouts for OPMN as we believed that OPMN was just incorrectly “seeing” the containers as down and restarting them. OPMN restarts them by shutting them down first and then starting them.

We filed cases with Oracle support to no avail–they didn’t come up with any useful suggestions in a week or more. They were trying, but didn’t come up with the solution.

The system administrator developed a theory based on what he believed was a pattern. Every time he did a deployment, he would notice a crash of all the non-Oracle default containers. That is, the home and OC4J_WebCenter containers didn’t crash.

The deployment process that he followed resulted in him connecting to the server using remote desktop. His remote desktop client was configured with the /console option which was required by some other servers he managed, more about that later.

Once he was able to demonstrate that he could make the containers crash each time he logged off, we started testing variations using the system console, the remote desktop client with and without the /console option and found a pattern. The remote desktop client without the /console option did not cause a crash, but all other combinations did. Through all of this, the home and OC4J_WebCenter containers remained up and running.

Bottom line: Read Metalink Note 245609.1 which documents the apparently, well-known fact that logging out from the Windows console causes JVM termination. The very simple fix is to start the containers with the “-Xrs” option which tells the JVM to ignore certain signals from the OS.

The really terrible thing about all this is that Oracle puts the -Xrs option on the containers deployed during the installation, but the OEM tool doesn’t add them to the container startup parameters for the custom containers. Easy to fix, even easy to find once you know what to look for.

This begs two questions:

  1. Why doesn’t Oracle add -Xrs to the startup options for the containers created after the initial installation? That would have avoided all the problems and there’s apparently no negative side effect–at least not that we’ve seen.
  2. How could an SR analyst not find this Metalink note and refer us to the simple solution? Granted, we didn’t find it easily in our searches either, but eventually it was one of us that found the article and solution. Now that we know the fix, a simple search for -Xrs on Metalink gets plenty of hits. As they say, hindsight is 20/20.

Hopefully, this information will help some of you that are lucky enough to work on OC4J deployments on Windows.