I have been working on a customer project involving new installations of Oracle Database (a two-node failover cluster using Oracle Clusterware–good stuff) and two load-balanced BEA Oracle Weblogic servers for the middle tier. The middle tier environment runs on Windows Server 2003 Enterprise x64 Edition and is managed by an outsourced hosting facility.
I had worked with Weblogic before this project, but not on Windows and I was a little surprised by the difficulty finding relatively simple information on BEA’s support site as well as in their documentation.
<rant>I have to say that I’m pleased that Oracle closed this deal so that (I never thought I’d say this) they can make BEA’s support as good as Oracle Support and Metalink. I know it will take quite a while–probably 6-12 months–but it absolutely needs to happen. It’s little reminders like these that make me glad I work with Oracle (or the companies they ultimately buy) and not other software. I’m as guilty as anyone of not knowing how good we Oracle professionals have it when it comes to good, well-organized documentation, a solid support site with a killer knowledge repository (with an awesome search), and a warm, active community of users (too many “thanks to” to mention at this point).</rant>
Back to the reason you’re reading: this post will summarize what turns out to be the relatively easy changes required to ensure that your Windows-based Weblogic managed servers will start up at boot time.
When creating a new Weblogic Domain on Windows, you’ll sometimes have the opportunity to choose the Sun JDK or BEA JRockit JDK to use for your domain’s Java engine. Be conscious of which one you choose (for WL 10.0, it’s on the screen where you choose a Development or Production configuration). I chose BEA JRockit for my environment’s servers because on 64-bit Windows, it’s the only one supported. For 32-bit Windows, you have the option to use Sun’s JDK, but then we’d have a mix and more potential for new bugs to pop up due to the JDK differences. There’s a small change in this process when using Sun’s JDK versus BEA JRockit and I’ll note that as well.
To ensure that your Windows-based BEA Weblogic Managed Server(s) start at boot time, follow these steps:
- After creating the new domain and the new Managed Server (Managed Server is a Weblogic term to identify the differences between an Admin Server and the application server where applications should be deployed), modify the Managed Server settings.
- Lock & Edit the configuration and proceed to Environment > Servers > (name of your Managed Server) > Configuration tab > Server Start subtab. Then scroll down to the Arguments box and enter
-Xnohupin the box. If you’re using Sun’s JDK instead of BEA JRockit, enter-Xrsin that box instead. This is documented at http://edocs.bea.com/wls/docs100/server_start/nodemgr.html#wp1101004. - Once that’s in place, you can Activate Changes.
- The admin server says that no restarts are needed, but I would restart it just to make sure. I’m not sure how the admin server can change a JDK flag without restarting the application server. Maybe I’m just not knowledgeable enough to know how it works, but I think it’s just not smart enough to know that you *do* have to restart.
- The last change was the one that I missed initially. Briefly, the reason I missed it was that it has to do with crash recovery and in my opinion, a server reboot shouldn’t cause an application server to crash, so I ignored this part of the documentation. One has to ask why you wouldn’t want crash recovery enabled by default anyway, but that’s probably for another rant some other day. Anyway, the final change is to modify a property for the node manager process. Edit
BEA_HOME\wlserver_10.0\common\nodemanager\nodemanager.propertiesand setCrashRecoveryEnabled=true(it is in the file set to false by default). Save and exit the editor. - This is Windows, so go to the Services control panel and restart the BEA Products NodeManager service to put the change into effect.
- Ensure that your managed server is up and running. Then, test the changes you’ve made by rebooting the node and see that your managed server restarts after the reboot is complete (and you’ve given time for node manager to start the managed server).
If you have problems, check the node manager logfile (BEA_HOME\wlserver_10.0\common\nodemanager\nodemanager.log) as it will be most useful in determining what happened. If you don’t see any hint that it even tried to restart the server after the reboot, then it’s probably because the crash recovery setting is not enabled–make sure you changed the right thing in the right file.
I didn’t test to see if this process will restart the admin server as well, but I think it probably will or at least should. With a production configuration, you have to enter the username/password for the admin server when starting it, so you may have to store that in the admin server configuration, but that should be a relatively easy fix. In my case, we didn’t want our admin server running all the time and only start it when needed, so having it start after a reboot wasn’t necessary or desired.
Hey Dan,
Thanks a lot for a detailed post. It worked for me But i have a question.
After i rebooted my server the managed server instance state was:Running but its health was Not reachable!!!!
Any idea how to fix this and what would be the error!
I presume that the normal state before reboot was that the health was fine and everything working well. If you're not sure, confirm that first. Assuming that I was right, then it seems at least possible that you have some firewall or other networking issue that may have been introduced by the reboot? That's just one idea.
The logfiles I mentioned in the post may provide some additional information about why the server is unreachable, so it'd also be good to see what specific error messages are in them.
Hey,
Thanks for prompt reply!!
I do not see any thing significant in the nodemanager and msi logs. In the console the health of that msi is blank, the state is running whereas admin servers health seems to be ok and state is running.
Here is what my nodemanager.log says:
<Aug 14, 2008 3:49:08 PM> <INFO> <Secure socket listener started on port 5556>
<Aug 15, 2008 8:56:36 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 8:57:14 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 8:57:42 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 8:57:59 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:36:42 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:38:51 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:39:25 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:40:01 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:40:14 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:40:31 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:40:50 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:40:55 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:41:13 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:43:10 AM> <INFO> <Loading domains file: D:BeaWL10weblogiccommonnodemanagernodemanager.domains>
<Aug 15, 2008 9:43:11 AM> <INFO> <Loading identity key store: FileName=D:/BeaWL10/weblogic/serverlibDemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<Aug 15, 2008 9:43:11 AM> <INFO> <Loaded node manager configuration properties from 'D:BeaWL10weblogiccommonnodemanagernodemanager.properties'>
<Aug 15, 2008 9:43:12 AM> <Info> <ECMS> <DRS> <Startup configuration properties loaded from “D:BeaWL10user_projectsdomainsECMSserversDRSdatanodemanagerstartup.properties”>
<Aug 15, 2008 9:43:12 AM> <Info> <ECMS> <DRS> <Resume monitoring of active server process (pid = 3380)>
<Aug 15, 2008 9:43:12 AM> <Info> <ECMS> <DRS> < Initializing ServerMonitor for weblogic.nodemanager.server.ServerManager@2d073c : with config : weblogic.nodemanager.common.StartupConfig@2d0f6d>
<Aug 15, 2008 9:43:12 AM> <Info> <WEBTOP> <webtop_msi> <Startup configuration properties loaded from “D:BeaWL10user_projectsdomainsWEBTOPserverswebtop_msidatanodemanagerstartup.properties”>
<Aug 15, 2008 9:43:12 AM> <Info> <WEBTOP> <webtop_msi> <Resume monitoring of active server process (pid = 3392)>
<Aug 15, 2008 9:43:12 AM> <Info> <WEBTOP> <webtop_msi> < Initializing ServerMonitor for weblogic.nodemanager.server.ServerManager@2dad65 : with config : weblogic.nodemanager.common.StartupConfig@2db884>
<Aug 15, 2008 9:43:14 AM> <INFO> <Secure socket listener started on port 5556>
<Aug 15, 2008 9:45:32 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
Any idea?
I think you'll have to change the logging level to get more details. Looks like you're using the INFO level now. Crank it up to DEBUG and you'll see lots of noise, but hopefully something there will indicate some sort of issue you can use to troubleshoot further. There's nothing in this output that indicates what the problem may be.
Hey,
Thanks for prompt reply!!
I do not see any thing significant in the nodemanager and msi logs. In the console the health of that msi is blank, the state is running whereas admin servers health seems to be ok and state is running.
Here is what my nodemanager.log says:
<Aug 14, 2008 3:49:08 PM> <INFO> <Secure socket listener started on port 5556>
<Aug 15, 2008 8:56:36 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 8:57:14 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 8:57:42 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 8:57:59 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:36:42 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:38:51 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:39:25 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:40:01 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:40:14 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:40:31 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:40:50 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:40:55 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:41:13 AM> <INFO> <WEBTOP> <States = {webtop_msi=RUNNING, WEBTOP_ADMIN=UNKNOWN, domain_bak=UNKNOWN}>
<Aug 15, 2008 9:43:10 AM> <INFO> <Loading domains file: D:BeaWL10weblogiccommonnodemanagernodemanager.domains>
<Aug 15, 2008 9:43:11 AM> <INFO> <Loading identity key store: FileName=D:/BeaWL10/weblogic/serverlibDemoIdentity.jks, Type=jks, PassPhraseUsed=true>
<Aug 15, 2008 9:43:11 AM> <INFO> <Loaded node manager configuration properties from 'D:BeaWL10weblogiccommonnodemanagernodemanager.properties'>
<Aug 15, 2008 9:43:12 AM> <Info> <ECMS> <DRS> <Startup configuration properties loaded from “D:BeaWL10user_projectsdomainsECMSserversDRSdatanodemanagerstartup.properties”>
<Aug 15, 2008 9:43:12 AM> <Info> <ECMS> <DRS> <Resume monitoring of active server process (pid = 3380)>
<Aug 15, 2008 9:43:12 AM> <Info> <ECMS> <DRS> < Initializing ServerMonitor for weblogic.nodemanager.server.ServerManager@2d073c : with config : weblogic.nodemanager.common.StartupConfig@2d0f6d>
<Aug 15, 2008 9:43:12 AM> <Info> <WEBTOP> <webtop_msi> <Startup configuration properties loaded from “D:BeaWL10user_projectsdomainsWEBTOPserverswebtop_msidatanodemanagerstartup.properties”>
<Aug 15, 2008 9:43:12 AM> <Info> <WEBTOP> <webtop_msi> <Resume monitoring of active server process (pid = 3392)>
<Aug 15, 2008 9:43:12 AM> <Info> <WEBTOP> <webtop_msi> < Initializing ServerMonitor for weblogic.nodemanager.server.ServerManager@2dad65 : with config : weblogic.nodemanager.common.StartupConfig@2db884>
<Aug 15, 2008 9:43:14 AM> <INFO> <Secure socket listener started on port 5556>
<Aug 15, 2008 9:45:32 AM> <INFO> <ECMS> <States = {ECMS_ADMIN=UNKNOWN, DRS=RUNNING, domain_bak=UNKNOWN}>
Any idea?
I think you'll have to change the logging level to get more details. Looks like you're using the INFO level now. Crank it up to DEBUG and you'll see lots of noise, but hopefully something there will indicate some sort of issue you can use to troubleshoot further. There's nothing in this output that indicates what the problem may be.