Just a quick one during my lunch hour …. ran into an issue yesterday at my current client that shows once more that when you do not work with a specific OS for a while, you really loose your touch for the small details.

The Saga of WAS, AIX and the damn Java Cache

We installed iFixes yesterday and that all went well. However the syncing of the nodes (kicked off from the Dmgr console) took forever, and then one of the app clusters on one of the nodes would not restart (it eventually did after 4 hours).

To clean the system and get rid of any old temp files we:

• Stopped all WebSphere servers:  /WAS_Profile/bin/stopServer.sh xxx -user xxx -password ****
• Stopped the NodeAgent:  /WAS_Profile/bin/stopNode.sh -username xxxx -password ****
• Cleaned all temp files /WAS_Profile/temp  and /wstemp (everything inside of both folders)
• Ran  /WAS_Profile/bin/osgiCfgInit.sh
• Ran  /WAS_Profile/bin/clearClassCache.sh

Note: you can also use the command “./stopNode.sh -stopservers -username xxxx -password ****” to shut down the node agent AND the servers at the same time. We wanted to see the individual servers come down as we had issues with one of them.

We then tried to restart the node agent ….. and it failed. We found this in the startserver.log for the node agent:

ADMU3011E: Server launched but failed initialization

Damn, nothing worked … re-cleaned, checked, cursed, cried ….. and then opened a Sev 1 ticket with IBM support online. (had a REALLY fast response – thanks guys!)

The Cavalry to the Rescue …

The Connections support guy had a look at the logs and brought in a WAS support specialist who had me repeat the clean-ups steps above AND clean this location as well (everything in this folder, but not delete the folder itself):

/tmp/javasharedresources

The IBM tech thinks we had a corrupted system level java cache that was causing the issue.  After that a ./startNode.sh worked like a charm and the servers started fine as well.

Total Clean-up

Incidentally, we ended up shutting each AIX WAS server (including the Dmgr) down one by one so we do not have a service outage and ran the above maintenance once more. On the nodes we also ran a “./synchNode.sh” with the node agent turned off – just to eliminate any possibility of the nodes maybe being out of synch (thanks for the idea Stuart).

We will also be going through our automated scripts to test adding some more items to them (email notifications when individual steps are done, add the “/temp/javasharedresources” to the list of folders to be cleaned,  etc.).

Lessons to be learned:

• When you don’t work with an OS for a while you forget the important SMALL stuff (/tmp/javasharedresources) – I had run into this very issue a few years ago and totally forgot about it. I actually did not remember it until this morning, the day after.
• When in doubt – call support RIGHT AWAY, if for no other reason than to validate your thought process is correct and you are not barking up the wrong tree. We did not wait very long to call, but sometimes even 5 minutes can mean the difference between failure and success.

Recent Blog Posts

35

IBM Open Mic Webcast Invitation: IBM Lotus Notes & Domino Out Of Office Functionality – 22 May 2012 Tue, May 15th 2012 6:12a Victor Toal You are invited to join an Open Mic Webcast on the topic "IBM Lotus Notes & Domino Out Of Office Functionality." This event will be held Tuesday, May 22nd at 11:00 AM EDT (15:00 UTC, or GMT -4), for 60 minutes. After a short presentation, you will have the opportunity to ask questions directly of IBM Lotus Developers and Support Engineers. See more details below. (Note: To view this information online, go to http://www.ibm.com/support/docview.wss?uid=swg27024609) When Date: 22 May 2012 [read] Keywords: collaboration domino ibm ldd lotus notes facebook office twitter

67

IBM Open Mic Webcast invitation: Troubleshooting Notes/Domino NRPC Mail Routing issues – 9 May 2012 Tue, May 1st 2012 6:12a Victor Toal You are invited to join an Open Mic Webcast on the topic "Troubleshooting Notes/domino NRPC Mail Routing issues". After a short presentation, you will have the opportunity to ask questions directly of IBM Lotus Developers and Support Engineers. This event will take place on Wednesday, 9 May 2012, at 11:00AM EDT (16:00 UTC, or GMT -5). You can post questions in advance as a response to this entry in the Notes/Domino 8.5 forum. Bookmark Webcast Technote #7023930 Title: Open Mic Webcast: Trou [read] Keywords: collaboration domino ibm ldd lotus notes notesdomino facebook twitter

79

IBM Open Mic Webcast: What’s the Latest with Notes Traveler? – 24 April 2012 Wed, Apr 18th 2012 12:12p Victor Toal I just received this in my in-box – definitely looks like something everybody should put into their calendars: You are invited to join an Open Mic Webcast on the topic, “What’s the Latest with Notes Traveler?” ” After a presentation, you’ll have the opportunity to ask questions directly of IBM developers and support engineers. You can post questions in advance of the call as a response to this entry in the Notes/Domino/Traveler forum. The event will take pl [read] Keywords: collaboration domino ibm ldd lotus notes traveler exchange exchange facebook mobile twitter

89

FLASH: Fix Available: WebSphere Plugin Personal Certificate and password will expire on April 26, 2012 (2012.04.13) Fri, Apr 13th 2012 5:12p Victor Toal 13 Apr 2012 Dear Subscriber (vtoal), Here are your bulletin email notifications for your subscriptions at IBM My notifications. Visit the IBM Electronic Support site to get connected with our powerful online tools, tips, and resources. Lotus Quickr for WebSphere Portal: Flashes • Fix Available: WebSphere Plugin Personal Certificate and password will expire on April 26, 2012 The WebSphere Support team has released two Flash technotes: 1) The personal certificate [read] Keywords: ibm lotus quickr email facebook password profile twitter websphere websphere portal

89

Get Free Training! New Complimentary IBM WebSphere Education Courses Available from the Global WebSphere Community Wed, Apr 11th 2012 9:12a Victor Toal I think everybody following me knows that I am telling everybody to learn WebSphere – it will come your way like it or not. Here are some free training courses/videos that you can look at – all you have to do is become a member of the Global WebSphere Community – and PRESTO – free training for you. I will be looking at this later tonight … Dear WebSpherians,Last year, we polled our Global WebSphere Community members asking you what WebSphere topics you were mos [read] Keywords: ibm application community email facebook linkedin server twitter websphere

60

IBM Open Mic Webcast Invitation – Configuring and Troubleshooting the Resource Reservation database in Lotus Notes – 10 April 2012 Tue, Apr 3rd 2012 2:12p Victor Toal You are invited to join an Open Mic Webcast on the topic “Configuring and Troubleshooting the Resource Reservation database in Lotus Notes.” This event will be held Tuesday, 10 April 2012 at 11:00 AM EDT (15:00 UTC, or GMT -4), for 60 minutes. After a short presentation, you will have the opportunity to ask questions directly of IBM Lotus Developers and Support Engineers. See more details below. (Note: To view this information online, go to http://www.ibm.com/support/docview.wss?uid= [read] Keywords: domino ibm ldd lotus notes database facebook reservation twitter

74

April 5 webcast: The World’s First DFDL Implementation – Modeling Text and Binary Data Using WebSphere Message Broker V8 Tue, Apr 3rd 2012 2:12p Victor Toal Dear WebSpherian, Don’t miss the upcoming webcast: The World’s First DFDL Implementation – Modeling Text and Binary Data Using WebSphere Message Broker V8 Date: April 5, 2012 Title: The World’s First DFDL Implementation – Modeling Text and Binary Data Using WebSphere Message Broker V8 Time: 11 AM EDT (GMT -4) Register Now! DFDL (Data Format Description Language, often pronounced “daffodil”) is a new standard for modeling and parsing all kinds of genera [read] Keywords: ibm community eclipse email facebook linkedin twitter websphere xml

87

Connections 3.0.1.1 and Sametime Proxy Server 8.5.2 IFR1 fix OBEN-8SDLGS – RECALLED Tue, Mar 27th 2012 7:12a Victor Toal I just got a mail from IBM support on the Sametime Proxy Server 8.5.2 IFR1 fix OBEN-8SDLGS that is mandatory for awareness between Connections 3.0.1 FP1 and Sametime. It appears this HF was recalled (not sure what he exact problem is) which explains why I have not been able to get awareness to work with Connections 3.0.1.1 – no mater what I do. That there is no updated, specific documentation for Connections 3.0.1.1 and ST awareness is not a good sign either but that is why I opened a [read] Keywords: connections ibm lotus sametime sametimelotus facebook server twitter

117

Linux Kernel 3.3 Released, will support Android Apps Wed, Mar 21st 2012 1:14p Victor Toal Linux and Android are going to be on friendly terms with each other again – it has been a long time. the article has all the details but to keep it short – you will be able to run Android apps on Linux machines with the 3.3 kernel and expect them to run .. more or less. Again, this is fun … it’s upgrade time again soon! Filed under: Linux Related Tagged: Android, Linux [read] Keywords: facebook google linux twitter

71

End of Support for IBM WebSphere Application Server V6.1 is 30 Sept 2012 Wed, Mar 21st 2012 11:12a Victor Toal This came across my desk the other day – if you are still running WebSphere servers in the 6.1 code stream you will need to upgrade son. I don’t think many of those servers are still out there unless you are still running a few applications that require older versions of WAS to run. This is fun …. it’s upgrade time again!!!! Filed under: Websphere [read] Keywords: ibm application applications facebook server twitter websphere