Posts Tagged ‘AIX 5.3’

How to Clear a BladeCenter LED Amber Bang (!) Light on a JS21 Blade

Wednesday, August 13th, 2008

My IBM JS21 blade (type 8844), which is running AIX 5.3, had several error lights lit.  They were the amber light with an exclamation mark.  I saw no errors in the BladeCenter Advanced Management Module (AMM) event log.  There were no hardware errors when I type in “errpt” in the AIX prompt. Strange.

There were four LED lights lit:

  1. On the physical blade
  2. On the physical BladeCenter chasis
  3. In the BladeCenter AMM GUI > Monitors > LED Media Tray and Rear Panel LEDs > System Error
  4. In the BladeCenter AMM > Monitors > LEDs > Blade LEDs

These error lights meant one of two things, since no errors were showing:

  • They’re just false alarms.

OR

  • There are hidden errors stored somewhere, and we needed to reset various system components to flush the errors out.

It my case it was (1).  But it was a good idea to follow this process just to be certain.

How to Clear the LED Bang Lights

1) Restart the Blade System Management Processor. On the BladeCenter AMM, navigate to Blade Tasks > Power/Restart.  Select your blade and click Restart Blade System Mgmt Processor.  Don’t worry.  Your blade will continue running.

My result: The Blade Center errors (errors 3 and 4, above)

2) Check for errors. BladeCenter AMM > Event Log.

AIX# errpt

My result: I found no errors.

3) Restart Management Module. BladeCenter AMM > MM Control > Restart MM.  This will cause you to lose connection to the BladeCenter.  Your blade servers will continue running.

My result: the physical chasis LED turned off.  Bang light #1, on the physical blade remained.  I had to schedule an outage and clear the NVRAM.

How to Clear NVRAM on a System P Blade Server

1) Schedule an outage.

2) Click Restart Blade and clear NVRAM, if you have this option.  It should be in BladeCenter AMM > Blade Tasks > Power/Restart.  Look on the bottom of the page.  I, unfortunately, didn’t have this option.  I just had Restart Blade.  I restarted my blade for kicks.

My result: No dice.  Still no errors.  Error light 1, on my physical blade remained on.

3) Power off the server. It should be in the Power/Restart screen of your Blade Center.

4) Remove the battery from the server for five minutes. Slide out the blade, open the server, attach an anti-static wristband (if you worry about these things), and remove the nickel-sized battery.  A small flathead screwdriver would be useful.

Warning:  This will reset any custom BIOS settings, return you to the orginal BIOS firmware level, and reset your time clock. If you pretty much use default settings, like me, you’d probably be fine.

5) Open up a console window. If you’re using a JS20 or JS21, you’ll need these instructions.  Otherwise, if you have a graphics card, you can use the BladeCenter Remote Control.

6) Put the server back together and power it up.

7)  Follow the prompt and press 0 to select the console on your console window,.  This is a good indication that the NVRAM has been cleared.

My result as it was booting:  The physical blade LED turned off.  But error lights 2, 3, and 4 all turned back on.  This was like a bad joke.

8) Reset the time when you can log in.

The time command to set it for Aug 13 (0813), 6:52 pm (1852), 2008 (08):

AIX# time 0813185208

9) Wait for everything to come up.  This is a good time to double check the error logs again.

10) Restart the Blade System Management Processor again. On the BladeCenter AMM, navigate to Blade Tasks > Power/Restart.  Select your blade and click Restart Blade System Mgmt Processor.

My result: All LED bang lights turned off.  Finally.

How Do I Install AIX 64-bit?

Wednesday, August 6th, 2008

The simple answer is you don’t. You install AIX normally, which by default installs a 32-bit kernel (at least 32-bit is default for AIX 5.3). Then you change your kernel and reboot.

I have AIX running on top of (as a client partition of) Virtual I/O (VIO) Server, which is IBM’s virtualization version of VMware.  Does VIO server allow 64-bit client OS installs?  Yes.

Does My System Support AIX 64-bit?

Check whether your kernel is currently 32-bit:

AIX# bootinfo -K
32

Your AIX version must be 5.1 or above:

AIX# oslevel
5.3.0

Verify that your hardware supports 64 bits:

AIX# bootinfo -y
64

How to Change Your 32-bit AIX Kernel to 64-bit

1) Create/overwrite your links to the Unix kernel:
ln -sf /usr/lib/boot/unix_64 /unix
ln -sf /usr/lib/boot/unix_64 /usr/lib/boot/unix

2) Verify:
ls -l /unix
ls -l /usr/lib/boot/unix
lsvg -l rootvg

3) Create your boot image and restart:
lslv -m hd5
bosboot -ad /dev/ipldevice
shutdown -Fr

Note: If you have LSOF installed, make sure you’re running the 64-bit version.  Check out the last part of my How to Install LSOF post.

4) After you reboot, verify your OS Kernel:

bootinfo -K

How to Revert Back from 64-bit to 32-bit:

From 64-bit to 32-bit:
ln -sf /usr/lib/boot/unix_mp /unix
ln -sf /usr/lib/boot/unix_mp /usr/lib/boot/unix
lslv -m hd5
bosboot -ad /dev/ipldevice
shutdown -Fr

How to Install LSOF on AIX

Friday, August 1st, 2008

The LSOF command, which lists open files, shows the processes connected to a particular file. It’s a very useful command.

The name’s a little bit misleading. But remember, Unix treats everything (devices, filesystems) as files. So think of lsof as the command to “show the process that’s using a particular resource”.

Example of Using lsof

For example, I’d like to unmount /dev/cd0, which is on /mnt. But I can’t unmount, because I’m in the /mnt directory:

AIX# cd /mnt
AIX# umount /mnt
umount: 0506-349 Cannot unmount /dev/cd0: The requested resource is busy.

Suppose I didn’t realize my shell is keeping the /mnt directory busy. But the LSOF command can verify that it’s my bash session preventing the umount:

AIX# lsof /mnt
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 975102 root cwd VDIR 15,0 /mnt (/dev/cd0)

AIX# cd /
AIX# umount /mnt

Steps to Install lsof on AIX

1) Obtain the fileset. It’s in the AIX 5L V5.3 Expansion Pack which shipped with your AIX system. Supposedly, you can call IBM and they will ship you the CDs if you have support with them. Otherwise, try downloading it from their website. I had to create an IBM login to get the file I needed.

I personally installed this using the CDs, so those are the instructions I’m giving.

2) Insert CD, “AIX Expansion Pack”

3) Run SMIT

AIX# smit

4) Navigate to Install Software

Software Installation and Maintenance > Install and Update Software > Install Software

5) Specify your device as your CD (i.e. /dev/cd0)

6) Scroll to SOFTWARE to install. Press F4 or esc+4.

7) Find lsof.base and lsof.man.en_US. Select these using F7 or esc+7

8) Hit enter a few times and it should install in /usr/sbin/lsof.

9) Test the command as root:

AIX# lsof

Uh-Oh. lsof: FATAL: compiled for a kernel of 32 bit size.

You have a 64-bit machine. You’re running the 32-bit version. It’s okay. You probably have both versions installed. To check:

AIX# lsof64

If that command runs without any errors:

AIX# which lsof
/usr/sbin/lsof

AIX# cd /usr/sbin
AIX# mv lsof lsof32
AIX# ln -s lsof64 lsof
AIX# lsof

How to Upgrade Mirrored IBM AIX with Minimal Disruption

Wednesday, December 19th, 2007

AIX provides a very nice way to upgrade the OS while the system us up. It will require two reboots, but the time consuming parts can be done first. This is the process I used to upgrade AIX 5.3, technology level (TL) 4 to AIX 5.3 TL7.

0. Backup the OS

AIX# mksysb

1. Download updates. I downloaded it to an NFS-mounted filer, and it worked okay.

2. See which hdisks are in rootvg. In my case, hdisk0 and hdisk1 were part of rootvg.

AIX# lsvg
AIX# lspv
AIX# lspv -l disk1
AIX# lspv -l disk0

3. Remove one disk from the rootvg mirror:

AIX# unmirrorvg -c 1 rootfg hdisk0 #this unmirrors hdisk0, hdisk1 remains
AIX# reducevg rootvg hdisk0 # this removes hdisk0 from rootvg complete. Don’t use -d or -f
AIX# lspv -l hdisk0 #check to ensure hdisk0 is free

4. Install upgrade on the free disk. This takes a while (1-2 hours for me). This command copies rootvg (hdisk1) to hdisk0, and then applies the updates to disk0:

AIX# alt_disk_copy -d hdisk0 -b update_all -l

5. Reboot. This will boot off of hdisk0; hdisk0’s volume group will be renamed old_rootvg. Test and verify it works. Check OS level after reboot:

AIX# oslevel -r

If everything works properly, re-sync mirrors on rootvg

You can leave things in the two-disk state for a while. I left it for a few days. Once you’re sure you want to keep the changes, re-sync the mirrors.

IBM’s document on mirroring rootvg:
http://www-1.ibm.com/support/docview.wss?uid=isg3T1000358

AIX# alt_rootvg_op -X old_rootvg #This removes old_rootvg; hdisk1 is now accessable
AIX# extendvg rootvg hdisk1 #Add hdisk1 to rootvg; I had to use the -f force option
AIX# mirrorvg rootvg

Rebuild boot list; add hdisk0 and hdisk1 to bootlist; and reboot to disable the quorum:

AIX# bosboot -a
AIX# bootlist -m normal hdisk0 hisk1
AIX# reboot

Search for your disk (hdisk1 for me) in the boot log:

AIX# alog -o -t boot | less

Look at boot order:

bootlist -m normal -o #shows boot order

If things don’t look right, boot off of old_rootvg

Boot off of the old_rootvg disk (hdisk1). After a reboot, you should be able to see rootvg (which was the old one) and alt_rootvg (the new one that didn’t work).

Check which disk is in which vg, edit the boot order to make hdisk1 first, and then reboot:

AIX# lspv
AIX# bootlist -m normal -o hdisk1 hdisk0
AIX# reboot
AIX# oslevel -r