Ryan’s Tech Notes

I’m running Red Hat Linux Enterprise (RHEL) 4 on an IBM BladeServer HS20 blade. The machine was throwing a bunch of errors about the battery. After much diagnosis and the compulsory exercise of upgrading the firmware, IBM came out and replaced the system board.

After the replacement, the system booted up fine. But the network connection wouldn’t come up. I tried to bring it up.

These command should work on Fedora, CentOS, and other Red Hat-flavored distributions.  I believe that these commands are also similar on Debian distros like Ubuntu.  How about Solaris, HP-UX, or AIX?  Sorry, ifconfig works, but you need to look elsewhere for the rest of the commands.

Red Hat Linux Commands to Start Network Connection

Linux command to start the network connection:

RHEL4# /etc/init.d/network start
Bringing up interface eth0: Device eth0 has different MAC address than expected, ignoring. [FAILED]

Linux command to find the hardware/MAC address of eth0:

RHEL4# ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:14:5E:AE:2D:B1

Update this line in ifcfg-eth0 (replace eth0 with your interface, of course):

RHEL4# grep HWADDR /etc/sysconfig/network-scripts/ifcfg-eth0
HWADDR=00:14:5E:AE:2D:B1

RHEL4# /etc/init.d/network start

There are always a bunch of strange commands to connect a server to SAN disks. In my case, I’m running Solaris 9, using QLogic Fibre Channel cards, and connecting to an IBM DS4300.

1) Connect the Solaris server SAN disks. After I made the connection, the GUI that allows me to zone the SAN recognized the QLogic connections, and I zoned the LUNs.

If you need more detailed instructions, here are some potentially useful posts: How to Zone a Brocade SAN Switch and How to Zone IBM DS4000 SAN Disks.

2) Scan your disks, and it should show up as a new disk when you run “format”.

Solaris# devfsadm
Solaris# format

My result: No new disks.  Sigh.

3) Run a bunch of cryptic but useful diagnostic commands:

To see your HBA ports and whether you’re connected:

Solaris# luxadm -e port
Found path to 3 HBA ports
/devices/pci@8,700000/SUNW,qlc@2/fp@0,0:devctl        CONNECTED
/devices/pci@8,700000/SUNW,qlc@2,1/fp@0,0:devctl     CONNECTED
/devices/pci@8,600000/SUNW,qlc@4/fp@0,0:devctl        CONNECTED

To see your disks:

Solaris# luxadm probe

To see your HBA ports (type fc-private, below) and their connected disks (type disk):

Solaris# cfgadm -al

Ap_Id                         Type           Receptacle  Occupant Condition
c8                               fc-private  connected configured unknown
c8::200800a0b8199b3b  disk           connected configured unknown
c9                               fc-private  connected configured unknown
c9::200900a0b8199b3b   disk          connected configured unknown

4) Force Fibre Channel SAN disk rescan, since everything looks connected and okay.  Use your device path from “luxadm -e port” output.

Solaris# luxadm -e forcelip /devices/pci@8,700000/SUNW,qlc@2/fp@0,0:devctl
Solaris# luxadm -e forcelip /devices/pci@8,700000/SUNW,qlc@2,1/fp@0,0:devctl
Solaris# luxadm -e forcelip /devices/pci@8,600000/SUNW,qlc@4/fp@0,0:devctl

5) Rerun format command.

Solaris # format

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w500000e0107111e1,0
1. c1t1d0 t2
/pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w500000e01070d761,0
2. c7t600A0B801019B1B2002032A5489C60F3d0
/scsi_vhci/ssd@g600a0b801019b1b2002032a5489c60f3

Voila! There it is, disk #2.

My IBM JS21 blade (type 8844), which is running AIX 5.3, had several error lights lit.  They were the amber light with an exclamation mark.  I saw no errors in the BladeCenter Advanced Management Module (AMM) event log.  There were no hardware errors when I type in “errpt” in the AIX prompt. Strange.

There were four LED lights lit:

1. On the physical blade
2. On the physical BladeCenter chasis
3. In the BladeCenter AMM GUI > Monitors > LED Media Tray and Rear Panel LEDs > System Error
4. In the BladeCenter AMM > Monitors > LEDs > Blade LEDs

These error lights meant one of two things, since no errors were showing:

• They’re just false alarms.

OR

• There are hidden errors stored somewhere, and we needed to reset various system components to flush the errors out.

It my case it was (1).  But it was a good idea to follow this process just to be certain.

How to Clear the LED Bang Lights

1) Restart the Blade System Management Processor. On the BladeCenter AMM, navigate to Blade Tasks > Power/Restart.  Select your blade and click Restart Blade System Mgmt Processor.  Don’t worry.  Your blade will continue running.

My result: The Blade Center errors (errors 3 and 4, above)

2) Check for errors. BladeCenter AMM > Event Log.

AIX# errpt

My result: I found no errors.

3) Restart Management Module. BladeCenter AMM > MM Control > Restart MM.  This will cause you to lose connection to the BladeCenter.  Your blade servers will continue running.

My result: the physical chasis LED turned off.  Bang light #1, on the physical blade remained.  I had to schedule an outage and clear the NVRAM.

How to Clear NVRAM on a System P Blade Server

1) Schedule an outage.

2) Click Restart Blade and clear NVRAM, if you have this option.  It should be in BladeCenter AMM > Blade Tasks > Power/Restart.  Look on the bottom of the page.  I, unfortunately, didn’t have this option.  I just had Restart Blade.  I restarted my blade for kicks.

My result: No dice.  Still no errors.  Error light 1, on my physical blade remained on.

3) Power off the server. It should be in the Power/Restart screen of your Blade Center.

4) Remove the battery from the server for five minutes. Slide out the blade, open the server, attach an anti-static wristband (if you worry about these things), and remove the nickel-sized battery.  A small flathead screwdriver would be useful.

Warning:  This will reset any custom BIOS settings, return you to the orginal BIOS firmware level, and reset your time clock. If you pretty much use default settings, like me, you’d probably be fine.

5) Open up a console window. If you’re using a JS20 or JS21, you’ll need these instructions.  Otherwise, if you have a graphics card, you can use the BladeCenter Remote Control.

6) Put the server back together and power it up.

7)  Follow the prompt and press 0 to select the console on your console window,.  This is a good indication that the NVRAM has been cleared.

My result as it was booting:  The physical blade LED turned off.  But error lights 2, 3, and 4 all turned back on.  This was like a bad joke.

8) Reset the time when you can log in.

The time command to set it for Aug 13 (0813), 6:52 pm (1852), 2008 (08):

AIX# time 0813185208

9) Wait for everything to come up.  This is a good time to double check the error logs again.

10) Restart the Blade System Management Processor again. On the BladeCenter AMM, navigate to Blade Tasks > Power/Restart.  Select your blade and click Restart Blade System Mgmt Processor.

My result: All LED bang lights turned off.  Finally.

Zoning a DS4800, DS4200, DS4300, or any DS4000 is quite easy. Once you get the terminology (of which there’s quite a bit), the process is quite straightforward.

Quick Start

1) Designate a few of your physical disks to be hot spares.
2) Assign your physical disks to a logical array.
3) Split up your logical arrays into logical drives.
4) Map each logical drive to a logical unit number (LUN) and set access permissions.
5) From your server, scan for new disks.  Your LUNs should appear like a newly-installed local disk.

Prerequisites

Before zoning, make sure you have already

• Physically installed your DS4000.  For help see this post: How To Install DS4000 Disks.
• Installed IBM Storage Manager Client.
  • How to Install Storage Manager on AIX
  • How to Install Storage Manager on Microsoft Windows
• Installed your HBA/Fibre Channel card on your host (i.e. the server connecting to the disks)
• Physically connected your host to your DS4000.

What am I Looking At?

Launch Storage Manager.  It show your DS4000 devices.  Double click your device. (If your device is not shown, you might need to run Automatic Discovery.  From the menu choose Tools > Automatic Discovery).

It should bring up a screen this:

At the top, there are two tabs.  One says “Logical/Physical View”.  The other says “Mappings View”.  For now, let’s focus on the “Logical/Physical View”.

On the right side of the screen depicts your physical system.  Each little rectangle represents an individual disk.

• The white-and-gray rectangles are not-yet assigned or “unconfigured” disks.
• The fully gray rectangles are assigned disks.
• The white-and-red ones are hot spares.
• The white dotted line rectangles indicates an empty slot without a disk.
• Each row of disks is an enclosure or a shelf.
• Each enclosure has an ID number.  If you look at your physical DS4000, the enclosure numbers are on the back of the shelf.

On the left side of the screen depicts the logical assignment of your system.

• The double cylinder represents a logical array (not to be confused with a physical enclosure).
• The single cylinder represents a logical drive.

1) Designating Hot Spares

1.1) Navigate to Drive > Hot Spare coverage in Storage Manager
1.2) Choose Automatically Assign Drives
1.3) Adjust the layout/assignment of hot spares based on your preferences, if you want

When a physical disk fails, a hot spare will automatically jump in and take the place of the failed disk.  The automatic assignment of hot spares is a good starting point.  Adjust from there depending on your risk tolerance and your planned RAID type (i.e. mirrored arrays are “safer” than RAID 5 ones).

2) Assign Physical Disks to Logical Arrays

2.1) Right click on the Unconfigured Capacity icon on the left side of the screen, and select Create Logical Drive, follow the wizard to the Specify Array screen.
2.2) Choose your RAID level
2.3) Choose your Array Capacity and select Automatic drive selection.
2.4) Create your first logical drive.  Decide on its size.  (It does not need to fill the entire array)
2.5) Name your logical drive
2.6) Choose Map Later using the Mappings View
2.7) Your new Logical Array should appear on the left side of the screen.  Click on it and examine your physical mapping.  If you don’t like it, now is the time to make your changes.

A few tips on logical arrays:

• Each disk can only be in one logical array
• It’s preferable to have your arrays made of similar disks (i.e. all SATA or all FC, and all the same size)
• If you don’t like your array, it’s easier to delete and recreate it than to change its size — as long as the array is not used.
• When you click on your logical array, the blue/purple dots indicates the physical disks that are assigned to that particular array.
• In the past, I’ve had trouble with large drives (over 1 TB).
• You can expand a logical array, but you cannot shrink it.

3) Split up your Logical Arrays into Logical Drives

3.1) If you have extra space left on a logical drive, there should be an icon that says Free Capacity.
3.2) Right click on that icon and choose Create Logical Drive
3.3) Follow the wizard and create the drives, as you did in steps 2.4 through 2.6.

Tips on logical drives:

• Each logical drive must be in exactly one array.  It cannot span multiple arrays, nor can it be larger than its parent array.
• You can expand a logical drive, but you cannot shrink it.
• It’s preferable to create the drive at the right size, rather than counting on expanding it — based on personal experience, of course.

4) Map each Logical Drive to a LUN and Set Permissions

4.1) Click on your Mappings View tab
4.2) If the left side of your screen doesn’t show your host (the server that will connect to the SAN), create your host and host groups.
4.3) Expand Undefined Mappings on the left side of your screen.  All of your unmapped logical drives should be here.
4.4) Right click on one of your logical drives and select Define Additiona Mapping…
4.5) Select the host or host group that should have access to this logical drive
4.6) Assign a logical unit number (LUN) and press add.

Congratulations.  Your logical drive is now mapped to your selected host(s).

5) Scan for new disks from your server

Log into your server and scan for new hardware.

In AIX:

AIX# cfgmgr
AIX# fget_config -Av

In Solaris:

Solaris# devfsadm
Solaris# cfgadm -a
Solaris# format  (this should show your new drive)

The simple answer is you don’t. You install AIX normally, which by default installs a 32-bit kernel (at least 32-bit is default for AIX 5.3). Then you change your kernel and reboot.

I have AIX running on top of (as a client partition of) Virtual I/O (VIO) Server, which is IBM’s virtualization version of VMware.  Does VIO server allow 64-bit client OS installs?  Yes.

Does My System Support AIX 64-bit?

Check whether your kernel is currently 32-bit:

AIX# bootinfo -K
32

Your AIX version must be 5.1 or above:

AIX# oslevel
5.3.0

Verify that your hardware supports 64 bits:

AIX# bootinfo -y
64

How to Change Your 32-bit AIX Kernel to 64-bit

1) Create/overwrite your links to the Unix kernel:
ln -sf /usr/lib/boot/unix_64 /unix
ln -sf /usr/lib/boot/unix_64 /usr/lib/boot/unix

2) Verify:
ls -l /unix
ls -l /usr/lib/boot/unix
lsvg -l rootvg

3) Create your boot image and restart:
lslv -m hd5
bosboot -ad /dev/ipldevice
shutdown -Fr

Note: If you have LSOF installed, make sure you’re running the 64-bit version.  Check out the last part of my How to Install LSOF post.

4) After you reboot, verify your OS Kernel:

bootinfo -K

How to Revert Back from 64-bit to 32-bit:

From 64-bit to 32-bit:
ln -sf /usr/lib/boot/unix_mp /unix
ln -sf /usr/lib/boot/unix_mp /usr/lib/boot/unix
lslv -m hd5
bosboot -ad /dev/ipldevice
shutdown -Fr

The IBM Storage Manager Client allows you to control and manage your DS4800, DS4700, DS4300 disk systems.  This guide describes how to install on a Windows machine.  For my purposes, I’m upgrading from Storage Manager v9.60 to v10.10.  I’ve also written instructions for Installing Storage Manager on AIX.

Also, see that guide for basic concepts and terminology such as

• Enclosure Firmware
• Storage Manager Client
• SMagent Software
• SMruntime and SMesm

How to Install or Upgrade Storage Manager Client

1) Download the software.

For Windows XP, 2000, 2003, or 2008, on a 32-bit platform, click here.
For Windows Vista , 2003, 2008, on a 64-bit platform, click here
For Windows Vista 32-bit, click here.

If these links don’t work for you, try navigating IBM’s site:

www.ibm.com > support & downloads > fixes, updates, and drivers
Category > (under SYSTEMS) system storage
Product Family > disk systems, Product > DS4800 or whichever DS4000
Select Operating System and Click GO
Click Downloads on the Support and Downloads box.

2) Unzip your download. Keep a copy of these install files around.  The scripts folder will likely come in handy.

3) Run your installation executable file.  In my case it’s SMIA-WS32-10.10.35.06.exe.

4) Select your language, agree to IBM’s legal stuff.

5) Select your Installation type.  To select the right type, answer this question.  What computer are you installing Storage Manager on?

If it’s the server connecting to your DS4000 disks, select Full Installation.
If it’s just a desktop workstation or a laptop, select Management Station.

6) Select Do not Automatically Start the Monitor

You should designate one machine, preferably a server that’s always running and connected to the disks by fibre channel, to act as the monitor.  For me, that’s our AIX machine.

7) Review the disk space requirements and click Install.

The LSOF command, which lists open files, shows the processes connected to a particular file. It’s a very useful command.

The name’s a little bit misleading. But remember, Unix treats everything (devices, filesystems) as files. So think of lsof as the command to “show the process that’s using a particular resource”.

Example of Using lsof

For example, I’d like to unmount /dev/cd0, which is on /mnt. But I can’t unmount, because I’m in the /mnt directory:

AIX# cd /mnt
AIX# umount /mnt
umount: 0506-349 Cannot unmount /dev/cd0: The requested resource is busy.

Suppose I didn’t realize my shell is keeping the /mnt directory busy. But the LSOF command can verify that it’s my bash session preventing the umount:

AIX# lsof /mnt
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
bash 975102 root cwd VDIR 15,0 /mnt (/dev/cd0)

AIX# cd /
AIX# umount /mnt

Steps to Install lsof on AIX

1) Obtain the fileset. It’s in the AIX 5L V5.3 Expansion Pack which shipped with your AIX system. Supposedly, you can call IBM and they will ship you the CDs if you have support with them. Otherwise, try downloading it from their website. I had to create an IBM login to get the file I needed.

I personally installed this using the CDs, so those are the instructions I’m giving.

2) Insert CD, “AIX Expansion Pack”

3) Run SMIT

AIX# smit

4) Navigate to Install Software

Software Installation and Maintenance > Install and Update Software > Install Software

5) Specify your device as your CD (i.e. /dev/cd0)

6) Scroll to SOFTWARE to install. Press F4 or esc+4.

7) Find lsof.base and lsof.man.en_US. Select these using F7 or esc+7

8) Hit enter a few times and it should install in /usr/sbin/lsof.

9) Test the command as root:

AIX# lsof

Uh-Oh. lsof: FATAL: compiled for a kernel of 32 bit size.

You have a 64-bit machine. You’re running the 32-bit version. It’s okay. You probably have both versions installed. To check:

AIX# lsof64

If that command runs without any errors:

AIX# which lsof
/usr/sbin/lsof

AIX# cd /usr/sbin
AIX# mv lsof lsof32
AIX# ln -s lsof64 lsof
AIX# lsof

It’s hard to find old versions of Java on Sun’s website. So it may be worthwhile for a Linux/Unix administrator to keep old copies at hand. Also, keeping several versions installed is relatively easy — once you know the version of Java you want.

Different Versions to Install

The Linux versions of JDK come in several forms. There’s the 32-bit and 64-bit version. For each of these, there’s the .bin version and the rpm.bin version.

Installing the .bin version is very easy. Just run the jdk-*-linux-i586.bin or the jdk-*-linux-amd64.bin file as root. Agree to Sun’s legal jargon, and it will create a version-specific jdk folder in your current directory. Move it anywhere you please (i.e. /usr/local/jdk).

Installing from RPM

Since I’m using a Red Hat Linux machine, I’ll go into detailed instructions for the rpm install. I’m upgrading from JDK 1.5.0_12 to JDK 1.5.0_16. J2SE 5.0 is reaching its end-of-life (EOL), but that’s a different story.

For this install, I’ll place the various versions of java in /usr/local/jdk

1) Download the correct version of Java. I’m using a 64-bit machine, so I downloaded jdk-1_5_0_16-linux-amd64-rpm.bin. If you’re using a 32-bit machine, it’d probably be labeled jdk-1_5_0_16-linux-i586.rpm.bin. I like to keep a copy in /usr/local/src.

2) Check your current version of Java

# which java
/usr/bin/java

# java -version

3) Set up your Java as a set of links, as follows:

# ls /usr/bin/java
lrwxrwxrwx … /usr/bin/java -> /usr/local/java/bin/java

# ls -l /usr/local/java
lrwxrwxrwx … /usr/local/java -> jdk1.5.0_12

4) Validate, install package in /usr/local, and verify your installation

# cd /usr/local/src
# rpm –checksig jdk-1_5_0_16-linux-amd64.rpm
# mkdir /usr/java
# rpm -i –relocate /usr/java=/usr/local jdk-1_5_0_16-amd64.rpm
# rpm -q jdk

See the files associated with your jdk install
# rpm -ql jdk

5) Point your default Java executable to your new install

# cd /usr/local
# rm java
# ln -s jdk1.5.0_16 java
# ls -l java
# java -version

The IBM BladeCenter is pretty flexible in letting you mount CDs and DVDs. You can share the CD player on your own workstation or an ISO file saved on a disk somewhere. With these additional options comes complexity… and confusion.

If You Can Physically Access the BladeCenter

The BladeCenter has a shared CD/DVD drive for all of the blades. Use this drive.

1) Login to the BladeCenter

2) Select Blade Tasks > Remote Control

3) Click Start Remote Control

4) A pop-up should, well… pop up

5) In the “Media Tray” drop down select your blade server.

6) Refresh your Remote Control Status page. The “Media tray owner” should indicate your selected blade

7) Place your CD/DVD in the blade center drive

8) Login to your AIX server blade as root. If you need bladecenter console access, you can use SOL. Follow these directions: How to Get Non-Graphical Console Access to a BladeCenter Server

9) Run this command:

# mount -v cdrfs -o ro /dev/cd1 /mnt

10) The CD should be mounted in the /mnt directory

If You CANNOT Physically Access the BladeCenter

I have been able to mount to Linux blade servers, but have not done this on AIX. Either way, this should work. Follow the steps above, with a few changes.

<Follow steps 1-5 above>

5.1) On the remote control pop up screen, select your workstation’s CD-ROM drive in the available resources box.

5.2) Click the double arrow (>>)

5.3) Click “Mount all”

<Follow step 6 above>

7) Place the CD/DVD in your own workstation’s drive

<Follow step 8 above>

9) The CD should be in /dev/cd0 instead of /dev/cd1. Run this command:

# mount -v cdrfs -o ro /dev/cd0/mnt

<Follow step 10 above>

How To Remove a CD on AIX

11) To unmount the CD

umount /mnt

12) If you get a command that says “Cannont umount /dev/cd0: the requested resource is busy”, you probably need to change directory

# cd /

# umount /mnt

Juniper Networks, doesn’t officially support Firefox 3 as of yet. But here’s an unofficial workaround.

Prerequisite: A Recently Upgraded Juniper VPN Device

You (or your network administrator) need to upgrade the Juniper VPN software to one of the latest versions. Juniper’s software downloads can be found here and I’ve written upgrade instructions here: How to Upgrade Juniper SSL VPN. I got this working on version 6.0R6.

How To Login From Firefox 3

1) Login to the VPN using your user name and password (not as administrator).

2) Your browser will hang at a URL that looks like this: https://vpn.twosmallcoins.com/dana/home/starter0.cgi?check=yes

3) Edit your URL to remove the “0″ between “starter” and “cgi”. It should look like this:
https://vpn.twosmallcoins.com/dana/home/starter.cgi?check=yes

4) Press enter and your VPN client should load properly.