IBM* PCI Hot-Plug Controller Driver v1.02 for SUSE 8.0/RedHat 2.1 Advanced Server /RedHat 7.3 and Installation Instructions. Readme.txt Version 1.04 CONTENTS --------- 1.0 Hot-Plug PCI Installation Overview for SUSE 8.0 and RedHat 2.1 Advanced Server/Redhat 7.3 1.1 Overview 1.2 Limitations 1.3 Dependencies: 2.0 Change History 3.0 Installation and Setup Instructions for SUSE 8.0 and RedHat 2.1 Advanced Server/Redhat 7.3 3.1 Kernel configuration 3.1.1 For SUSE 8.0 3.1.2 For RedHat 2.1 Advanced Server 3.1.3 For RedHat 7.3 3.2 Driver installation 4.0 How to perform Hotplug PCI Operations (command-line) 4.1 Required steps in Hotplug Operations 4.1.1 Removing an adapter 4.1.2 Adding a new adapter to an empty slot 4.1.3 Changing the state of attention LED 4.1.4 Other files in the slot directory 4.2 Warnings about incorrect usage of Hotplug PCI. 5.0 Installation and Setup instructions for Graphical User Interface 6.0 Troubleshooting hot plug PCI operations 6.1 Basic Troubleshooting 6.2 Attention Indicator LED 6.3 Interrupt handling 1.0 Hot-Plug PCI Installation Overview for SUSE 8.0 and RedHat 2.1 Advanced Server/Redhat 7.3 1.1 Overview The Hot Plug PCI support for SUSE 8.0 and RedHAt 2.1 Advanced Server/RedHat 7.3 provided by IBM is compliant with many different specifications including full compliance with the PCI Hot-Plug v2.2 specification. The IBM Hot Plug PCI system support consists of an interlock switch and a set of two LED's for each hot plug PCI slot. One LED will remain on when power is ON to the given slot. The other LED will indicate that attention is required to that slot or will be used to indicate which slot was chosen for a hotplug action. Slots that do not have these devices are NOT hot plug slots. All hotplug operations must be controlled through a command line interface provided by the Linux kernel. IN NO CASE SHOULD PHYSICAL HOT PLUG OPERATIONS BE DONE WITHOUT FIRST REMOVING POWER FROM THE SLOT. IF A SLOT DOES NOT HAVE A LATCH,DO NOT REMOVE THE ADAPTER. DOING SO CAN CAUSE SERIOUS DAMAGE TO THE SYSTEM AND ADAPTER. The order of events in any hot-plug operation are: - Load the kernel PCI hotplug core driver. - Mount the PCI hotplug file system. - Load the IBM PCI hotplug controller driver to provide hot plug PCI support. - For hot-add operation, open the latch and insert an adapter and any necessary cabling. Close latch. - At command prompt, echo the requested values into the hotplug file system to power on or power off the desired PCI slots. - For hot-remove operation, When the adapter power is OFF, open the latch and remove, an adapter and any necessary cabling. Close latch. NOTE: The power to a slot cannot be turned ON until the latch is closed for the given slot. The position of the latch (opened or closed) is not important if no adapter is in the given slot. - If selected, the adapter power is turned ON and the adapter is configured. At this point, you may need to manually load the drivers associated with the newly added adapter card. 1.2 Limitations - Cannot load the IBM PCI hotplug controller driver multiple times within same boot session. If the driver is loaded, adapters are hot-added or hot-removed, then the driver is unloaded and re-loaded, erroneous behavior will occur. - This driver supports (1)single x-Series e-Server 440 systems and RXE100, Multiple e-Server systems require additional software; (2) x-Series e-Server 235; - This driver Will not support onboard PCI-PCI bridges. Additionally, only adapters cards with a single bus behind its onboard bridge are supported. - This driver does not support hot-adding/removing a nested bridge device, a bridge by itself, or any graphical devices. - This driver will not function properly if there is no APIC in the system. - Opening the latch while the adapter card is active may cause the system to reboot or hang. Under no circumstances should the latch be opened while power is applied to an active PCI slot. - Multi-function PCI devices are supported as long as the multi-function capability is not provided with or through a PCI-to-PCI bridge. - Devices that are not PCI v2.2 compliant or that do not implement the PCI presence detection pins are not supported. - Non-PCI devices are not hot pluggable. 1.3 Dependencies: - All files that are required by the Hotplug controller driver are provided in the standard SUSE 8.0/ RedHat 2.1 Advanced Server system and included RPM (ibmphp-[suse/redhat]-[8.0/2.1AS]-'VERSION_NUMBER'-src.rpm). - For RedHat 7.3, the Hotplug controller driver is provided in the standard RedHat 7.3 applying the latest kernel errata, and included source RPM 2.0 Change History of ibmphp ------------------------------ ************************* * ibmphp v0.5 * * June, 2002 * ************************* - Added support for x-Series 235 - Fixed some bugs - Release of binary RPM for RedHat 7.3 ************************* * v0.2 Changes * * March/April, 2002 * ************************* - Added bus speed limitations - Fixed polling - Added support for RXE100 - Release of RPMs for Suse 8.0 ************************* * v0.1 Changes * * January 14, 2002 * ************************* - Made formatting changes to comply with the Linux community. - Updated the readme file to remove incorrect information as well as to add additional instructions and clarifications. ************************* * v1.00 Changes * * December 14, 2001 * ************************* - Original Release 3.0 Installation and Setup Instructions for SUSE 8.0 ----------------------------------------- NOTE: To successfully install provided RPM, make sure that the source code for linux tree is installed on the system. You can get the source code by finding the correct RPM kernel-source-'VERSION-NUMBER'.rpm. 3.1 Kernel configuration ------------------------- 3.1.1 For SUSE 8.0 -------------- If SUSE 8.0 is installed on IBM x-series 440/235, the kernel options 64GB-SMP are selected, therefore, before installing ibmphp RPM, you will need to do the following: rpm -qa | grep kernel Sample result (NOTE: Results will vary with different kernel versions): kernel-source-2.4.18.SuSE-35 SuSE is configured with the Yast tool. Use it install the appropriate kernel source files. After installation configure the kernel sources by executing the following commands: cd /lib/modules/`uname -r`/build cp -f /boot/vmlinuz.version.h /lib/modules/`uname -r`/build/include/linux/version.h cp -f /boot/vmlinuz.config /lib/modules/`uname -r`/build/.config make cloneconfig make dep 3.1.2 For RedHat 2.1 Advanced Server ------------------------------------ rpm -qa | grep kernel Sample result (NOTE: Results will vary with different kernel versions): kernel-headers-2.4.2-2 kernel-source-2.4.2-2 kernel-smp-2.4.2-2 kernel-2.4.2-2 Note: On x-Series 440, make sure you are running summit kernel. NOTE: NORMALLY THE STEPS OUTLINED BELOW WILL WORK, UNFORTUNATELY, THIS PROCESS IS BROKEN ON REDHAT ADVANCED SERVER 2.1, AND THEREFORE WE ARE PROVIDING A BINARY RPM DIRECTLY. YOU CAN SKIP THIS SECTION AND STEPS 1 AND 2A IN THE SECTION 3.2: DRIVER INSTALLATION. Make sure that kernel-headers and kernel-source rpm packages are present and that they match your kernel version number. Once you have the appropriate kernel sources installed you will need to configure them. Execute the following commands: cd /lib/modules/`uname -r`/build make mrproper make menuconfig Go down in the list and choose an option 'Load an Alternate Configuration File', and choose ONE of the following configurations (depending on your hardware configuration) (exact file names may vary): Typing 'uname -r' and `uname -m` at command prompt might help you decide which configuration file you should load. configs/kernel-2.4.9-i586.config configs/kernel-2.4.9-i586-smp.config configs/kernel-2.4.9-i686-enterprise.config configs/kernel-2.4.9-i686-summit.config Choose and save configuration file. Execute the following command: make dep 3.1.3 For RedHat 7.3 ------------------------------------- Install RedHat 7.3 and latest kernel errata using RPM. Make sure you install kernel-source and rpm-build packages, and that they match your kernel version number. Type the following commands at the prompt: rpm -qa | grep kernel rpm -qa | grep rpm Sample result (NOTE: Results will vary with different kernel versions): kernel-smp-2.4.18-3 kernel-source-2.4.18-3 Once you have the appropriate kernel sources installed, you will need to configure them. Execute the following commands: cd /lib/modules/`uname -r`/build make mrproper ls configs Examples of config files you might see: configs/kernel-2.4.18-i686.config configs/kernel-2.4.18-i686-smp.config NOTE: Combination of `uname -r` and `uname -m` will help you decide which configuration file you should copy. Then copy the configuration file: cp configs/ .config Go to the directory where ibmphp RPM is stored. Type the following to extract a file: rpm2cpio ibmphp-src-redhat-'VERSION_NUMBER'-7.3.i386.rpm | cpio -ivd ./usr/src/redhat/SOURCES/Makefile.script Make sure your bash version is > 2.0. You can verify that by typing bash --version at the prompt. Type the following to launch the script: ./usr/src/redhat/SOURCES/Makefile.script Note: This script modifies the Makefile in /lib/modules/`uname -r`/build directory to match your running kernel. The command above (rpm2cpio) created a file tree in the directory where your RPM is contained. To remove it, type command: rm -rf ./usr MAKE SURE YOU ARE IN THE SAME DIRECTORY YOU WERE WHILE YOU ISSUED RPM2CPIO COMMAND, AND NOTICE THE LEADING . (dot) IN FRONT OF /usr, IMPORTANT: Make sure you are NOT in the / directory. cd /lib/modules/`uname -r`/build make oldconfig make dep 3.2 Driver installation ------------------------ Note 1: If this is an upgrade to an existing IBM PCI Hotplug driver, then previously installed packages must first be removed. Depending upon your system's configuration, you may see messages about missing files when running the "rpm -e" commands. You may safely ignore those messages. Type the following two commands at a shell prompt to remove the old driver source and binaries (if you have them): To find out which ones your system currently has, type rpm -qa | grep ibmphp at the command prompt. To remove, type: rpm -e ibmphp rpm -e ibmphp-src-[redhat/suse] rpm -e ibmphp-src-gui-[redhat/suse] 1. Go to the directory where ibmphp rpm package is stored, at the command prompt, type rpm -ivh ibmphp-src-[redhat/suse]-'VERSION_NUMBER'-[8.0/2.1AS/7.3].i386.rpm This rpm will compile the needed files and the binary RPM will be located at /usr/local/ibmphp NOTE: If you want to use GUI instead of command line interface, you will need to rpm ibmphp-src-[suse/redhat]-'VERSION_NUMBER'-[8.0/7.2AS/7.3]-src-gui.rpm instead (if available), please see step "5.0 Installation and Setup instructions for Graphical User Interface" for details. 2. At the command prompt, type 2A. cd /usr/local/ibmphp 2B. rpm -ivh ibmphp-[suse/redhat]-'VERSION_NUMBER'-[8.0/2.1AS/7.3].rpm 3. This binary RPM will put the binary object files in the /lib/modules/... directory and add a comment into /etc/fstab for mounting. The hotplug directory is located by default at /hotplug. 4. Add the ability to use the separate debug option. This can be done by adding the following line to /etc/syslog.conf: "*.debug /var/log/debug" In order for this change to take effect, please type the following commands at the prompt: touch /var/log/debug service syslog restart NOTE: The above section is only required to be performed prior to the initial use of the driver. This section can be skipped after the first installation/compilation. To install the IBM PCI hotplug driver, perform the following: Note: With the current version of RedHat 2.1 Advanced Server/RedHat 7.3, you do NOT need to perform the first 2 steps. Therefore, if you are running RedHat Advanced Server 2.1/RedHat 7.3, please skip to step 3. 1. Insert the PCI hotplug core module: "insmod pci_hotplug" 2. Mount the hotplug file system directory "mount /hotplug/" 3. Load the IBM PCI hotplug driver: "insmod ibmphp" NOTE: To view any errors due to improper operation, please view /var/log/messages or type "dmesg" at the command prompt. NOTE: If there is an unexpected failure, please power down the machine, reboot, and reload the IBM PCI hotplug driver with an additional debug option. Place a diskette into the floppy drive to save the error logs. The debug information can then be copied and saved for debugging purposes: "insmod ibmphp debug=1" "mount /mnt/floppy" "cp /var/log/messages /mnt/floppy/messages.txt" "dmesg >& /mnt/floppy/dmesg.txt" "cp /var/log/debug /mnt/floppy/debug.txt" "umount /mnt/floppy" 4.0 How to perform Hot-Plug PCI Operations (command line interface) 4.1 Required steps in Hot-Plug Operations NOTE: Opening an adapter latch will turn off the power to the slot. If a device driver was loaded for that slot, the system will most likely hang, requiring the system to be rebooted. 4.1.1 Removing an adapter 1. Verify that the pci_hotplug and ibmphp drivers are loaded. This can be done by issuing the "lsmod" command. 2. Verify that the /hotplug file system is mounted. This can be done by issuing the "mount" command. 3. Issue the power off command to the desired PCI hotplug capable slot. "echo 0 > /hotplug//power" Ex. "echo 0 > /hotplug/2/power" to turn power off to slot 2. 4. Verify that the power LED for the slot is OFF. If the LED is still ON and the attention LED is OFF, go back to the command line and verify the proper command was issued. NOTE: NEVER REMOVE AN ADAPTER FROM A SLOT WITH THE SLOT POWER STILL ON. THIS COULD RESULT IN A SYSTEM HANG AND/OR SERIOUS DAMAGE TO THE ADAPTER CARD AND/OR SYSTEM UNIT. 5. Open the latch and remove the adapter from the slot. 6. Close latch after adapter has been removed. 4.1.2 Adding a new adapter to an empty slot 1. Verify that the pci_hotplug and ibmphp drivers are loaded. This can be done by issuing the "lsmod" command. 2. Verify that the /hotplug file system is mounted. This can be done by issuing the "mount" command. 3. Open the latch, place the new adapter in the desired slot and close the latch. 4. Attach any necessary cables that the newly inserted adapter may require. 5. Issue the power on command to the desired PCI hotplug capable slot where the new adapter will be inserted. "echo 1 > /hotplug//power" Ex. "echo 1 > /hotplug/2/power" will turn power on to slot 2. 6. Verify that the power LED for the selected slot is now on. NOTE: If any error occurs, the IBM PCI hotplug controller driver will turn the attention LED on for the selected slot. To check the corresponding error message, please read /var/log/messages or /var/log/debug (if the debug=1 option was provided). 4.1.3 Changing the state of attention LEDs Note: In addition to internal indications that the operation is currently being performed or an error occured, ibmphp driver lets user change the state of attention LEDs. 1. To turn on attention LED for a slot, type at the command prompt echo 1 > /hotplug//attention 2. To turn off attention LED for a slot, at the command prompt type echo 0 > /hotplug//attention 3. To make the attention LED blink for a slot, at the command prompt type echo 2 > /hotplug//attention 4.1.4 Other files in the slot directory You can find the status of a slot by going going to the directory corresponding to that slot cd /hotplug// Each directory contains several files. You can do cat for each particular file. - latch file contains 1 if the latch of the slot is open, 0 if the latch is closed - attention file contains 0 if attention is off, 1 if attention is on, 2 if attention is blinking - power file contains 0 if the power is off, 1 if power is on - adapter file contains 1 if there is an adapter in the slot, 0 otherwise - test file is not used 4.2 Warnings about incorrect usage of Hotplug PCI. Below are several warnings about hot plug PCI operations: 1. Do not remove an adapter from a non-hotplug PCI slot. 2. Grounding equipment, including wrist straps, should be used when working inside any system unit to protect against electrostatic discharge (ESD) that could damage system components and adapter cards. 3. Make sure that any adapter card hot added to the system is fully seated before attempting to turn on power. 4. Opening an adapter latch will turn off the power to the slot. If a device driver was loaded for that slot, the system could hang, requiring the system to be rebooted. 5. Do NOT remove the plastic cover over the system components. Removal of the plastic cover exposes system components that may be shorted by adapter brackets during hot plug PCI operations. 5.0 Installation and Setup instructions for Graphical User Interface -------------------------------------------------------------------- NOTE: In the future, a gui rpm will be provided that will install a customized version of Graphical User Interface tailored just for IBM servers. For now, you would need to download the latest version from http://www.kroah.com/linux/hotplug/ (for example, www.kroah.com/linux/hotplug/pcihpview-0.3-1.i386.rpm) 1. Make sure you have gtk and gtk-devel packages installed on your system 2. If you are running RedHat Advanced Server 2.1, make sure you have pygnome-gtkhtml package installed on your system. If you are running SUSE 8.0, make sure you are running GNOME and not a KDE session. 3. To configure and use the pcihpview, please type the following in the directory where pcihpview-0.3-1.i386.rpm resides rpm -ivh pcihpview-0.3-1.i386.rpm 4. This will produce an executable pcihpview. You need to be in X-environment in order to use it. To use it, type pcihpview /hotplug & Note: 0.3 version writes all the messages into a log file (usually /var/log/messages), and this operation might take a few minutes before GUI window pops out. The order of operations is the same as for command-line interface. To operate on a slot, highlight it and right-mouse click to see the available operations. Limitations: the current version of pcihpview (0.3) does not automatically update if a change has been detected from polling (such as a latch opening or closing, or if you issue a command through command line interface). However, you can choose Refresh option to refresh your screen. There is also a limit of 1 pcihpview running at the same time on a system. 6.0 Troubleshooting Hotplug PCI operations ------------------------------------------- 6.1 Basic Troubleshooting There are several troubleshooting techniques that can be used to determine why a hot plug PCI operation failed. Two LED's are provided for each hotplug PCI capable slot. The attention LED blinks to indicate that an operation is currently being performed and is solid if attention is required. The second LED indicates power state. Error messages are stored in /var/log/messages and /var/log/debug (if the debug=1 option was provided). 6.2 Attention LED The attention LED is controlled by the IBM PCI hotplug controller driver. The attention LED is used by the device driver to let the user know that an operation is currently ongoing on that particular slot or if an error has occurred. When powering a slot on, you will see the attention indicator flash momentarily while the newly inserted adapter is configured. The user can manually turn the attention LED on by issuing the attention on command: "echo 1 > /hotplug//attention" Ex. "echo 1> /hotplug/2/attention" will turn the attention LED on for slot 2. Similarly, the user can turn the attention LED off by issuing the off command: "echo 0 > /hotplug//attention" Ex. "echo 0 > /hotplug/2/attention" will turn the attention LED off for slot 2. 6.3 Interrupt handling Interrupts are handled in the following manner: - Interrupts are preserved from the MPS table located in the extended BIOS data area (EBDA) of the system.