To: Users of the DEC 21050 PCI-PCI Bridge From: Todd Comins, DEC Consulting Engineer Subject: Host Bus Bridge and PCI to PCI Bridge Compatibility Issues Overview We have been receiving reports of incompatibilities between PCI to PCI bridges and some x86 processor host bus bridges. The purpose of this memo is to summarize what we believe is the cause of these problems and to discuss a work around to them. We believe that the incompatibility problem is the result of the behavior of certain host bus bridges and cannot be corrected by modifications of PCI to PCI bridge designs. Fundamentally, we believe that the problem can occur with all PCI to PCI Bridges and is not specific to Digital's DECchip 21050. One work around to this problem is known and can be employed in those systems which have host bus bridges which are incompatible with PCI to PCI bridges. However, this work around negatively impacts performance when used. Description of Problem The incompatibility problem exhibits itself as a bus deadlock. The following sequence of events will trigger this deadlock condition in those systems which use incompatible host bus bridges. 1. A bus master device downstream of the PCI to PCI bridge (PPB) generates a memory write transaction (Memory Write or Memory Write and Invalidate) targeting system memory. The PPB completes the transaction with the bus master and posts the memory write data in an internal write buffer. 2. Before the posted memory write data can be flushed (written to system memory) by the PPB, the processor initiates a non-postable transaction which accesses a device downstream of the PPB. Non-postable PCI transactions include: Memory Read Memory Read Line Memory Read Multiple I/O Read I/O Write Configuration Read Configuration Write Note: PCI to PCI Bridges do not respond to PCI Special Cycle or Interrupt Acknowledge transactions. At this point the conditions which are necessary to trigger the deadlock have been established. The PCI to PCI Bridge Architecture Specification requires the PPB to terminate the non-posted access from the processor by signaling retry while the posted write data remains in its write buffer. This requirement preserves the x86 processor architecture ordering rules necessary for correct operation of device drivers and hardware. The posted write data must be flushed (written to system memory) before allowing the non-posted transaction to proceed. Host bus bridges which are incompatible with PCI to PCI bridges may have one of two problems. Deadlock Case 1 In the most severe case, the host bus bridge will not grant the PCI bus to any other PCI bus master until the PCI transaction initiated by the processor completes. The host bus bridge will simply repeat the retried transaction continuously. Each time the PPB will terminate the transaction by signaling retry since it has been unable to flush its write buffer. This will repeat forever. The PCI Local Bus Specification requires the bus master of a transaction which terminates with retry to relinquish its bus request for two cycles thus allowing other bus masters to utilize the bus. Host bus bridges which comply with this requirement do not exhibit this particular deadlock condition. Deadlock Case 2 In the second case, the host bus bridge relinquishes its request after the cpu initiated access is retried by the PPB. However the host bus bridge terminates all accesses to system memory with retry until it is able to complete the processor initiated request. In these systems the PPB is able to arbitrate for the PCI bus. However, when the PPB is granted the PCI bus it attempts to write the posted data to system memory but the host bus bridge terminates the transaction by signaling retry. The PPB relinquishes its request for the bus as a result of the retry. The host bus bridge then attempts to complete the cpu initiated transaction which the PPB retries because it has not been able to empty its write buffer. This sequence will repeat forever. The host bus bridge must allow accesses to system memory to complete independent of the completion status of any processor initiated PCI transactions. Solution These problems appear to be common on chipsets designed for 486 processors. It is believed that the chipset must make use of the BOFF# signal provided by the 486 processors in order to avoid these deadlock conditions. If the chipset does not connect to this signal then it is believed that the chipset will exhibit one of the deadlock scenarios outlined above. During the PCI configuration the BIOS in systems that use host bus bridges which are incompatible with PCI to PCI bridges must disable posting in all PCI to PCI bridges found in the system. This will prevent the deadlock from occurring but will negatively impact performance. The default condition for Digital's DECchip 21050 is to have write posting enabled. Write posting in the DECchip 21050 can be software controlled (provided the S_DISPST_L pin is grounded). To disable posting in the DECchip 21050 the following conditions must be established. 1. Required for both deadlock case 1 and deadlock case 2: Ground the S_DISPST_L pin (pin 90) on the DECchip 21050 This pin must be grounded to enable software to control the write posting function of the DECchip 21050. 2. Required for both deadlock case 1 and deadlock case 2: Set bit 1 in configuration register 40 H This bit must be set to 1 to disable write posting. The default state of this bit is 0 which enables write posting. 3. Required for deadlock case 1, optional for deadlock case 2: Byte 0 and byte 1 in configuration register 44 H (byte addresses 44H and 45H) must be set to non-zero values. These values specify the number of clocks that the DECchip 21050 will hold a transaction in wait states while trying to arbitrate for the PCI bus on the opposite interface of the PPB. Byte 0 applies to the primary interface and byte 1 applies to the secondary interface. If the number of wait states exceeds the specified value the DECchip 21050 will terminate the transaction by signaling retry. The transaction will complete on a subsequent attempt. Note: One suggestion for these register values are byte 0 - 14H and byte 1 - 10H. Different values can be used to optimize throughput for specific configurations. These values are best determined through experimentation. Conclusion We have been in contact with the major chipset vendors while investigating solutions to this issue. We will continue to work with these chipset vendors to encourage that modifications be made to host bus bridges to insure interoperability with PCI to PCI Bridges. In addition, we will be working with system vendors, chipset vendors as well as BIOS vendors to insure that the systems incorporating chipsets which are incompatible with PCI to PCI bridges include the necessary BIOS modifications outlined in this memo, thereby minimizing the impact this issue has regarding PCI compatibility.