The zero buffer-to-buffer credits nightmare

Recently I came across a performance problem most likely caused by a congestion of the SAN and especially the lack of buffer-to-buffer credits. So I dusted off the old theories that date back to the origins of SAN, but that in certain critical situations should not be completely neglected.

Basics of Fibre Channel Flow Control and Buffer Credit Theory
The basic information carrier in Fibre Channel is the frame, and all information is contained within frames like when you send a letter encapsulated within an envelope. When you want to send data thru a SAN, your data is encapsulated within a frame.

Buffer credits, also called buffer-to-buffer credits are used as a flow control method by Fibre Channel technology and represent the number of frames a port can store.
Each time a port transmits a frame that port’s BB Credit is decremented by one; for each R RDY received, that port’s BB Credit is incremented by one. If the BB Credit is zero the corresponding node cannot transmit until an R_RDY is received back.
Each of these credits represents the ability of the device to accept additional frame(s). If a recipient issues no credits to the sender, no frames can be sent. Pacing the transport of subsequent frames on the basis of this credit system helps prevent the loss of frames and reduces the frequency of entire fibre channel sequences needing to be retransmitted across the link.
This mechanism to prevent a target device (either host or storage) from being overwhelmed with frames is known also as BB_credit flow control.
If a FC frame arrives while the receiver is already processing one first frame, a second receive buffer is needed to hold this new frame. Unless the receiver is capable of processing frames as fast as the transmitter is capable of sending them, if the receiver will not have a receive buffer available the next frame will be lost. To prevent this type of condition, the Fibre Channel architecture provides a two level flow control mechanism that allows the receiver to control when the transmitter may send frames.
The receiving port controls the frame transmission by giving the sending port permission to send one or more frames to the receiving port in question. This permission is called a credit. The actual credit(s) are granted during the login process between two ports. The credit value is decremented when a frame is sent and replenished when a response is received. If the available credits for a given port reaches zero, the supply of credits is said to be exhausted. Further transmission of frames with that port is then suspended until the amount of credits can be replenished to a non-zero value.

There are two types of flow control mechanisms in FC:
•    End-to-End Flow Control.
•    Buffer-to-Buffer Flow Control.

End-to-End Flow Control.
Transmission credit is initially established when two communicating nodes log in and exchange their respective communication parameters. The nodes monitor end-to-end flow control themselves. Switches or directors do not participate in EE Credit and End-to-End Flow Control is always managed between a specific pair of node ports. Therefore, an individual node port may have many different end-to-end credit values, each corresponding to a different destination node port.

Buffer to Buffer Flow Control
Buffer-to-Buffer Flow Control is flow control between adjacent ports in the I/O path.
In this case a separate, independent pool of credits is used to manage Buffer-to-Buffer Flow Control where a sending port using its available credit supply is waiting to have the credits replenished by the port on the opposite end of the link.
An end node attached to a FC director or switch will establish its BB Credit during login to the fabric. A communicating partner attached elsewhere on the same director or switch will establish its own and most likely different BB Credit value to the director during its login process.

This system may affect overall performance and efficiency. Considering that light takes approximately 5usec to travel 1km and a typical FC frame at 2Gbits is 1 Km long, this behavior becomes even less efficient and more of a performance drag on longer distance  or when traveling through complex topologies that contribute significant delivery latencies.
So, to achieve the higher performance we need to use BB Credit values>1 and if a sending port is allowed to send more than one frame without having to wait for a response to each, performance can be improved until link utilization reaches 100%. When the link is thus fully utilized, frames can be sent as rapidly as allowed but additional credits will not help matters.

In now days Fibre Channel fabrics have evolved and are containing thousands FC ports,  hundreds of hosts and several storage ports are now common. I/O levels and consequently traffic volumes, particularly across fabric, are much higher then in the past. Workload has also become much more complex and this makes much more difficult to isolate application problems when application performance becomes a problem.
Storage virtualization has created its own special I/O requirements, adding a degree of complexity to the I/O complex .
Rogue or badly behaving devices have much more impact on production environment than previously.
When application performance problems become obvious the storage complex is frequently blamed. The impact of such outages ranges from an inconvenience to a massive outage where mission critical application availability is compromised and the enterprise is seriously affected.

Diagnose a exhausted  buffer-to-buffer credits problem is a long, complex activity and requires appropriate tools. These tools must be capable of measuring the lack of buffer credits from the device or host point of view, connected to the SAN and must also be capable of measuring the lack of buffer credits from the SAN director or switch point of view.
A great help come from Storage performance analysis tool or from SAN brand tool like Brocade Bottleneck Detection or CISCO proprietary tools.
In my case, because the SAN was based on Brocade products I found very helpful the following guide:

Bottleneck Detection Best Practices Guide GA-BP-383-00 http://tinyurl.com/8aolmo8

and

Fabric Resiliency Best Practices IBM redpaper http://tinyurl.com/8sphtvb

and
Thanks to Sebastian for his blog http://tinyurl.com/8uwc7df

Advertisements

One thought on “The zero buffer-to-buffer credits nightmare

Leave a comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s