myspace profile view counter
| About BytePile.com | Contact | News@BytePile.com |

Tue 26-September-2017



 Mass Storage Services:
  RAID Arrays - Storage
  SAN & NAS Storage - Services
  Tape Systems - Services



 Hosting & Web Services:
  Shared Hosting
  Dedicated Hosting
  Collocation Services



 Networks & Technology:
  BytePile Network Maps
  System Hardware
  Software Suites



 Company:
  About BytePile.com
  News@BytePile.com



 Support:
  Definitions & Terms
  Frequently Asked Questions



 Tech - White Papers:
  CAT-6 Tutorial by Lucent (pdf)
  Data Speed Table
  Dbase Conf. SAN or NAS (pdf)
  DSL Types & Categories
  Ethernet (UTP) CAT Cables
  Fibre Channel Overview
  RAID 3 vs. RAID 5 in HPC
  RAID Types & Categories
  T1 & T3 RJ-48 Cables
  The SAN Book 3.0 (7MB pdf)


 Legal:
  Acceptable Use Policy
  Privacy Statement
  Service License Agreement



CaseLabs, The Next Generation of Enthusiast Cases

True Crypt - Free Open Source On - The Fly Encryption

Phisical Psience ΦΨ

StatCounter




RAID 3 vs. RAID 5 In A HPC Environment

TV Array - Japan


RAID 3 vs. RAID 5 in a HPC Environment by  

Text by Chris Wood

This paper will address the application and file system performance differences between RAID 3 and RAID 5 when used in a high performance computing environment. The reader is expected to have a working knowledge of RAID architecture and parity management for the different RAID types.

RAID 3:

RAID 3 typically organizes data by segmenting a user data record into either bit or byte sized chunks and evenly spreading the data across "N" drives in parallel. One of the drives is a parity drive. In this manner, every record that is accessed is delivered at the full media rate of the "N" drives that comprise the stripe group. The drawback is that every record I/O must access every drive in the stripe group.

Additionally, the stripe width chosen forces the user record length to be a multiple of what each disks underlying atomic format is. Assume each disk has an atomic 4 KByte format. Atomic format, in this sense, is the blocksize on the disk, (e.g. the data you get when you issue a single atomic read or write operation). An 8+P array would have 32 KByte logical blocks. A 5+P would have 20 KByte logical blocks and so on. You may not write to a RAID 3 except in full stripe logical blocks. This limits application design flexibility and limits the users ability to change the physical array characteristics.

Multiple Process Problems

For a single long request delivering a single stream of sequential data, RAID 3 is excellent. For multiple requests RAID 3 performs very poorly. A single record request (e.g. file system metadata) would tie up 100% of the disks and, more importantly, pull the disk heads away from the actual user data to read the metadata. Multiple requests from two or more active processes (jobs) will only serve to exacerbate this problem. The disk heads are being pulled from one user's data area to the other every time each process makes a request. Each head movement requires a "seek" and a disk latency. Often, this is on the order of 15-20 ms. Two processes typically require four (4) movements each time an I/O request ping-pongs between the two jobs; two metadata requests and two data transfer requests.

The resulting disk head positioning time of possibly up to 80 ms. coupled with the latencies built into any controller contribute to poor overall throughput. In RAID 3 mode, disk latency is not 1/2 a revolution like a single disk, but is statistically (N-1)/N, where N is the width of the stripe group. For example, an 8 way stripe would have a latency of (8-1)/8 or 7/8th of a revolution. Additionally, on a 1.063 GBit FC-AL loop or a 100 MBytes/sec. HIPPI channel, sustaining 75 MBytes/sec. data transfers, just two operations, each reading a megabyte of data with a 20 ms latency to switch between them, would result in an average throughput of about 30 MBytes/sec. (i.e. 20 ms position, 13.333 data transfer, 20 ms position, 13.333 transfer and so on). This example where we lost 45 MBytes/sec. of potential throughput assumed NO metadata fetches. If you add metadata I/O, assuming each metadata fetch was less than 64 KByte, then the throughput drops to between 10 and 15 MBytes/sec. Any host induced delays will further reduce this number. A good rule of thumb is to assume that a good RAID 3 controller can sustain about 60-65 unique short record I/Os per second (60 x 15 ms = 900 ms) or less. Long I/Os decrease this rate by the function of the transfer time involved for each request.

It is clear that RAID 3 is not well suited for either multiple process I/O (long or short) and especially not suited for any application that requires a high I/O per second rate (IOPS) with any degree of randomness. On the other hand, RAID 3 will deliver excellent performance for single process/single stream long sequential I/O requests.

Note: Long "skip sequential" processing (regular stride access) is a special case and is covered in the RAID 5 section for both types of arrays.

RAID 3 Performance Mitigation

Some users have proposed to mitigate the head movement problem and limited I/O per second rate of RAID 3 by making only very large requests and caching vast quantities of user data in main memory. This method has been shown to work in some cases where there is a high degree of locality of reference and data reuse, but is not very cost effective given the 100X cost delta between RAM and disk storage. Most users have realized that it is much more cost effective to use a flexible RAID 5 architecture rather than try to hide the shortcomings of the RAID 3 architecture. They would rather use expensive main memory for application space and data manipulation vs. allocating giant in-memory disk caches.

It is for these reasons of overall inflexibility and cost that the industry has gravitated to full function RAID 5 type devices in order to try to better handle both long and short requests, better multiplex many concurrent users and still deliver the very high data rates desired for long requests.

RAID 5:

RAID 5 devices interleave data on a logical block by block basis. (Logical block is usually defined in this sense as what the user application or file system sees and expects to perform atomic I/O on.) Logical blocks may be any size that is a multiple of the physical disk block. Most high performance RAID 5s allow the user to specify the logical blocksize as a multiple of the underlying physical disk block. This design allows a single logical block read to access only a single disk. A 16 block read would access all disks twice in a 8+P configuration. Parity is typically distributed (rotated) across all the disks to eliminate "hot spots" when accessing parity drives. The most common method is called "right orthogonal" parity wherein the parity block starts on the right most drive for track zero and rotates to the left on a track or cylinder basis. Numerous other schemes have been supported, but the right orthogonal scheme seems to be the most prevalent. MAXSTRAT's Gen5 family of storage servers uses a variant of this method.

Not all RAID 5s are alike...

Within the RAID 5 category, there are two fundamental types of RAID 5s:

  1. The "full function" variety where the array recognizes large requests and dynamically selects and reads as many disks in parallel as are required to answer the request, up to the physical width of the stripe group. This type of array typically has a separate lower interface port for every drive in the stripe group.
  2. The "entry level" model that can not perform parallel stripe reads, but has to read the data off each disk in a serial sequence. This type of array usually "daisy chains" multiple drives a small number of lower interfaces to save on cost at the expense of performance and parallelism.

These "entry" arrays are typically found in small SCSI arrays designed for PC servers and UNIX workstations that cannot use the data rate but want the data protection offered by parity arrays. Larger hosts require the "full function" variety since they want RAID 3 speed for long requests but also want the improved short, random block performance offered by RAID 5s.

Skip Sequential (Stride) Operations

Skip sequential operations (regular stride access) degrades the performance of RAID 3s directly in proportion to the "stride" length. If you read one block and skip two (stride of 3) then your data rate will be 1/3 of theoretical burst rate. A stride of 4 delivers 1/4th the rate and so on. A RAID 5, on the other hand will access every third disk in parallel and in the case of a stride of 3 mapped to a 8+P array would sustain the full data rate. (Disks 1, 4, & 7 hold requests 1, 2 & 3. Disks 2, 5 & 8 hold records 4, 5 & 6 and disks 3 & 6 hold records 7 & 8.) Other strides and other stripe widths map in a similar manner but may not always deliver 100% of the potential bandwidth due to resultant topology of the data.

The Read-Modify-Write Cycle

Short block writes are a special case for RAID 5s. In order to make the parity stay consistent when an entire stripe is not completely updated, (e.g. fewer records are written than the width of the stripe group) a function called "read-modify-write" must take place. This function requires that the old data block, the new data block and the old parity all be XOR'd together to create the new parity block. In the worst case, this can cause four (4) disk operations (2 reads/2 writes) to logically perform a single block write.

Modern RAID 5s tend to have built in caching to hold the old data in internal cache and possibly pre-read the old parity so that when the new write block is sent to the unit, all the necessary disk blocks are in cache and do not have to be fetched to do the three-way XOR. The MAXSTRAT Gen5 does this via 24 MBytes of write behind cache and up to 96 MBytes of read ahead/store-through cache. Obviously, the cache hit ratio is paramount to the performance of a RAID 5 when there are many short, random write updates.

As a second rule of thumb, high performance, full function RAID 5s are capable of sustaining between 500 and 2000 short random disk read operations a second. The Gen5 is at the high end of this range. Note: These numbers represent actual disk reads, not cache hits. Cache hits typically are higher. Short writes operate at about half this speed. Long writes (greater than a stripe width) perform in a manner similar to RAID 3 since parity can be generated on the fly and complete stripes, including new parity, can be written out in parallel.

Read to Write Ratios

Given the effect of the read-modify-write cycle, the actual read to write I/O ratio becomes important to any user considering a RAID 5 device. Most commercial processing has at least a 4-1 read to write ratio (IBM study) and scientific processing typically is higher on the read side.

Intermediate "swap" I/Os typically are large so the read-modify-write case is not of interest during job execution. Final output, now that visual vs. textual output is the rule, virtually eliminates short end of job writes. In summary, the read-modify-write overhead may be of less concern than previously thought.

Multiple Processes - RAID 5

Going back to our examples of single and multiple processes that we used above in the RAID 3 discussion, the case of a single process is very similar. In a full function RAID 5 array, a long request will cause all the disks to be accessed in parallel thus resulting in approximately the same data rate that RAID 3 offers. (Entry level RAID 5 arrays will be gated by the serial nature of their ability to access more than one disk.) Accessing metadata will tend to have a significantly lesser effect on overall performance since only one disk (vs. all the disks in the RAID 3 case) will receive a sudden command to move its read head to a different position. If this is not the disk that is next in the chain to transfer data for the long read, then the effect of head movement may be masked by the statistical location of the long read block transfer start location as measured within the time domain of the entire operation. This "mask" is statistically half of the stripe group width since we can never predict which disk in the stripe group actually holds the first block of the request.

In any given specific case, the mask time is both a function of file system design and metadata layout as well as the width of the stripe group. For instance, file system layout may or may not attempt to issue well formed (complete stripe group) I/O. To the extent that the metadata fetch operation can be masked, we recognize that we have, on the average, the time it takes to read the data off half of a stripe group to recover from a metadata induced disk head movement. This time is a disk rotational latency period plus the disk transfer time for one logical block of data. Assuming 64 KByte logical disk blocks and 7200 RPM drives, you would require about 11.7 ms. to perform this operation.

We can calculate that the average time required to return the disk head on the metadata drive (to a position where it will be able to access user data) will be approximately 12.2 ms. (4.2 ms. latency and 8 ms. average seek). Note that this number is less than the RAID 3 number since latency is always 1/2 a revolution for RAID 5 arrays vs. 7/8th of a revolution in the RAID 3 example. This value tells us that in most cases a fast RAID 5 controller can recover from a metadata fetch without impacting user data transfers. Unfortunately, our analysis above also shows us that this "mask" window is only available 50% of the time. However, half loaf is better than none!

Given the assumptions, we can then extrapolate that the "two process" performance hit will occur about half of the time on a well designed RAID 5. The data rate will not degrade to 33 MBytes/sec from 75 MBytes/sec, as in the RAID 3 example, but only to about 54 MBytes/sec. This is quite an improvement! A four process access will scale in the same manner. Experience has shown these assumptions to be correct.

Given the above, it is difficult to understand why anybody would choose RAID 3 architecture over a full function RAID 5 design except in the case where the user is virtually guaranteed that there will be only a single, long process accessing sequential data. The RAID 3 is so limited that it becomes a poor choice in this case.

Chris Wood
email: chrisw@maxstrat.com



BytePile.com has Dedicated Servers for your Hosting needs.
Please inquire at support@BytePile.com.





MySQL Database Powered Powered by Apache Full with PHP Modules Powered by Perl linux-logo
Last Update - 10 April 2012 All Rights Reserved. Copyright © 2002 BytePile.com Inc.