PCIe Flash – Part 1

One of the side-benefits of being around for a few decades is the joy of reminiscing about old technology and foibles of the past with friends from said era.

I ran into an old friend I have not seen in over ten years – one thing led to another and we were on the third round of Guinness and somehow the talk veered to the exciting world of flash memory.

And flash trends. Like…
…why a segment of the trade-press have fallen so much in love with PCI-e attached flash memory over the last few years. Pause to consider this for a moment of why PCI-e attached flash is considered the next best thing in computer architecture since Apple…

A. It is PCI-e attached – therefore, it is blindingly fast. Since, everybody knows, being outside the server is “slow”
B. It is on PCI-e – and NOT SATA (shudder – how plebian!). Therefore, it’s not cheap and common. Since everybody KNOWS SATA is slow.

Let’s look at the  two types of PCI-e SSD cards:

  • Single-stage PCI-e SSD: Cards with a PCI-e attached controller directly attached to flash chips
  • Dual-stage PCI-e SSD: Cards with a SAS HBA controller attached to multiple sub-systems – each with a traditional SSD controller attached to flash chips

You can find equally happy users for each type. You can also find users that are blissfully unaware that they have a dual-stage PCI SSD as opposed to single-stage.

Hey – so what if almost 10% of that premium unleaded you are buying @ $5/Gallon for your BMW is essentially industrial corn-hooch (a few cents a gallon in its native form) – it feels like the Ultimate Driving Machine – no?

Single-stage PCI-e SSD

This genre of products essentially started with Fusion-IO’s products.  These types of cards have a controller that attaches to PCI-e on one side and directly to flash devices on the other side.

Take a look (http://www.fusionio.com/platforms/iodrive/)  – 

It has become a true workhorse during its existence.

Another one (this one from Micron)

Again – the same lineup. PCI-e connects to controller that connects to Flash.

The next one here is from Virident – (http://www.virident.com/products/flashmax/)

This card not only has a very impressive list of specifications – it looks pretty stylish, too.

Here are some pros and cons of single-stage PCI-e SSDs:



Simplistic Controller Design – Most of the compute and data complexity can be left to the CPU. This is an advantage to the vendor and not necessarily to the end user Server RAM based buffering – These cards use server (host) memory and host CPU cycles to do wear-leveling and as page and data buffers.  The vendor argument here is that the host typically has enough RAM and CPU cycles and typically nobody misses them!
Wide parallel stripe – The controller can operate over much wider channels of flash chips Flash Chip changes force controller spin – As you go from one gen to another – it requires a different controller

Dual-stage PCI-e SSD

This type of PCI-e SSDs usually use a SAS Controller as first stage and then hang multiple SSD subsystems from that SAS/SATA controller. The SSD subsystem functionally looks like bare SATA SSD. That’s the second stage.

LSI SAS controllers are very popular as first stage. They have been around a long time, have great driver support and have mature interfaces up to x8 PCI-Gen2.

What about the second stage controller? Marvel and Sandforce are the two top popular choices. Check out the nifty Sun/Oracle F20 PCI-e Card. This card has a lot of working experience in Oracle environments. The Marvel controllers are visible on the SSD subsystems.

Here is another example from the good folks of OCZ –

This is a well-executed design with four SSD modules.

Below are some advantages of dual-stage designs:

  1. Good DMA and RAID Performance – For users that want to run RAID1 inside ONE single card the first stage controllers provide proven performance. Also, the typical SAS Controllers used in this stage have long and evolved history.
  2. Parallel Operation in Second Stage – The SSD controllers in the second stage operate only smaller numbers of chips at a time and work in parallel.
  3. Better scaling – Larger number of SSD modules can be attached in the second stage, potentially providing scalable performance for large capacity modules.

Coming soon –

The myths, the stats and fables of PCI-e Flash… does it really matter if you have a single or dual-stage card?


One thought on “PCIe Flash – Part 1

  1. Pingback: GridIron CTO Talks About PCIe Flash « Sattler's Online Diary

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s