PowerEdge R510 with PERC H200/H700 with ZFS

[Geoff Nordli]

Anyone have any experience with a R510 with the PERC H200/H700 controller
with ZFS?

My perception is that Dell doesn't play well with OpenSolaris.

Thanks,

Geoff

[Geoff Nordli]

-----Original Message-----
Sent: Saturday, August 07, 2010 8:10 AM
To: Geoff Nordli
Subject: Re: [zfs-discuss] PowerEdge R510 with PERC H200/H700 with ZFS

Post by Geoff Nordli
Anyone have any experience with a R510 with the PERC H200/H700
controller with ZFS?

Not that particular setup, but I do run Solaris on a Precision 690 with

PERC 6i

controllers.

Post by Geoff Nordli
My perception is that Dell doesn't play well with OpenSolaris.

What makes you say that? I've run Solaris on quite a few Dell boxes and

have

never had any issues.
-brian
--

Hi Brian.

I am glad to hear that, because I would prefer to use a dell box.

Is there a JBOD mode with the PERC 6i?

It is funny how sometimes one forms these views as you gather information.

Geoff

[Terry Hull]

Date: Sat, 7 Aug 2010 08:39:46 -0700
Subject: Re: [zfs-discuss] PowerEdge R510 with PERC H200/H700 with ZFS

-----Original Message-----
Sent: Saturday, August 07, 2010 8:10 AM
To: Geoff Nordli
Subject: Re: [zfs-discuss] PowerEdge R510 with PERC H200/H700 with ZFS

Post by Geoff Nordli
Anyone have any experience with a R510 with the PERC H200/H700
controller with ZFS?

Not that particular setup, but I do run Solaris on a Precision 690 with

PERC 6i

controllers.

Post by Geoff Nordli
My perception is that Dell doesn't play well with OpenSolaris.

What makes you say that? I've run Solaris on quite a few Dell boxes and

have

never had any issues.
-brian
--

Hi Brian.
I am glad to hear that, because I would prefer to use a dell box.
Is there a JBOD mode with the PERC 6i?
It is funny how sometimes one forms these views as you gather information.
Geoff

It is just that lots of the PERC controllers do not do JBOD very well. I've
done it several times making a RAID 0 for each drive. Unfortunately, that
means the server has lots of RAID hardware that is not utilized very well.
Also, ZFS loves to see lots of spindles, and Dell boxes tend not to have
lots of drive bays in comparison to what you can build at a given price
point. Of course then you have warranty / service issues to consider.

--
Terry Hull
Network Resource Group, Inc.

[Geoff Nordli]

-----Original Message-----
Sent: Saturday, August 07, 2010 1:12 PM

Date: Sat, 7 Aug 2010 08:39:46 -0700

Sent: Saturday, August 07, 2010 8:10 AM

Post by Geoff Nordli
Anyone have any experience with a R510 with the PERC H200/H700
controller with ZFS?

Not that particular setup, but I do run Solaris on a Precision 690 with

PERC 6i

controllers.

Post by Geoff Nordli
My perception is that Dell doesn't play well with OpenSolaris.

What makes you say that? I've run Solaris on quite a few Dell boxes and

have

never had any issues.
-brian

Hi Brian.
I am glad to hear that, because I would prefer to use a dell box.
Is there a JBOD mode with the PERC 6i?
It is funny how sometimes one forms these views as you gather

information.

Geoff

It is just that lots of the PERC controllers do not do JBOD very well.

I've done it

several times making a RAID 0 for each drive. Unfortunately, that means

the

server has lots of RAID hardware that is not utilized very well.
Also, ZFS loves to see lots of spindles, and Dell boxes tend not to have

lots of

drive bays in comparison to what you can build at a given price
point. Of course then you have warranty / service issues to consider.
--
Terry Hull

Terry, you are right, the part that was really missing with the Dell was the
lack of spindles. It seems the R510 can have up to 12 spindles.

The online configurator only allows you to select SLC SSDs, which are a lot
more expensive than the MLC versions. It would be nice to do MLC since that
works fine for L2ARC.

I believe they have an onboard SD flash connector too. It would be great to
be able to install the base OS onto a flash card and not waste two drives.

Are you using the Broadcom or Intel NICs?

For sure the benefit of buying name brand is the warranty/service side of
things, which is important to me. I don't want to spend any time
worrying/fixing boxes.

Thanks,

Geoff

[Terry Hull]

Date: Sat, 7 Aug 2010 14:11:37 -0700
Subject: RE: [zfs-discuss] PowerEdge R510 with PERC H200/H700 with ZFS

[stuff deleted]

Terry, you are right, the part that was really missing with the Dell was the
lack of spindles. It seems the R510 can have up to 12 spindles.
The online configurator only allows you to select SLC SSDs, which are a lot
more expensive than the MLC versions. It would be nice to do MLC since that
works fine for L2ARC.
I believe they have an onboard SD flash connector too. It would be great to
be able to install the base OS onto a flash card and not waste two drives.
Are you using the Broadcom or Intel NICs?
For sure the benefit of buying name brand is the warranty/service side of
things, which is important to me. I don't want to spend any time
worrying/fixing boxes.

I understand that one.

I have been using both Intel and Broadcom NICs successfully. My gut tells
me I like the Intel better, but I can't say that is because I have had
trouble with the Broadcom. It is just a personal preference that I probably
can't justify.

--
Terry

[v***@gmail.com]

I've been running OpenSolaris on my Dell Precision Workstation T7500 for 9 months, and it works great. It's my default desktop operating system, and I've had zero problems with hardware compatibility.

I also have installed EON 0.600 on my Dell PowerEdge T410 (not so different from your R510). A few words of caution:

1. Beware of the onboard controllers. The "RAID" controller on that motherboard only works in Windows; neither Linux nor OpenSolaris can recognize drives attached to it at all. So I was stuck running in "ATA" mode at the beginning, which is awful in terms of performance.

2. I'd also recommend avoiding the PERC cards, in particular since it makes drives attached to it impossible to transport to another system. Instead, I use the SAS 6i/R controller. That's built into the motherboard on the PW T7500, and I got one separately for the PE T410. That works well, and is completely fine with OpenSolaris. I'd recommend those, because then you can be sure to get the cabling from Dell (which in the case of the PowerEdge, was completely nonstandard). And if the card fails, they'll replace it ASAP, which isn't necessarily the case with other vendors' cards.

So aside from the RAID controller and cabling issues on the PE T410, I've had nothing but good experiences in terms of Dell Precision workstations and PowerEdge servers, running OpenSolaris.

[Lasse Osterild]

Post by v***@gmail.com
2. I'd also recommend avoiding the PERC cards, in particular since it makes drives attached to it impossible to transport to another system. Instead, I use the SAS 6i/R controller. That's built into the motherboard on the PW T7500, and I got one separately for the PE T410. That works well, and is completely fine with OpenSolaris. I'd recommend those, because then you can be sure to get the cabling from Dell (which in the case of the PowerEdge, was completely nonstandard). And if the card fails, they'll replace it ASAP, which isn't necessarily the case with other vendors' cards.

The PERC H200's have full pass-through, completely bypassing the raid controller. http://tinyurl.com/yl6zupj

- Lasse

[Edward Ned Harvey]

Post by v***@gmail.com
2. I'd also recommend avoiding the PERC cards, in particular since it
makes drives attached to it impossible to transport to another system.

This is not true. You can transport these drives to another system with
either a Perc, or any other form of LSI MegaRaid.

What's more, I'll advise staying away from generic SATA, because they're
impossible to transport to any system which is more advanced than generic
SATA. ;-) See? It works two-ways. And the same is true of all the other
HBA's out there which have any intelligence in them, too ... RealTek and so
forth. It's all not compatible with each other. Realtek drives have to
stay with realtek controllers. LSI drives with LSI controllers. Generic
SATA with generic controllers. Etc.

One advantage of using the Perc, or some other HBA with battery backed
write-back cache, is a really big performance benefit for sync writes...
But not quite as big a benefit as using the dedicated log device ... But the
HBA is far lower cost than the SSD solution, and doesn't take up a drive
slot (or two).

[Mariusz]

Hi.
I install solaris 10 x86 on PowerEdge R510 with PERC H700 without problem.
8HDD configured with RAID 6.
Only question is how to monitor this controller?

Do you have any tools which allow you to monitor this controller?
Get HDD status.

Thank you for help.

PS.
I know this is OpenSolaris not solaris group but maybe I get help here.

[Kyle McDonald]

It is just that lots of the PERC controllers do not do JBOD very well. I've
done it several times making a RAID 0 for each drive. Unfortunately, that
means the server has lots of RAID hardware that is not utilized very well.

Doing that lets you use the cache, which is the only part of the RAID
HW that I'd worry about wasting.

Also, ZFS loves to see lots of spindles, and Dell boxes tend not to have
lots of drive bays in comparison to what you can build at a given price
point.

I've found the R515 (the R510's cousing with AMD processors) to be
very interesting in this regard. It has many more drive bays than most
Dell boxes.

I've also priced out the IBM x3630 M3, even more drive bays in this one.
for about %20 more.

Of course then you have warranty / service issues to consider.

I don't know what you're needs are, but I found dell's 5yr onsite 10x5
NBD support to be priced very attractively. But I can live with a
machine being down till the next day, or through a weekend.

-Kyle

--
Terry Hull
Network Resource Group, Inc.
_______________________________________________
zfs-discuss mailing list
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

[Edward Ned Harvey]

Post by Geoff Nordli
Anyone have any experience with a R510 with the PERC H200/H700
controller
with ZFS?
My perception is that Dell doesn't play well with OpenSolaris.

I have an R710... Not quite the same, but similar.

When you say "doesn't play nice with opensolaris," you're hitting a grain of
truth but blowing it out of proportion. It's true that some stuff might not
brainlessly work the way you'd like. And it's true that you'll be lacking
some functionality that you'd probably like. But it's easy to quantify and
describe, and easy to describe the options of dealing with it, too:

#1 Optical drive, by default, in AHCI mode. For some reason, apparently
using a chipset that osol can't handle. So put the optical drive into ATA
or Legacy mode before booting. Or use an external optical drive for
installation.

#2 On my R710, the perc driver wasn't included in the sol10 DVD by default,
so I had to download a special one from support.dell.com, and hit "5 -
insert driver disk" or whatever, during sol10 installation. This was not an
issue for osol installation.

#3 OMSA provides the web-based gui to manage your raid card. Not available
in sol10/osol. Instead, find the appropriate MegaCLI for your device ...
which is really tough to do. This is needed if you want the ability to
replace a failed drive without rebooting.

Once you overcome these things, it works great.

[Geoff Nordli]

Sent: Sunday, August 08, 2010 8:34 PM

Post by Geoff Nordli
Anyone have any experience with a R510 with the PERC H200/H700
controller with ZFS?
My perception is that Dell doesn't play well with OpenSolaris.

I have an R710... Not quite the same, but similar.
When you say "doesn't play nice with opensolaris," you're hitting a grain

of truth

but blowing it out of proportion. It's true that some stuff might not

brainlessly

work the way you'd like. And it's true that you'll be lacking some

functionality

that you'd probably like. But it's easy to quantify and describe, and easy

to

#1 Optical drive, by default, in AHCI mode. For some reason, apparently

using a

chipset that osol can't handle. So put the optical drive into ATA or

Legacy mode

before booting. Or use an external optical drive for installation.
#2 On my R710, the perc driver wasn't included in the sol10 DVD by

default, so I

had to download a special one from support.dell.com, and hit "5 - insert

driver

disk" or whatever, during sol10 installation. This was not an issue for

osol

installation.
#3 OMSA provides the web-based gui to manage your raid card. Not

available in

sol10/osol. Instead, find the appropriate MegaCLI for your device ...
which is really tough to do. This is needed if you want the ability to

replace a

failed drive without rebooting.
Once you overcome these things, it works great.

Thanks Edward.

What did you end up using for the L2ARC? The SSDs shown in the online
configurator are SLC based.

Did you use the Broadcom or the optional Intel NIC?

Geoff

[Edward Ned Harvey]

Post by Geoff Nordli

Post by Edward Ned Harvey
I have an R710... Not quite the same, but similar.

Thanks Edward.
What did you end up using for the L2ARC? The SSDs shown in the online
configurator are SLC based.
Did you use the Broadcom or the optional Intel NIC?

We opted not to use L2ARC in the dell server. Because the Sun server is our
primary server, and the Dell is only for backup. That being said, for most
purposes I would usually recommend investing in a log device before a cache
device, and that's what we're using on the Sun server. But we did some
performance benchmarks on the Dell including the SSD before putting it into
production.

Do you have something against SLC?

We got the broadcom based nic. I think it was the default. No thought was
put into this decision - If the intel nic has some advantage to offer, you
might consider it. In my experience, as long as a nic works, there's very
little difference from one to the next.

[Geoff Nordli]

Sent: Monday, August 09, 2010 4:25 AM

Post by Geoff Nordli

Post by Edward Ned Harvey
I have an R710... Not quite the same, but similar.

Thanks Edward.
What did you end up using for the L2ARC? The SSDs shown in the online
configurator are SLC based.
Did you use the Broadcom or the optional Intel NIC?

We opted not to use L2ARC in the dell server. Because the Sun server is our
primary server, and the Dell is only for backup. That being said, for most
purposes I would usually recommend investing in a log device before a cache
device, and that's what we're using on the Sun server. But we did some
performance benchmarks on the Dell including the SSD before putting it into
production.
Do you have something against SLC?

Cost of SLC is considerably more than an MLC version, which I don't think is
justified for an L2ARC.

This server is going to be used to host VMs via iSCSI in a lab/test system.

We are going to use dedup with the table stored on SSD.

The strategy I am looking at is using slower 7200 RPM disks to hold rarely
used data, and then get the "working" set running from SSDs.

We got the broadcom based nic. I think it was the default. No thought was

put

into this decision - If the intel nic has some advantage to offer, you

might

consider it. In my experience, as long as a nic works, there's very little

difference

from one to the next.

Thanks for the feedback.

Geoff