Geek projects, tidbits and things I'm working on at the moment…

Last year I had an odd issue when installing CentOS 6.0 on a Supermicro motherboard with an Intel PCIe quad port GB nic using an 82576 chip. I got by the initial kernel panic by using pci=noaer as a kernel option but later encountered really weird kernel panics regarding the Intel nic again which required an additional pcie_aspm=off in my grub file. Pretty odd stuff really since this was a server and I wasn’t worried about managing my power consumption since we’re a small company and I simply want the server to stay on. There’s a number of good articles about ASPM & Linux over on the Phoronix.com website which I found very interesting.

Initial bug report from RHEL: https://bugzilla.redhat.com/show_bug.cgi?id=704758

A number of Kernel parameters one can use at boot time to help troubleshoot things: http://www.kernel.org/doc/Documentation/kernel-parameters.txt

Initial Phoronix News article that got me wondering if Linux’s implementation of ASPM might be my problem: http://www.phoronix.com/scan.php?page=news_item&px=MTAwMjg

I can see how ASPM would be really important to laptop users sure, but when I’m installing a server I don’t need stuff shutting down on me when not in use. (Of course this is different for datacenters!)

Like most geeks I’ve got lots of files and a need for spacious network shares. However, as soon as you go over a 2TB partition, you’ll encounter problems w/ how to correctly partition a drive or RAID volume using the newer 4096 sector spec called simply Advanced Format. I initially stumbled upon a series of articles written by Roderick W. Smith over on IBM’s DeveloperWorks when searching how to properly check partition alignment under Linux for SSDs but quickly had an “Uh Oh” moment when I realized SSDs weren’t my only problem and I had in fact been flying blind on larger than 2TB partitions under Linux. Thankfully I didn’t have systems in production where this was a problem but I was literally building a system at work where this could bite me in the rear!

To bottom line it for you, if you’re using software RAID or a true HW RAID setup, you need to stripe your file system across the partition in accordance to the number of discs involved, type of RAID and the stripe size in KBs. My example below is strictly regarding Areca ARC RAID cards and might not be applicable to your setup.

I’m using an ARC-1680ix-24 w/ 4GB of ram on board. I’ve got 24 – 500GB HDs with one volume using all HDs in a RAID 6 raidset. So, for me the math is such when using EXT4 filesystem:

chunk size = 128KB (For Areca ARC-1680s it's your Stripe size.)
block size = 4KB (My desired partition is over 2TB. 6.5TB to be exact.)
stride = chunk / block 
       = 128KB / 4KB 
       = 32
Stripe-width = stride * (( # disks in RAID) - # of RAID parity disks)
             = 32 * ((using all 24 disks) - RAID 6 uses 2 parity disks)
             = 32 * (24 - 2)
             = 32 * 22
             = 704

so in other words, once I’ve created the basic partition on my device and I’m ready to format it as EXT4 I’ll end up using:

mkfs.ext4 -v -m .1 -b 4096 -E stride=32,stripe-width=704 /dev/sdd1

I’m not going to go into every detail of the line so please read RAID Setup over on Kernel.org for the skinny.

 
So from Soup to Nuts:

Use parted to create the partition table on my new volume & create a massive single partition at the right offset:

parted /dev/sdd

Once in the parted tool:

mklabel gpt
unit s
mkpart primary 2048s 100%
name 1 BFS
quit

Next, you’ll need to create an EXT4 filesystem inside your newly created partition from above:

mkfs.ext4 -v -m .1 -b 4096 -E stride=32,stripe-width=704 /dev/sdd1

Now you’ll want to add it to your fstab file so let’s grab the UUID from blkid next:

blkid -o list

And edit your /etc/fstab file so it will automount it at boot time

.
UUID=8e0a7d10-blah-blah-tomatoes-are-yummy-b4a0f6a13c15 /bfs ext4 defaults  1 2
.

Finally, create the mount point and mount it.

sudo mkdir /bfs
mount /bfs

Here’s a couple of links I found note worthy while going down this rabbit hole:
https://raid.wiki.kernel.org/index.php/RAID_setup#Calculation (shows the actual formula shown above & values)
http://insights.oetiker.ch/linux/raidoptimization.html (a great read)
http://ubuntuforums.org/showthread.php?t=1715375  (Rod weighs in with helpful advice in the Ubuntu forums)
http://www.gnu.org/software/parted/manual/parted.html  (parted’s man page over on GNU.ORG)
http://en.wikipedia.org/wiki/Ext4
http://lwn.net/Articles/377897/  (talks about 4KB sector size disks and let the panic ensue!)
http://whattheit.wordpress.com/2011/08/23/linux-aligning-partitions-to-a-hardware-raid-stripe/  (lots of theory but looks incomplete)
Last 4 links are from Rod Smith:
http://www.rodsbooks.com/gdisk/advice.html  (using gdisk but applicable to parted)
http://www.rodsbooks.com/gdisk/index.html  (main gdisk site!)
http://www.ibm.com/developerworks/linux/library/l-4kb-sector-disks/  (talks about the severe performance effects if you gloss over this stuff!)
http://www.ibm.com/developerworks/linux/library/l-gpt/  (good overview of GPT & understanding why they’re moving away from MBR)

(Personal Note: By finally posting this up on my blog I can close 10 tabs I’ve had open since middle of 2011!)

I’m currently working on building my own iSCSI SAN w/ SCST & a modified Linux kernel. Things are going pretty well but I needed to also bond out the 5 Gb nics so my VMHosts can maximize their connections. In Linux, you bond out an interface by creating a new network interface and telling the physical nics to register w/ that virtual network interface. There’s also 7 different types of bonding modes but the most common one is mode 6 or balance-ALB (active load balancing.)

If you’re using CentOS or RHEL like I am you cd to /etc/sysconfig/network-scripts/ and vi ifcfg-bond0.

Once you’ve got vi open, you dump the following in (this is an example only. Please change your IPADDR, NETMASK and NETWORK parameters as needed.)

DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
NETWORK=10.0.0.0
NETMASK=255.255.0.0
IPADDR=10.0.0.100
USERCTL=no

From there, you then modify your individual eth# interface config files so they point to the bonded device. As an example, here’s my /etc/sysconfig/network-scripts/ifcfg-eth0 interface file.

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no

And oh ya, don’t forget to modify your modprobe.conf file to specify the loading of the bond kernel module and what options it will needed @ run-time.

alias bond0 bonding 
options bond0 mode=6 miimon=100

Couple of useful links. There’s tons of good info on bonding w/ Linux. Keep in mind, some distros don’t want you to modify the individual interface config files so use their administrator control panel first if possible.

http://www.how2centos.com/how-to-set-up-network-bonding-with-centos-5x/
http://www.howtoforge.com/network_card_bonding_centos

I’m a real big fan of nVidia’s Linux drivers for their GPUs and have installed them numerous times with CentOS 5. I just recently installed them under Red hat 6 (RHEL 6) and happened to notice a warning by the nVidia driver that Nouveau was already installed and needed to be removed first. Further investigation reveals that Fedora & the latest version of RHEL 6 are including Nouveau at the initial ramdisk so not only do you need to create your own modprobe.conf blacklist file but you also need to add a line on the end of your grub.conf file to tell the ramdisk not to load Nouveau at boot up.

First, creating a modprobe blacklist file to prevent the kernel from loading the Nouveau kernel module. You could easily add blacklist nouveau to any ole modprobe.conf file but you never know when your linux distribution may overwrite your modified file w/ the next update. You’re better off to just create your own modprobe blacklist file and dropping it inside /etc/modprobe.d/ folder. I went ahead and followed some directions I stumbled upon on the interwebs and created a file called /etc/modprobe.d/disable-nouveau.conf.

I then added the following to that file…

blacklist nouveau options nouveau modeset=0

Once that was done, I then needed to modify my /etc/grub.conffile so the initial ramdisk would stop loading Nouveau too. To do this I added rdblacklist=nouveauto my kernel vmlinuz line like such…

.
.
.
kernel /vmlinuz-2.6.32-71.14.1.el6.x86_64 ro root=UUID=209502fb-f4f0-4755-a275-de807916fb76 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us crashkernel=128M rhgb rdblacklist=nouveau
.
.
.
.

Oh ya, I always remove the “quiet” at the end of the line too. Something about watching all of the modules load…

Here’s a couple links as usual:

http://us.download.nvidia.com/XFree86/Linux-x86_64/260.19.36/README/commonproblems.html
http://www.if-not-true-then-false.com/2009/howto-install-nvidia-driver-on-fedora-and-just-disable-the-nouveau-driver/