ZFS Server build notes

ZFS Filesystem backup

Hostname	xxxx
OS	centos6.2 64
10GB	Yes
MTU	9000
IP Address	xx.xx.xx.xx
VLAN	backup & archive
Location	xxxxx

Storage Layout

Device	Size	Type	Compression	Comments
Raid6_sys	120GB	Raid6	No	Hardware Raid on Adaptec Raid Card - 1 hot spare
Raid_data	2.5TB	Raid6	No	Hardware Raid on Adaptec Raid Card - 1 hot spare
SATA	1TB	EXT4	No	Clone of System
RaidZFS	XXTB	ZFS	Yes	Main Backup Storage

Installation Notes

• Centos requires the Adaptec module to be installed during the install process in order to see the local raid

http://ask.adaptec.com/app/answers/detail/a_id/17150

• The Mellonx 10G card needs a drive compiled and installed (make sure gcc, make, rpm_build_tools etc is installed!). The install files are in the homedir of root.
• BIOS is password locked with password
• Adaptec Raid Controller is password locked with password
• Full yum update performed - 21.11.2013
• Mellonx En driver doesn't load on it's own at start up. Fix this by adding the following to /etc/rc.local - modprobe mlx4_en
• Installed Megacli to talk to the LSI raid controller - rpm -ivh MegaCli-8.07.08-1.noarch.rpm
• Disable selinux - it's not supported with zfs at the moment.

LSI Controller Admin

We will use MegaCLI to communicate with the LSI controller

https://calomel.org/megacli_lsi_commands.html

 lsi.sh help

The controller is set to mail alerts. This is done via the root crontab

00   */2  *   *   *   /usr/local/bin/lsi.sh checkNemail

View enclosures

/opt/MegaRAID/MegaCli/MegaCli64 -EncInfo -aALL

Creating the raid0 devices to present each disk to the OS

i=0; while [ $i -le 23 ] ; do /opt/MegaRAID/MegaCli/MegaCli64 -cfgldadd -r0[8:${i}] WB RA Cached CachedBadBBU -strpsz512 -a0 -NoLog ; i=`expr $i + 1`; done
i=0; while [ $i -le 23 ] ; do /opt/MegaRAID/MegaCli/MegaCli64 -cfgldadd -r0[9:${i}] WB RA Cached CachedBadBBU -strpsz512 -a0 -NoLog ; i=`expr $i + 1`; done

Note that the enclosure ID's are 8 and 9. We can find this doing.

/opt/MegaRAID/MegaCli/MegaCli64 -EncInfo -aALL

Check the status of the drives

lsi.sh status
lsi.sh drives

Information on a disk (in this case Enclosure 8, disk 20)

/opt/MegaRAID/MegaCli/MegaCli64 -PDInfo -PhysDrv [8:20] -a0

Rebuild disk (in this case Enclosure 8, disk 20)

/opt/MegaRAID/MegaCli/MegaCli64 -PDRbld -Start -PhysDrv [8:20] -a0

Misc. commands

Enable controller alarm

/opt/MegaRAID/MegaCli/MegaCli64 -AdpSetProp AlarmEnbl -aALL

Disable controller alarm

/opt/MegaRAID/MegaCli/MegaCli64 -AdpSetProp AlarmDsbl -aALL

Good cheat sheet of MegaCLI commands: http://www.damtp.cam.ac.uk/internal/computing/docs/public/megacli_raid_lsi.html

Install ZFS

yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release-1-2.el6.noarch.rpm
yum install dkms gcc make kernel-devel perl
yum install spl zfs
chkconfig zfs on

Create ZFS Raid

zpool create -f tank /dev/sdf /dev/sdg /dev/sdh /dev/sdk /dev/sdj /dev/sdi /dev/sdl /dev/sdn /dev/sdm /dev/sdp /dev/sdo /dev/sdq /dev/sdr /dev/sds /dev/sdt /dev/sdu /dev/sdv /dev/sdx /dev/sdw /dev/sdy /dev/sdz /dev/sdaa /dev/sdab /dev/sdac /dev/sdad /dev/sdae /dev/sdaf /dev/sdai /dev/sdah /dev/sdaj /dev/sdag /dev/sdal /dev/sdam /dev/sdak /dev/sdan /dev/sdaq /dev/sdao /dev/sdar /dev/sdat /dev/sdas /dev/sdav /dev/sdap /dev/sdau /dev/sdaw /dev/sdax /dev/sday
zpool add tank spare /dev/sdd /dev/sdb
zfs create tank/projects
zfs set compression=lzjb tank/projects
zfs set dedup=on tank/projects
zfs set atime=off tank
zfs set atime=off tank/projects

What is looks like

# zfs list
NAME            USED  AVAIL  REFER  MOUNTPOINT
tank           1.38M   164T   144K  /tank
tank/projects   136K   164T   136K  /tank/projects

Common Tools

Check dedupe ratio

zpool get dedupratio tank

Snapshots

Running from cron

# ls -l /etc/cron.* |grep zfs
-rw-r--r--. 1 root root  79 Nov 21 17:24 zfs-auto-snapshot.cron.daily
-rw-r--r--. 1 root root  80 Nov 21 17:24 zfs-auto-snapshot.cron.hourly
-rw-r--r--. 1 root root 81 Nov 21 17:26 zfs-auto-snapshot.cron.monthly
-rw-r--r--. 1 root root 79 Nov 21 17:26 zfs-auto-snapshot.cron.weekly

These crons run

/usr/local/bin/zfs-auto-snapshot

Current snapshot schedule is to keep 12 months of snapshots. These are located in .zfs of each folder.

Display snapshots config

zpool get listsnapshots tank

List snapshots

zfs list -r -t snapshot -o name,creation tank  
zfs list -t snapshot

List space

zfs list -o space

Advanced reading https://pthree.org/2012/12/17/zfs-administration-part-x-creating-filesystems/

Devices

Disk /dev/sdd: 4000.2 GB, 4000225165312 bytes
Disk /dev/sde: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdf: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdg: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdh: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdk: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdj: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdi: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdl: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdn: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdm: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdp: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdo: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdq: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdr: 4000.2 GB, 4000225165312 bytes
Disk /dev/sds: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdt: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdu: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdv: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdx: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdw: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdy: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdz: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdaa: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdab: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdac: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdad: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdae: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdaf: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdai: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdah: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdaj: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdag: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdal: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdam: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdak: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdan: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdaq: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdao: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdar: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdat: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdas: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdav: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdap: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdau: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdaw: 4000.2 GB, 4000225165312 bytes
Disk /dev/sdax: 4000.2 GB, 4000225165312 bytes
Disk /dev/sday: 4000.2 GB, 4000225165312 bytes)

Create Bootable USB from ISO on OSX

I found this page very useful

GNG Electric Mid Motor bearing replacement - more info

Changing the bearings on my two week old GNGElectric bicycle motor.

Stereo xorg.conf - Centos playback machine to a Barco DP90 using SDI

=== Playback Hardware ====

The playback machine in the theater is called rodoric. It has 2 SSD 500GB drives, hardware raided.

  /dev/sda1             937G  614G  277G  69% /ssd

The graphics card is a Nvidia 4000. It has 3 connections, only two are in use at one time.

  * In Mono mode it will use a DVI to the KVM extender, and a DVI to a DVI-Ramp which in turn converts it's signal to SDI into the Barco (Input A)
  * In Stereo mode it will output Left and Right eye signals using both display ports through DVI into a single DVI-Ramp which in turn converts it's signals to SDI into the Barco (SDI input A and B)

Users can switch between Stereo and Mono using the xorgswitch file below.

=== Xorg Switch ===

Type xorgswitch in a terminal to switch between Stereo and Mono Display modes
xorgswitch

#!/bin/bash

#switch the xorg files for stereo and regular playback
#v1.0 - Dylan Penhale 04-11-2013


X11PATH=/etc/X11
XORG3D=xorg.conf.rv.3d.WORKING.04.11.2013
XORG2D=xorg.conf.safe.31.10.2013

echo "1. Stereo Mode - Barco only"
echo "2. Mono - Barco and LCD display"
echo -n "Enter: [1 or 2]:"
read choice

if [ $choice -eq 1 ]; then
        echo "Switching xorg to Stereo mode"
        rm -rf $X11PATH/xorg.conf
        cp $X11PATH/$XORG3D $X11PATH/xorg.conf
        echo "Now restart X Server {Cntl+Alt+Backspace)"
        exit 0
else
  if [ $choice -eq 2 ]; then
        echo "Switching xorg to Mono mode"
        rm -rf $X11PATH/xorg.conf
        cp $X11PATH/$XORG2D $X11PATH/xorg.conf
        echo "Now restart X Server {Cntl+Alt+Backspace)"
        exit 0
  fi
fi

done

=== Stereo ===

/etc/X11/xorg.conf.rv.3d.WORKING.04.11.2013

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
    Option         "Xinerama" "0"
EndSection

Section "Files"
    FontPath        "/usr/share/fonts/default/Type1"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/input/mice"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"

    # generated from data in "/etc/sysconfig/keyboard"
    Identifier     "Keyboard0"
    Driver         "keyboard"
    Option         "XkbLayout" "us"
    Option         "XkbModel" "pc105"
EndSection

Section "Monitor"

    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "OEM DVI-Ramp2"
    HorizSync       1.0 - 250.0
    VertRefresh     10.0 - 250.0
    ModeLine       "2048x1080_24.00" 66.02 2048 2080 2280 2512 1080 1081 1084 1095 -hsync +vsync
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro 4000"
EndSection

Section "Screen"

# Removed Option "Stereo" "4"
# Removed Option "metamodes" "DFP-1: 2048x1080_24 +0+0, DFP-2: 2048x1080_24 +0+0; DFP-1: nvidia-auto-select +0+0, DFP-2: nvidia-auto-select +0+0"
# Removed Option "metamodes" "DFP-0: nvidia-auto-select +64+0, DFP-2: 2048x1080_24.00 +0+60; DFP-2: nvidia-auto-select +0+0"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option   "CIOverlay" "on"
    Option   "Overlay" "on"
    Option         "TwinViewOrientation" "Clone"
    Option         "ExactModeTimingsDVI" "TRUE"
    Option         "FlatPanelProperties" "Scaling = Native"
    Option         "TwinView" "1"
    Option         "nvidiaXineramaInfoOrder" "DFP-0"
    Option         "Stereo" "4"
    Option         "metamodes" "DFP-1: 2048x1080_24.00 +0+0, DFP-2: 2048x1080_24.00 +0+0; DFP-2: nvidia-auto-select +0+0"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection
Section "Extensions"
   Option "Composite" "Disable"
Endsection

=== Mono ===

/etc/X11/xorg.conf.safe.31.10.2013

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
    Option         "Xinerama" "0"
EndSection

Section "Files"
    FontPath        "/usr/share/fonts/default/Type1"
EndSection

Section "InputDevice"
    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/input/mice"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"
    # generated from data in "/etc/sysconfig/keyboard"
    Identifier     "Keyboard0"
    Driver         "keyboard"
    Option         "XkbLayout" "us"
    Option         "XkbModel" "pc105"
EndSection

Section "Monitor"
    # HorizSync source: edid, VertRefresh source: edid
    Identifier     "Monitor0"
    VendorName     "Unknown"
    ModelName      "Eizo S2411W"
    HorizSync       31.0 - 76.0
    VertRefresh     59.0 - 61.0
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro 4000"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "Stereo" "0"
    Option         "nvidiaXineramaInfoOrder" "DFP-0"
    Option         "metamodes" "DFP-0: 1920x1200 +0+0, DFP-2: 2048x1080_24 +1920+60"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Howto repair XT shifters.

My first tech blog

Short XC Single Track Loop - Belrose 720P

Quarry Track (Sydney) MTB

Playing with the new helmet Cam... The mount needs some work!

Find duplicates in bash

find_dupes.sh

 #! /bin/bash
OUTF=rem-duplicates.sh;
echo "#! /bin/sh" > $OUTF;
echo ""                >> $OUTF;
find "$@" -type f -print0 | xargs -0 -n1 md5sum | sort --key=1,32 | uniq -w 32 -d --all-repeated=separate | sed -r 's/^[0-9a-f]*( )*//;s/([^a-zA-Z0-9./_-])/\\\1/g;s/(.+)/#rm \1/' >> $OUTF;
chmod a+x $OUTF

Run the script:

./find_dupes.sh /path/to/check .\

Which will output a file called rem-duplicates.sh which can then be edited as you wish.

I stole this from here. Credit to Marco Fioretti, who in turn used code from J. Elonen

LED lights on my Vespa

Getting started with Linux in VFX

Useful entry level reading for someone that's never used linux before....

http://www.creativecrash.com/tutorials/the-unix-shell-for-vfx-artists#tabs

VFX pipeline diagram - real world example

Taken from a pipeline design meeting. 

Credit to Simon Maddison I think

How to setup a reverse ssh tunnel

Ever wanted to know how you can ssh into your work desktop/server? Well do not worry any longer.....

1. All you need to do is create a reverse tunnel from your Work Desktop/Server to your home server.
   
   
   • ssh -nNT -R 5000:local_server:22 username@remote_server
   • Now from the remote_server run this
     ssh -p5000 localhost.... now we are in our work desktop/server.
2. Lets brake this down a bit..
   
   
   • The -n option Redirects stdin from /dev/null. This must be used when ssh is run in the background.
   • The -N option does not execute a remote command. This is useful for just forwarding ports.
   • The -T option disables pseudo-tty allocation.
   • The -R option does tha job of setting up the reverse tunnel.
   • Port 5000 is the port that will be listening on the remote_server (this can be any random port over 1024, if you want to run this as a non-root user).
   • local_server is the desktop/server that you are creating the connection from.
   • Port 22 is the port that you are making the ssh connection to.
   • user_name@remote_server is where you are making the ssh connection to for the reverse tunnel.
We need to make sure we keep this connection open.
• In /etc/ssh/sshd_config we need to make sure this is set TCPKeepAlive yes.\

This page was stolen from somewhere else, I can't remember where. Sorry to the original author..

Running a script as root

The problem:

In theory, running a script as it's owner (in our case root) is simple, all you have to do is set the 'setuid' bit (sometimes mistakenly called the 'sticky' bit) by using 'chmod 4755' (or chmod 4***, depending on what mode you need). The problem is, that the 'setuid' bit functionality has been disabed on most Linux based distros for security reasons. This affects only scripts, not compiled code which should still work.

Solutions:

binary wrapper

The 'setuid' bit still works for compiled files. So what you can do is write a little c wrapper that will execute the script you want to run as root.

#include 
#include 
#include 
#include 

using namespace std;

int main(int argc, char* argv[]) {
    setuid( 0 );
    string cmd = "/opt/local/cmd.sh";
    for (int i = 1; i < argc; i++) {
        string arg (argv[i]);
        cmd += " ";
        cmd += arg;
    }
    int exitcode;
    exitcode = system(cmd.c_str());
    return WEXITSTATUS(exitcode);
}

Safe this as 'runscript.c', then compile and do the following:

gcc runscript.c -o runscript
sudo chown root:root runscript
sudo chmod 4750 runscript

You should now be able to run this command as root.

sudo wrapper

Add the following line to your sudoers file, with 'group' being the group you want to give access to your script.

group    ALL = NOPASSWD: /path/to/script

Then write a wrapper, executing you script with sudo.

sudo /path/to/script

Also your script will have to perform a 'setuid(0)' to make sure that all child processes are executed as root.

A couple of things to note:

• it's not enough to do 'chmod 4***' on the binary, you also have to use 'setuid' inside the code.
• an additional problem on Mac OS X is, that by default the 'setuid' bit is disabled on NFS mounts.

x forward session with xdmcp

XDMCP allows logging into a remote host, but displaying the desktop on your own host.

1) edit /etc.gdm/custom.conf

[xdmcp]
Enable=true

[security]
DisallowTCP=false
AllowRemoteRoot=true

2) restart X (telinit 3; telinit 5)

3) login as root and run:

gdmsetup

4) On the remote tab, change 'style' to 'same as local'

5) On the security tab, tick 'allow local system administrator login' and 'allow local system administrator login'

6) restart X

To test, log off your host, and press F10 in the login GDM login window. You can enter the name of the remote host to login too here.

Forward user x sessions

When logging into a remote host as root, you lose the ability to forward x sessions after an su to another user.

To allow x-forwarding after you perform an su, you need to request the cookie for the current server:

ssh root@somehost

xauth list

returns:

 somehost.somedomain/unix:10  MIT-MAGIC-COOKIE-1  d9fc3512df83a70ea8054dff21f58e28

Now, su to the user:
 

su - someuser

xauth add somehost.somedomain/unix:10  MIT-MAGIC-COOKIE-1  d9fc3512df83a70ea8054dff21f58e28

now x apps will forward correctly - test with xterm.

20MB Boot Floppy Image

These days BIOS and IPMI (and other no doubt) updates are getting larger.

Most of the boot floppies you can download are 1.4MB which makes upgrading some firmware modules difficult when PXE booting as the images wont fit.

I was able to build a 20MB image using these commands.

wget http://itblog.su/downloads/W98SE_R.img
rpm -ivh ms-sys-2.2.0-1.el5.rf.i386.rpm
dd if=/dev/zero of=floppy.img bs=1024 count=20000
mkdosfs floppy.img
ms-sys -1 -f floppy.img
mount -o loop floppy.img /mnt/floppy
mount /W98SE_R.img /mnt/dos
cp /mnt/dos/* /mnt/floppy/
umount /mnt/dos/

IPMI Upgrade through PXE

The Supermicro nodes have IPMI modules built in. These can be accessed using either the dedicated IPMI NIC, or by sharing the primary NIC with the host OS. We have chosen to use the shared option to reduce cabling.

Out of the box the machines came with a version of IPMI that didn't work for us so we had to update the IMPI firmware using the yafukcs utility.

The standard approach is to perform this via USB key but due to the number of nodes this is not practical. Therefore I created a freedos image containing the required firmware and configuration files that can be PXE booted to the nodes in one go.

This process saves around 4 days of manual labour. We still have to manually gather the MAC addresses of the new IPMI interfaces and add them to DHCP, but there are ways we can speed this process up using FDB in the switches.

Once we have our bootable image we add the required firmware update files

autoexec.bat

@echo off set PATH=.;\;\LOCALE
echo Using US-English keyboard with US-English codepage [437]
ipmicfg.exe -m
1.bat
2.bat
fdapm coldboot 

1.bat

yafukcs.exe -full X8DTT_~1.IMA 

Although it was quicker to update only the required modules using the -auto option we had better success with re-updating all the modules using the -full option.

2.bat

ipmicfg.exe -m
ipmicfg.exe -dhcp on
ipmicfg.exe -dhcp
ipmicfg.exe -r 

Once we run the update we configure IPMI with ipmicfg.exe to DHCP and very importantly perform a BMC cold reset. Then we add the image to the PXE config on our boot server (puppet in this case)

The relevant lines in the /var/lib/tftpboot/pxelinux.cfg/

LABEL Supermicro IPMI Firmware Update
KERNEL /sysresccd/isolinux/memdisk
APPEND initrd=/sysresccd/isolinux/floppy.img 

And copy the image (floppy.img) to /var/lib/tftpboot/sysresccd/isolinux/

We can now boot into the new image to ensure it works by selecting the new menu option from the PXE boot menu.

hot new club