<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>conrey.org</title>
	<atom:link href="http://conrey.org/feed/" rel="self" type="application/rss+xml" />
	<link>http://conrey.org</link>
	<description>fermentation and virtualization madness</description>
	<lastBuildDate>Tue, 21 May 2013 17:36:32 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.5.1</generator>
		<item>
		<title>BitTorrent Sync as Geo-Replication for Storage</title>
		<link>http://conrey.org/2013/05/21/bittorrent-sync-as-geo-replication-for-storage/</link>
		<comments>http://conrey.org/2013/05/21/bittorrent-sync-as-geo-replication-for-storage/#comments</comments>
		<pubDate>Tue, 21 May 2013 17:35:47 +0000</pubDate>
		<dc:creator>theron</dc:creator>
		
		<guid isPermaLink="false">http://conrey.rwx3.com/?p=1969</guid>
		<description><![CDATA[A friend pointed me at a new bittorrent lab project, Sync. While it took me a few minutes to wrap my head around what was going on and how this is being used, I&#8217;d have to say I&#8217;m impressed. The concept is simple, using a local client on your desktop or laptop Sync will synchronize [...]]]></description>
				<content:encoded><![CDATA[<p>A friend pointed me at a new bittorrent lab project, <a href="http://labs.bittorrent.com/experiments/sync.html">Sync</a>.  While it took me a few minutes to wrap my head around what was going on and how this is being used, I&#8217;d have to say I&#8217;m impressed.</p>
<p>The concept is simple, using a local client on your desktop or laptop Sync will synchronize the contents of the selected folder to other remote Sync clients sharing the same key. Synchronization is done securely via an encrypted (AES) bittorrent session. This ends up being effective for moving a lot of data across multiple devices and while I think it was initially designed for secure private dropbox style replication, I&#8217;ve been testing this as an alternative method of geo-replication between <a href="http://www.gluster.org">glusterfs</a> clusters on <a href="http://fedoraproject.org">Fedora</a>.<br />
<span id="more-1969"></span></p>
<p>Leveraging bittorrent, one of the more interesting features is the optional use of a tracker service.  This helps with peer discovery, letting the tracker announce SHA2(secret):IP:port to help peers connect directly. They tracker service also serves acts as a STUN server, helping with NAT traversal for peers that can&#8217;t directly see each other behind firewalls. This is interesting for environments where you don&#8217;t want to have to deal with reconfiguration of firewalls.  It&#8217;s important to note thought that even leveraging the tracker service all transmission of data is encrypted in flight.</p>
<h3>Getting Started</h3>
<p>For quick testing, find a couple of boxes that you want to get replication moving between.  These could be minimal install linux boxes, SAMBA servers for your SMB, webservers (backup replication?), or in my case, a single node of a gluster cluster.</p>
<p>Identify the directory you want to replicate, and go <a href="http://labs.bittorrent.com/experiments/sync/technology.html#downloadpop">download the client</a> from bittorrent labs for your installation.  For me it was the <a href="http://btsync.s3-website-us-east-1.amazonaws.com/btsync_x64.tar.gz">x64 Linux client</a>.</p>
<h3>Configuration</h3>
<p>First we&#8217;ll need to untar the download and get some config files ready to go.  Additionally, we&#8217;ll want to build an init.d script to ensure the client is running on startup.  You don&#8217;t have to do all of this, but I wanted to have it available to manage as a service. </p>
<p><code>$ tar -xf btsync.tar.gz</code></p>
<p>We&#8217;ll want to move the binary to a better location<br />
<code>$ sudo mv btsync /usr/bin</code></p>
<p>Next, create a directory for the configuration and generated storage files<br />
<code>$ sudo mkdir /etc/btsync</code></p>
<p>We should identify or create the directory we want to use as a replication target as well.  As an example I&#8217;ll create a new directory&#8230;.<br />
<code>$ sudo mkdir /replication</code></p>
<p>With our directories created and in place it&#8217;s time to generate the initial config file and edit it appropriately.<br />
<code>$ sudo btsync --dump-sample-config > /etc/btsync/btsync.conf</code></p>
<p>Using your favorite text editor, edit the following lines&#8230;</p>
<p><code>"device name": "My Sync Device",</code><br />
to<br />
<code>"device name": "whateveryourhostnameis",</code></p>
<p><code>"storage path" : "/home/user/.sync",</code><br />
to<br />
<code>"storage path" : "/etc/btsync",</code></p>
<p><code>// "pid_file" : "/var/run/syncapp/syncapppid.pid",</code><br />
to<br />
<code>"pid_file" : "/var/run/btsync.pid",</code></p>
<p>As we&#8217;re going to identify the replicated folders via the conf file it&#8217;s important to note that the webui that is normally available for the linux client will be disabled.  First thing you&#8217;ll need to do is generate a &#8220;secret&#8221; that you&#8217;re going to use for your share.  from the command line:<br />
<code>$ sudo btsync --generate-secret</code><br />
will give you a secret you can use,but I find it easier to just go ahead and dump the secret at the bottom of the conf file I&#8217;m going to use and just move it around from there.<br />
<code>$ sudo btsync --generate-secret >> /etc/btsync.conf</code><br />
In the shared folder section look for the following line:<br />
<code>"secret" : "MY_SECRET_1", // * required field</code><br />
and replace MY_SECRET_1 with the secret you generated.  As an example:<br />
<code>"secret" : "GYX6MWA67INIBN5XRHBQZRTGYX6MWA67XRHPJOO6ZINIBN5OQA", // * required field</code></p>
<p>you&#8217;ll want to change the directory line as well&#8230;</p>
<p><code>"dir" : "/home/user/bittorrent/sync_test", // * required field</code><br />
to<br />
<code>"dir" : "/replication",       // * required field</code></p>
<p>In the shared folders section either edit or comment out the known host section.  The easiest thing is to comment out the examples provided. Change&#8230;<br />
<code>  "192.168.1.2:44444", </code><br />
<code>  "myhost.com:6881"</code></p>
<p>to<br />
<code>//  "192.168.1.2:44444",</code><br />
<code>//  "myhost.com:6881"</code></p>
<p><strong>IMPORTANT:</strong> You&#8217;ll need to remove the leading /* and trailing */ of the shared folders section.</p>
<p>With the config file set start bittorrent sync using the config set.</p>
<p><code>btsync --config /etc/btsync.conf</code></p>
<p><b3>Other nodes and additional thoughts</h3>
<p>With the above in place you&#8217;ll want to configure additional nodes with the same secret used above.  The folder path can be different, but the secret has to be the same.  This will allow for replication to start amongst the identified folders.  Ideally, once we want to move beyond tinkering we&#8217;d create an init script to make things easier and serviceable via existing tools.  It&#8217;s a bit frustrating that the bittorrent sync tool is not an open tool.  While it&#8217;s available at no cost, I&#8217;d have liked to have the ability to distribute it bundled into a scripted install option for storage appliances.  Grabbing it after the initial install works too though. </p>
]]></content:encoded>
			<wfw:commentRss>http://conrey.org/2013/05/21/bittorrent-sync-as-geo-replication-for-storage/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>command nuggets : rsync</title>
		<link>http://conrey.org/2013/03/13/command-nuggets-rsync/</link>
		<comments>http://conrey.org/2013/03/13/command-nuggets-rsync/#comments</comments>
		<pubDate>Wed, 13 Mar 2013 18:08:17 +0000</pubDate>
		<dc:creator>theron</dc:creator>
		
		<guid isPermaLink="false">http://conrey.rwx3.com/?p=1934</guid>
		<description><![CDATA[This one is short and sweet. I needed to upload a LARGE file over a crappy connection that: dropped out regularly was inherently insecure broke if I maxed up the upload pipe. the command used to solve this: rsync --partial --progress --bwlimit=KBPS --rsh=ssh file.to.push user@remotehost:/path/to/dir/ Breaking this down: &#8211;partial will pick up where it left [...]]]></description>
				<content:encoded><![CDATA[<p>This one is short and sweet.<span id="more-1934"></span> I needed to upload a <strong>LARGE</strong> file over a crappy connection that:</p>
<ol>
<li>dropped out regularly</li>
<li>was inherently insecure</li>
<li>broke if I maxed up the upload pipe.</li>
</ol>
<p>the command used to solve this:</p>
<p><code>rsync --partial --progress --bwlimit=KBPS --rsh=ssh file.to.push user@remotehost:/path/to/dir/</code></p>
<h4>Breaking this down:</h4>
<ul>
<li>&#8211;partial will pick up where it left off if the file exists</li>
<li>&#8211;progress will display progress as the file uploads</li>
<li>&#8211;bwlimit will limit I/O bandwidth, KBytes per second</li>
<li>&#8211;rsh=ssh pushes the traffic through an ssh connection</li>
</ul>
<p>Problem solved! now just to wait for this file to upload&#8230;..</p>
]]></content:encoded>
			<wfw:commentRss>http://conrey.org/2013/03/13/command-nuggets-rsync/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Converged Infrastructure prototyping with Gluster 3.4 alpha and QEMU 1.4.0</title>
		<link>http://conrey.org/2013/02/26/converged-infrastructure-prototyping-with-gluster-3-4-alpha-and-qemu-1-4-0/</link>
		<comments>http://conrey.org/2013/02/26/converged-infrastructure-prototyping-with-gluster-3-4-alpha-and-qemu-1-4-0/#comments</comments>
		<pubDate>Tue, 26 Feb 2013 21:32:59 +0000</pubDate>
		<dc:creator>theron</dc:creator>
		
		<guid isPermaLink="false">http://conrey.rwx3.com/?p=1863</guid>
		<description><![CDATA[I just wrapped up my presentation at the Gluster Workshop at CERN where I discussed Open Source advantages in tackling converged infrastructure challenges. Here is my slidedeck. Just a quick heads up, there&#8217;s some animation that&#8217;s lost in the pdf export as well as color commentary during almost every slide. During the presentation I demo&#8217;ed [...]]]></description>
				<content:encoded><![CDATA[<p>I just wrapped up my presentation at the <a href="http://www.gluster.org/community/documentation/index.php/Planning/CERN_Workshop">Gluster Workshop at CERN</a> where I discussed Open Source advantages in tackling converged infrastructure challenges.  Here is my <a href="http://www.rwx3.com/downloads/CI_presentation_26Feb2013.pdf">slidedeck</a>.  Just a quick heads up, there&#8217;s some animation that&#8217;s lost in the pdf export as well as color commentary during almost every slide.</p>
<p>During the presentation I demo&#8217;ed out the new QEMU/GlusterFS native integration leveraging libgfapi.  For those of you wondering what that means, in short, there&#8217;s no need for FUSE anymore and QEMU leverages GlusterFS natively on the back end.  Awesome.</p>
<p>So for my demo I needed two boxes running QEMU/KVM/GlusterFS.  This would provide the compute and storage hypervisor layers.  As I only have a single laptop to tour Europe with, I obviously needed a nested KVM environment.<br />
<span id="more-1863"></span></p>
<p>If you&#8217;re got enough hardware feel free to skip the Enable Nested virtualization section and skip ahead to the Base OS installation.</p>
<p>This wasn&#8217;t as easy envionment to get up and running, this is alpha code boys and girls so expect to roll your sleeves up.  Ok with that out of the way, I&#8217;d like to walk through the steps I did in order to get my demo envionment up and running.  These installation assumes you have Fedora 18 installed and updated with virt-manager and KVM installed.</p>
<h3>Enable Nested Virtualization</h3>
<p>Since we&#8217;re going to want to install an OS on our VM running on our Gluster/QEMU cluster that we&#8217;re building, we&#8217;ll need to enable Nested Virtualization.</p>
<p>Let&#8217;s first check and see if nested virtualization is enabled.  If it responds with an N then No.  If yes, skip this section to the install.<br />
<code><br />
$ cat /sys/module/kvm_intel/parameters/nested<br />
N<br />
</code><br />
If it&#8217;s not we&#8217;ll need to load a KVM specific module with the nested option loaded.  The easist way to change this is using the modprobe configuration files:<br />
<code><br />
$ echo “options kvm-intel nested=1″ | sudo tee /etc/modprobe.d/kvm-intel.conf<br />
</code></p>
<p>Reboot your machine once the changes have been made and check again to see if the feature is enabled:<br />
<code><br />
$ cat /sys/module/kvm_intel/parameters/nested<br />
Y<br />
</code><br />
That&#8217;s it we&#8217;re done with prepping the host.</p>
<h3>Install VMs OS</h3>
<p>Starting with my base Fedora laptop, I&#8217;ve installed virt-manager for VM management.  I wanted to use Boxes, but it&#8217;s not designed for this type of configuration. So. Create your new VM, I selected the &#8220;Fedora http install&#8221; as I didn&#8217;t have an iso laying around. Also http install=awesome.<br />
<a href="http://conrey.org/files/2013/02/gluster01.png"><img src="http://conrey.org/files/2013/02/gluster01-150x150.png" alt="gluster01" width="150" height="150" class="aligncenter size-thumbnail wp-image-1897" /></a></p>
<p>To do this, select the http install option and enter the nearest available location.  </p>
<p><a href="http://conrey.org/files/2013/02/gluster02.png"><img src="http://conrey.org/files/2013/02/gluster02-150x150.png" alt="gluster02" width="150" height="150" class="aligncenter size-thumbnail wp-image-1898" /></a></p>
<p>For me this was the Masaryk University, Brno (where I happened to be sitting during <a href="http://www.devconf.cz/">Dev Days 2013</a>)<br />
<code></p>
<p>http://ftp.fi.muni.cz/pub/linux/fedora/linux/releases/18/Fedora/x86_64/os/</p>
<p></code><br />
I went with an 8 gig base disk to start (we&#8217;ll add another one in a bit), gave the VM 1G of ram and a default vCPU.  Start the VM build and install.</p>
<p><a href="http://conrey.org/files/2013/02/gluster03.png"><img src="http://conrey.org/files/2013/02/gluster03-150x150.png" alt="gluster03" width="150" height="150" class="aligncenter size-thumbnail wp-image-1899" /></a></p>
<p>The install will take a bit longer as it&#8217;s downloading the install files during the intial boot.</p>
<p><a href="http://conrey.org/files/2013/02/gluster04.png"><img src="http://conrey.org/files/2013/02/gluster04-150x150.png" alt="gluster04" width="150" height="150" class="aligncenter size-thumbnail wp-image-1900" /></a></p>
<p>Select the language you want to use and continue to the installation summary screen.  Here we&#8217;ll want to change the software selection option.</p>
<p><a href="http://conrey.org/files/2013/02/gluster05.png"><img src="http://conrey.org/files/2013/02/gluster05-150x150.png" alt="gluster05" width="150" height="150" class="aligncenter size-thumbnail wp-image-1901" /></a></p>
<p>and select the minimal install:</p>
<p><a href="http://conrey.org/files/2013/02/gluster06.png"><img src="http://conrey.org/files/2013/02/gluster06-150x150.png" alt="gluster06" width="150" height="150" class="aligncenter size-thumbnail wp-image-1902" /></a></p>
<p>during the installation, go ahead and set the root password:</p>
<p><a href="http://conrey.org/files/2013/02/gluster07.png"><img src="http://conrey.org/files/2013/02/gluster07-150x150.png" alt="gluster07" width="150" height="150" class="aligncenter size-thumbnail wp-image-1903" /></a></p>
<p>Once the installation is complete, the VM will reboot.  Once done, power it down.  Although we&#8217;ve enable Nested Virtualization, we need to pass the CPU flags onto the VM.</p>
<p>In the virt-manager window right click on the VM, and select open. In the VM window, select view > details.  Rather than guessing the cpu architecture, select the copy from host and select Ok.</p>
<p><a href="http://conrey.org/files/2013/02/gluster08.png"><img src="http://conrey.org/files/2013/02/gluster08-150x150.png" alt="gluster08" width="150" height="150" class="aligncenter size-thumbnail wp-image-1904" /></a></p>
<p>While you&#8217;re here go ahead and add an additional 20 gig virtual drive.  Make sure you select virtio for the drive type!</p>
<p><a href="http://conrey.org/files/2013/02/gluster09.png"><img src="http://conrey.org/files/2013/02/gluster09-150x150.png" alt="gluster09" width="150" height="150" class="aligncenter size-thumbnail wp-image-1905" /></a></p>
<p>Boot your VM up and let&#8217;s get started.</p>
<h3>Base installation components</h3>
<p>You&#8217;ll need to install some base components before you get started installing GlusterFS or QEMU.</p>
<p>After logging in as root,<br />
<code><br />
yum update<br />
</code><br />
<code><br />
yum install nettools wget xfsprogs binutils<br />
</code></p>
<p>Now we&#8217;re going to create the mount point and format the additional drive we just installed.<br />
<code><br />
mkdir -p /export/brick1<br />
</code><br />
<code><br />
mkfs.xfs -i size=512 /dev/vdb<br />
</code><br />
We&#8217;ll need to edit our fstab and add this as well, so that it will remain persistent going forward after any reboots.</p>
<p>add the following line to /etc/fstab<br />
<code><br />
/dev/vdb /export/brick1 xfs defaults 1 2<br />
</code></p>
<p>Once you&#8217;re done with this, let&#8217;s go ahead and mount the drive.<br />
<code><br />
mount -a &#038;&#038; mount<br />
</code></p>
<h3>Firewalls.  YMMV</h3>
<p>it may be just me (I&#8217;m sure it is) but I struggled getting gluster to work with firewalld on fedora 18.  This is not reccomeneded in production envionments, but for our all in VM on a laptop deployment, I just disabled and removed firewalld.<br />
<code><br />
yum remove firewalld<br />
</code></p>
<h3>Gluster 3.4.0 Alpha Installation</h3>
<p>First thing we&#8217;ll need to do on our VM is configure and enable the gluster repo.<br />
<code><br />
wget http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.4.0alpha/Fedora/glusterfs-alpha-fedora.repo<br />
</code><br />
and move it to /etc/yum.repos.d/<br />
<code><br />
mv glusterfs-alpha-fedora.repo /etc/yum.repos.d/<br />
</code></p>
<p>Now we enable the repo and install glusterfs:<br />
<code><br />
yum update<br />
</code><br />
yum install glusterfs-server glusterfs-devel<br />
</code></p>
<p>Important to note here we need the gluster-devel package for the QEMU integration we'll be testing.  Once done we'll start the  glusterd service and verify that it's working.</p>
<h3>break break 2nd VM</h3>
<p>Ok folks, if you've made it here, get a coffee and do the install again on a 2nd VM.  You'll need the 2nd replication VM target before you proceed.<br />
<code></end coffee break></code></p>
<h3>break break Network Prepping both VMs</h3>
<p>As we're on the private nat'd network on our laptop that virt-manager is managing, we'll need to update our VMs we create and assign static addresses, as well as editing the /etc/hosts file to add both servers with thier addresses.  We're not proud here people, this is a test envionment, if you want to use proper DNS, I won't judge if you don't.</p>
<p>1) change both VMs to using static addresses in the nat range.<br />
2) change VMs hostnames<br />
3) update both VMs /etc/hosts to include both nodes.  This is hacky but expedient.</p>
<h3>back to Gluster</h3>
<p>start and verify the gluster services on both VMs.<br />
<code>service glusterd start</code><br />
<code>service glusterd status</code></p>
<p>On either host, we'll need to create the gluster volume and set it for replication.<br />
<code>gluster volume create vmstor replica 2 ci01.local:/export/brick1 ci02.local:/export/brick1</code></p>
<p>Now we'll start the volume we just created<br />
<code>gluster volume start vmstor</code></p>
<p>Verify that everything is good, if this returns fine, you're up and running with GlusterFS!<br />
<code>gluster volume info</code></p>
<h3>building QEMU dependancies</h3>
<p>let's get some prereqs for getting the latest qemu up and running</p>
<p><code>yum install lvm2-devel git gcc-c++ make glib2-devel pixman-devel</code></p>
<p>Now we'll download QEMU:</p>
<p><code>git clone git://git.qemu-project.org/qemu.git</code></p>
<p>The rest is pretty standard compiling from source.  you'll start with configuring your build.  I'll trim the target list to save time as I know I'm not going to use many of the QEMU supported architectures.<br />
<code>./configure --enable-glusterfs --target-list=i386-softmmu,x86_64-softmmu,x86_64-linux-user,i386-linux-user</code> </p>
<p>With that done everything on this host is done, and we're ready to start building VMs using GlusterFS natively bypassing fuse and leveraging thin provisioning. W00!</p>
<h3>Creating Virtual Disks on GlusterFS</h3>
<p><code>qemu-img create gluster://ci01:0/vmstor/test01?transport=socket 5G</code></p>
<p>Breaking this down, we're using qemu-img to create a disk natively on GlusterFS that's five gigs in size.  I'm looking for some more information about what the transport socket is, expect an answer soonish.</p>
<h3>Build a VM and install an OS onto the GlusterFS mounted disk image</h3>
<p>At this point you'll want something to actually install on your image.  I went with TinyCore because as it is I'm already pushing up against the limitations of this laptop with nested virtualization.  You can download TinyCore Linux <a href="http://distro.ibiblio.org/tinycorelinux/">here</a>.	</p>
<p><code>qemu-system-x86_64 --enable-kvm -m 1024 -smp 4 -drive file=gluster://ci01/vmstor/test01,if=virtio -vnc 192.168.122.209:1 --cdrom /home/theron/CorePlus-current.iso</code></p>
<p>This is the quickest way to get this moving, I skipped using Virsh for the demo, and am assigning the VNC IP and port manually.  Once the VM starts up you should be able to connect to the VM from your external host and start the install process.</p>
<p><a href="http://conrey.org/files/2013/02/gluster10.png"><img src="http://conrey.org/files/2013/02/gluster10-150x150.png" alt="gluster10" width="150" height="150" class="aligncenter size-thumbnail wp-image-1906" /></a></p>
<p>To get the install going, select the harddrive that was build with qemu-img and follow the OS install procedures.</p>
<h3>finish</h3>
<p>At this point you're done and you can start testing and submitting bugs!</p>
<p>I'd expect to see some interesting things with OpenStack in this space as well as tighter oVirt integration moving forward.</p>
<p>Let me know what you think about this guide and if it was useful.</p>
<h3>side note</h3>
<p>Also, something completely related.  I'm pleased to announce that I've joined the Open Source and Standards team at Redhat working to promote and assist making upstream projects wildly successful.  If you're unsure what that means or you're wondering why Red Hat cares about upstream projects, PLEASE reach out and say hello.</p>
<h3>References:</h3>
<p><a href="http://www.rdoxenham.com/?p=275">nested KVM</a><br />
<a href="http://www.cyberciti.biz/faq/linux-kvm-vnc-for-guest-machine/ ">KVM VNC</a><br />
<a href="http://www.youtube.com/watch?v=JG3kF_djclg">Using QEMU to boot VM on GlusterFS</a><br />
<a href="http://qemu-project.org/Download">QEMU downloads</a><br />
<a href="http://raobharata.wordpress.com/2012/10/29/qemu-glusterfs-native-integration/">QEMU Gluster native file system integration</a></p>
]]></content:encoded>
			<wfw:commentRss>http://conrey.org/2013/02/26/converged-infrastructure-prototyping-with-gluster-3-4-alpha-and-qemu-1-4-0/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Is converged infrastructure a crutch?</title>
		<link>http://conrey.org/2013/01/31/converged-infrastructure-moving-forward/</link>
		<comments>http://conrey.org/2013/01/31/converged-infrastructure-moving-forward/#comments</comments>
		<pubDate>Thu, 31 Jan 2013 15:58:57 +0000</pubDate>
		<dc:creator>theron</dc:creator>
		
		<guid isPermaLink="false">http://conrey.rwx3.com/?p=1865</guid>
		<description><![CDATA[This started as a response to a twitter conversation with @DuncanYB and @joshobrien77  re: converged infrastructure. Duncan recently posted a great blog post about Converged compute and storage. Go read that first. I&#8217;ll wait here. Welcome back! Ok, to start I agree with Duncan&#8217;s comments that Nutanix is certainly in the leader group for what&#8217;s [...]]]></description>
				<content:encoded><![CDATA[<p>This started as a response to a twitter conversation with <a href="http://twitter.com/DuncanYB">@DuncanYB</a> and <a href="http://www.twitter.com/joshobrien77">@joshobrien77</a>  re: converged infrastructure. Duncan recently posted a <a href="http://www.yellow-bricks.com/2013/01/28/converged-compute-and-storage-solutions/">great blog post about Converged compute and storage</a>. Go read that first. I&#8217;ll wait here.</p>
<p>Welcome back! Ok, to start I agree with Duncan&#8217;s comments that <a href="http://www.nutanix.com/">Nutanix</a> is certainly in the leader group for what&#8217;s viewed today as &#8220;Converged Infrastructure&#8221;, in that it&#8217;s delivering a whole stack solution. The other company mentioned in Duncan&#8217;s blogpost is <a href="http://simplivity.com/OmniCube/overview.html">Simplivity</a>. Both companies are doing awesome stuff, and have figured out ways to solve REALLY complex problems.</p>
<p><span id="more-1865"></span></p>
<p>Taking a step back and thinking about what they&#8217;re both delivering, both companies have a hardware solution that bundles virtualized network, compute and storage capabilities. Fundamentally, they&#8217;re both delivering x86 platforms with this new methodology for what I&#8217;m calling &#8220;complete virtualization<strong>*</strong>&#8220;. Converged Infrastructure today implying a specific hardware vendor component to deliver.</p>
<p>If we look at what problems virtualization solved in the enterprise, separating the compute from the hardware really pushed forward in an evolutionary way. With old school virtualization (vSphere 101, compute only) old towers were brought down and in some cases, new towers were built up. Storage and Network hardware and designs remained relatively unaffected, but certainly leveraged in new ways.</p>
<p>Inserting a shim between the Storage hardware and virtualizing storage is moving forward quickly. This isn&#8217;t as simple as virtualizing traditional storage architectures. While you can throw traditional storage in a VM and leverage it (think a virtual <a href="http://nickapedia.com/2011/04/08/new-uber-model-uber-vnx-nfs-v1/">EMC</a>, <a href="http://www.wooditwork.com/2012/09/07/netapp-releases-its-virtual-storage-appliance-data-ontap-edge/">Netapp</a>, or <a href="http://www.nexenta.com">Nexenta</a> virtual appliances, this doesn&#8217;t give a full featured CI answer to storage. This is a key component in CI solutions today providing stability, replication and scalability needed to host enterprise applications. There are other places outside of CI where this is happening in IT today. Looking at <a href="http://www.gluster.org">Gluster</a>, <a href="http://www.ceph.com">Ceph</a>, or even <a href="http://www.emc.com/domains/isilon/index.htm">Isilon</a>, these solutions scale out storage effectively. If these can be instantiated virtually it allows for the colllapse onto a single compute, storage, and network &#8220;node&#8221;.  I&#8217;d say this allows for an existing precedent for these types of potential &#8220;CI&#8221; style storage solutions, and assume they&#8217;ll continue to gain traction.</p>
<p>While it&#8217;s all x86 commodity hardware with a shiny nameplate, there&#8217;s a final ingredient in the secret sauce. Automation. This is critical for scaling out a complete solution. I see these challenges day in and day out already with customers in the field. <a href="http://www.puppetlabs.com">Puppet</a>, <a href="http://www.opscode.com/chef/">Chef</a>, <a href="http://ansible.cc/">Ansible</a> and others are already starting to solve these challenges in the enterprise. I don&#8217;t think anyone sees this going away.</p>
<p>&#8220;Complete virtualization&#8221; boils down to a single software stack delivering this solution regardless of the box or unit it&#8217;s delivered on. I&#8217;d say that Nutanix delivers this. Does this mean that I have to buy my solution from Nutanix? In what is called &#8220;converged infrastructure&#8221; I do. What added value does Nutanix hardware provide vs commodity hardware? (Support would be my first thought, known hardware configs etc&#8230; more on this later) The flipside would be, does this type of solution lock us into a familiar hardware paradigm that VMware and virtualization in general worked to free us from?</p>
<p>Having said all of that, I can&#8217;t see a reason that this can&#8217;t be bundled into a software package and still allow for full functionality. Tying hardware support and software support is a step backwards for where we&#8217;re going. It&#8217;s going to work for some customers, but I believe theres a risk for artificial value in hardware that is completely commoditized today. The value is in the software, and I&#8217;d assume it will be the continued path forward. <a href="http://www.vmware.com">VMware</a>, <a href="http://www.redhat.com">Redhat</a>, and even <a href="http://www.microsoft.com">MS</a> really nailed that point home.</p>
<p>As <a href="http://www.openstack.org">openstack</a> continues to make inroads, I would hope to see full &#8220;complete virtualization&#8221; product(s) coming out soonish.  There&#8217;s certainly CI solutions in the works, <a href="http://www.nebula.com">nebula</a> jumps to mind.</p>
<p>Trying to summarize before I ramble off, my initial thought is that Enterprise IT will still want choice for hardware.  Nutanix is delivering an <strong>AMAZING</strong> solution because the different software solutions aren&#8217;t tied together natively to make this an easy problem to solve. Trying to force a bundled solution of hardware and software to just deliver functionality that leverages commodity x86 hardware seems awkward.  Additionally, requiring new support chains within larger Enterprises can create more challenges can make adaption harder than it needs to be. I expect to see open source solutions (from Redhat, VMware or others) to deliver these types of solutions faster and gain additional traction in this space to server as an infrastructure foundation for Open/Cloudstack style products.  I think CI will fade away to &#8220;complete virtualization&#8221; software solutions.</p>
<p>All in all, I think this is a product wrapping around the <a href="http://stu.radnidge.com/post/30819681149">idea of &#8220;software defined datacenters&#8221;</a>.</p>
<p>* Complete virtualization is a horrible name for this. Ideas welcome.</p>
]]></content:encoded>
			<wfw:commentRss>http://conrey.org/2013/01/31/converged-infrastructure-moving-forward/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>#ProjectSputnik &#8211; first impressions</title>
		<link>http://conrey.org/2012/09/05/projectsputnik-first-impressions/</link>
		<comments>http://conrey.org/2012/09/05/projectsputnik-first-impressions/#comments</comments>
		<pubDate>Wed, 05 Sep 2012 07:30:26 +0000</pubDate>
		<dc:creator>theron</dc:creator>
		
		<guid isPermaLink="false">http://conrey.rwx3.com/?p=1625</guid>
		<description><![CDATA[Dell has recently kicked off #ProjectSputnik, a beta program to deliver a Linux based laptop to developers featuring a tweaked out version of Ubuntu 12.04 LTS for a Dell XPS laptop with all the drivers ready to go. Most intriguing was the idea that additional tools for devs specifically cloud tools that enable you to [...]]]></description>
				<content:encoded><![CDATA[<p>Dell has recently kicked off <a href="http://content.dell.com/us/en/enterprise/d/campaigns/sputnik">#ProjectSputnik</a>, a beta program to deliver a Linux based laptop to developers featuring a <a href="http://hwe.ubuntu.com/uds-q/dellxps/">tweaked out version of Ubuntu 12.04 LTS</a> for a Dell XPS laptop with all the drivers ready to go.  </p>
<p>Most intriguing was the idea that additional tools for devs specifically cloud tools that enable you to create &#8220;microclouds” on your laptop, simulating an at-scale environment, and then deploy that environment seamlessly to the cloud would be provided. Now, I get this is some of the work around Ubuntu juju, but I haven&#8217;t had a chance to dig in.  I was hooked.<br />
<span id="more-1625"></span><br />
I applied, was accepted to participate as a <del datetime="2012-09-05T20:03:49+00:00">beta users</del> cosmonaut in the beta field trial, and ordered up my kit.  </p>
<p>I was excited to be selected for the beta program but I wasn&#8217;t prepared for the sheer awesomeness that is Ubuntu on this notebook.  Before I go any further, a big thanks to <a href="http://bartongeorge.net/2012/08/27/project-sputnik-beta-cosmonauts-chosen/">Barton George</a> and the rest of the Dell crew for letting me participate.  I know there will be more than one segment to this, so this post will cover my first impressions and getting a base working environment up and running.</p>
<h3>Packaging</h3>
<p>Someone inside the Dell design group took a note or two from the fruit company and they&#8217;ve upped their game for packaging.  This was the first Dell product I&#8217;ve opened carefully, with concern about ripping the tape (cheesy I know). Seriously well done.</p>
<table border="0" width="100%" cellpadding="0" cellspacing="0">
<tr>
<td><a href="http://conrey.org/files/2012/09/XPS-photo1.png"><img src="http://conrey.org/files/2012/09/XPS-photo1-150x150.png" alt="" title="XPS-photo1" width="150" height="150" size-thumbnail wp-image-1664" /></a></td>
<td><a href="http://conrey.org/files/2012/09/XPS-photo2.png"><img src="http://conrey.org/files/2012/09/XPS-photo2-150x150.png" alt="" title="XPS-photo2" width="150" height="150" size-thumbnail wp-image-1665" /></a></td>
<td><a href="http://conrey.org/files/2012/09/XPS-photo3.png"><img src="http://conrey.org/files/2012/09/XPS-photo3-150x150.png" alt="" title="XPS-photo3" width="150" height="150" size-thumbnail wp-image-1666" /></a></td>
</tr>
</table>
<h3>Beta Only Woes</h3>
<p>While this goes to market later this year, the intent is to have this XPS notebook sold preloaded with Ubuntu.  For us lucky few in the beta we get to load up the customized Ubuntu 12.04 iso ourselves.  Interestingly enough for the life of me I couldn&#8217;t get the Ubuntu published instructions to work for <a href="https://help.ubuntu.com/community/Installation/FromUSBStick#From_Mac_OSX">creating a bootable USB drive on a Mac</a>.  <a href="http://afreshcup.com/home/2012/9/4/project-sputnik-report-1.html">I apparently wasn&#8217;t the only one</a>. Rather than waste time, I booted the downloaded ISO into a new VM with <a href="www.vmware.com/VMwareFusion5">VMware Fusion</a>, and used the built in Ubuntu tools to create a bootable usb drive.  A bit circuitous, but familiar territory.</p>
<table border="0" width="100%" cellpadding="0" cellspacing="0">
<tr>
<td><a href="http://conrey.org/files/2012/09/photo0.png"><img src="http://conrey.org/files/2012/09/photo0-150x150.png" alt="" title="photo0" width="150" height="150" class="aligncenter size-thumbnail wp-image-1649" /></a></td>
<td><a href="http://conrey.org/files/2012/09/XPS-photo10.png"><img src="http://conrey.org/files/2012/09/XPS-photo10-150x150.png" alt="" title="XPS-photo10" width="150" height="150" class="aligncenter size-thumbnail wp-image-1673" /></a></td>
</tr>
</table>
<p>With a bootable usb drive ready, I powered on the laptop and without any fanfare installed Ubuntu.  It should be pointed out here that having a custom build for this hardware made NOT digging in and having to tweak really nice.  Once the installation was done, the reboot was fast and EVERYTHING JUST WORKS.</p>
<li>Wifi
<li>monitor out
<li>keyboard lights
<li>function buttons
<li>volume buttons
<li>every keyboard button&#8230;nicely done.
<p>The only thing I&#8217;ve had to do was increase the mouse (touchpad) sensitivity to max.  Realistically, I could go higher but it&#8217;s good.  With that done, It was time to actually take a solid look at this beast.</p>
<h3>hardware awesomeness</h3>
<p>It&#8217;s hard to say how thin this notebook is.  I wish I had an Air to take some photos next to the XPS.  It&#8217;s thin. It&#8217;s interesting that for how thin it is, it still feels really solid. The keyboard is nice, and inside there&#8217;s an Intel i7 quad core processor, 4G of ram, and a 256G SSD drive.</p>
<table border="0" width="100%" cellpadding="0" cellspacing="0">
<tr>
<td><a href="http://conrey.org/files/2012/09/XPS-photo4.png"><img src="http://conrey.org/files/2012/09/XPS-photo4-150x150.png" alt="" title="XPS-photo4" width="150" height="150" class="aligncenter size-thumbnail wp-image-1667" /></a></td>
<td><a href="http://conrey.org/files/2012/09/XPS-photo5.png"><img src="http://conrey.org/files/2012/09/XPS-photo5-150x150.png" alt="" title="XPS-photo5" width="150" height="150" class="aligncenter size-thumbnail wp-image-1668" /></a></td>
<td><a href="http://conrey.org/files/2012/09/XPS-photo6.png"><img src="http://conrey.org/files/2012/09/XPS-photo6-150x150.png" alt="" title="XPS-photo6" width="150" height="150" class="aligncenter size-thumbnail wp-image-1669" /></a></td>
</tr>
<tr>
<td><a href="http://conrey.org/files/2012/09/XPS-photo7.png"><img src="http://conrey.org/files/2012/09/XPS-photo7-150x150.png" alt="" title="XPS-photo7" width="150" height="150" class="aligncenter size-thumbnail wp-image-1670" /></a></td>
<td><a href="http://conrey.org/files/2012/09/XPS-photo8.png"><img src="http://conrey.org/files/2012/09/XPS-photo8-150x150.png" alt="" title="XPS-photo8" width="150" height="150" class="aligncenter size-thumbnail wp-image-1671" /></a></td>
<td><a href="http://conrey.org/files/2012/09/XPS-photo9.png"><img src="http://conrey.org/files/2012/09/XPS-photo9-150x150.png" alt="" title="XPS-photo9" width="150" height="150" class="aligncenter size-thumbnail wp-image-1672" /></a></td>
</tr>
</table>
<h3>tools migration</h3>
<p>This is going to be an easy migration as most of the tools I use today are cross platform.  I can&#8217;t think of an app that I use daily (sometimes almost constantly) that I don&#8217;t already have a replacement for or in some cases, something better.</p>
<p><strong>Browser:</strong><br />
Chrome &#8211; > Chrome<br />
firefox /w firebug -> firefox /w firebug<br />
<strong>Communication:</strong><br />
Colloquy &#8211; > XChat<br />
tweetdeck &#8211; > gwibber?<br />
Skype &#8211; > Skype<br />
Adium -> unity? ( I was going to fix this note from an earlier draft to Empathy but I left it to point out that chat is so well integrated into unity that I honestly couldn&#8217;t recall what client it is. Nicely done ubuntu folks)<br />
n/a &#8211; > SFLPhone  Holy crap this is the best SIP softphone I&#8217;ve ever used. On any platform and I&#8217;ve used a lot of them.  I&#8217;m actually looking forward to using this again.  Can&#8217;t wait. Stop laughing.<br />
<strong>File Replication:</strong><br />
Dropbox &#8211; > Dropbox<br />
<strong>Text Editor:</strong><br />
Sublime Text2 &#8211; > <a href="http://www.webupd8.org/2011/03/sublime-text-2-ubuntu-ppa.html">Sublime Text2</a><br />
Vim &#8211; > Vim<br />
<strong>Quick Launch:</strong><br />
QuickSilver &#8211; > native unity<br />
<strong>Shell:</strong><br />
Terminal (with zsh) &#8211; > Terminal (with zsh)<br />
<strong>Mail:</strong><br />
Mail is going to be interesting.  Apparently I was in a small camp that LOVED evolution.  <a href="https://help.ubuntu.com/community/EmailClients">You can still install evolution, it&#8217;s just not the default option</a>.  I&#8217;m thinking about going all in and buying support for this laptop from Canonical at some point, so I may give the default Thunderbird a go.</p>
<h3>The elephant in the room</h3>
<p>Windows Office is a sticky point that I&#8217;m going to have to tackle at some point. This isn&#8217;t hopefully an issue in most dev shops, but it&#8217;s certainly an issue for my company. I tried hard to not use Office until I had to put together a presentation with meat.  Let lightning strike me down, powerpoint is good at what it does.  Keynote rocks as well, but working in a mixed platform shop and shipping files around causes some headaches.  Hopefully I can get a VDI image somewhere for me to connect to via <a href="http://code.google.com/p/vmware-view-open-client/">VMware&#8217;s open PCoIP client</a>.  Honestly, this may be my biggest hurdle for leveraging this beast as a &#8220;Company&#8221; laptop.  If you&#8217;re lucky enough to work in a sizable enough company to have a proper VDI or Citrix deployment in place, you&#8217;re set.</p>
<p>As we&#8217;re in that odd middle size, I&#8217;d like to say my company is leveraging google docs, <a href="http://www.sliderocket.com/">sliderocket</a>, or any next gen presentation products.  We&#8217;re not.</p>
<p><a href="http://conrey.org/files/2012/09/XPS-photo11.png"><img src="http://conrey.org/files/2012/09/XPS-photo11-150x150.png" alt="" title="XPS-photo11" width="150" height="150" class="aligncenter size-thumbnail wp-image-1705" /></a></p>
<h3>Summary</h3>
<p>After initial load and getting my standard working environment up and running, I&#8217;ve got to say this little beast is pretty amazing.  From the tight OS integration to the feel of the laptop, it looks and feels like a solid build.  I&#8217;m going to be busy working with OpenStack over the coming months and I&#8217;m excited to see how closely integrated I can get my build envionment on this laptop to the ubuntu server I&#8217;m using for testing.  After <a href="http://youtu.be/W2LGmY796wQ">watching Mark Shuttleworth talk this year at Oscon about JuJu and #ProjectSputnik</a>, I&#8217;m pretty sure it&#8217;s going to be amazing.  More blog posts to follow.</p>
]]></content:encoded>
			<wfw:commentRss>http://conrey.org/2012/09/05/projectsputnik-first-impressions/feed/</wfw:commentRss>
		<slash:comments>7</slash:comments>
		</item>
		<item>
		<title>Howto add the RPMForge repo in Centos 6</title>
		<link>http://conrey.org/2012/04/25/howto-add-the-rpmforge-repo-in-centos-6/</link>
		<comments>http://conrey.org/2012/04/25/howto-add-the-rpmforge-repo-in-centos-6/#comments</comments>
		<pubDate>Wed, 25 Apr 2012 03:32:49 +0000</pubDate>
		<dc:creator>theron</dc:creator>
		
		<guid isPermaLink="false">http://conrey.rwx3.com/?p=1616</guid>
		<description><![CDATA[This one is a quick one. If you&#8217;re looking for a package and you can&#8217;t find it in the main CentOS repos, there&#8217;s a good chance it&#8217;s over in the RPMforge repository. However, adding additional repos like this can be problematic during upgrades as software version can conflict. Even though you can&#8217;t always avoid the [...]]]></description>
				<content:encoded><![CDATA[<p>This one is a quick one. If you&#8217;re looking for a package and you can&#8217;t find it in the main CentOS repos, there&#8217;s a good chance it&#8217;s over in the RPMforge repository.  However, adding additional repos like this can be problematic during upgrades as software version can conflict.</p>
<p>Even though you can&#8217;t always avoid the problem, you CERTAINLY can try to minimize the potential damage that you can do.</p>
<p><span id="more-1616"></span></p>
<h3>Download the RPM Forge Repository Package</h3>
<p><code>wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-1.el6.rf.x86_64.rpm</code></p>
<h3>Install the RPM Forge Repository</h3>
<p><code>rpm -ivh rpmforge-release-0.5.2-1.el6.rf.x86_64.rpm</code></p>
<p>You&#8217;ll get a warning, but it will proceed based on the flags that you set.</p>
<p><code>warning: rpmforge-release-0.5.2-1.el6.rf.x86_64.rpm : Header V3 DSA/SHA1 Signature, key ID 6b8d79e6: NOKEY</code><br />
<code>Preparing... ########################################### [100%]</code><br />
<code>1:rpmforge-release ########################################### [100%]</code></p>
<h3>Set YUM priorities</h3>
<p>Now you need to edit /etc/yum.repos.d/CentOS-Base.repo and add these priorities to the bottoms of each of the following repo sections:</p>
<p><code>[base]<br />
priority=1</p>
<p>[updates]<br />
priority=1</p>
<p>[extras]<br />
priority=1</code></p>
<p>Save your changes on this file and edit the rpmforge.repo file.  You&#8217;ll only need to edit the one repository section.</p>
<p><code>[rpmforge]<br />
priority=5</code></p>
<p>Save your changes and you&#8217;re finished.</p>
<p>Setting this repo to a higher priority ensures that this repo is only used if the other repos don&#8217;t have what you&#8217;re looking for.  Like Pure-FTPd. <img src='http://conrey.org/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' />   The above assumes you&#8217;re using 64bit Centos 6.  if you&#8217;re using 32bit today stop and ask yourself why.  Then, use the link and substitute the 32 bit links instead.</p>
<h3>Install a test package</h3>
<p>In this case we don&#8217;t see Pure-FTPd in the stock CentOS repos.  let&#8217;s update yum and then install PureFTPd.</p>
<p><code> yum update</code><br />
<code> yum install pure-ftpd</code></p>
<p>viola! you should see the package install now and you&#8217;re pulling this from the newly install RPM Forge repository.</p>
]]></content:encoded>
			<wfw:commentRss>http://conrey.org/2012/04/25/howto-add-the-rpmforge-repo-in-centos-6/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>SparkleShare on illumian Howto &#8211; a privately hosted dropbox replacement</title>
		<link>http://conrey.org/2012/03/12/sparkleshare-on-illumian-howto-a-privately-hosted-dropbox-replacement/</link>
		<comments>http://conrey.org/2012/03/12/sparkleshare-on-illumian-howto-a-privately-hosted-dropbox-replacement/#comments</comments>
		<pubDate>Mon, 12 Mar 2012 07:20:02 +0000</pubDate>
		<dc:creator>theron</dc:creator>
		
		<guid isPermaLink="false">http://conrey.rwx3.com/?p=1512</guid>
		<description><![CDATA[In my ever lengthening wait for VMware&#8217;s project octopus to be released into public beta, I&#8217;ve been trying other dropbox like applications for feature functionality. One of the ones that I&#8217;ve been most impressed by is SparkleShare. With the announcement today that they&#8217;ve released their first windows client, I thought I&#8217;d post this quick tutorial [...]]]></description>
				<content:encoded><![CDATA[<p>In my ever lengthening wait for VMware&#8217;s project octopus to be released into public beta, I&#8217;ve been trying other <a href="http://www.dropbox.com">dropbox</a> like applications for feature functionality.  One of the ones that I&#8217;ve been most impressed by is <a href="http://sparkleshare.org">SparkleShare</a>. <a href="http://twitter.com/SparkleShare/status/179013183275667456">With the announcement today that they&#8217;ve released their first windows client</a>, I thought I&#8217;d post this quick tutorial on how to get sparkleshare up and running leveraging illumian as your data repository, and also test it out with the new windows client.  There&#8217;s already a mac and linux client available and both work great.</p>
<p><span id="more-1512"></span></p>
<h3>What is SparkleShare?</h3>
<p>From the sparkleshare website: &#8220;SparkleShare is a collaboration and sharing tool that is designed to keep things simple and to stay out of your way.&#8221;  Feature wise, it acts very similarly to dropbox, but provides some additional functionality. It leverages a Git repository on the back end, and is extremely flexible.</p>
<p>My only concern with it, or any hosted data story, is ensuring that my data is always available and more importantly, that it doesn&#8217;t corrupt over time. That&#8217;s why I&#8217;m using illumian.</p>
<h3>What is Illumian?</h3>
<p>illumian was started by Nexenta to create an illumos distribution with Debian style package management tools for familiarity and compatibility with existing tools. This distribution uses the same versions of externally maintained packages as the OpenIndiana project uses today.  Where NCP was based on OpenSolaris and some interesting ways to get debian packages ported, illumian is a based on the work in the continued new work in the illumos project.  The recent 1.0 release contains only a small number of maintained packages and a simple command line installer.  No GUI required.</p>
<h3>Download illumian</h3>
<p>you can download the illumian iso from <a href="http://www.illumian.org">illumian.org</a></p>
<h3>Base VM configuration</h3>
<p>On the road and working in a hotel, I&#8217;m using VMware workstation for this installation and have assigned the following items to the VM:</p>
<ul>
<li>2 8G disks for the syspool (OS install disks)</li>
<li>2 16G disks for data (I&#8217;ll mirror these as well)</li>
<li>2G of RAM</li>
<li>1 vCPU</li>
</ul>
<h3>Install Illumian</h3>
<p>Boot from your illumian ISO and start the install process.</p>
<p>select language, keyboard and timezone.</p>
<p>You&#8217;ll be prompted to select the installation drives, very similar to a Nexenta installation, I&#8217;ve selected the two 8G drives as shown below.</p>
<p>disk layout<br />
<a href="http://conrey.org/files/2012/03/illumian01.png"><img src="http://conrey.org/files/2012/03/illumian01-300x172.png" alt="" title="illumian01" width="300" height="172" class="aligncenter size-medium wp-image-1517" /></a></p>
<p>It will ask if you would like to add an additional drive as a hot spare.  I selected no.<br />
Next it will prompt you to make sure you want to erase all the data on the drives, select yes.</p>
<p>Once the installation is complete, it will prompt you to enter the root password for the system.  </p>
<p><a href="http://conrey.org/files/2012/03/illumian02.png"><img src="http://conrey.org/files/2012/03/illumian02-300x166.png" alt="" title="illumian02" width="300" height="166" class="aligncenter size-medium wp-image-1518" /></a></p>
<p>Enter the password twice you want to use and select next.</p>
<p>Create a user account.</p>
<p><a href="http://conrey.org/files/2012/03/illumian03.png"><img src="http://conrey.org/files/2012/03/illumian03-300x169.png" alt="" title="illumian03" width="300" height="169" class="aligncenter size-medium wp-image-1519" /></a></p>
<p>enter the password twice for the new user, and select next.<br />
Commit the changes for the new user.</p>
<p>Enter the hostname for the new system, select next, and commit the changes for the hostname.</p>
<p><a href="http://conrey.org/files/2012/03/illumian04.png"><img src="http://conrey.org/files/2012/03/illumian04-300x165.png" alt="" title="illumian04" width="300" height="165" class="aligncenter size-medium wp-image-1520" /></a></p>
<p>The next step is configuring the network adapter.  Illumian (and Nexenta) allow for some pretty serious network configurations out of the box, but for this one we&#8217;re going to just use dhcp for getting up and running.  We can go back and change it as needed later.</p>
<p>You should be presented for the network interface that you&#8217;ve assigned or is installed in your system.  Select Yes to configure it.</p>
<p>When prompted if you want to enable DHCP, select yes.<br />
When prompted if you want to enable ipv6 I selected no.  <insert long drawn out ipv6 tirade here><br />
Select yes that you&#8217;re done configuring the device and the installation will proceed.</p>
<p>With the installation completed, disconnect your cdrom and reboot into your new illumian installation.</p>
<p><a href="http://conrey.org/files/2012/03/illumian05.png"><img src="http://conrey.org/files/2012/03/illumian05-300x167.png" alt="" title="illumian05" width="300" height="167" class="aligncenter size-medium wp-image-1521" /></a></p>
<p>Like ubuntu, the first user created has sudo privileges. First thing after logging in as your user you&#8217;ll want to do is update the system.</p>
<p><code>sudo apt-get update</code></p>
<p>Once that&#8217;s complete, we can install git</p>
<p><code>sudo apt-get install git</code></p>
<p>Before we actually get started with sparkleshare we want to create a new mirrored zpool with the additional 16G drives we added to the VM.  With illumian, (and Nexenta) we can grow this zpool as needed when we need additional space.  I don&#8217;t expect to need more than 16G to start, but I want to keep my options open.</p>
<p>First let&#8217;s get a look at what the installer did:<br />
<code>zpool status</code></p>
<p><a href="http://conrey.org/files/2012/03/illumian06.png"><img src="http://conrey.org/files/2012/03/illumian06-300x150.png" alt="" title="illumian06" width="300" height="150" class="aligncenter size-medium wp-image-1522" /></a></p>
<p>with that listed we can see what drives are being used by the syspool.</p>
<p>The easiest way to get a drive list is via format.<br />
<code>sudo format</code></p>
<p><a href="http://conrey.org/files/2012/03/illumian07.png"><img src="http://conrey.org/files/2012/03/illumian07-300x118.png" alt="" title="illumian07" width="300" height="118" class="aligncenter size-medium wp-image-1523" /></a></p>
<p>This shows us the initial two 8G drives that we saw in the initial zpool as well as the additional two drives that aren&#8217;t currently in use.<br />
press:<br />
<code>ctrl+c</code><br />
to exit format.</p>
<p>Now we&#8217;ll create our new zpool for the sparkleshare repository.</p>
<p><code> sudo zpool create sparkle mirror c1t2d0 c1t3d0</code></p>
<p>another:</p>
<p><code> zpool status</code></p>
<p>should show us our new zpool:</p>
<p><a href="http://conrey.org/files/2012/03/illumian08.png"><img src="http://conrey.org/files/2012/03/illumian08-300x293.png" alt="" title="illumian08" width="300" height="293" class="aligncenter size-medium wp-image-1524" /></a></p>
<p>One thing to remember if you&#8217;re new to illumian or ZFS in general, when you create a zpool there is automatically a zfs filesystem created on it. The mountpoint defaults to the poolname. So my pool &#8220;sparkle&#8221; is mounted as a zfs filesystem at /sparkle by default.</p>
<p><a href="http://conrey.org/files/2012/03/illumian09.png"><img src="http://conrey.org/files/2012/03/illumian09-300x94.png" alt="" title="illumian09" width="300" height="94" class="aligncenter size-medium wp-image-1525" /></a></p>
<p>Now we can configure illumian for hosting the sparkleshare repository.</p>
<p>Create our user<br />
<code>sudo useradd git</code><br />
Set the password for the user as a value that cannot be used for login, but does not lock the account.<br />
<code>sudo passwd -N git</code><br />
Create our project folder<br />
<code>sudo mkdir /sparkle/git</code><br />
Change the permissions on the project folder<br />
<code>sudo chown -R git.staff /sparkle/git</code><br />
<code>cd /sparkle/git</code><br />
create the project<br />
<code>sudo git init ––bare MyProject</code><br />
create and change dir to the users home directory<br />
<code>sudo mdkir /home/git</code><br />
<code>cd /home/git</code><br />
create the ssh folder<br />
<code>sudo mkdir .ssh</code><br />
create the authorized keys file<br />
<code>sudo touch .ssh/authorized_keys</code><br />
set the permissions on git&#8217;s home directory<br />
<code>sudo chown -R git.staff /home/git</code></p>
<p>On the Client System<br />
Install sparkleshare for you OS, I&#8217;m going to be testing with the new Windows client.  Head over to sparkleshare.org and download the windows client.</p>
<p>Start the installer.</p>
<p><a href="http://conrey.org/files/2012/03/illumian11.png"><img src="http://conrey.org/files/2012/03/illumian11-300x234.png" alt="" title="illumian11" width="300" height="234" class="aligncenter size-medium wp-image-1528" /></a></p>
<p>It&#8217;s a simple accept the license next, next, install, finish affair.</p>
<p>Start SparkleShare and enter your name and email address. </p>
<p><a href="http://conrey.org/files/2012/03/illumian12.png"><img src="http://conrey.org/files/2012/03/illumian12-300x207.png" alt="" title="illumian12" width="300" height="207" class="aligncenter size-medium wp-image-1529" /></a></p>
<p>Continue through the installer to finish the installation.</p>
<p>Once done, you&#8217;ll see a SparkleShare icon in your taskbar.</p>
<p>Right click on it and open your SparkleShare folder.</p>
<p><a href="http://conrey.org/files/2012/03/illumian13.png"><img src="http://conrey.org/files/2012/03/illumian13-237x300.png" alt="" title="illumian13" width="237" height="300" class="aligncenter size-medium wp-image-1530" /></a></p>
<p>During the install SparkleShare will have created a key for you to use.  This key will be in your SparkleShare directory. Copy the contents of the key file.  </p>
<h3>Back on the Server</h3>
<p>Open up the .ssh/authorized_keys file we created earlier in a text editor and paste the key that you previously copied at The client side step on a new line at the end of the file. This gives the client access to the git repository. </p>
<p>Now lock down the ssh directory and the authorized key file.</p>
<p><code>sudo chmod 700 .ssh</code><br />
<code>sudo chmod 600 .ssh/authorized_keys</code><br />
<code>sudo chown -R git.staff . </code></p>
<h3>Add a project to the client</h3>
<p>With that done, move or delete the key in the sparkleshare directory.  Notice that before moving (or deleting) this file, the option to &#8220;Add Hosted Project&#8221; is grayed out.  The key file needs to be moved or deleted before you are allowed to add a project.</p>
<p><a href="http://conrey.org/files/2012/03/illumian14.png"><img src="http://conrey.org/files/2012/03/illumian14-234x300.png" alt="" title="illumian14" width="234" height="300" class="aligncenter size-medium wp-image-1532" /></a></p>
<p>Right click on the sparkleshare folder and select &#8220;Add Hosted Project&#8221;</p>
<p>In the Setup screen, select &#8220;On my own Server&#8221; and enter the address and remote path you created for the project.</p>
<p><a href="http://conrey.org/files/2012/03/illumian15.png"><img src="http://conrey.org/files/2012/03/illumian15-300x207.png" alt="" title="illumian15" width="300" height="207" class="aligncenter size-medium wp-image-1534" /></a></p>
<p>Select Add and when complete, Finish!</p>
<h3>You&#8217;re up and running!</h3>
<p>From here you can add files to your project folder and it will sync the data with the server automatically.</p>
<p>If you want to give more people access to your project, just paste their keys on a new line at the end of the authorized_keys file.  Don&#8217;t share your client key, append it.  This will allow you to add a lot of users based on the project as needed.</p>
<p>On top of this you can add multiple projects and servers to sync to. This opens up a range of ideas, from projects down to the team level, to organization wide distribution.  </p>
<p>On top of this you get all the benefits of ZFS at the file system layer managing your data.</p>
<p>Let me know if you have any questions, or if this works for you.  SparkleShare is a great looking project and when you add in the data security that ZFS provides it makes a pretty solid combination.</p>
<p>When you&#8217;re ready, continue on to part two.</p>
]]></content:encoded>
			<wfw:commentRss>http://conrey.org/2012/03/12/sparkleshare-on-illumian-howto-a-privately-hosted-dropbox-replacement/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>Nexenta features and why they matter.</title>
		<link>http://conrey.org/2011/12/30/nexenta-features-and-why-they-matter/</link>
		<comments>http://conrey.org/2011/12/30/nexenta-features-and-why-they-matter/#comments</comments>
		<pubDate>Fri, 30 Dec 2011 14:46:27 +0000</pubDate>
		<dc:creator>theron</dc:creator>
		
		<guid isPermaLink="false">http://www.conrey.org/?p=1477</guid>
		<description><![CDATA[Initially I was going to post this article as part of an install and configure series that I&#8217;m writing, but quickly realized that it&#8217;s going to be a probably going to be a reference article for other stuff as well. Hopefully, as I continue to learn and pick things up, this article will expand as [...]]]></description>
				<content:encoded><![CDATA[<p>Initially I was going to post this article as part of an install and configure series that I&#8217;m writing, but quickly realized that it&#8217;s going to be a probably going to be a reference article for other stuff as well. Hopefully, as I continue to learn and pick things up, this article will expand as well.<br />
<span id="more-1477"></span></p>
<h3>Why Nexenta is different</h3>
<p>If you&#8217;re coming from the standard storage arena, it&#8217;s hard to put a bead on Nexenta. First off, it&#8217;s just software. But more than that, at it&#8217;s core is open source software. Think of it as a <a href="http://en.wikipedia.org/wiki/Gobstopper">gobstopper</a> with three layers. At it&#8217;s core it&#8217;s based off of illumos and ZFS. Added on top is NexentaOS, a ubuntu userland that makes software management easy. If you&#8217;re ever tried Ubuntu, you know that it&#8217;s pretty simple to install updates. The third and outermost layer to Nexenta are the additional modules that add additional enterprise functionality to the storage appliance.</p>
<h3>What is illumos</h3>
<p>The illumos project started as a binary compatible distribution of OpenSolaris, the comunity supported distribution of Solaris.  Years ago, for me anyway, it was the most cost effective way to learn Solaris.  The intent was to create a straight fork that would stay compatible with any possible future releases that Oracle makes to the community.  It turned out this was a pretty wise move.  The OpenSolaris project within Sun didn&#8217;t adjust well to the transition [1].  Now illumos stands at the center of a new OS community, with a <strong>SOLID</strong> enterprise class foundation.  Many of the folks that were contributing/creating to the solaris codebase continue to contribute to illumos, and it will continue to be an interesting project to watch.</p>
<h3>What is ZFS</h3>
<p>In short, ZFS is an enterprise grade file system. It was designed by Sun to be a modern filesystem to handle modern hardware. To understand why ZFS is important you also need to understand how modern harddrives work:</p>
<blockquote><p>A modern hard disk devotes a large portion of its capacity to error detection data. Many errors occur during normal usage, but are corrected by the disk&#8217;s internal software, and thus are not visible to the host software. A tiny fraction of errors are not corrected. For example, a modern Enterprise SAS disk specification estimates this fraction to be one uncorrected error in every 1016 bits, or approximately one in every 1.2 PB.[19] A smaller fraction of errors are not even detected by the disk firmware or the host operating system. This is known as &#8220;silent corruption&#8221;. In a recent study, <a href="http://indico.cern.ch/getFile.py/access?contribId=3&amp;sessionId=0&amp;resId=1&amp;materialId=paper&amp;confId=13797">CERN found this issue to be problematic</a>.</p></blockquote>
<p>As hard drives get bigger, we&#8217;re all seeing data corruption become more of an issue. The changes of corruption, both noticed and silent, increase. In vmware envionments silent data corruption can ruin your day. I&#8217;m actually not sure how common it is, but in a service provider environment I worked at we had VMs that would not let us do anything with the vmdk. It seemed &#8220;stuck&#8221;. We couldn&#8217;t storage vmotion it, and it would continually give us grief. After troubleshooting for couple days, the only thing we could do was clone it and delete the original VM. That&#8217;s worked fine, but it could have been a lot worse. As storage envionments get bigger, silent data corruption will eventually happen in your environment unless you protect against it.</p>
<p>ZFS checksums every block that is written and automatically repairs blocks where corruption occurs. This provides end to end data integrity allowing you to be confident there won&#8217;t be a problem with the data when you need it.</p>
<p>Beyond this feature, ZFS allows for heterogeneous block and file replication, <strong>UNLIMITED</strong> snapshots and clones, compression, deduplication, non-disruptive volume increases and a host of other features. I&#8217;ll work through (and link to from here) these features in the future, but it&#8217;s important to get some other base functions explained.</p>
<h3>What is the ARC</h3>
<p>As legacy storage vendors will tell you; Cache is king.  The ability to cache data efficiently or &#8220;move blocks&#8221; or &#8220;other fancy caching name here&#8221; basically does one thing.  It moves data that is most accessed to a faster tier, or more responsive, layer.  ZFS leverages the ARC as it&#8217;s front line cache for most accessed blocks.  This means if your Nexenta appliance has 24G of RAM, it&#8217;s going to get 23 or so gigs to use to cache the most heavily used blocks.  As RAM is added, the ARC cache grows.  RAM being A LOT faster than SSD&#8217;s this proves invaluable for providing FAST response times and lower latency to the client devices.</p>
<h3>What is the L2ARC</h3>
<p>Below the ARC ZFS can leverage additional cache devices, in most configurations today this is SSD drives.  Cheaper than RAM, but still way faster than spinning tin.  This is the 2nd level of cache that Nexenta appliances use for caching &#8220;warm&#8221; data.  This hybrid storage model allows for Nexenta to use slow SLOOOOOWWWW disks for it&#8217;s main storage pool of the working size set (average amount of blocks that are constantly active) fit inside both the L2ARC and ARC cache.</p>
<div></div>
<h3>What is the ZIL</h3>
<p>The ZIL (ZFS intent Log) is the write cache.  Random R/W is the bane of every VMware administrators life.  Not only does Nexenta leverage a read cache, we can cache writes to more efficiently lay down blocks on the slow disks to better handle write workloads.</p>
<h3>How does all of this tie together?</h3>
<p>For our lab environment, if you can move the L2ARC and the ZIL to an SSD drive, you&#8217;ll already start being able to use some of the performance gains that Nexenta brings to the table. Even in a two drive configuration, if one of them is SSD, and there is plenty of Ram available, Nexenta can start helping increase performance through it&#8217;s caching technology.  But that&#8217;s not it, Nexenta brings to the table a host of other features.</p>
<ol>
<ol>
<li>Nexenta provides a software stack that can deliver NAS (NFS/CIFS) and/or SAN (iSCSI/FC)<br />
storage that allows for high performance allowing users to leverage current commodity technologies (ex SSD for performance).</li>
<li>It provides UNLIMITED snapshots and clones</li>
<li>It can be configured in a fully active/active HA configuration.</li>
<li>It provides block and file level replication that can be leveraged for easy DR configurations either in a syncronous or asynchronous configuration.</li>
<li>It&#8217;s vmware certified</li>
<li>It provides compession, dedupe and inline virus scanning. (not post process!)</li>
<li>metro clustering</li>
<li>global namespace</li>
<li>(this goes on and on)</li>
</ol>
</ol>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>[1] The opensolaris project wasn&#8217;t alone. Oracle&#8217;s purchase of Sun resulted in the forking of <a href="http://www.libreoffice.org/">LibreOffice</a> from OpenOffice.org, and <a href="http://mariadb.org/">MariaDB</a> from MySQL. <a href="http://www.youtube.com/watch?v=-zRN7XLCRhc">Bryan Cantrill&#8217;s talk at LISA</a>  this year pretty much sums up oracle in a nutshell.</p>
<p>&nbsp;</p>
]]></content:encoded>
			<wfw:commentRss>http://conrey.org/2011/12/30/nexenta-features-and-why-they-matter/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>looking forward to 2012</title>
		<link>http://conrey.org/2011/11/21/2011-wrap-up-and-looking-forward-to-2012/</link>
		<comments>http://conrey.org/2011/11/21/2011-wrap-up-and-looking-forward-to-2012/#comments</comments>
		<pubDate>Mon, 21 Nov 2011 13:00:18 +0000</pubDate>
		<dc:creator>theron</dc:creator>
		
		<guid isPermaLink="false">http://www.conrey.org/?p=1389</guid>
		<description><![CDATA[I&#8217;ve spent the last few years working at a service provider working on client facing VMware projects, rolling out out a VMware vCloud environment, and helping shape how VMware can be leveraged internally. One of great things for me professionally was a chance to get my hands deeper into VMware&#8217;s product catalog from about every [...]]]></description>
				<content:encoded><![CDATA[<p>I&#8217;ve spent the last few years working at a service provider working on client facing VMware projects, rolling out out a <a href="http://www.vmware.com">VMware</a> <a href="http://www.vmware.com/products/vcloud/overview.html">vCloud</a> environment, and helping shape how VMware can be leveraged internally. One of great things for me professionally was a chance to get my hands deeper into VMware&#8217;s product catalog from about every angle imaginable.<br />
It was awesome getting a look behind the curtains at how a really great datacenter and internet company works to constantly deliver solid solutions to their customers.   With this year drawing to a close soon, I&#8217;m eagerly looking forward to new projects and new challenges to tackle.</p>
<p><span id="more-1389"></span></p>
<p>Outside the office I had the opportunities to get involved elsewhere as well.  <a href="http://www.vmworld.com">VMworld</a> this year was awesome and I&#8217;m still watching presentations I didn&#8217;t watch live. <a href="http://www.vmunderground.com">VMunderground</a> this year was pretty amazing.  We had our biggest event ever. As always we had great sponsors and we were really impressed with the support and turnout from the community. I was invited to attend TechFieldDay6 but due to an unruly internal organ, attended <a href="http://techfieldday.com/2011/tfd7/">TechFieldDay7</a> instead. Stephen Foskett runs a great event, and I was honored to be invited.</p>
<p>Locally, and against better judgement, I was voted in as president for my local LUG, and even though I didn&#8217;t have any intentions of running, I&#8217;m looking forward to it nonetheless.  I&#8217;m also excited that we may be able to start more coordinated and focused efforts with the Iowa Ubuntu LoCo, as I&#8217;m moving into a more active role there as well.</p>
<p>One of the more interesting things I&#8217;ve done this year was take some time to attend the <a href="http://www.conrey.org/?p=1414">Open Storage Summit</a> in San Jose.  Great week, great presentations, and I took away a ton of information.  Maybe more importantly, I took away the knowledge that these guys (and gals) are really building something different, something open, and something with community at the forefront.  The team around illumos is really focused on making a difference.  <a href="http://www.nexenta.com">Nexenta</a> has been a big supporter of VMUGs around the globe, and of course <a href="http://www.vmunderground.com">VMunderground</a> as well.</p>
<p>The storage market has got to be a tough nut to crack.  <em>Buying your storage from a software company</em> is something new in the minds of many companies and IT management, but it isn&#8217;t a new idea.  Most of us in the industry today are completely ok with <em>buying our compute and OS resource management from a software company</em>. VMware has effectively moved the cheese for many MANY hardware companies.  I don&#8217;t need to buy &#8220;windows specific&#8221; or linux optimized&#8221; hardware.  Solid hardware that supports VMware is enough.  This change isn&#8217;t done yet.  I think Stephen Foskett&#8217;s assessment of <a href="http://blog.fosketts.net/2011/10/31/vmware-oedipus-server-virtualization-change-storage/">VMware as Oedipus</a> has legs.  This doesn&#8217;t mean that traditional storage vendors are going to toss in the towel, there is significant innovation going on <em>industry wide</em> and they&#8217;ve got huge war chests to spend.  It just means, like always, things are changing.  This is a really exciting time to be involved in technology.</p>
</p>
<p>With the adaption of vmware&#8217;s VSA it&#8217;s important to realize, now more than ever, you don&#8217;t have to wrap mostly expensive tin around your software <em>anywhere</em>, anymore.  You <strong>can</strong> leverage commodity hardware.  You <strong>can</strong> build what you need, <strong>when</strong> you need it. </p>
<p>Which brings me back to the beginning of this tirade.  New projects and new challenges.  At this point it should be pretty evident where I&#8217;m placing my chips.  It was a tough decision but I&#8217;ve decided to leave <a href="http://www.lightedge.com">LightEdge</a> and jump feet first into the open storage arena.  I&#8217;ll be starting soon with <a href="http://www.nexenta.com">Nexenta</a> helping get open storage based solutions deployed to customers around the globe.  <a href="http://www.lightedge.com">Lightedge</a> has been a great company to work for and I wish them the best of luck as they move forward.</p>
<p>In charting a new course I&#8217;m looking forward to new and exciting challenges.  Wish me luck!</p>
]]></content:encoded>
			<wfw:commentRss>http://conrey.org/2011/11/21/2011-wrap-up-and-looking-forward-to-2012/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>building virtual appliances on ubuntu</title>
		<link>http://conrey.org/2011/11/14/building-virtual-appliances-on-ubuntu/</link>
		<comments>http://conrey.org/2011/11/14/building-virtual-appliances-on-ubuntu/#comments</comments>
		<pubDate>Tue, 15 Nov 2011 00:07:21 +0000</pubDate>
		<dc:creator>theron</dc:creator>
		
		<guid isPermaLink="false">http://www.conrey.org/?p=1435</guid>
		<description><![CDATA[Ubuntu to me is still the best OS to use for virtual appliances. It may not be for the reasons that you&#8217;re thinking about so here&#8217;s my short list. it&#8217;s wildly successful to me this means lots of updates and support for packages both from the vendor (canonical) as well as the community. support available [...]]]></description>
				<content:encoded><![CDATA[<p><a href="http://www.ubuntu.com">Ubuntu</a> to me is still the best OS to use for virtual appliances. It may not be for the reasons that you&#8217;re thinking about so here&#8217;s my short list.</p>
<ul>
<li>it&#8217;s wildly successful</li>
</ul>
<p>to me this means lots of updates and support for packages both from the vendor (canonical) as well as the community.</p>
<ul>
<li>support available from ONE install</li>
</ul>
<p>What other distribution has one install for the community AND commercial support? Yea. I can&#8217;t think of one either. This allows users to choose how much support they want/need from one virtual appliance/installation.</p>
<ul>
<li>small minimal virtual machine footprint</li>
</ul>
<p>The ovf will fit neatly on a 512M usb drive when complete.<br />
<span id="more-1435"></span></p>
<h3>why another virtual appliance tutorial?</h3>
<p>I know that building virtual appliances is old hat for many of you, but I keep getting the question about how to build virtual appliances for distribution. In a recent scenario at my work, they were looking for a splunk virtual appliance that members of the consulting group could quickly leave on site. I thought this would be a good walkthrough. Let me know if you have any questions.</p>
<h3>let&#8217;s get started!</h3>
<p>First things first we need to download a copy of <a href="http://www.ubuntu.com/download/server/download">ubuntu server</a>. Once you&#8217;ve got your iso downloaded, go ahead and build out the base VM that you&#8217;re going to use. For these types of deployments, I go ahead and configure the appliance inside a vApp as we are starting to deploy more of these on vCloud deployments.</p>
<h3>build your vApp and VM</h3>
<ul>
<li>In vSphere, right click on your cluster or host and select New vApp.</li>
<li>Name your vApp, select next.</li>
<li>If you need/want to isolate resources specifically for this vApp you can do so on the Resource Allocation screen. I&#8217;ll go ahead and reserve 1G of Memory cause I&#8217;m greedy like that. Select next.</li>
<li>Select Finish and find your vApp in the VM list.<br />
Now we need to add the VM to the vApp.</li>
<li>Right click on the vApp and select New Virtual Machine.</li>
<li>Name your virtual machine and select Next (this demo will be for spunk so I&#8217;ll use the ever so cunning name splunk. Select Next.</li>
<li>select which datastore you want this VM to temporarily live, select next.</li>
<li>select the Guest Operating System for this VM, in this case, select the Linux radio button, and select Ubuntu Linux (64-bit) in the Version dropdown window. Select Next. (you did download the 64bit version right?!)</li>
<li>Set the size of your virtual appliance, I&#8217;ll crank this down to 10G, as the splunk data directories will be remote NFS mounted in most cases. I&#8217;m a huge fan of thin provisioning. check it if it&#8217;s appropriate for your workload. select Next.</li>
<li>Select Finish.<br />
<h3>install ubuntu</h3>
<p>mount the iso to your vm, and start &#8216;er up. Once you&#8217;ve selected your language, on the initial install screen&#8230;</li>
<li>select f4</li>
<li>
<p style="clear: both;">select Install a minimal virtual machine [1] <img src="http://conrey.org/files/2011/11/virtappliance01.png" alt="" width="519" height="471" /></p>
</li>
<li>select enter (this takes you back to the main screen)</li>
<li>select Install Ubuntu Server (should be checked by default) Select Next</li>
<li>select your language again (don&#8217;t ask) Hit Enter</li>
<li>select your location. hit Enter</li>
<li>select No for keyboard layout</li>
<li>select your Keyboard language (English (US) is default)</li>
<li>select your Keyboard layout (English (US) is default)</li>
<li>The system will load additional components as needed</li>
<li>Network configuration. I SERIOUSLY recommend DHCP for just about everything. It&#8217;s 2011 folks, DNS and DHCP works.</li>
<li>give your appliance a hostname (splunkappliance in my case) Select next.</li>
<li>it&#8217;ll start up the clock and check your time zone. change it if needed or select Yes.</li>
<li>For partitioning, I select Guided &#8211; use entire disk and set up LVM to make it easier to expand the install as needed down the road if required. Select this and hit enter</li>
<li>default vm will have one disk listed, select it and hit enter.</li>
<li>Select Yes to write the changes to disk, and hit enter</li>
<li>If using LVM you&#8217;ll be prompted with the maximum amount of space in the volume group. Select Continue and hit enter.</li>
<li>Select Yes to write the changes to disk and hit enter</li>
<li>Enter your default full username, select continue.</li>
<li>Enter your default user name, select continue.</li>
<li>Enter your default password, select continue.</li>
<li>You&#8217;ll be prompted with &#8220;How do you want to manage upgrades on this system?&#8221; I select install security updates automatically so I don&#8217;t have to worry about being blamed for something that is WAY outside my control 6 months from now. It&#8217;ll still happen, but I&#8217;ll sleep easier. Select which of the first two you want, and hit enter.</li>
<li>Choose software to install. The only thing I select here is OpenSSH server. that&#8217;s it. select it and select continue.</li>
<li>This will start the package install, and it should go relatively fast</li>
<li>select Yes to install the GRUB boot loader to the master boot record.</li>
<li>Select Continue to finish your installation</li>
<li>unmount the iso from vSphere client</li>
<li>The above will install a really base system for your virtual appliance. I go ahead and install openssh-server on the vm so that it doesn&#8217;t need unproxied internet access during it&#8217;s first boot to get it installed. As I&#8217;m regenerating the ssh keys anyway, I don&#8217;t worry about it too much. This VM should boot REALLY really fast.<br />
<h3>/etc/issue</h3>
<p>/etc/issue.net is the file that handles what is displayed before a user logs in. like this:<br />
<img src="http://conrey.org/files/2011/11/virtappliance02.png" alt="" width="468" height="281" /></p>
<p>that&#8217;s pretty boring right? what we want for our appliance is to have something awesome like this:</p>
<p><img src="http://conrey.org/files/2011/11/virtappliance03.png" alt="" width="408" height="269" /></p>
<p>To get this we&#8217;ll need to do a couple of things. we&#8217;ll want to create a new script that is run at bootup <strong>every time</strong>. This will generate the /etc/issue.net for us after we get our ip address. Here&#8217;s the contents of a generic one I used for this demo. Adjust as needed.</p>
<p><sourcecode language="text"><br />
#! /bin/bash<br />
#<br />
# issue-reset set ipaddr in /etc/issue<br />
#<br />
# chkconfig: 2345 90 60<br />
# description: this will echo the ip address to /etc/issue for display<br />
#<br />
# Source function library.<br />
#. /etc/init.d/functions<br />
RETVAL=0<br />
#<br />
# See how we were called.<br />
#<br />
prog=&#8221;issue-reset&#8221;<br />
start() {<br />
#<br />
#change /etc/issue to display ip address<br />
#<br />
echo &#8220;AWESOMETACULAR Splunk Appliance&#8221; &gt; /etc/issue<br />
echo &#8220;Powered by Ubuntu Linux and caffeine&#8221; &gt;&gt; /etc/issue<br />
echo &#8221; &#8221; &gt;&gt; /etc/issue<br />
netcount=$(/sbin/ifconfig | grep -o &#8220;inet&#8221;)<br />
if [[ $netcount &gt; 1 ]]<br />
then<br />
echo &#8220;To manage this virtual appliance please use a web &#8221; &gt;&gt; /etc/issue<br />
echo &#8220;browser on a different system and navigate to &#8221; &gt;&gt; /etc/issue<br />
echo &#8221; &#8221; &gt;&gt; /etc/issue<br />
/sbin/ifconfig | grep &#8220;inet addr&#8221; | grep -v &#8220;127.0.0.1&#8243; | awk &#8216;{ print $2 }&#8217; | awk -F: &#8216;{ print &#8220;http://&#8221;$2&#8243;:8000&#8243; }&#8217;&gt;&gt; /etc/issue<br />
echo &#8221; &#8221; &gt;&gt; /etc/issue<br />
else<br />
echo &#8220;This appliance does not have networking configured. Please log in to configure networking&#8221; &gt;&gt; /etc/issue<br />
fi<br />
echo &#8221; &#8221; &gt;&gt; /etc/issue<br />
cp /etc/issue /etc/issue.net<br />
# RETVAL=$?<br />
echo<br />
[ $RETVAL -eq 0 ]<br />
return $RETVAL<br />
}<br />
case &#8220;$1&#8243; in<br />
start)<br />
start<br />
;;<br />
*)<br />
echo $&#8221;Usage: $0 {start}&#8221;<br />
exit 1<br />
esac</p>
<p>exit $?<br />
</sourcecode></p>
<p>Once you&#8217;ve downloaded this, we&#8217;re going to move it to the /etc/init.d directory and rename it banner:<br />
<code>mv ./banner.txt /etc/init.d/banner</code><br />
change the owner to root, and make it executable.<br />
<code>sudo chmod +x /etc/init.d/banner &amp;&amp; sudo chown root.root /etc/init.d banner</code><br />
add /etc/init.d/banner to be run at startup:<br />
<code>sudo update-rc.d banner defaults</code></p>
<p>that&#8217;s it! You&#8217;ll want to edit that file to make it reflect what you want it to say, but it&#8217;s pretty self explanatory.</p>
<h3>firstboot startup script</h3>
<p>There are a couple of ways to handle first boot only startup scripts. The easiest way I&#8217;ve found is to edit the rc.local file to run a script, and when complete, clean up after itself. Here&#8217;s what my firstboot script looks like:</p>
<p><sourcecode language="perl"><br />
#This script is run the first time a user logs in<br />
echo &#8220;Your Splunk appliance is almost finished being deployed&#8221;</p>
<p>INSTALL=&#8221;apt-get install -y&#8221;<br />
REMOVE=&#8221;apt-get remove -y&#8221;<br />
SPLUNK_HOME=&#8221;/opt/splunk&#8221;<br />
apt-get update</p>
<p># email<br />
$INSTALL exim4-daemon-heavy</p>
<p># expire the user account passwd<br />
passwd -e defaultuser</p>
<p># new ssh key generation<br />
rm /etc/ssh/ssh_host*key*<br />
dpkg-reconfigure -f noninteractive -p critical openssh-server</p>
<p># splunk<br />
dpkg -i /root/splunk-4.2.4-110225-linux-2.6-amd64.deb</p>
<p>#start splunk server for the first time<br />
$SPLUNK_HOME/bin/splunk start &#8211;accept-license</p>
<p>#configure splunk to autostart at boot<br />
$SPLUNK_HOME/bin/splunk enable boot-start</p>
<p># clean up<br />
sed -i &#8216;s_/root/firstboot.sh_exit 0_&#8217; /etc/rc.local</p>
<p>echo &#8220;firstboot complete&#8221;<br />
</sourcecode></p>
<p>You&#8217;ll want create a file with that as the content, save it, and put it somewhere where it can be run during boot. For ease of this demo, we&#8217;re going to put it in our home folder. Don&#8217;t Judge, this is just a demo.</p>
<p><code>sudo mv ./firstboot.txt /root/firstboot.sh &amp;&amp; sudo chmod +x /root/firstboot.sh &amp;&amp; sudo chown root.root /root/firstboot.sh</code></p>
<p>Now we&#8217;re going to setup this script to be run during first boot. using whatever editor you want edit /etc/rc.local and remove the exit 0 line and substitute it with the following line:</p>
<p><code>/root/firstboot.sh</code></p>
<p>Let&#8217;s walk through firstboot.sh and take a look at what we&#8217;re doing. In our example, we&#8217;re creating a splunk virtual appliance, so from the top:</li>
<li>we&#8217;re updating our repositories</li>
<li>we&#8217;re installing an MTA for splunk to use.</li>
<li>we&#8217;re expiring the password for our default user account. Change this to reflect whatever username you selected.</li>
<li>we&#8217;re deleting and regenerating ssh keys</li>
<li>we&#8217;re installing splunk. For this example, you&#8217;ll need to get the splunk .deb package downloaded and put somewhere accessible. It&#8217;s free, go grab it now.</li>
<li>Now we&#8217;re going to do some startup specific commands for splunk, first we&#8217;re going to set splunk to start for the first time and accept the license.</li>
<li>Next we&#8217;re going to go ahead and configure splunk to start every time the server starts.</li>
<li>after that&#8217;s complete, we&#8217;re going to clean up /etc/rc.local to not run this script again.That&#8217;s it for our startup script. pretty straightforward.<br />
<h3>Test your appliance</h3>
</li>
<li>power off the VM and create a snapshot.</li>
<li>Power the vm on and test the startup, the install of our splunk package, and the firstboot configuration steps.</li>
<li>If everything works, go ahead and roll back our changes and create your ovf below</li>
<li>if it doesn&#8217;t power back on and dig in <img src='http://conrey.org/wp-includes/images/smilies/icon_smile.gif' alt=':)' class='wp-smiley' /><br />
<h3>add some info to your appliance</h3>
<p>If you&#8217;re planning on posting updates to this, or you want to provide the default username and password information directly in the ovf file, you&#8217;ll want to edit some of the details of your vApp.</li>
<li>Right-click on your vApp and select Edit Settings.</li>
<li>On the options tab, select Advanced.</li>
<li>Enter your appliance name, website, and any other information that you want to provide.</li>
<li>
<h3>export to ovf</h3>
</li>
<li>From the vSphere client, select the vApp, and in the File menu select Export, Export OVF Template.</li>
<li>pick a directory where you can store your appliance</li>
<li>export!</li>
<li>
<h3>test appliance</h3>
<p>one last test of your appliance and you&#8217;re ready for deployments. This last check for me usually provides a sanity check for the finished appliance. to deploy from vSphere client just:</li>
<li>select file</li>
<li>deploy OVF template</li>
<li>browse to the directory and select the OVF file</li>
<li>the only thing you should have to do other than picking a datastore, is selecting a network</li>
<li><a href="http://twitter.com/zenbiking/status/136209420320374784">Go get a cup of coffee while the deployment occurs</a></li>
<li>done!</li>
<li>
<h3>conclusion</h3>
<p>This walkthrough for a splunk appliance is pretty simple but if done correctly virtual appliances deployments can shave hours off of simple traditional application installations for your team. Getting more advanced versions of this, you could get fancy and add additional layers like puppet autoconfigs to further automate package deployments. Let me know what you think and if this worked for you!</p>
<p>[1] this is what used to be the <a href="https://help.ubuntu.com/community/JeOS">ubuntu JeOS install</a>.</li>
</ul>
]]></content:encoded>
			<wfw:commentRss>http://conrey.org/2011/11/14/building-virtual-appliances-on-ubuntu/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>
