What is SparkleShare?

From the sparkleshare website: “SparkleShare is a collaboration and sharing tool that is designed to keep things simple and to stay out of your way.” Feature wise, it acts very similarly to dropbox, but provides some additional functionality. It leverages a Git repository on the back end, and is extremely flexible.

My only concern with it, or any hosted data story, is ensuring that my data is always available and more importantly, that it doesn’t corrupt over time. That’s why I’m using illumian.

What is Illumian?

illumian was started by Nexenta to create an illumos distribution with Debian style package management tools for familiarity and compatibility with existing tools. This distribution uses the same versions of externally maintained packages as the OpenIndiana project uses today. Where NCP was based on OpenSolaris and some interesting ways to get debian packages ported, illumian is a based on the work in the continued new work in the illumos project. The recent 1.0 release contains only a small number of maintained packages and a simple command line installer. No GUI required.

Download illumian

you can download the illumian iso from illumian.org

Base VM configuration

On the road and working in a hotel, I’m using VMware workstation for this installation and have assigned the following items to the VM:

• 2 8G disks for the syspool (OS install disks)
• 2 16G disks for data (I’ll mirror these as well)
• 2G of RAM
• 1 vCPU

Install Illumian

Boot from your illumian ISO and start the install process.

select language, keyboard and timezone.

You’ll be prompted to select the installation drives, very similar to a Nexenta installation, I’ve selected the two 8G drives as shown below.

disk layout

It will ask if you would like to add an additional drive as a hot spare. I selected no.
Next it will prompt you to make sure you want to erase all the data on the drives, select yes.

Once the installation is complete, it will prompt you to enter the root password for the system.

Enter the password twice you want to use and select next.

Create a user account.

enter the password twice for the new user, and select next.
Commit the changes for the new user.

Enter the hostname for the new system, select next, and commit the changes for the hostname.

The next step is configuring the network adapter. Illumian (and Nexenta) allow for some pretty serious network configurations out of the box, but for this one we’re going to just use dhcp for getting up and running. We can go back and change it as needed later.

You should be presented for the network interface that you’ve assigned or is installed in your system. Select Yes to configure it.

When prompted if you want to enable DHCP, select yes.
When prompted if you want to enable ipv6 I selected no.
Select yes that you’re done configuring the device and the installation will proceed.

With the installation completed, disconnect your cdrom and reboot into your new illumian installation.

Like ubuntu, the first user created has sudo privileges. First thing after logging in as your user you’ll want to do is update the system.

sudo apt-get update

Once that’s complete, we can install git

sudo apt-get install git

Before we actually get started with sparkleshare we want to create a new mirrored zpool with the additional 16G drives we added to the VM. With illumian, (and Nexenta) we can grow this zpool as needed when we need additional space. I don’t expect to need more than 16G to start, but I want to keep my options open.

First let’s get a look at what the installer did:
zpool status

with that listed we can see what drives are being used by the syspool.

The easiest way to get a drive list is via format.
sudo format

This shows us the initial two 8G drives that we saw in the initial zpool as well as the additional two drives that aren’t currently in use.
press:
ctrl+c
to exit format.

Now we’ll create our new zpool for the sparkleshare repository.

sudo zpool create sparkle mirror c1t2d0 c1t3d0

another:

zpool status

should show us our new zpool:

One thing to remember if you’re new to illumian or ZFS in general, when you create a zpool there is automatically a zfs filesystem created on it. The mountpoint defaults to the poolname. So my pool “sparkle” is mounted as a zfs filesystem at /sparkle by default.

Now we can configure illumian for hosting the sparkleshare repository.

Create our user
sudo useradd git
Set the password for the user as a value that cannot be used for login, but does not lock the account.
sudo passwd -N git
Create our project folder
sudo mkdir /sparkle/git
Change the permissions on the project folder
sudo chown -R git.staff /sparkle/git
cd /sparkle/git
create the project
sudo git init ––bare MyProject
create and change dir to the users home directory
sudo mdkir /home/git
cd /home/git
create the ssh folder
sudo mkdir .ssh
create the authorized keys file
sudo touch .ssh/authorized_keys
set the permissions on git’s home directory
sudo chown -R git.staff /home/git

On the Client System
Install sparkleshare for you OS, I’m going to be testing with the new Windows client. Head over to sparkleshare.org and download the windows client.

Start the installer.

It’s a simple accept the license next, next, install, finish affair.

Start SparkleShare and enter your name and email address.

Continue through the installer to finish the installation.

Once done, you’ll see a SparkleShare icon in your taskbar.

Right click on it and open your SparkleShare folder.

During the install SparkleShare will have created a key for you to use. This key will be in your SparkleShare directory. Copy the contents of the key file.

Back on the Server

Open up the .ssh/authorized_keys file we created earlier in a text editor and paste the key that you previously copied at The client side step on a new line at the end of the file. This gives the client access to the git repository.

Now lock down the ssh directory and the authorized key file.

sudo chmod 700 .ssh
sudo chmod 600 .ssh/authorized_keys
sudo chown -R git.staff .

Add a project to the client

With that done, move or delete the key in the sparkleshare directory. Notice that before moving (or deleting) this file, the option to “Add Hosted Project” is grayed out. The key file needs to be moved or deleted before you are allowed to add a project.

Right click on the sparkleshare folder and select “Add Hosted Project”

In the Setup screen, select “On my own Server” and enter the address and remote path you created for the project.

Select Add and when complete, Finish!

You’re up and running!

From here you can add files to your project folder and it will sync the data with the server automatically.

If you want to give more people access to your project, just paste their keys on a new line at the end of the authorized_keys file. Don’t share your client key, append it. This will allow you to add a lot of users based on the project as needed.

On top of this you can add multiple projects and servers to sync to. This opens up a range of ideas, from projects down to the team level, to organization wide distribution.

On top of this you get all the benefits of ZFS at the file system layer managing your data.

Let me know if you have any questions, or if this works for you. SparkleShare is a great looking project and when you add in the data security that ZFS provides it makes a pretty solid combination.

When you’re ready, continue on to part two.

Why Nexenta is different

If you’re coming from the standard storage arena, it’s hard to put a bead on Nexenta. First off, it’s just software. But more than that, at it’s core is open source software. Think of it as a gobstopper with three layers. At it’s core it’s based off of illumos and ZFS. Added on top is NexentaOS, a ubuntu userland that makes software management easy. If you’re ever tried Ubuntu, you know that it’s pretty simple to install updates. The third and outermost layer to Nexenta are the additional modules that add additional enterprise functionality to the storage appliance.

What is illumos

The illumos project started as a binary compatible distribution of OpenSolaris, the comunity supported distribution of Solaris.  Years ago, for me anyway, it was the most cost effective way to learn Solaris.  The intent was to create a straight fork that would stay compatible with any possible future releases that Oracle makes to the community.  It turned out this was a pretty wise move.  The OpenSolaris project within Sun didn’t adjust well to the transition [1].  Now illumos stands at the center of a new OS community, with a SOLID enterprise class foundation.  Many of the folks that were contributing/creating to the solaris codebase continue to contribute to illumos, and it will continue to be an interesting project to watch.

What is ZFS

In short, ZFS is an enterprise grade file system. It was designed by Sun to be a modern filesystem to handle modern hardware. To understand why ZFS is important you also need to understand how modern harddrives work:

A modern hard disk devotes a large portion of its capacity to error detection data. Many errors occur during normal usage, but are corrected by the disk’s internal software, and thus are not visible to the host software. A tiny fraction of errors are not corrected. For example, a modern Enterprise SAS disk specification estimates this fraction to be one uncorrected error in every 1016 bits, or approximately one in every 1.2 PB.[19] A smaller fraction of errors are not even detected by the disk firmware or the host operating system. This is known as “silent corruption”. In a recent study, CERN found this issue to be problematic.

As hard drives get bigger, we’re all seeing data corruption become more of an issue. The changes of corruption, both noticed and silent, increase. In vmware envionments silent data corruption can ruin your day. I’m actually not sure how common it is, but in a service provider environment I worked at we had VMs that would not let us do anything with the vmdk. It seemed “stuck”. We couldn’t storage vmotion it, and it would continually give us grief. After troubleshooting for couple days, the only thing we could do was clone it and delete the original VM. That’s worked fine, but it could have been a lot worse. As storage envionments get bigger, silent data corruption will eventually happen in your environment unless you protect against it.

ZFS checksums every block that is written and automatically repairs blocks where corruption occurs. This provides end to end data integrity allowing you to be confident there won’t be a problem with the data when you need it.

Beyond this feature, ZFS allows for heterogeneous block and file replication, UNLIMITED snapshots and clones, compression, deduplication, non-disruptive volume increases and a host of other features. I’ll work through (and link to from here) these features in the future, but it’s important to get some other base functions explained.

What is the ARC

As legacy storage vendors will tell you; Cache is king.  The ability to cache data efficiently or “move blocks” or “other fancy caching name here” basically does one thing.  It moves data that is most accessed to a faster tier, or more responsive, layer.  ZFS leverages the ARC as it’s front line cache for most accessed blocks.  This means if your Nexenta appliance has 24G of RAM, it’s going to get 23 or so gigs to use to cache the most heavily used blocks.  As RAM is added, the ARC cache grows.  RAM being A LOT faster than SSD’s this proves invaluable for providing FAST response times and lower latency to the client devices.

What is the L2ARC

Below the ARC ZFS can leverage additional cache devices, in most configurations today this is SSD drives.  Cheaper than RAM, but still way faster than spinning tin.  This is the 2nd level of cache that Nexenta appliances use for caching “warm” data.  This hybrid storage model allows for Nexenta to use slow SLOOOOOWWWW disks for it’s main storage pool of the working size set (average amount of blocks that are constantly active) fit inside both the L2ARC and ARC cache.

What is the ZIL

The ZIL (ZFS intent Log) is the write cache.  Random R/W is the bane of every VMware administrators life.  Not only does Nexenta leverage a read cache, we can cache writes to more efficiently lay down blocks on the slow disks to better handle write workloads.

How does all of this tie together?

For our lab environment, if you can move the L2ARC and the ZIL to an SSD drive, you’ll already start being able to use some of the performance gains that Nexenta brings to the table. Even in a two drive configuration, if one of them is SSD, and there is plenty of Ram available, Nexenta can start helping increase performance through it’s caching technology.  But that’s not it, Nexenta brings to the table a host of other features.

1. Nexenta provides a software stack that can deliver NAS (NFS/CIFS) and/or SAN (iSCSI/FC)
   storage that allows for high performance allowing users to leverage current commodity technologies (ex SSD for performance).
2. It provides UNLIMITED snapshots and clones
3. It can be configured in a fully active/active HA configuration.
4. It provides block and file level replication that can be leveraged for easy DR configurations either in a syncronous or asynchronous configuration.
5. It’s vmware certified
6. It provides compession, dedupe and inline virus scanning. (not post process!)
7. metro clustering
8. global namespace
9. (this goes on and on)

[1] The opensolaris project wasn’t alone. Oracle’s purchase of Sun resulted in the forking of LibreOffice from OpenOffice.org, and MariaDB from MySQL. Bryan Cantrill’s talk at LISA  this year pretty much sums up oracle in a nutshell.

Outside the office I had the opportunities to get involved elsewhere as well. VMworld this year was awesome and I’m still watching presentations I didn’t watch live. VMunderground this year was pretty amazing. We had our biggest event ever. As always we had great sponsors and we were really impressed with the support and turnout from the community. I was invited to attend TechFieldDay6 but due to an unruly internal organ, attended TechFieldDay7 instead. Stephen Foskett runs a great event, and I was honored to be invited.

Locally, and against better judgement, I was voted in as president for my local LUG, and even though I didn’t have any intentions of running, I’m looking forward to it nonetheless. I’m also excited that we may be able to start more coordinated and focused efforts with the Iowa Ubuntu LoCo, as I’m moving into a more active role there as well.

One of the more interesting things I’ve done this year was take some time to attend the Open Storage Summit in San Jose. Great week, great presentations, and I took away a ton of information. Maybe more importantly, I took away the knowledge that these guys (and gals) are really building something different, something open, and something with community at the forefront. The team around illumos is really focused on making a difference. Nexenta has been a big supporter of VMUGs around the globe, and of course VMunderground as well.

The storage market has got to be a tough nut to crack. Buying your storage from a software company is something new in the minds of many companies and IT management, but it isn’t a new idea. Most of us in the industry today are completely ok with buying our compute and OS resource management from a software company. VMware has effectively moved the cheese for many MANY hardware companies. I don’t need to buy “windows specific” or linux optimized” hardware. Solid hardware that supports VMware is enough. This change isn’t done yet. I think Stephen Foskett’s assessment of VMware as Oedipus has legs. This doesn’t mean that traditional storage vendors are going to toss in the towel, there is significant innovation going on industry wide and they’ve got huge war chests to spend. It just means, like always, things are changing. This is a really exciting time to be involved in technology.

With the adaption of vmware’s VSA it’s important to realize, now more than ever, you don’t have to wrap mostly expensive tin around your software anywhere, anymore. You can leverage commodity hardware. You can build what you need, when you need it.

Which brings me back to the beginning of this tirade. New projects and new challenges. At this point it should be pretty evident where I’m placing my chips. It was a tough decision but I’ve decided to leave LightEdge and jump feet first into the open storage arena. I’ll be starting soon with Nexenta helping get open storage based solutions deployed to customers around the globe. Lightedge has been a great company to work for and I wish them the best of luck as they move forward.

In charting a new course I’m looking forward to new and exciting challenges. Wish me luck!

overview

First off, if you’re interested in the rapidly changing world around open storage software solutions and/or ZFS based storage, you should try to go next year. seriously. Start thinking about this now. This year had three great tracks and solid keynotes around the evolving world of storage and the datacenter, and it all made for a great show.

event organization

As a regular vmworld attendee, my bar is set pretty high about scale and size. This conf is much MUCH smaller. I quickly made my way through registration, and wandered around the partner area. The “people herding” side of running an event always seems to be where it falls apart, but this event was well organized and ran smoothly.

Thumbing through the schedule one of the things that stuck me was that all of the tracks had solid presentations, for both days. Like vmworld, when I wasn’t sure, I just dropped in on the session that I’d have the most questions to ask at the end of the presentation, hoping that the videos will be made available after the fact (Hey Nexenta, they are going to be made available right?)

Lunch was handled efficiently, and the dining area was large separate room.

keynotes

Keynotes were well planned and Nexenta did a great job getting speakers lined up.

That was just a few of them. All in all, an impressive lineup.

sessions

Out of all of the sessions that I sat through, I’ve got to say three of them stuck with me for different reasons.

I wasn’t expecting what I got in this session. at all. ACTIVE / ACTIVE / ACTIVE metro cluster (yea that’s three way) up and running today on nexenta with vmware. HOLY FRAKING AWESOME.

Sitting through this really made a few lightbulbs go off. the premise of this is why ZFS makes a silly awesome VSA for VDI workloads. I’ll write up something about this here in the coming week or so, it really deserves it’s own blogpost.

This was a closing presentation right before the summit ended. This was your typical “tuning” presentation, except for a few things:

1. it ran long…. like over an hour long.
2. no one got up and left. There were rears in chairs for the whole thing.

parties

You’d have been shocked if I hadn’t mentioned the parties right? Nexenta and sponsors threw a REALLY solid party at the Loft Bar and Bistro. While it wasn’t an official party around Open Storage Summit, some of the vmware folks in the area threw a #vBeers event on thursday night that was well attended and it was great seeing the nexenta team show up in force also. Thanks again to @KokopeIIi for organizing the silicon valley vbeers events.

closing thoughts

The three sessions highlighted above really solidified what I see going on in the open storage arena today:

I’ll be adding some photos that I took later this week. look for them here.