July 25, 2008 – 5:54 am

Disclaimer: This is probably a bad idea in motion, I make no claims as to how well this could hold up in the long run or real world - I have some personal concerns about how Amazon is able to scale their S3 service effectively, and the recent downtime should give cause to think first before putting this into production.

If you want to be a cool geek these days, you absolutely have to insert the words ‘cloud computing’ into every other sentence. It’s no suprise then that one of the more adopted and interesting toys in the last year or two has been the Amazon Simple Storage Service (S3 for short).

It’s a “cloud” based storage solution, where you can upload files onto the cloud relatively cheaply. It’s being used in places Akamai used to be used for mirroring and proxying of frequently accessed files. (Even Linden Lab uses it for distrobuting their updates.)

The price is right, and the solution is fully programmable via a convenient web API, so it’s snuck itself into all sorts of places. What I’d like to experiment here is using it as a complete grid asset server, for those who havnt toyed much with the OpenSim asset server under the hood - it’s effectively a webserver with two supported operations: “PUT” and “GET” - funnily enough this overlaps nicely with any network accessible storage system, S3 included.

Building a new asset storage module from scratch

First thing we need to do is go hunting for another asset storage module in OpenSim so we can take a look at the interface we are required to implement. After a quick hunt for “asset_database” (the OpenSim.ini setting) it turns up that we need to create a class that derives from AssetServerBase which in turn derives from IAssetServer.

public interface IAssetServer { void SetReceiver(IAssetReceiver receiver); void RequestAsset(LLUUID assetID, bool isTexture); void StoreAsset(AssetBase asset); void UpdateAsset(AssetBase asset); void Close(); }

This looks relatively painless so far, we need to make a class which implements all five of these functions. The next step is to go take a look around and see if there is a decent .NET library for interfacing with S3. Turns out there is - .NET S3. (Post-authoring note: Apparently Amazon have an official library which looks easier to use, for one it doesn’t require the delegate hackery seen in the GetAsset function)

This library, while not fantastically documented does seem to do the job quickly and easily.

Implementing the asset client

Now we are at the point where we can actually write our class, using the above library, I’ve skipped ahead and written the class as presented below:

using System.Collections.Generic; using System.IO; using System.Reflection; using System.Xml.Serialization; using log4net; using S3Client; namespace OpenSim.Framework.Communications.Cache { public class S3AssetClient : AssetServerBase { private static readonly ILog m_log = LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType); private S3Connection con; private string dir; // For converting assets to/from XML // (convenient for us, although a binary serialiser will be cheaper storage-space wise) private XmlSerializer xs = new XmlSerializer(typeof(AssetBase)); public S3AssetClient(string s3id, string s3key, string s3dir) { m_log.Info("[S3 Asset Store] Initialising Amazon Asset Client"); con = new S3Connection(s3id, s3key, true); dir = s3dir; // Find our directory. (I CAN HAZ MY BUKKIT?) List x = con.ListAllBuckets(); bool found = false; foreach (BucketInfo info in x) { if(info.Name == dir) { found = true; } } // No luck? Create a new bucket/directory. if (!found) con.CreateBucket(dir); } public override void StoreAsset(AssetBase asset) { m_log.Info("[S3 Asset Store] Saving asset..."); MemoryStream tw = new MemoryStream(); // Convert the asset to XML and throw it into memory. xs.Serialize(tw, asset); string xml = tw.ToString(); string md5 = Util.Md5Hash(xml); con.PutObject(dir, asset.FullID.ToString(), xml.Length, "text/xml", md5, null, delegate(Stream s) { tw.WriteTo(s); return null; }); } protected override AssetBase GetAsset(AssetRequest req) { Stream v = null; bool done = false; m_log.Info("[S3 Asset Store] Retrieving asset..."); con.GetObject(dir, req.AssetID.UUID.ToString(), delegate(Stream s) { v = s; done = true; return null; }); while(!done) { System.Threading.Thread.Sleep(500); } return (AssetBase) xs.Deserialize(v); } } }

As you can see, the meat of the function sits in the StoreAsset and GetAsset functions, in this case we’ve overridden them to save to an EC2 Bucket (”directory”). We’ve made the directory a parameter of the constructor, as long as each region on the “grid” uses the same directory, they will act as one giant asset database.

Final Steps: Hooking it up

If we head back to OpenSimBase.cs where we first went looking for “m_assetStorage” originally, on line 412 we see:

IAssetServer assetServer; if (m_assetStorage == "grid") { assetServer = new GridAssetClient(m_networkServersInfo.AssetURL); }

Below this, let’s add a new statement

else if (m_assetStorage == "s3") { assetServer = new S3AssetClient("id", "key", "directory"); }

Now, if we change the ini setting to read “s3″ our brand spanking new S3 client should fire up.

In Conclusion

I need to make some notes here, first hard coding classes like the above is a bad idea. We shouldnt be doing it for the grid asset server (and instead should be loading a class via reflection). Second we shouldnt be hard-coding the login keys - moving that up to an ini setting is preffered (and not too difficult to do properly). Third this is a untested bit of code, use it at your own risk, this was done as an experiment in approximately 45 minutes.

However, for all the above warts - this shows off the power of OpenSim for rapid prototyping, one of the great features of OpenSim is just how flexible it can be, and how little work is needed to do something completely crazy with it quickly and relatively easily.

Tags: amazon, assets, code, OpenSim, prototyping, s3, storage