Leveraging StorNext’s Self-Describing Objects

Want to run a cloud-based application against data that StorNext has stored in the cloud? Now you can.  Need access to that data from other sites? Now it’s possible. Looking to share files with business partners via the cloud? You’ve got it. Even if you just want insurance that you can always get your data back from the cloud, through disaster or technology obsolescence, it’s here.

StorNext has been ‘cloud-aware’ for a very long time, making it easy to use public cloud destinations or on-prem object stores to store and protect warm and cold data. With the release of v6.4, StorNext can now store additional metadata with objects, including the full path and filename of the source file.  Previously, this information was only available by querying StorNext directly. Storing the path and filename in the cloud with the object makes these objects “self-describing.”

To enable massive scale, StorNext stores the source path and filename in a user-defined metadata field.  You can see the contents of this field using tools like the AWS Console, S3 Browser or Cyberduck. That’s fine if you only have a few objects to manipulate, but if you need to handle a lot of objects, it’s better to write a script.

I am not a programmer. I have done some coding, but mostly in C back when I was an EE student, and that was almost 30 years ago.  So, believe me when I tell you that if I can do this, you can do this.

Below is a sample script that will copy every object from a StorNext-controlled AWS bucket down to the local machine, naming the resulting files using the paths and filenames stored by StorNext with the objects.

I used PowerShell on Windows because it seemed minimally intimidating, and because Amazon has something called AWS Tools for PowerShell that provides easy access to AWS APIs. In case you didn’t know, PowerShell is available for Linux and MacOS too, and the basic concepts I used could easily be translated to your favorite development environment.

Here’s the script:

# Credentials must have been established prior to running this script using 'Set-AWSCredential' directly or recalling a stored profile
 
# Set source bucket and destination local path
$bucket = '228868asn64tapeaws'
$localPath = 'C:\test'
 
# Scan all objects in bucket, load metadata into array 'objects'
$objects = Get-S3Object -BucketName $bucket
 
# Loop through object metadata, pull object keys into array 'fileName'
foreach ($object in $objects) {
$fileName = $object.Key
 
# Loop through array 'fileName', & for each object key do a few things:
foreach ($file in $fileName) {

# Get object metadata
$Metadata = Get-S3ObjectMetadata -Bucketname $bucket -Key $fileName -Select 'Metadata’

# Set 'localFileName' to SN stored path
$localFileName = $Metadata.Item('x-amz-meta-path')

# Create valid local path including filename
$localFilePath = Join-Path $localPath $localFileName

# URL Decode local path
$localFilePath = [System.Web.HttpUtility]::UrlDecode($localFilePath)

# Download object to local path & name
Read-S3Object -BucketName $bucket -Key $fileName -File $localFilePath }}

That’s it. Less than a dozen lines of code, and I bet someone smarter could make it even simpler. When you run it, you get a result that looks like this:

It is much more useful than what you get if you simply download the objects, which looks like this:

This was a dirt-simple example, (and it comes with absolutely no warranty), but even I can see that the possibilities for customization are endless. We have plans to make it easier to browse and re-use objects for ad-hoc use cases, but for handling lots of objects or where complex filtering is required, scripting will always be the best way. I hope I’ve convinced you that it doesn’t have to be hard. For complete details on how I set up AWS Tools for PowerShell and additional information, please see our Tech Brief entitled “Accessing StorNext Self-Describing Objects with PowerShell” available here.

Final note: Thanks to everyone who shares their PowerShell knowledge on the ‘Net. This blog in particular provided a critical jump start to my thinking.