The Journey to a Hybrid Software Defined Storage Infrastructure S01E07

This is the episode number 7, the last one of this Journey named “The Journey to a Hybrid Software Defined Storage Infrastructure”.

It is a IBM TEC Study made by Angelo Bernasconi, PierLuigi Buratti, Luca Polichetti, Matteo Mascolo, Francesco Perillo.

To read previous episode check here:







Enjoy your reading!

This is the final episode of this Study; thus, we should come to some conclusions, but more we and you read the previous episodes we’ll definitely made some questions and we did!

The current level of technologies does not have all the answers, nor all the solution will fit all the customers’ needs or use case.

We will continue to look at on what technology enhancement will be available in the future, and how that will cover the missing piece.

By the way we are not considering this study completed but just the beginning of a new journey in a new Storage Technology era.

Here some open questions about the actual technologies:

  1. What state of readiness of new functionalities and the related market perception?
    • What will be the future next steps of this technology?
    • When those functions will be effectively developed, and delivered?
    • When the required functions can be really adopted in production?
  1. What is the risk to move data to a non-well known storage architecture?
  1. What could be potential risk to move Storage to the cloud?
    • Is data within the cloud more exposed to theft?
    • How meet regulations with data? (banking and health industry much more exposed)
  1. Storage performance constraints on Cloud:
    • How does Cloud provider infrastructure impacts the Storage performance?
      • Compute scalability
      • iSCSI and/or Fiber Channel and/or FCOE
      • LAN bandwidth and latency
      • Security: LAN Segregation and Overlays
      • How does the connectivity to Cloud impact the Storage performance?
        • Accessibility
        • Bandwidth
        • Latency
  1. Storage economics on Cloud:
    • What is the difference in costs among a traditional vs SDS on-premise and SDS on cloud?
    • Is Cloud assuring lower cost and lower TCO?

At the end of this episodes it seems slogan or choruses of which you are likely to forget the meaning and scope.

All of us who work in Information Technology at the bottom should make examine our conscience.

How many times have we too strategic approach or even evangelical and how many times we put ourselves on the side of businesses?

How can we help our customer to overcome the resistance and concerns about the change?

And again, are we really able to offer them the path and the best solutions or more suitable to their needs?

In this complex scenario it is obvious that the real protagonists of the companies are the Data.

The Data growing should be read and interpreted like something that gives value to the business, which must be managed and which must be used when needed and where needed.

Here some questions received from some readers and some answer based on our point of view:

  1. What is today the Storage role in the success of this big change?

    The role of the Storage may sound side than the trendiest topics such as the Internet of Things, Cognitive, Big Data, Analytics and Hybrid Cloud, but it is crucial. As per analysts, by 2020 we will have 42ZB data to manage. All these data will still end up somewhere … isn’t it?

    The truth is that the Storage can be both a commodity and a crucial factor for business efficiency. The traditional Storage, the one used for traditional workloads, is set to change from its traditional rotating disk shape to All Flash. This is currently happening with Solid State Disk (SSD) first, Flash later and now with Read Intensive SSD.

By 2020 more than 74% of the disk space will be on SSD / Flash technology.

But technology does not stop and we are bound to see an evolution of storage to be used for new workloads, and this will come more and more from big data, analytic, mobile and  Internet of Things workloads that will best fit with the cloud solutions and/or hyper-converged environment. This will most likely be driven by the evolution of new technologies like NVMe (Non-Volatile Memory Express) that will exploit the IP connectivity and FC together with RDMA architecture (remote direct memory access).

  1. What environment will be required?

    Definitely a virtualized environment, agile, efficient, hybrid, more and more driven by the software that is aligned with the Software Defined Infrastructure. Then you can make use of all opportunities of Cognitive Storage. A smart storage capable to understand the value of data and placing them on the most suitable media at their probable utilization. This will happen dynamically, depending on the needs and interests of the business. The Cognitive Storage will enable a highly efficient data management and an evident cost saving, because it will avoid to use advanced solutions for the management of little-used data and perhaps useless.

  1. What workload will be moved to the cloud and which one will be kept on a traditional infrastructure?

    It is possible that the traditional workload will remain in “the house” for a while, or at least until the application layer will not change so as it will be able to talk directly with the Storage layer to make self-provisioning or de-committing, orchestration and Disaster Recovery.

It will be different for new applications that will born to be placed in the cloud and it will be designed to take advantage of new storage technologies at the most since the beginning.

One of the drivers will be given by the total cost. Bring data in the cloud will not necessarily mean to save money, but definitely invest in a different way to get something better.

  1. What solutions and choice, the customer will need to take seen that the needs could change quickly?

    Definitely a mixed, hybrid solution that offers a wide technological coverage and not niche. There is not and there will not be a single solution for all, that is why IBM offers a portfolio of diverse storage solutions capable of covering all application requirements, architectural and economic. It also gives you the opportunity to use it in different ways: as integrated solutions, such as software or as a cloud service, depending on the business requirements and budget types (Capex and Opex). All of this, obviously,  will mean great freedom of choice and flexibility for customers.


The Journey to a Hybrid Software Defined Storage Infrastructure S01E06

This is the sixth episode of “The Journey to a Hybrid Software Defined Storage Infrastructure”. It is a IBM TEC Study made by Angelo Bernasconi, PierLuigi Buratti, Luca Polichetti, Matteo Mascolo, Francesco Perillo.

This episode has been already published on Linkein on November 8th 2016 as pilot Episode here:

To read previous episode check here:






Enjoy your reading!

Software Defined Storage (SDS) provides the possibility to spin-up, on standard components, a look-alike storage technology to enable data mirror and to manage the entire data replication process using the same tools and the same look-and-feel as the traditional disk mirroring functions among two Storage controllers.

Using standard components, means reduce the TCO by leveraging on common compute and storage blocks, while traditional storage is (almost) always a specialised-built infrastructure. Common Compute and Storage blocks also brings a “standardisation” in the support and skills required to manage the infrastructure and again this goes into a simplification of the IT operation, hence in the direction of the TCO reduction.

On the other hands, standardised components mean standardised reliability and performance, thus a SDS solution normally doesn’t perform at the same level as the correspondent storage subsystem.

To summarise, the use of an SDS solution into a Disaster Recovery brings costs reduction, at the expense of reliability and performance that the Storage will have. Does this matter? In day by day, when the solution only replicates data, and maybe performs DR Tests, this reduction in reliability and performance is normally not an issue, but in an emergency condition, when your infrastructure is requested to perform (or out-perform) the production infrastructure, it might.

To further contain costs at DR Site, a solution known as Hyper-Converged Infrastructure (HCI) topology can be adopted. In the HCI topology, the SDS function delivering Storage capability is hosted on the same Hypervisors as the DR VMs.

The DR solution can also have Recovery Tiers, basically your customer can split the IT infrastructure in waves (Tiers or Recovery Zones) with a homogeneous Recovery Time Objective (RTO) requirement.

Typically, a three-tier approach is used:

  • Tier-1 applications, with an immediate or near-immediate restart (0-12h), which requires dedicated resources
  • Tier-2 applications, with a restart within 1-4 days, could leverage on shared or re-usable assets at the alternate site. Since the time is short, usually this spare capacity must be already there, and cannot be acquired at time of disaster (ATOD).
  • Tier-3 applications, with a restart greater than 5 days, might be covered by additional Compute power that can be freed-up in the alternate site, by closing Dev/Test environments and re-purposing those Compute resources to the DR solution. These Dev/Test environments might be restarted later, as soon as the additional Compute resources are re-provisioned in the alternate site, or when the emergency is over and the production moved back to the original site.

All the above is a general definition; your customer might have more tiers or different combinations or requirements.

Other considerations on a DR Strategy are related to the distance among the primary and secondary sites; distance among the two sites, brings network latency in consideration, thus an Asynchronous replication might be the best option to contain performance impacts on production applications.

Also, the “specialisation” of the sites plays a great role in the decision and on the complexity of the solution. Dealing (managing and controlling) a uni-directional Disk Mirroring is far more simple than doing the same on a bi-directional Disk Mirroring, when it comes to “data consistency”.

In this article, we will have a view of possible DR Use Cases related to SDS, in a one-to-one topology.

In a one-to-one topology, four possible combinations exist:

  1. Prod on Premise; DR on a proprietary Alternate Site
  2. Prod on Premise; DR on a Cloud Provider
  3. Prod on a Cloud Provider; DR on another Cloud Provider site (or an alternate Cloud Provider)
  4. Prod on a Cloud Provider; DR on a proprietary Alternate Site

What-ever is the combination, there are common pit-falls that you need to be aware of.

 Common DR pit-falls

Plan and Design for the “best conditions” – It is important that your solution and your testing methodology are planned and designed to mimic what the real condition might be. Design for “best case”, of course contains costs, but leaves you in the hope that it would work.

Having Single-Point-Of-Failure (SPOF) – SPOF can be anywhere in a solution, not only on the technology side. Your solution might be dependent on people, vendors, providers, and other external dependencies. It is essential that you know and identify clearly in advance most of your SPOFs at least, and have a plan to mitigate your dependencies. Be also prepared to discover SPOFs during the initial sessions of your DR Test.

Have a Plan-A only – Any solution has a residual risk. Risks cannot be eliminated; they can only be mitigated. The more risks you mitigate with a single solution, the more the solution would cost. Having a Plan-B with a different RTO, might help to mitigate additional risks, and not adding too much costs too. Your RTO might not be the one expected, meaning you will need more time to restart your operations, and thus you will have more impact by the emergency, but restarting later might be far better than not restarting at all. A possible Plan-B, in a two sites topology, might be to have a periodic backup secured off-site from the two sites, and at a distance to be reasonable considered safe.

DR Testing – an un-tested DR solution is like running blind in a wood; chances are high that you will encounter an obstacle that stop your running. Moreover, you discover that only during the most important run of your life. If Testing is essential to guarantee you have a valid solution, also the condition you are going to test are important. If you plan to perform a DR Test, by doing a pre-ordered close of operation on one site and a well-organised restart in the other, you are sure your DR Tests works, but is this what will happen during an emergency? You should design your Tests to mimic as much as possible the possible emergency conditions, by simulating a so called “rolling disaster condition”, where your IT service is impacted progressively by the emergency. This is the best way to Test your solution and have a reasonable understanding whether it is resilient (ability to resist to stress conditions).

Case 1: Production on-premise, DR on another “traditional” site

In this topology, the Storage used by IT on-premise is usually provided by traditional storage subsystems.

At the DR Site, to contain costs, instead of adopting the same storage subsystem, your customer can spin up the storage requirement adopting the corresponding SDS solution, which can also be implemented as a Hyper-Converged Infrastructure (HCI) topology.

SDS in this topology will help in containing TCO as you can reuse your existing storage that might be resulting from a technology change and deploy that as the storage used by the SDS in the alternate site. Also, the Compute you need for the SDS solution might be resulting from a technology upgrade of the Compute in your primary site.

How a Three-tier DR solution can be implemented

Tier-1 applications, could be hosted on the same Hyper-Converged Infrastructure with the SDS solution.

Tier-2 applications, needs to have additional spare Compute capacity at the alternate site.

Tier-3 applications, might be covered by additional Compute power that can be freed-up in the alternate site, by closing Dev/Test environments and re-purposing those Compute resources to the DR solution.

Attention Points

Virtualisation & Standardisation – virtualisation plays a great role, as it masks the difference in the physical layer, making more easy to restart on a different hardware set, while physical servers requires more customisation during the recovery/restart process to adapt the software layer to the different physical layer the Compute has in the alternate site. In the same way, standardise your IT infrastructure on a small set of components helps in reducing the permutations of combination you must deal with.

Automation – DR operations are complex and requires a high level of technical interactions, so Automation can help in reducing the dependency on critical resources.

DR Provider resource syndication – most DR Providers actively use resource syndication to contain costs. Resource syndication means the same resource is used to provide service to different customer that might not be exposed to the same concurrent event. In other words, the Compute that you use for your DR solution might also be used by other customers of the DR provider, but not at the same time as you. It’s is important to understand this methodology and evaluate how the DR provider applies resource syndication. Resource syndicated among customers at 1000’s KM/miles offers a different (better) “usability ratio” compared to Resources syndicated among customers in the same city or city-block.

Case2: Production On-premise, DR to a Cloud Provider

In this topology, the SDS provides abstraction from the underline technologies used by cloud provider and creates the look-alike storage to enable a bi-directional data replication process (on-premise to the cloud and cloud to the on-premise).

Like the previous topology, you might adopt a Hyper-Converged Infrastructure (HCI) topology to contain costs.

How a Three-tier DR solution can be implemented

Tier-1 applications, could be hosted on the same Hyper-Converged Infrastructure with the SDS solution, eventually.

Tier-2 applications, can leverage on the Cloud Provider on-Demand resources to integrate additional Compute resources as and when required.

Tier-3 applications too, can be requested on-Demand to the Cloud Provider.

Attention Points

Provisioning Time– Cloud Providers are not created equally, so you need to evaluate the different offerings, and the SLA associated to the provisioning time of On-Demand resources, as they impact the RTO of your DR solution. Another important point is the commitment offered by the Cloud Provider on the SLA, and what limitations might be there (does it commit to provide on-Demand resources, per the SLA and independent on the number of concurrent requests?).

Resource Pricing– on-demand resources are cheaper, as they are usually billed by usage. It is important to evaluate all the caveats associated with the usage billing, and how that might impact your TCO with hidden and unplanned costs.

Compatibility with the Cloud Provider– doing DR on a Cloud Provider from the on-premise, has the same requirements and challenges as a migration project to the Cloud. You must verify that what is going to be deployed on the Cloud Provider can run on that infrastructure.

Constrains on the Cloud Provider – despite all the abstractions the cloud provider can do, at the end you might be still facing some constrains in your design. An example is the limitation you might have to boot a Virtual Machine from a replicated disk (LUN), when that VM runs in the in Cloud Provider managed hypervisor infrastructure. Other examples are in the area of LAN and WAN, where the Software Defined Network (SDN) provides some flexibilities, but not the full flexibility you have in your own infrastructure.

Cloud Resource Scaling – having a DR on Cloud, might provide the false expectation that cloud scalability is infinite. It is not, and it is not particularly in the Cloud Data Center where you have decided to replicate your data.

Cloud Networking (LAN) – having a DR on Cloud, might also imply that you need to re-build your network structures on the Cloud Provider. In some cases, you will be forced to use Software Defined Network (SDN) and Network Functions Virtualisation (NFV), to reproduce your complex enterprise network layout. A careful planning and evaluation of performance and scalability limitation is essential to assure your re-provisioned compute can work properly.

Cloud Networking (WAN) – all the external networks connected to your primary (failed) site must be re-routed to the alternate Cloud-Site. Different options are available, and you need to evaluate can plan what options best fit your requirements. Consider the charges associated to the use of the network (download/upload charges) when using the Cloud Provider resources, as they might add a substantial cost to your DR TCO.

Case 3: Production on Cloud, DR on Cloud

In a Cloud to Cloud topology, the SDS provides abstraction from the underline technologies used by Cloud Providers and creates the look-alike storage to enable a bi-directional data replication process. On the same provider (different cloud sites), SDS might provide a disk replication technology that offers better options compared to the native feature available by the provider.

Like the previous topology, also here to contain costs, you might adopt a Hyper-Converged Infrastructure (HCI) topology.

How a Three-tier DR solution can be implemented

Tier-1 applications, could be hosted on the same Hyper-Converged Infrastructure with the SDS solution, eventually.

Tier-2 applications, can leverage on the Cloud Provider on-Demand resources to integrate additional Compute resources as and when required.

Tier-3 applications too, can be requested on-Demand to the Cloud Provider.

Attention Points

Cloud Resource Scaling – having a DR on Cloud, might provide the false expectation that cloud scalability is infinite. It is not, and it is not particularly in the Cloud Data Center where you have decided to replicate your data.

Provisioning Time – Cloud Providers are not created equally, so you need to evaluate the different offerings, and the SLA associated to the provisioning time of on-Demand resources, as they impact the RTO of your DR solution. Another important point is the commitment offered by the Cloud Provider on the SLA, and what limitations might be there (does it commit to provide on-Demand resources, per the SLA and independent on the number of concurrent requests?).

Cloud Provider Compatibility (when different providers) – on Different providers, you would expect to have different technologies. Like in the On-Premise to cloud topology, moving your workload from a provider to another, is equivalent and has the same complexity as a cloud migration project from the on-premise. Attention must be placed in investigating the different methodologies, interfaces and services offered by the two Cloud Providers (Monitoring, API, Networks, Provisioning Time)

Resource Pricing (when different providers) – it is important to evaluate and understand all the caveats associated with the usage billing of the two providers, and how that might impact your TCO with hidden and unplanned costs.

Case 4: Production on Cloud, DR to On-premise

Although in nowadays “get on cloud” mood this option seems odd, if you give Data the right importance they have, this option might not sound odd anymore.

A company that want to maintain flexibility in its cloud strategy, should think to maintain a copy of the company data in their own site.

SDS enables this flexibility and option, and it can be easily implemented, by reversing the flow of the data replication, once everything has on-boarded the Cloud.

To leverage on this topology as a DR solution, you must maintain all your Compute locally and not getting too much dependent on services or features available at the Cloud Provider only, as they will not be available at your on-premise.

As in the other topology, even here you can opt for a Hyper-Converged Infrastructure, in which case you probably transform your traditional storage into a SDS solution you host on-premises on the Compute servers.

In terms of DR Tiering, you can implement the same DR Tiering as in the On-Premise to On-Premise, as the target is your site.

Attention Points

Compatibility with the Cloud Provider – doing DR from a Cloud Provider to the on-premise has challenges on the compatibility side. You might need to mimic the hypervisor infrastructure you have used on the cloud to avoid hypervisor migrations on-the-fly.

Dependencies from the Cloud Provider – you should also mind the possible dependencies on services or features that your applications have started using on the cloud provider (DNS, monitoring, logging, …), as they might not be available in your on-premise.


The Journey to a Hybrid Software Defined Storage Infrastructure S01E05

This is the fifth episode of “The Journey to a Hybrid Software Defined Storage Infrastructure”. It is a IBM TEC Study made by Angelo Bernasconi, PierLuigi Buratti, Luca Polichetti, Matteo Mascolo, Francesco Perillo.

Some episodes will follow and may be a S02 next year as well.

To read previous episode check here:





Enjoy your reading!

9.    #8 Cloud Storage Problems: How to Avoid Them

Moving storage to the cloud offers some enticing benefits, but only if you can avoid the common cloud storage problems. Here are some of the biggest cloud storage problems you need to be aware of before moving your invaluable data to cloud storage.

  1. Not choosing the right cloud storage provider

The old adage is that no-one got fired for choosing IBM, and when it comes to cloud storage it’s tempting to choose one of the two biggest cloud providers: AWS or Microsoft Azure.

But while they may well be the best choice for many companies, they may not be the best choice for yours. Depending on the size of your organizations it may make sense to look at smaller storage providers who will be able to give you more attention.

The things to look for with other storage providers include:

  • Downtime history, to get an idea of how reliable they have been in the past – and therefore an indication of how reliable they may be in the future.
  • Data accessibility, including what bandwidth they have within their data center, between their data centers and to the Internet.
  • Their pricing structure, including fixed charges and bandwidth charges to move data in and out. A common cloud storage problem is to neglect to establish how easily you can scale your requirements up and down. For example, are you committed to a certain amount of storage every month, or can pay only for what you use each day, week or month?
  • Familiarity with your industry vertical. Choosing a storage provider that understands your business and your likely data requirements can make life much easier for you, and failing to choose a good provider that specializes in your industry is a clouds storage pitfall that could put you at a disadvantage compared to your competitors. That’s because service providers familiar with your industry may be better equipped to accommodate your industry’s usage patterns and performance requirements and to demonstrate compliance with relevant industry regulations.
  1. Neglecting connectivity

You may have a state of the art network in your data center running at 100Gbps or 10Gbps, with perhaps 10Gbps, 1Gbps or even 100Mbps in the rest of the organization. But when it comes to connectivity with the Internet your bandwidth will likely be much slower – perhaps as low as 10Mbps – and it may well be asymmetric (meaning uploads to a cloud storage provider will be much slower than downloads from it.)

Cloud storage gateways and other WAN optimization appliances can help alleviate the problem, but if the connectivity to your cloud storage provider is not sufficient then a move to cloud storage is unlikely to enable high enough storage performance to get many of the potential benefits.

  1. Not getting the service level agreement (SLA) right

Most cloud storage providers will offer you a boilerplate SLA outlining their obligations to you and what they will do if things go wrong. But there is no reason why you have to accept it – IDC estimates that about 80% of cloud customers accept the boilerplate SLA they are offered, but 20% negotiate alterations to this boilerplate to ensure that it more closely meets their needs.

For example, a provider may offer you “four nines” (i.e 99.99%) uptime guarantee, allowing 50 minutes’ downtime per year. But this may be calculated on an annual basis, so the service could be down for 50 minutes on the first day of the contract and you would have to wait until the end of the year to find out if the SLA had been breached and you were therefore entitled to any compensation.


In the meantime, you would have to bear any resultant losses yourself. To avoid this cloud storage problem, it may be possible to negotiate that while 50 minutes per year is permissible, there should be no more than (say) 15 minutes per month if that suits your business needs better.

  1. Overestimating the compensation, you might get if the provider breaches the SLA

It’s tempting to think of an SLA as some kind of insurance policy: your business can survive as long as the terms of the SLA are met, and if they are not you’ll be OK because your cloud storage provider will provide compensation that is tied to the impact on your business of the breach.

But that is simply not the case and it’s a common cloud storage problem. In most cases breach penalties come in the form of service credits (i.e. free storage for a few months), and in the case of a serious breach – such as all your data being lost – the most you should hope for is a monetary payment of three or four times your annual contract value. In many cases that will be nothing like the cost to your business of losing so much data.

Of course, it may be possible to negotiate higher compensation payments from your cloud storage provider, but then it’s likely you will have to pay much more for your storage. In most cases, it would work out cheaper to buy insurance cover from a third party.

  1. Failing to monitor your SLA effectively

Working with a cloud storage provider adds another layer of complexity between the business users who use corporate data and the data itself. The IT department, which monitors the SLA, is somewhere in the middle.

A common cloud storage pitfall when it comes to data access problems is that users or business units may bypass the IT department and go directly to the cloud storage provider’s help desk to resolve issues when they occur. If that happens then you can’t necessarily rely on the provider to record every problem that occurs, and that means accurate monitoring of the SLA is effectively impossible. Avoiding this cloud storage pitfall comes down to educating users that your IT helpdesk should be their first point of contact in all cases.

  1. Failing to get a clear understanding of how to get your data back or move it to another provider

Cloud storage providers may fall over themselves to make it easy for you to give them your data in the first place perhaps by collecting physical media such as hard disk drives from your data center or offering free data ingress over a network connection. But if you decide that you no longer want to use the provider’s services it can often prove unexpectedly difficult or expensive to get it back.

To avoid this cloud storage pitfall, it’s important to get satisfactory answers to the following questions:

  • How will your data be made available – over a network connection or can it be placed on physical storage media for collection?
  • How soon will it be available – will you be expected to wait for days or weeks?
  • How much bandwidth will be available if you plan to download your data? That’s important because even with a 1Gbps link, it would take almost two weeks to get 150TB of data back from a cloud storage provider to your data center.
  • What bandwidth costs will be involved if you move your data back over a network, and what are the costs of having it put on physical media?
  • How long will it take for copies and backups of your data to be deleted, and what formal confirmation can you expect that all copies have been deleted?
  • In what format, will data be made available – will it be provided in a .csv file or in some other more closed format?


  1. If using a cloud, storage provider absolves you of all security responsibilities

Cloud providers are meant to be experts at what they do, including keeping their clouds and the data within it secure. But if there is a data security breach then it is you that your customers will hold responsible and seek compensation from, and it is you that will suffer the embarrassment, loss of reputation and possible loss of business.

That means that to avoid this cloud storage problem it is up to you to do due diligence and satisfy yourself that the security offered by the cloud storage provider is good enough for your needs. To do this you will need to find out as much as possible about the security arrangements that are in place, and what guidelines and regulations (think HIPPA, PCI-DSS, SSAE 16) it has been certified to comply with.

  1. Fixating on costs without considering other factors

For many companies one of the key drivers for moving to the cloud is reduced costs, or at the very least a switch from a single large capital expenditure to small regular operating expenditures. While that may be beneficial from a business point of view it’s important to remember that as well as changing how you pay, you are also paying for something fundamentally different.

Cloud storage, in other words, is not the same as your existing data center storage, and as well as new security, compliance and accessibility challenges there are also new performance characteristics to consider. What this boils down to is that some applications that you run in your data center aren’t performance sensitive and are well suited to being used in conjunction with cloud storage. For other applications that’s not the case.

That means that if you decide to use cloud storage for these latter applications then the applications themselves may also have to run in the cloud, close to the cloud storage. And that in turn means that moving your data to cloud storage may need to be part of a far larger consideration of the viability of moving some or all your applications to the cloud.

This is the end of Episode #5. Next Episode will come on 2017 because this Blog will step into the Holiday Season like I hope you will soon!!

Thank you for reading…Stay tuned and see you in 2017!

The Journey to a Hybrid Software Defined Storage Infrastructure S01E04

This is the fourth episode of “The Journey to a Hybrid Software Defined Storage Infrastructure”. It is a IBM TEC Study made by Angelo Bernasconi, PierLuigi Buratti, Luca Polichetti, Matteo Mascolo, Francesco Perillo.

Some episodes will follow and may be a S02 next year as well.

To read previous episode check here:




Enjoy your reading!

8.    The SDS Hybrid Architecture

Moving to a SDS Hybrid Infrastructure is a journey and it needs to go through some specific steps.

Consolidate and Virtualize: Storage Virtualization is strategic and a Storage Monitor System to discover storage resources, check their dependency and track the changes is imperative as well.

Only through a standardized lifecycle management process we will be able to get:

  • Automated provisioning / de-provisioning
  • Virtual Storage Pools
  • Capture and catalog virtual images used in the data center
  • Management of the virtualized environment


Then Integrating virtualization management with IT service delivery processes the infrastructure can supply:

  • Elastic scaling
  • Pay for use
  • Self-service provisioning
  • Simplified deployment with virtual appliances
  • Workload / Virtual Servers provisioning and Workload Management
    • Virtual Servers / Hypervisors
    • Dedicated Storage
  • Integrated Infrastructure
    • Server, Storage and Network
    • Specialized storage services
    • Orchestration/Management of the virtualized environment
  • Hybrid Clouds
  • Business Policy Driven
  • Dynamic Infrastructure
  • Data and workload services
  • Based on business policy
  • QOS driven
  • Regulatory compliance
  • Cost and performance optimization
  • Extended Enterprise Infrastructure

SDS Hybrid infrastructure leverage the Cloud Services


At the end of the Journey the main goal of a SDS Hybrid Infrastructure is to get a Storage for Workload Optimized System, in other words a Storage Infrastructure able to match the requirements of the workload. screenhunter_05-dec-13-15-10A simple SDS Hybrid Cloud picture can be depicted as follow:

screenhunter_06-dec-13-15-15This architecture applying different data transfer solution will be able for block data to:

  • Backup / Archive to the Cloud – physical media
  • Backup / Archive to the Cloud by network – restore from the Cloud to on premise
  • Pre-position data in the Cloud
  • Migrate workloads to the Cloud – run against pre-positioned data



And for NAS

  • Backup / Archive to the Cloud by network – restore from the Cloud to on premise
  • Pre-position data in the Cloud with Object Store gateway
  • Pre-position data in the Cloud with AFM
  • Migrate workloads to the Cloud – run against pre-positioned data


In a Cloud Environment, we can define some Storage Class as seen by Guest VM. Each storage class could have one or more tiers of storage behind it.


An agnostic view of what is the Storage Technology for Cloud able to match Cloud Storage Classes, Cloud Storage Platform Services and Customer Workload can be summarized as following:


As long as this study aim to show how is possible to build a SDS Hybrid Infrastructure, its target is to show how IBM SDS portfolio can match and achieve this goals as well.

In the next picture the IBM SDS Technology for Cloud Product Selection is shown.


8.1    IBM Storage for the Cloud and Cognitive era

This flexibility makes hybrid cloud the ideal platform on which to build cognitive solutions. And because data is the resource on which all cognitive solutions depend, IBM storage is the foundation for hybrid cloud and cognitive solutions. It protects that data, delivering it when, where, and how it’s needed with the efficiency, agility, and performance that cognitive solutions demand.

IBM storage delivers a host of powerful capabilities that enable an enterprise to easily exploit the full value of its data while simultaneously reducing the cost of its management to achieve optimal data economics throughout its lifecycle. These are just a few. There are many more. The point is, whatever the use case, IBM storage offers a robust solution that satisfies it.

The key to agility, efficiency and performance in the modern data center is software defined flexibility.

Software Defined Storage takes the intelligence that is in traditional storage systems (which are a combination of proprietary hardware and software) and makes it available to run on commodity hardware.

By decoupling the software from the underlying hardware, the capabilities of a particular software stack can then be deployed wherever and consumed however they are needed – on premises or in the cloud –  as a fully-integrated solution, an appliance, software, a cloud service, or various combinations.

The next paragraph will describe the software defined storage solutions in IBM SDS portfolio.


8.1.1    Storage Infrastructure

IBM Spectrum Virtualize helps simplify storage management by virtualizing storage in heterogeneous environments. Among other benefits, virtualization simplifies deployment of new applications and new storage tiers, eases movement of data among tiers, and enables consistent, easy-to-use optimization technologies across multiple storage types.

IBM Spectrum Accelerate™, is a highly flexible storage solution that enables rapid deployment of block storage services for new and traditional workloads, on-premises, off-premises and in a combination of both. Designed to help enable cloud environments, it is based on the proven technology delivered in IBM XIV Storage System and in use on more than 100,000 servers worldwide.

IBM® Spectrum Scale™ is scale-out file storage for high performance, large scale workloads either on-premises or hybrid cloud. It unifies storage for cloud, big data and analytics workloads to accelerate insights and deliver optimal cost and ease of deployment. It combines enterprise features with performance-aware intelligence to position data across disparate storage hardware, making data available in the right place at the right time.

IBM Cloud Object Storage enables storing and retrieving object data on-premises, in-the-cloud, or both with the ability to easily and transparently move data between them.

8.1.2    Management

IBM® Spectrum Control provides storage and data optimization using monitoring, automation and analytics. It enables organizations to make an easy transition to virtualized, cloud-enabled, and software defined storage environments— because it provides a storage management solution for all types of data and storage. It helps significantly reduce storage costs by helping optimize storage tiers, and simplify capacity and performance management, storage provisioning and performance troubleshooting.

IBM Spectrum Protect provides a single platform for managing backups for virtual and physical machines, and cloud data.  Modern capabilities, such as scalable deduplication and cloud storage access, are delivered entirely in software, eliminating the requirement for deduplication appliances and cloud gateways in many instances.

IBM Spectrum Archive gives organizations an easy way to use cost-effective IBM tape drives and libraries within a tiered storage infrastructure. By using tape libraries instead of disks for Tier 2 and Tier 3 data storage—data that is stored for long-term retention organizations can improve efficiency and reduce costs.


8.1.3    IBM Storage for Cloud and Cognitive era

It’s from this software defined capabilities that four key storage platforms emerge for Virtualized Storage, Cloud, Big Data, and Business Critical storage needs. These are the cornerstones of a cognitive storage infrastructure.


And because of their software defined flexibility, they are available in a range of deployment models including fully-integrated solutions, software, cloud services, and appliances. Notice also that we have all-flash offerings in every platform.


Combined with the rest of our storage portfolio, they provide capabilities that enable a business to be more than digital but to marshal valuable data assets with the efficiency, performance, and agility required to be a truly cognitive enterprise.

IBM storage solutions offer flexibility in deployment to make data available where and how it’s needed, in the form most easily consumed by the applications that depend on it, and with the best data economics possible, whether on-premises or off.


Cloud-Scale solutions based on IBM Spectrum Accelerate software are purpose-engineered for the demands of cloud deployments with strong support for multi-tenancy and Quality-of-Service, and deliver consistent high performance even with unpredictable workloads. They dramatically simplify scale-out and management by eliminating tuning, load-balancing, and most other storage management activities.  They offer extreme ease of use and task automation, reducing administrative overhead, and scale management to many petabytes in a single environment, and come with advanced mirroring, security and other enterprise capabilities including remote replication, multi-tenancy, snapshots, monitoring.

Versatile integration options make cloud infrastructure easy with rich integrations for the cloud, like a REST API, a thorough command line interface, OpenStack Cinder, and deep VMware and Microsoft integrations

The benefits of IBM Spectrum Accelerate are available as software, as a cloud-service, or in these fully-integrated solutions:

  • The field-proven and much-loved XIV Gen3 storage system which is our capacity-optimized offering.
  • IBM FlashSystem A9000, which integrates the extreme performance of IBM FlashCore technology, a full-featured data management stack, and flash-optimized data reduction in one very simple and efficient, all-inclusive 8U solution for cloud deployments.
  • And IBM FlashSystem A9000R, designed for the global enterprise with data-at-scale challenges. It is a grid-scale, highly parallel, all-flash storage platform designed to drive business into the cognitive era with performance, MicroLatency response time and the reliability, usability and efficiency needed by today’s enterprise businesses.


All the products of the IBM Storage Portfolio will match the SDS Hybrid Infrastructure requirements and goals:

  1. To be ready for Private, Public and Hybrid Cloud
  2. Be flexible and agnostics thanks to Storage Virtualization layer
  3. Leverage Flash Technology for Business Critical and Analytics applications
  4. Match back up and DR customer requirements
  5. Easily managed with a top down view in a single pane product.


8.1.4    Transparent Cloud Tiering (aka Multi Cloud Storage Gateway)

Basing on the fact that the Storage in Cloud will play a crucial role in the Storage future, the Cloud or Multi Cloud Storage Gateway (TCT for IBM) will be one of the facility that will drive the data from on premise to off premise.

An agnostic vision about how the Multi Cloud Storage Gateway will give benefits is shown in the next picture:


IBM has some technology formally called already present in some product (Spectrum Scale) and that will be ready on other SDS products as well in a short future. Currently this technology is called Transparent Cloud tiering (TCT), potentially in the future will have different name.




This technology potentially (Roadmap need to be confirmed time to time), will be present in any IBM SDS Portfolio products and in other storage subsystem that currently represent the IBM Enterprise Storage Subsystem as well, like DS8000 and XIV family.


The MCStore technology is already available on IBM Spectrum Scale Technology and can give benefits in the following Use Case:

  • Enable a secure, reliable, transparent cloud storage tier in Spectrum Scale with single namespace
    • Based on GPFS Information LifeCycle Management (ILM) policies
    • Leveraging GPFS Light-Weight Event technology (LWE)
  • Supported Clouds
    • AWS S3 (Amazon) and Swift


screenhunter_18-dec-13-15-24This solution will do a couple of things for you.

  1. Because we are looking at the last read date, data that is still needed but the chance you will read it is highly unlikely can be moved automatically to the cloud. If a system needs the file/object there is no re-coding that needs to be done as the namespace doesn’t change.
  2. If you run out of storage and need to ‘burst’ out because of some monthly/yearly job you can move data around to help free up space on-perm or write directly out to the cloud.
  3. Data protection such as snapshots and backups can still take place. This is valuable to many customers as they know the data doesn’t change often but like the idea they do not have to change their recovery process every time they want to add new technology.
  4. Cheap Disaster Recovery. Scale does have the ability to replicate to another system but as these systems grow larger and beyond multiple petabytes, replication becomes more difficult. For the most part you are going to need to recover the most recent (~90 Days) of data that runs your business. Inside of Scale is the ability to create mirrors of data pools. One of those mirrors could be the cloud tier where your most recent data is kept in case there is a problem in the data center.
  5. It allows you to start small and work your way into a cloud offering. Part of the problem some clients have is they want to take on too much too quickly. Because Scale allows customers to have data in multiple clouds, you can start with a larger vendor like IBM and then when your private cloud on OpenStack is up and running you can use them both or just one. The migration would be simple as both share the same namespace under the same file system. This frees the client up from having to make changes on the front side of the application.


Today this feature is offered as an open beta only. The release is coming soon as they are tweaking and doing some bug fixes before it is generally available. Here is the link to the DevWorks page that goes into more about the beta and how to download a VM that will let you test these features out.

Addressing a solution with Data replicated in Cloud using MCStore or Multi Cloud Storage Gateway, more than Disaster Recovery, it is correct to talk about Back Up in Cloud solution as long as the data replicated, moved or backed up in cloud will be “Object”, hence their usage can’t be immediate for DR purposes with usual minimum RPO or RTO, but they will be available for restore purposes as needed.

Talking about IBM Spectrum Virtualize what we are expecting in a short future with V7.8 is shown in the following picture, where:





  • User configures Back Up / DR for production volumes via Spectrum Virtualize GUI
  • MCS Gateway runs alongside Spectrum Virtualize and pulls full or incremental FlashCopy snapshots from Volumes
  • MCS Gateway applies encryption, integrity protection etc. as configured
  • MCS Gateway stores data and metadata to remote object store(s), thus snapshots can be incremental forever
  • User restores snapshot from cloud to original or new Volume via Spectrum Virtualize GUI

 A kind of built-in easy-to-use “cloud based time machine for enterprise block storage” for the following Use Case:

Built-in easy-to-use feature for various use cases:

  • Backup
  • Disaster recovery
  • Data sharing
  • Migration/archiving
  • Compliance/auditing


This technology, as already said, is something that is already present and something that will be available in a short term or medium term future.

By the way it is something need to take in consideration because its future evolution will be really interesting and will give tremendous benefit for Hybrid SDS Cloud infrastructures.

This is the end of Episode #4. Next Episode will come shortly

Thank you for reading…Stay tuned!