N1005 IBM Research: openstack

Deep dive insights into Swift

<div class="MsoNormal" style="line-height: normal; margin-bottom: 6.0pt; mso-layout-grid-align: none; text-autospace: none;"><i><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">Editor’s note: This blog entry is by IBM Fellow <a href="http://researcher.watson.ibm.com/researcher/view.php?person=il-FACTOR" target="_blank">Michael Factor</a> and IBM storage research engineer <a href="http://researcher.watson.ibm.com/researcher/view.php?person=il-DMITRYS" target="_blank">Dmitry Sotnikov</a> from IBM Research – Haifa. The work was done together with <a href="http://researcher.watson.ibm.com/researcher/view.php?person=il-YARON" target="_blank">Yaron Weinsberg</a> from IBM Research - Haifa.</span></i><br /><br /></div><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGCV0rQE75GSaOb0OGThCssam9wIW5qeDdvv93DK7q3-MUjOcpGwmuacVgir_WDr8uKtShCQvHJmyBlZz_yBIe_RbVnn5GKB2475RZnuW7qjwUyOIirJPiXuCNgB79g2jdPAzt8PefXHM/s1600/michael4_sm.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="173" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGCV0rQE75GSaOb0OGThCssam9wIW5qeDdvv93DK7q3-MUjOcpGwmuacVgir_WDr8uKtShCQvHJmyBlZz_yBIe_RbVnn5GKB2475RZnuW7qjwUyOIirJPiXuCNgB79g2jdPAzt8PefXHM/s200/michael4_sm.jpg" width="200" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Michael Factor, IBM Fellow</td></tr></tbody></table><div class="MsoNormal"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;">When companies deploy a system, they have specific performance, durability, and cost objectives in mind. So, before physical deployment, they will run models and simulations to get a close approximation of how the system will meet those objectives. But after deployment, the system must be checked. This holds true for even open source systems </span><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;">—</span> which have become more and more popular because of their flexibility in terms of available, interchangeable hardware and software. Our team at IBM Research-Haifa, building on open source tools, has been gaining experience and developing techniques to observe and monitor one of the most-popular options, OpenStack <a href="https://wiki.openstack.org/wiki/Swift" target="_blank">Swift</a>. It's the leading open source object storage system that runs in public and private clouds.</span><br /><a name="more"></a><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;"><br /><b> </b></span><br /><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;"><b>Let’s dive into Swift and what we learned from the data collected during monitoring. </b><br /><br /><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLCTJHAoko05CpmnbCsrio8oYYrwdcY5zDQ5gEJmNCRoZx9uGPrZLELhGI7hAzB0_DLaQfVNJ6Q19wddPJvuqrftewn6swr6WwmwIvF46UtKr2E-PtZ1Jt4yO-C5WgdD7ljwenjdG97hs/s1600/Dima1.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="196" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLCTJHAoko05CpmnbCsrio8oYYrwdcY5zDQ5gEJmNCRoZx9uGPrZLELhGI7hAzB0_DLaQfVNJ6Q19wddPJvuqrftewn6swr6WwmwIvF46UtKr2E-PtZ1Jt4yO-C5WgdD7ljwenjdG97hs/s200/Dima1.jpg" width="200" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Dmitry Sotnikov, IBM storage engineer</td></tr></tbody></table>Increasing system complexity means increased monitoring complexity — since huge amounts of data need to be analyzed to find out what’s really going on. That’s where our work comes in. Our methodology  uses an open source toolbox and enables understanding the behavior of Swift clusters by examining the data collected during monitoring.<br /><br />Today, performance monitoring and troubleshooting of a running cloud-based object storage is as much an art as a science. Although there are a plethora of open source monitoring tools to gather system metrics, the real challenge is how to use them to find the root cause of a problem. <br /><br />We developed a general, open-source-based, step-by-step methodology to understand performance bottlenecks in a Swift system. Our solution uses standard tools including <a href="https://www.elastic.co/products/logstash" target="_blank">Logstash</a>, <a href="https://collectd.org/" target="_blank">collectd</a>, <a href="https://codeascraft.com/2011/02/15/measure-anything-measure-everything/" target="_blank">StatsD</a>, <a href="https://www.elastic.co/" target="_blank">Elasticsearch</a>, <a href="https://www.elastic.co/products/kibana" target="_blank">Kibana </a>and <a href="http://graphite.wikidot.com/" target="_blank">Graphite</a>. It also includes an additional simple Swift middleware we developed to gain further insights into the source of system bottlenecks.  </span></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4spDf38YlwBo9umkLg7djk8blFxzKEHXBV-RMjL6Ih6c1wFjVIUMI7V9HzzwhUIn_di9pk2LlslD-c_zndA6dIK6GNLoHwLxPtGcCMEGmM_rkdJk1dX75r6YPiN4fxeun-lifixUGTfM/s1600/swiftflowmonitoring.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="282" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4spDf38YlwBo9umkLg7djk8blFxzKEHXBV-RMjL6Ih6c1wFjVIUMI7V9HzzwhUIn_di9pk2LlslD-c_zndA6dIK6GNLoHwLxPtGcCMEGmM_rkdJk1dX75r6YPiN4fxeun-lifixUGTfM/s320/swiftflowmonitoring.jpg" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Swift monitoring flow</td></tr></tbody></table><div class="MsoNormal"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;">Our methodology helps validate the correctness of a Swift cluster configuration, and identifies which important data should be presented in the visualization of the Swift parameters together with the system’s statistics. This visualization helps users gain a better understanding of the internals of Swift and its behavior; it also enables them to see potential problems and misconfigurations. <br /><br />For example, if validation of Swift’s network configuration is required, for instance to understand unexpectedly low performance, , it can be done by using our methodology and the open source toolkit on which it is based. This can be seen in the following charts, which present the network utilization between the Proxy and the Object servers, as well as the Proxy’s public network utilization for a write only workload. Based on these charts one can easily validate that all the data received by the Proxy is replicated three times and sent to the Object servers. </span></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3MnD1N_YIcblLbdH9arLJu80vCE0YGdEsdmc6MT6M1IB2l9LuEeXKm_lLqu50OUONSupqE9HeC6ZvafC8BcvhQiVUEPfteqLenwA-Atly4ZTE2Viys6NFGQ-VK_ANmcGUjXHeGxlppUA/s1600/swiftvisualization.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3MnD1N_YIcblLbdH9arLJu80vCE0YGdEsdmc6MT6M1IB2l9LuEeXKm_lLqu50OUONSupqE9HeC6ZvafC8BcvhQiVUEPfteqLenwA-Atly4ZTE2Viys6NFGQ-VK_ANmcGUjXHeGxlppUA/s400/swiftvisualization.jpg" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Data visualizations</td></tr></tbody></table><div class="MsoNormal"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;">   <br />At the <a href="https://www.openstack.org/summit/vancouver-2015/" target="_blank">OpenStack summit on May 20</a> in Vancouver, we will be <a href="https://openstacksummitmay2015vancouver.sched.org/event/ba79147a4be111341bf7a3d91f504a40#.VUtInZMj9JI" target="_blank">demonstrating </a>the results obtained from our approach, used with an internal deployment of Swift. You are welcome to join us at 1:50 pm, room 109. </span><br /><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;"></span></div>

Editor’s note: This blog entry is by IBM Fellow Michael Factor and IBM storage research engineer Dmitry Sotnikov from IBM Research – Haifa...

Storlets: From research prototype to open source technology

8:30 PM N1000 0

<br /><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs1tf1YT-VtgO5V9ftIFwb3GyuuBk24QSwOdmwaS03j46aJsjxq5Ir27KO43ghLoo_2QeoOPm6IDUnSqfsREZ7vR9R9Y8E0uJxidUHR-ivNtBv1cDLXlXCYa69dJljBe357XBTJyDAiPQ/s1600/DSC_0064_Fotor_Fotor.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs1tf1YT-VtgO5V9ftIFwb3GyuuBk24QSwOdmwaS03j46aJsjxq5Ir27KO43ghLoo_2QeoOPm6IDUnSqfsREZ7vR9R9Y8E0uJxidUHR-ivNtBv1cDLXlXCYa69dJljBe357XBTJyDAiPQ/s1600/DSC_0064_Fotor_Fotor.jpg" width="132" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Researcher Eran Rom</td></tr></tbody></table><i><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Editor’s note, this article is by Eran Rom, storage researcher at IBM Research – Haifa.</span></i><br /><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">In a <a href="http://ibmresearchnews.blogspot.co.il/2014/05/storlets-turning-object-storage-into.html">previous blog</a> post devoted to storlets, IBM Fellow Michael Factor highlighted how storlets can be used to turn a software-defined object store into a smart storage platform. This is done by allowing the computation to run near the data, rather than bringing the data to the servers doing the computations. Michael's post addresses the potential of storlets in cost reduction as well as in enabling new services.</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">While in this post we want to concentrate on the technology itself, here are a few things that have happened with storlets since they were a research prototype.</span><br /><a name="more"></a><br /><br /><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;"><b>Storlets advancements</b> </span></div><ul><li><span style="color: black; font-family: Symbol; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;"><span style="font: 7.0pt "Times New Roman";"></span></span></span><span dir="LTR"></span><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Storlets played a central role in a <a href="http://www.research.ibm.com/client-programs/foak/">first-of-a-kind</a>solution, called Active Media Store, developed with Radiotelevisione Italiana (RAI).</span></li><li><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">The <a href="https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/docker-meets-swift-a-broadcaster-and-39-s-experience">work with RAI</a> was presented at the Paris OpenStack Summit, and adopted by OpenStack as a "<a href="http://superuser.openstack.org/articles/docker-meets-swift-a-broadcaster-s-experience">superuser story</a>".</span></li><li><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">A reference implementation of storlets is now publicly available under 'Open-I-Beam' in <a href="https://github.com/Open-I-Beam/swift-storlets">github</a>.</span></li></ul><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">We started interactions with the OpenStack community on the question of ‘if and how’ to add storlet support into the official Swift release. We will be having a design session discussion on this topic at the upcoming <a href="https://wiki.openstack.org/wiki/Design_Summit/Liberty">Vancouver OpenStack Summit</a>. We encourage all interested parties to attend.<span style="mso-spacerun: yes;">  </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;"><span style="mso-spacerun: yes;"></span></span></div><br /><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;"><span style="mso-spacerun: yes;"> </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><b><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 12.0pt; mso-bidi-font-family: Helv;">Storlets on OpenStack Swift</span></b></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Our implementation of storlets is integrated with OpenStack Swift. Swift is an open source implementation of an object store and is behind several public object store services, including the IBM SoftLayer object store. Part of the idea behind storlets is to provide a flexible means of extending the function of the object store, by giving Swift users the ability to upload code to be executed near the data. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Running user-written code inside the storage system calls for adequate security and isolation measurements. This is where Docker comes into the picture. <a href="https://www.docker.com/">Docker</a> is a popular Linux container management framework.<span style="mso-spacerun: yes;">  </span>Linux containers (LXC) are similar to virtual machines, only instead of virtualizing the hardware they virtualize the operating system.</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">In addition to providing security and isolation, Docker has tools for packaging and deploying executable images. Using Docker our implementation allows the user to upload the storlet’s code, along with a tailored image where the storlet will execute.<span style="mso-spacerun: yes;">  </span>Thus, if a storlet relies on some non-trivial software stack, that stack can be packaged into a Docker image and deployed in a Swift cluster, to be later used for executing the user's storlets.</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Writing a storlet involves implementing a single method Java interface called IStorlet. That method - called invoke - has two major parameters: an input stream and an output stream. The input stream is used for consuming the data of an object on which the storlet is operating and the output stream is used to write the results of the storlet's computation. Storlets work in a streaming fashion, i.e., they start outputting data before reading all the input data. This is due to the synchronous fashion of storlets’ invocation as part of the upload or download operations as described next.</span><br /><br /><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Once the Docker images and storlets are deployed, they can be invoked on data objects in Swift. Storlets can be invoked in two ways:</span><br /><ol><li><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Invocation during object download. In this case the storlet transforms the object before it is returned to the user. This can be used for scenarios such as pre-filtering data being retrieved for an analytics engine or as an on-the-fly resolution reduction when downloading to a mobile device.</span></li><li><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Invocation during object upload. In this case the data stored is transformed from the data PUT by the user. One example use case is metadata enrichment, where a storlet can tag a data object with additional metadata while it is being uploaded. </span></li></ol></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; tab-stops: -36.0pt 0cm 36.0pt 72.0pt 108.0pt 144.0pt 180.0pt 216.0pt; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">In our current implementation, invoking a storlet during the upload or the download of an object involves adding a single header to the upload / download request.<span style="mso-spacerun: yes;">   </span>This header identifies the storlet to execute on the object that is the target of the request. Once the request is received by Swift, a pluggable middleware intercepts the request at the appropriate point: During download this point is when the response data is on its way back to the user, and during uploads this point is along the input path of the request. </span><br /><br /><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Our code then routes the data to the storlet using file descriptors that are passed over Linux domain sockets to the storlet code running inside the Docker container. Other than these file descriptors, the Docker container has no access to any I/O device. This means the storlet's code has no network access and no access to Swift's own disks. All I/O is done via the file descriptors provided by our Swift plugin.</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; tab-stops: -36.0pt 0cm 36.0pt 72.0pt 108.0pt 144.0pt 180.0pt 216.0pt; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; tab-stops: -36.0pt 0cm 36.0pt 72.0pt 108.0pt 144.0pt 180.0pt 216.0pt; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">For those interested in more, <a href="https://github.com/Open-I-Beam/swift-storlets">github</a> has comprehensive documentation that includes storlet samples and automated installation and configuration, which gives you a quick start.</span></div>   <br /><br />

Storlets: From research prototype to open source technology

Active Media Store cloud storage Docker Eran Rom IBM Research - Haifa openstack storlets Swift Storlets: From research prototype to open source technology

Researcher Eran Rom Editor’s note, this article is by Eran Rom, storage researcher at IBM Research – Haifa. In a previous blog post devoted...

Hybrid storage for the hybrid cloud

1:50 PM N1000 0

<br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB87TSkfXNMoMFeSBC-7fXZ7nICNP1WEy55FEGYzlD6n_9siKHqxt2v0tsMnaFFH7TY1gQ7QbTayxKtCcQ-66bMK6mXRNqb6Q_dlGEL2jZ6vOxhfluMBpOnLfeuawvu4lyhf7OjJrsdRy-/s1600/Krishna_Arvind_kickoff.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB87TSkfXNMoMFeSBC-7fXZ7nICNP1WEy55FEGYzlD6n_9siKHqxt2v0tsMnaFFH7TY1gQ7QbTayxKtCcQ-66bMK6mXRNqb6Q_dlGEL2jZ6vOxhfluMBpOnLfeuawvu4lyhf7OjJrsdRy-/s1600/Krishna_Arvind_kickoff.jpg" width="155" /></a></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">By Arvind Krishna, Senior Vice President and Director of IBM Research</span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">IBM Research has long played a pivotal role in the evolution of storage.<span style="mso-spacerun: yes;"> </span>In 1956 IBM researchers helped to create </span><span style="font-family: "Verdana",sans-serif; font-size: 12.0pt;"><a href="http://www-03.ibm.com/ibm/history/ibm100/us/en/icons/ramac/transform/"><span style="color: #0082bf; mso-bidi-font-family: Arial;">RAMAC</span></a></span><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">, the first magnetic hard disk.<span style="mso-spacerun: yes;"> </span>They also developed the giant magneto-resistive head for disk drives in the 1980s that still serves as the basis for all of today’s disk drives.<span style="mso-spacerun: yes;"> </span>In 1995 they helped IBM to win a National Medal of Technology for rewritable disks. Now it’s time for us to take another big step.</span><br /><a name="more"></a><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"> </span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">With data growing at 50 percent per year, IBM is investing $1 billion to manage this digital wellspring with storage software for the hybrid cloud. This five-year investment includes research and development of new cloud storage software, object storage and open standard technologies such as OpenStack. </span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">While perhaps not quite as captivating as watching RAMAC’s massive spinning disks must have been, the archiving capability for this new technology, called <a href="http://www-03.ibm.com/systems/storage/spectrum/" target="_blank">Spectrum Storage</a>, did win an </span><span style="font-family: "Verdana",sans-serif; font-size: 12.0pt;"><a href="http://www.research.ibm.com/articles/linear-tape-file-system.shtml"><span style="color: #0082bf; mso-bidi-font-family: Arial;">Emmy</span></a></span><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"> back in 2011. If you like having digital media instantly available on everything from your phone to your smart TV, you have IBM’s “Spectrum Archive” and our researchers to thank.</span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">Living up to our heritage in developing industry leading technology for IBM products, IBM Research played a significant role in this announcement by inventing four out of six IBM Spectrum Storage offerings and contributing heavily to the other two.</span></div><div class="MsoNormal"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><b><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">Unboxing the full spectrum of data storage</span></b></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><b><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"><br /></span></b></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">We predict that storage software will overtake storage hardware by 2020, by which time it will have to manage 40 zettabytes (40 sextillion bytes) of data. We believe most of that data will be in hybrid cloud because of the flexibility it offers businesses. A company that has your data, or data you want, will be able to manage, analyze, add to, and transfer it all from a single dashboard, something impossible to do today on storage hardware that sits alone in a datacenter. </span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"><br /></span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCHWgZ2fNgDTe87-8TK1bkYgqDskCYf2agO3PDgTi5ajQJa88OF77l6wErBx3OwYmPKyUzhzi_8NtQ6DZYB9CWriHw7Q7wujGk5JxY5SP6z0LhLEMsdAok0GmsByB0lXUTl9q1TugkfTju/s1600/storage2025.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCHWgZ2fNgDTe87-8TK1bkYgqDskCYf2agO3PDgTi5ajQJa88OF77l6wErBx3OwYmPKyUzhzi_8NtQ6DZYB9CWriHw7Q7wujGk5JxY5SP6z0LhLEMsdAok0GmsByB0lXUTl9q1TugkfTju/s1600/storage2025.jpg" /></a></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"><br /></span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"></span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">The other major benefit of storage software is that it can access and analyze any kind of data wherever it lives, no matter the hardware, platform, or format. So, from mobile devices linked to your bank, to servers full of unstructured social media information, data – via the cloud – can be understood. </span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">This technology is already demonstrating its value.<span style="mso-spacerun: yes;"> </span>For example, </span><span style="font-family: "Verdana",sans-serif; font-size: 12.0pt;"><a href="https://www-03.ibm.com/press/us/en/pressrelease/44866.wss"><span style="color: #0082bf; mso-bidi-font-family: Arial;">Caris Life Sciences</span></a></span><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"> is using a part of the Spectrum portfolio to speed up the company’s molecular profiling services for cancer patients. Scientists at DESY, a major research center out of Germany, use Spectrum to crunch more than 20 GB of data per second to study atomic structures. </span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">Beyond the next five years and all of its zettabytes, software-defined-storage can help lead us to new technologies like </span><span style="font-family: "Verdana",sans-serif; font-size: 12.0pt;"><a href="http://ibmresearchnews.blogspot.com/2014/07/40-year-old-challenge-solved-for-phase.html"><span style="color: blue; mso-bidi-font-family: Arial; text-decoration: none; text-underline: none;">phase-change memory</span></a></span><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"> (PCM), STT-RAM, and beyond. In fact, our scientists in Zurich made a breakthrough last year in the materials development of PCM, which promises to bridge the performance gap between the main memory and storage electronics from mobile phones to cloud data centers. And its unique physical properties make it ideal to serve as the memory for our work on </span><span style="font-family: "Verdana",sans-serif; font-size: 12.0pt;"><a href="http://research.ibm.com/cognitive-computing/neurosynaptic-chips.shtml#fbid=ybGSdbUeH13"><span style="color: blue; mso-bidi-font-family: Arial; text-decoration: none; text-underline: none;">brain chip architecture</span></a></span><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">.</span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">That’s what’s so exciting about the storage world – it’s always moving forward.<span style="mso-spacerun: yes;"> </span>As far as we’ve come in the storage evolution, the journey is just beginning.<span style="mso-spacerun: yes;"> </span>As it has done in the past, IBM Research will be there every step of the way.</span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div>

ibm research openstack spectrum storage storage Hybrid storage for the hybrid cloud

By Arvind Krishna, Senior Vice President and Director of IBM Research IBM Research has long played a pivotal role in the evolution of storag...