N1005 IBM Research: storage

Reflections en-route from the OpenStack Summit

<br /><br /><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRypNkD9GqFUitQ_faFZ61uAFHGLHmv-pfosxloB3VC9xmddcynWwMhfv5FGE3tBUIvgCHqcoabCjxnG0qhABpTyBxFRwBoDm2q9q2-buKZVbJeSiHvJcTBmnIwrX82ZfnKKB08egpBpc/s1600/michael4_sm.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="173" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRypNkD9GqFUitQ_faFZ61uAFHGLHmv-pfosxloB3VC9xmddcynWwMhfv5FGE3tBUIvgCHqcoabCjxnG0qhABpTyBxFRwBoDm2q9q2-buKZVbJeSiHvJcTBmnIwrX82ZfnKKB08egpBpc/s200/michael4_sm.jpg" width="200" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Michael Factor, IBM Research</td></tr></tbody></table><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"></span><i><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">Editor’s note: This post was authored by Michael Factor, IBM Fellow </span></i> </div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">As I sat on the plane during the return trip from the </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="https://www.openstack.org/summit/tokyo-2015/"><span style="mso-bidi-font-family: Helv;">OpenStack Summit in Tokyo</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">, I couldn’t help but reflect on the tremendous value of open source. One might ask: Why is IBM so interested in open source?</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">It’s not just about OpenStack and its various projects such as <a href="https://www-01.ibm.com/support/knowledgecenter/SST55W_4.3.0/liacc/liaccnova.html">Nova</a>, <a href="http://www.ibm.com/developerworks/cloud/library/cl-openstack-swift-cinder/">Cinder, or Swift</a>. There are so many benefits in the broad world of open source projects; some of which are being integrated with and used in IBM’s cloud, including OpenStack and others. Examples of these projects include <a href="http://www.ibm.com/analytics/us/en/technology/spark/">Spark</a>, <a href="https://developer.ibm.com/bluemix/2015/10/23/ibm-and-docker-bring-production-ready-containers-to-europe/">Docker</a>, <a href="https://www-01.ibm.com/support/knowledgecenter/SSPT3X_4.1.0/com.ibm.swg.im.infosphere.biginsights.product.doc/doc/bi_kafka.html">Kafka</a>, <a href="https://console.ng.bluemix.net/catalog/elasticsearch-by-compose/">ElasticSearch</a>, <a href="http://www.ibm.com/developerworks/data/library/techarticle/dm-1510-parquet-big-sql/index.html">Parquet</a>, among others. Each of these projects brings value on its own – not the least of which is enabling consistency and choice as to where the workload runs since these projects can all be deployed in local, dedicated, and public clouds. But the really big value comes from combining projects to address real world problems. It’s a definite case of the whole is greater than the sum of its parts. In short, 1+1=3.</span><br /><a name="more"></a></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">We can look at all of these services as puzzle pieces that when put together solve real world problems. In an ideal world, the different open source projects we need would all be available as services on public, private, and dedicated clouds. And it would be trivial to integrate the services by a point and a click. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">For example, we should be able to take a message bus like Kafka, and use a simple configuration command with no coding to have it archive the messages in an object store service. Once in that service, another point-and-click could pull the data into an analytics engine (like Apache Spark).<span style="mso-spacerun: yes;">  </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><b style="mso-bidi-font-weight: normal;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">The best (and worst) of open source times</span></b></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">It is the best of times for developers because of the wide array of open source puzzle pieces out there, which can be used to build a solution. It is also the worst of times since there is still work needed to make sure these pieces snap together easily, and too often this burdens the developer.</span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpUg7xgbtyI_dcZVnqN5oxAdukfJg89UITYrCfadVsXoCewnANca8mA22zGwNkRVcRsQO7Ti6LqahLBInEPLT0sOvsqdn0JSOpnmLoOoijKRWbpTBhMllopBp8NINdi-jhtcu311zw-1w/s1600/Putting+Pieces+Together+Nov+2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="97" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpUg7xgbtyI_dcZVnqN5oxAdukfJg89UITYrCfadVsXoCewnANca8mA22zGwNkRVcRsQO7Ti6LqahLBInEPLT0sOvsqdn0JSOpnmLoOoijKRWbpTBhMllopBp8NINdi-jhtcu311zw-1w/s400/Putting+Pieces+Together+Nov+2.jpg" width="400" /></a></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">But my colleagues at IBM’s Research lab in Haifa and I are working on integrating several of these projects – so we really can snap together those puzzle pieces!<span style="mso-spacerun: yes;">   </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">For example, my colleague <a href="http://researcher.watson.ibm.com/researcher/view.php?person=il-GILV">Gil Vernik</a>, a cloud storage, security and analytics expert in the lab, is </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="http://www.spark.tc/revving-up-performance-for-the-tachyon-file-system/"><span style="mso-bidi-font-family: Helv;">enabling Tachyon to use Swift</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"> as its persistent underlying storage system. <a href="http://ibmresearchnews.blogspot.co.il/2015/08/tachyon-for-ultra-fast-big-data.html">Tachyon</a>is a very active, but relatively new, open source project that provides an in-memory file system with automated tiering. While Tachyon is general purpose, one of its most known use cases is improving Spark’s performance over conventional stores. While Tachyon is in-memory, it needs some place to persist data when it is no longer being used, or if it runs out of memory in higher performance tiers. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">When Gil first started this work, the main choices for persisting data were <a href="http://www-01.ibm.com/software/data/infosphere/hadoop/hdfs/">HDFS</a> or Amazon S3 (other options exist today). Both are good solutions but each has its limitations: HDFS was not designed as a long term, multi-tenant store; Amazon’s S3 is part of a public cloud. By adding support for OpenStack Swift, there is now a multi-tenant, object store under Tachyon that can provide long term, cost-effective persistent store. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">And <a href="https://www.linkedin.com/profile/view?id=ACgAAAa-VpYBQF56yrrdDwbVLrKRiBeLm84evZs&authType=name&authToken=Of6l">Guy Hadash</a>, another colleague on our cloud team, is developing a solution to aggregate and store messages from the Kafka message bus in OpenStack Swift. Visual bookmark social media site Pinterest had a small project called </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="https://github.com/pinterest/secor"><span style="mso-bidi-font-family: Helv;">Secor</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"> which knows how to subscribe to Kafka, aggregate messages, and then put them in Amazon's S3. These messages are stored sequentially in the object. The object can later be retrieved for batch processing of the messages. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">Guy extended Secor so it could store messages via the Swift API. This gives Secor users the choice of using Kafka-Secor-Swift in all three deployment models: local, dedicated, and public. He then stored the data in </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="https://parquet.apache.org/"><span style="mso-bidi-font-family: Helv;">Parquet</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"> format, not just as a list of messages. Parquet is designed for objects or files that contain tabular data (like CSV). This data is stored in a columnar format, enabling efficient compression and the retrieval of only selected columns from the store. Lastly, Guy annotated the created objects with metadata, such as the minimum or maximum values of various columns in the table. We have already implemented prototypes of all three steps. A patch for the first step is already part of the community Secor code, and we are starting to work with the community on the second. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQ0NyvP4zJdJJqR8L1QjwDbOXVwTDIehq4s68WM5-PaWB0vTdZCYAYiEWXSHXKief1f85Yv3TcOiil0gDvim1QltqI_jCygCqyNFo4O1dqDpEUafSOIrIEbG0btpNaq7h7uIBOdQrvnSU/s1600/twitter_paula.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQ0NyvP4zJdJJqR8L1QjwDbOXVwTDIehq4s68WM5-PaWB0vTdZCYAYiEWXSHXKief1f85Yv3TcOiil0gDvim1QltqI_jCygCqyNFo4O1dqDpEUafSOIrIEbG0btpNaq7h7uIBOdQrvnSU/s320/twitter_paula.jpg" width="256" /></a></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">Then there’s what we are doing with our partners in the context of the EU </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="http://iot-cosmos.eu/"><span style="mso-bidi-font-family: Helv;">COSMOS</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"> project. <a href="https://spark-summit.org/eu-2015/speakers/paula-ta-shma/">Paula Ta-Shma</a>, a cloud security and analytics expert on our team, presented </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="https://spark-summit.org/eu-2015/events/how-spark-enables-the-internet-of-things-efficient-integration-of-multiple-spark-components-for-smart-city-use-cases/"><span style="mso-bidi-font-family: Helv;">this work</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"> at the recent Spark Summit in Amsterdam. The goal of this work is to help improve the timeliness of Madrid’s city buses by automating their reactions to changes in traffic. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">Data is available from thousands of static sensors. Using NodeRed (an open source project for defining data flows), the data is retrieved from the sensors and put on Kafka. As described above, Secor aggregates, formats, and annotates this data, before storing it in Swift. Spark is then used to analyze this data. Using standard Spark machine learning, the solution defines the expected values for speed and traffic density on different days and times. A complex event processing engine, which also subscribes to the messages from the sensors, takes the threshold values and can issue warnings to trigger corrective action, such as rerouting a bus or changing the traffic light pattern, when needed. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">These examples clearly show the tremendous value in putting puzzle pieces together from different open source projects to build something even better. We are continuing to work on integrating services and look forward to describing some of our other efforts in future blogs.<span style="mso-spacerun: yes;">  </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal"><br /></div>

Reflections en-route from the OpenStack Summit

Michael Factor, IBM Research Editor’s note: This post was authored by Michael Factor, IBM Fellow As I sat on the plane during the return tr...

Hybrid storage for the hybrid cloud

1:50 PM N1000 0

<br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB87TSkfXNMoMFeSBC-7fXZ7nICNP1WEy55FEGYzlD6n_9siKHqxt2v0tsMnaFFH7TY1gQ7QbTayxKtCcQ-66bMK6mXRNqb6Q_dlGEL2jZ6vOxhfluMBpOnLfeuawvu4lyhf7OjJrsdRy-/s1600/Krishna_Arvind_kickoff.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjB87TSkfXNMoMFeSBC-7fXZ7nICNP1WEy55FEGYzlD6n_9siKHqxt2v0tsMnaFFH7TY1gQ7QbTayxKtCcQ-66bMK6mXRNqb6Q_dlGEL2jZ6vOxhfluMBpOnLfeuawvu4lyhf7OjJrsdRy-/s1600/Krishna_Arvind_kickoff.jpg" width="155" /></a></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">By Arvind Krishna, Senior Vice President and Director of IBM Research</span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">IBM Research has long played a pivotal role in the evolution of storage.<span style="mso-spacerun: yes;"> </span>In 1956 IBM researchers helped to create </span><span style="font-family: "Verdana",sans-serif; font-size: 12.0pt;"><a href="http://www-03.ibm.com/ibm/history/ibm100/us/en/icons/ramac/transform/"><span style="color: #0082bf; mso-bidi-font-family: Arial;">RAMAC</span></a></span><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">, the first magnetic hard disk.<span style="mso-spacerun: yes;"> </span>They also developed the giant magneto-resistive head for disk drives in the 1980s that still serves as the basis for all of today’s disk drives.<span style="mso-spacerun: yes;"> </span>In 1995 they helped IBM to win a National Medal of Technology for rewritable disks. Now it’s time for us to take another big step.</span><br /><a name="more"></a><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"> </span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">With data growing at 50 percent per year, IBM is investing $1 billion to manage this digital wellspring with storage software for the hybrid cloud. This five-year investment includes research and development of new cloud storage software, object storage and open standard technologies such as OpenStack. </span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">While perhaps not quite as captivating as watching RAMAC’s massive spinning disks must have been, the archiving capability for this new technology, called <a href="http://www-03.ibm.com/systems/storage/spectrum/" target="_blank">Spectrum Storage</a>, did win an </span><span style="font-family: "Verdana",sans-serif; font-size: 12.0pt;"><a href="http://www.research.ibm.com/articles/linear-tape-file-system.shtml"><span style="color: #0082bf; mso-bidi-font-family: Arial;">Emmy</span></a></span><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"> back in 2011. If you like having digital media instantly available on everything from your phone to your smart TV, you have IBM’s “Spectrum Archive” and our researchers to thank.</span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">Living up to our heritage in developing industry leading technology for IBM products, IBM Research played a significant role in this announcement by inventing four out of six IBM Spectrum Storage offerings and contributing heavily to the other two.</span></div><div class="MsoNormal"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><b><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">Unboxing the full spectrum of data storage</span></b></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><b><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"><br /></span></b></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">We predict that storage software will overtake storage hardware by 2020, by which time it will have to manage 40 zettabytes (40 sextillion bytes) of data. We believe most of that data will be in hybrid cloud because of the flexibility it offers businesses. A company that has your data, or data you want, will be able to manage, analyze, add to, and transfer it all from a single dashboard, something impossible to do today on storage hardware that sits alone in a datacenter. </span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"><br /></span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCHWgZ2fNgDTe87-8TK1bkYgqDskCYf2agO3PDgTi5ajQJa88OF77l6wErBx3OwYmPKyUzhzi_8NtQ6DZYB9CWriHw7Q7wujGk5JxY5SP6z0LhLEMsdAok0GmsByB0lXUTl9q1TugkfTju/s1600/storage2025.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCHWgZ2fNgDTe87-8TK1bkYgqDskCYf2agO3PDgTi5ajQJa88OF77l6wErBx3OwYmPKyUzhzi_8NtQ6DZYB9CWriHw7Q7wujGk5JxY5SP6z0LhLEMsdAok0GmsByB0lXUTl9q1TugkfTju/s1600/storage2025.jpg" /></a></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"><br /></span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"></span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">The other major benefit of storage software is that it can access and analyze any kind of data wherever it lives, no matter the hardware, platform, or format. So, from mobile devices linked to your bank, to servers full of unstructured social media information, data – via the cloud – can be understood. </span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">This technology is already demonstrating its value.<span style="mso-spacerun: yes;"> </span>For example, </span><span style="font-family: "Verdana",sans-serif; font-size: 12.0pt;"><a href="https://www-03.ibm.com/press/us/en/pressrelease/44866.wss"><span style="color: #0082bf; mso-bidi-font-family: Arial;">Caris Life Sciences</span></a></span><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"> is using a part of the Spectrum portfolio to speed up the company’s molecular profiling services for cancer patients. Scientists at DESY, a major research center out of Germany, use Spectrum to crunch more than 20 GB of data per second to study atomic structures. </span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">Beyond the next five years and all of its zettabytes, software-defined-storage can help lead us to new technologies like </span><span style="font-family: "Verdana",sans-serif; font-size: 12.0pt;"><a href="http://ibmresearchnews.blogspot.com/2014/07/40-year-old-challenge-solved-for-phase.html"><span style="color: blue; mso-bidi-font-family: Arial; text-decoration: none; text-underline: none;">phase-change memory</span></a></span><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;"> (PCM), STT-RAM, and beyond. In fact, our scientists in Zurich made a breakthrough last year in the materials development of PCM, which promises to bridge the performance gap between the main memory and storage electronics from mobile phones to cloud data centers. And its unique physical properties make it ideal to serve as the memory for our work on </span><span style="font-family: "Verdana",sans-serif; font-size: 12.0pt;"><a href="http://research.ibm.com/cognitive-computing/neurosynaptic-chips.shtml#fbid=ybGSdbUeH13"><span style="color: blue; mso-bidi-font-family: Arial; text-decoration: none; text-underline: none;">brain chip architecture</span></a></span><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">.</span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana",sans-serif; font-size: 12.0pt; mso-bidi-font-family: Arial;">That’s what’s so exciting about the storage world – it’s always moving forward.<span style="mso-spacerun: yes;"> </span>As far as we’ve come in the storage evolution, the journey is just beginning.<span style="mso-spacerun: yes;"> </span>As it has done in the past, IBM Research will be there every step of the way.</span></div><div class="MsoNormal" style="mso-layout-grid-align: none; text-autospace: none;"><br /></div>

ibm research openstack spectrum storage storage Hybrid storage for the hybrid cloud

By Arvind Krishna, Senior Vice President and Director of IBM Research IBM Research has long played a pivotal role in the evolution of storag...

Storlets: Turning Object Storage into a Smart Storage Platform

6:00 AM N1000 0

<br /><div class="MsoNormal"></div><div class="MsoNormal"><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwU4RDCtEOxkH6uxGkWdTY_8oU9bNHuFNMIosRfsMRSgdad9Rv5sT4LNhUl8JhSBjJ0mg6iPbIkhpA2fDHDyo9hNHeSxYZhmGuSReFdUBWWcGafgDtNceXJVYuftwnXLWYPhLNDlIvw7Lx/s1600/michael5.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwU4RDCtEOxkH6uxGkWdTY_8oU9bNHuFNMIosRfsMRSgdad9Rv5sT4LNhUl8JhSBjJ0mg6iPbIkhpA2fDHDyo9hNHeSxYZhmGuSReFdUBWWcGafgDtNceXJVYuftwnXLWYPhLNDlIvw7Lx/s1600/michael5.jpg" width="191" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Michael Factor</td></tr></tbody></table><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;"><i>Editor’s note: This blog posting was authored by Michael Factor, Distinguished Engineer and expert on Storage Systems at IBM Research - Haifa</i></span></span></div><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="font-size: small;"><br /></span></span><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">Traditional file systems store our information in a tree structure. Although this works fine for small collections of data – like those on our local hard drive – they are not designed for the massive volumes of unstructured content most businesses are collecting, storing, and accessing on the cloud. </span></span><br /><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span></span><span style="font-size: small;"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">A new method of storing information is called object storage. This approach stores information as objects. Each object contains the data (the bits and bytes of our documents, movies, images, and so forth), together with metadata that holds user- and system-defined tags. These smart data objects include rich information – or metadata – that describes the content of the data, how the object is related to other objects, how the data should be handled, replicated, or backed up, and more.</span></span> </span><br /><div class="MsoNormal"><span style="font-size: small;"><br /></span></div><div class="MsoNormal"><div align="right"><table align="right" border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: medium none; margin-left: 6.75pt; margin-right: 6.75pt;"> <tbody><tr style="height: 71.5pt;">  <td style="border: 1pt solid white; height: 71.5pt; padding: 0in 5.4pt; width: 144.1pt;" valign="top" width="192"><div class="MsoNormal" style="line-height: 150%; margin-bottom: 0in;"><span style="font-size: x-large;"><b><span style="color: #0070c0; font-family: "Times New Roman"; line-height: 150%;">“But what if we could turn a software-defined object store into a smart storage platform?”</span></b></span></div></td> </tr></tbody></table></div><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;">Although object storage can store objects, manage them, protect them, and so on </span></span></span><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">– </span></span></span>it doesn’t by itself dramatically increase the rate at which we can extract value from objects. But what if we could turn a software-defined object store into a smart storage platform?</span></span></span></div><div class="MsoNormal"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span></span></div><div class="MsoNormal"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="line-height: 115%;">Storlets bring the computation to the data</span></b></span></span></div><div class="MsoNormal"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span></span></div><div class="MsoNormal"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;">A new research prototype called "storlets" holds the promise of greatly increasing the value we get out of storage and the speed at which we can access what we need. A new software-defined mechanism, storlets allow object storage to move the computation to the data, instead of the system having to move the data to a server to carry out the computation. </span></span></span><br /><br /><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;">Imagine if every time you wanted to cook a meal, you had to bring all your ingredients to a central neighborhood depot where stoves, appliances, and cooking utensils were available to "process" the food. That’s similar to the current situation with data on the cloud. Storlets come to remedy this situation by moving the heavy lifting to where it’s needed </span></span></span><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;">– </span></span></span></span></span></span>similar to allowing you to cook everything in your kitchen, where all your raw materials are already located. </span></span></span><br /><br /><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;">The impact of storlets is substantial. Stored data can be processed locally, and no longer needs to be transferred over the network to a remote computer, processed, and then put back onto the storage server – all of which incurs both network transfer latencies and thus real dollar costs.</span></span></span><br /><br /><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;">Our vision is to reduce costs, increase flexibility and improve security</span></span></span><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"> </span></span></span></span></span></span>by turning the object store into a platform, and allowing the functionality of the object store to be extended using software.  </span></span></span></div><div class="MsoNormal"><br /><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><b><span style="line-height: 115%;">Getting more value from data – much faster</span></b></span></span></div><div class="MsoNormal"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><br /></span></span></div><div class="MsoNormal"><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;">Aside from saving bandwidth by eliminating unnecessary data transfers, storlets are an ideal way to introduce new services. Storlets can analyze each object and extract its metadata, including size, subject, resolution, format, and more. Since storlets are dynamically loaded code, the function is only limited by the developer’s imagination.</span></span></span><br /><br /><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;">For example, a media company could upload a movie to an object store and have it automatically generate a representative image. A physician doing rounds could have the object store send only the portion of a patient’s x-ray needed to her wireless device for immediate viewing. A lawyer could request that the object store provide a document from a previous court case in a specific format. Or pathologists could have images analyzed in the object store itself. All this sophisticated computation is moved into the storage infrastructure via the software of dynamically loaded storlets, making it faster, more flexible, and far less expensive.</span></span></span></div><div class="MsoNormal"><br /><span style="font-size: large;"><span style="font-family: "Helvetica Neue",Arial,Helvetica,sans-serif;"><span style="line-height: 115%;">IBM researchers in Haifa, Israel have been working on prototyping storlets for several years in the context of European Union Research projects such as <a href="http://ensure-fp7-plone.fe.up.pt/site/">ENSURE</a>, <a href="http://www.visioncloud.eu/">VISION Cloud</a>, and <a href="http://www.forgetit-project.eu/">Forget IT</a>. In short, we’re looking forward to seeing more value from data in faster and more flexible ways.</span></span></span></div><div class="MsoNormal"></div>

Storlets: Turning Object Storage into a Smart Storage Platform

IBM Research - Haifa object storage SDS software-defined storage storage storlets systems storage Storlets: Turning Object Storage into a Smart Storage Platform

Michael Factor Editor’s note: This blog posting was authored by Michael Factor, Distinguished Engineer and expert on Storage Systems at IBM...