N1005 IBM Research: Eran Rom

<br /><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs1tf1YT-VtgO5V9ftIFwb3GyuuBk24QSwOdmwaS03j46aJsjxq5Ir27KO43ghLoo_2QeoOPm6IDUnSqfsREZ7vR9R9Y8E0uJxidUHR-ivNtBv1cDLXlXCYa69dJljBe357XBTJyDAiPQ/s1600/DSC_0064_Fotor_Fotor.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs1tf1YT-VtgO5V9ftIFwb3GyuuBk24QSwOdmwaS03j46aJsjxq5Ir27KO43ghLoo_2QeoOPm6IDUnSqfsREZ7vR9R9Y8E0uJxidUHR-ivNtBv1cDLXlXCYa69dJljBe357XBTJyDAiPQ/s1600/DSC_0064_Fotor_Fotor.jpg" width="132" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Researcher Eran Rom</td></tr></tbody></table><i><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Editor’s note, this article is by Eran Rom, storage researcher at IBM Research – Haifa.</span></i><br /><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">In a <a href="http://ibmresearchnews.blogspot.co.il/2014/05/storlets-turning-object-storage-into.html">previous blog</a> post devoted to storlets, IBM Fellow Michael Factor highlighted how storlets can be used to turn a software-defined object store into a smart storage platform. This is done by allowing the computation to run near the data, rather than bringing the data to the servers doing the computations. Michael's post addresses the potential of storlets in cost reduction as well as in enabling new services.</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">While in this post we want to concentrate on the technology itself, here are a few things that have happened with storlets since they were a research prototype.</span><br /><a name="more"></a><br /><br /><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;"><b>Storlets advancements</b> </span></div><ul><li><span style="color: black; font-family: Symbol; mso-bidi-font-family: Symbol; mso-fareast-font-family: Symbol;"><span style="mso-list: Ignore;"><span style="font: 7.0pt "Times New Roman";"></span></span></span><span dir="LTR"></span><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Storlets played a central role in a <a href="http://www.research.ibm.com/client-programs/foak/">first-of-a-kind</a>solution, called Active Media Store, developed with Radiotelevisione Italiana (RAI).</span></li><li><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">The <a href="https://www.openstack.org/summit/openstack-paris-summit-2014/session-videos/presentation/docker-meets-swift-a-broadcaster-and-39-s-experience">work with RAI</a> was presented at the Paris OpenStack Summit, and adopted by OpenStack as a "<a href="http://superuser.openstack.org/articles/docker-meets-swift-a-broadcaster-s-experience">superuser story</a>".</span></li><li><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">A reference implementation of storlets is now publicly available under 'Open-I-Beam' in <a href="https://github.com/Open-I-Beam/swift-storlets">github</a>.</span></li></ul><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">We started interactions with the OpenStack community on the question of ‘if and how’ to add storlet support into the official Swift release. We will be having a design session discussion on this topic at the upcoming <a href="https://wiki.openstack.org/wiki/Design_Summit/Liberty">Vancouver OpenStack Summit</a>. We encourage all interested parties to attend.<span style="mso-spacerun: yes;">  </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;"><span style="mso-spacerun: yes;"></span></span></div><br /><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;"><span style="mso-spacerun: yes;"> </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><b><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 12.0pt; mso-bidi-font-family: Helv;">Storlets on OpenStack Swift</span></b></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Our implementation of storlets is integrated with OpenStack Swift. Swift is an open source implementation of an object store and is behind several public object store services, including the IBM SoftLayer object store. Part of the idea behind storlets is to provide a flexible means of extending the function of the object store, by giving Swift users the ability to upload code to be executed near the data. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Running user-written code inside the storage system calls for adequate security and isolation measurements. This is where Docker comes into the picture. <a href="https://www.docker.com/">Docker</a> is a popular Linux container management framework.<span style="mso-spacerun: yes;">  </span>Linux containers (LXC) are similar to virtual machines, only instead of virtualizing the hardware they virtualize the operating system.</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">In addition to providing security and isolation, Docker has tools for packaging and deploying executable images. Using Docker our implementation allows the user to upload the storlet’s code, along with a tailored image where the storlet will execute.<span style="mso-spacerun: yes;">  </span>Thus, if a storlet relies on some non-trivial software stack, that stack can be packaged into a Docker image and deployed in a Swift cluster, to be later used for executing the user's storlets.</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Writing a storlet involves implementing a single method Java interface called IStorlet. That method - called invoke - has two major parameters: an input stream and an output stream. The input stream is used for consuming the data of an object on which the storlet is operating and the output stream is used to write the results of the storlet's computation. Storlets work in a streaming fashion, i.e., they start outputting data before reading all the input data. This is due to the synchronous fashion of storlets’ invocation as part of the upload or download operations as described next.</span><br /><br /><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Once the Docker images and storlets are deployed, they can be invoked on data objects in Swift. Storlets can be invoked in two ways:</span><br /><ol><li><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Invocation during object download. In this case the storlet transforms the object before it is returned to the user. This can be used for scenarios such as pre-filtering data being retrieved for an analytics engine or as an on-the-fly resolution reduction when downloading to a mobile device.</span></li><li><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Invocation during object upload. In this case the data stored is transformed from the data PUT by the user. One example use case is metadata enrichment, where a storlet can tag a data object with additional metadata while it is being uploaded. </span></li></ol></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; tab-stops: -36.0pt 0cm 36.0pt 72.0pt 108.0pt 144.0pt 180.0pt 216.0pt; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">In our current implementation, invoking a storlet during the upload or the download of an object involves adding a single header to the upload / download request.<span style="mso-spacerun: yes;">   </span>This header identifies the storlet to execute on the object that is the target of the request. Once the request is received by Swift, a pluggable middleware intercepts the request at the appropriate point: During download this point is when the response data is on its way back to the user, and during uploads this point is along the input path of the request. </span><br /><br /><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">Our code then routes the data to the storlet using file descriptors that are passed over Linux domain sockets to the storlet code running inside the Docker container. Other than these file descriptors, the Docker container has no access to any I/O device. This means the storlet's code has no network access and no access to Swift's own disks. All I/O is done via the file descriptors provided by our Swift plugin.</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; tab-stops: -36.0pt 0cm 36.0pt 72.0pt 108.0pt 144.0pt 180.0pt 216.0pt; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; tab-stops: -36.0pt 0cm 36.0pt 72.0pt 108.0pt 144.0pt 180.0pt 216.0pt; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; mso-bidi-font-family: Helv;">For those interested in more, <a href="https://github.com/Open-I-Beam/swift-storlets">github</a> has comprehensive documentation that includes storlet samples and automated installation and configuration, which gives you a quick start.</span></div>   <br /><br />

Storlets: From research prototype to open source technology

Researcher Eran Rom Editor’s note, this article is by Eran Rom, storage researcher at IBM Research – Haifa. In a previous blog post devoted...