N1005 IBM Research: Tachyon

<br /><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdXKb4-iraxLcymVH8BWczZChL0j15NvM2buQTwjqHuzTOKzSiN4Cyxf0gLtnlxaeQwRL-hemLNLszpRGqvfksLW4i3RI6ERsosgjKl-K1TrbFG1nj4GxSd-8DjTpR-dmCtqBwbW_LRYxp/s1600/gilvernik.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdXKb4-iraxLcymVH8BWczZChL0j15NvM2buQTwjqHuzTOKzSiN4Cyxf0gLtnlxaeQwRL-hemLNLszpRGqvfksLW4i3RI6ERsosgjKl-K1TrbFG1nj4GxSd-8DjTpR-dmCtqBwbW_LRYxp/s1600/gilvernik.jpg" /></a></div><i><span style="font-family: "verdana" , "sans-serif";">Editor’s note: This article is by c</span></i><span style="font-family: "verdana" , "sans-serif";"><i>loud analytics infrastructure expert <a href="http://researcher.watson.ibm.com/researcher/view.php?person=il-GILV">Gil Vernik</a><span class="MsoHyperlink">, IBM Research-Haifa.</span> </i></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><span style="color: black; font-family: "verdana" , "sans-serif";">Today's massive growth in data sets means that storage is increasingly becoming a critical bottleneck for system workloads. My storage team in Haifa, Israel wants to analyze and understand these massive volumes of data, and we need to store them somewhere reliable. Although disk space is an option, it's too slow to carry out fast Big Data processing. In-memory computing, which keeps the data in a server’s RAM for fast access and processing, offers a good solution for processing Big Data workloads </span><span style="color: black; font-family: "verdana" , "sans-serif";"><span style="font-family: "verdana" , "sans-serif";">–</span> but it’s limited and expensive.<span style="mso-spacerun: yes;">  </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><span style="color: black; font-family: "verdana" , "sans-serif";">Enter </span><span style="font-family: "verdana" , "sans-serif";"><a href="http://tachyon-project.org/master/index.html">Tachyon</a><span style="color: black;">, a memory-centric distributed storage system that offers processing at memory-speed and reliable storage. Its software works with servers in clusters so there’s plenty of room for storage, and a unique proprietary feature eliminates the need for replication to ensure fault tolerance. Now, we’ve connected Tachyon to Swift so it can work effortlessly with </span><a href="https://wiki.openstack.org/wiki/Swift">Swift</a><span style="color: black;"> and </span><a href="http://www.ibm.com/cloud-computing/us/en/softlayer.html">SoftLayer</a><span style="color: black;">. The result? Tachyon is even more flexibile and efficient. </span></span><br /><a name="more"></a></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><b><span style="color: black; font-family: "verdana" , "sans-serif";">Building efficiency into Big Data analytics </span></b></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><span style="font-family: "verdana" , "sans-serif";">Let’s take something like Facebook’s data. Tons of data need to be stored and analyzed, such as logs, activities, connections, media, locations, messages, and so forth. A good solution would be to store them as objects or files in an object store like Swift. Why as objects? Object storage offer two important things: low-cost storage and reliability; so even if my computer fails, I know my data is safe. <span style="mso-spacerun: yes;"> </span></span></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><span style="font-family: "verdana" , "sans-serif";">Several applications can analyze this data, including <a href="http://spark.apache.org/">Apache Spark</a> and <a href="https://flink.apache.org/">Apache Flink</a>. Often, while one set of analytics is being done, say to find out what ads to display on your news page, another user might analyze the same data set to find out which geographies you have visited most often. In short, we can have different instances of analytics workloads all reading and writing the same data. Tachyon uses memory aggressively and can serve up the results to the users so the work doesn’t have to be done separately, multiple times. </span></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><span style="font-family: "verdana" , "sans-serif";">The latest evaluations show that Tachyon outperforms in-memory <a href="http://www-01.ibm.com/software/data/infosphere/hadoop/hdfs/" target="_blank">HDFS</a> by 110x for writes. It also improves the end-to-end latency of a realistic workflow by 4x (<span style="font-family: "verdana" , "sans-serif";"><a href="http://dl.acm.org/citation.cfm?doid=2670979.2670985" target="_blank"><span style="font-family: "verdana" , "sans-serif";">s</span>ource</a>)</span>.</span></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /><b><span style="font-family: "verdana" , "sans-serif";">Faster, cheaper, and reliable, Tachyon solves the problem</span></b></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><span style="font-family: "verdana" , "sans-serif";">Tachyon, a project out of <a href="https://amplab.cs.berkeley.edu/" target="_blank">UC-Berkeley’s AMPLab</a>, is intended to help organizations quickly store and access all that information. (The term <a href="https://en.wikipedia.org/wiki/Tachyon" target="_blank">tachyon</a> refers to a particle that moves faster than light.) And it has been gaining momentum. We certainly recognize its tremendous potential for improving the efficiency and fault tolerance of computation frameworks, such as Spark and Flink. With the help of <span style="mso-bidi-font-style: italic;">Tachyon Nexus founder </span><a href="https://www.linkedin.com/in/haoyuanli1"><span style="mso-bidi-font-family: "Tms Rmn"; mso-bidi-font-style: italic;">Haoyuan Li</span></a> and the Tachyon community, we were able to turn this potential into reality.</span></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLqtySLvlwJ4Z1VOxqK8H3SpGO7kBuYymLkIEN4BeL-nbqwwHK4S7pU_761y7oRglnwq99NWe_y8_dxeu2iCNmkoKvxA_1X-FTbb7zlbRPEmfbPW9GrEKwAFGZ7rvZ33wmTorKtPe4dc4/s1600/tachyon.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLqtySLvlwJ4Z1VOxqK8H3SpGO7kBuYymLkIEN4BeL-nbqwwHK4S7pU_761y7oRglnwq99NWe_y8_dxeu2iCNmkoKvxA_1X-FTbb7zlbRPEmfbPW9GrEKwAFGZ7rvZ33wmTorKtPe4dc4/s320/tachyon.jpg" width="320" /></a></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><span style="font-family: "verdana" , "sans-serif";">Many frameworks like Spark take advantage of memory. When they share the data between different frameworks or jobs, they need to write the data to the different systems, which takes time. And making sure they are synchronized for write, is even more difficult. Tachyon helps achieve memory throughput without unnecessary replication and still provides reliability. </span></div><br /><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><span style="font-family: "verdana" , "sans-serif";">If the computer fails, the system re-computes the data using lineage, and in this way provides reliability through fault tolerance. Data lineage is generally defined as a kind of data life cycle that includes the data's origins and its transformations. Because it’s a distributed system, Tachyon works as a cluster, using the memory of a whole bunch of computers. So if one computer fails, there’s no problem. Even if the entire cluster fails, everything is backed up since Tachyon saves everything on disk from time to time.</span></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><span style="font-family: "verdana" , "sans-serif";">Tachyon is open source and already deployed in production at multiple companies. In addition, the project has more than 100 contributors from more than 30 institutions, including Yahoo, Tachyon Nexus, Redhat, Baidu, Intel, and of course, IBM. The project is the storage layer of the Berkeley Data Analytics Stack (BDAS) and also part of the Fedora distribution. </span></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><b><span style="font-family: "verdana" , "sans-serif";">What next?</span></b></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><span style="font-family: "verdana" , "sans-serif";">We’ll continue investing effort in Tachyon so that more organizations can take advantage of the performance boost offered. This collaboration goes a long way towards preventing repetitive work, improving memory utilization, and allowing processing to reach new levels of memory usage – especially when it comes to Big Data analytics frameworks like Spark and others.</span></div><div class="MsoNormal" style="margin-bottom: .0001pt; margin-bottom: 0cm;"><br /></div>

Tachyon for ultra-fast Big Data processing

Editor’s note: This article is by c loud analytics infrastructure expert Gil Vernik , IBM Research-Haifa. Today's massive growth in dat...