N1005 IBM Research: performance bottlenecks

<div class="MsoNormal" style="line-height: normal; margin-bottom: 6.0pt; mso-layout-grid-align: none; text-autospace: none;"><i><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">Editor’s note: This blog entry is by IBM Fellow <a href="http://researcher.watson.ibm.com/researcher/view.php?person=il-FACTOR" target="_blank">Michael Factor</a> and IBM storage research engineer <a href="http://researcher.watson.ibm.com/researcher/view.php?person=il-DMITRYS" target="_blank">Dmitry Sotnikov</a> from IBM Research – Haifa. The work was done together with <a href="http://researcher.watson.ibm.com/researcher/view.php?person=il-YARON" target="_blank">Yaron Weinsberg</a> from IBM Research - Haifa.</span></i><br /><br /></div><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGCV0rQE75GSaOb0OGThCssam9wIW5qeDdvv93DK7q3-MUjOcpGwmuacVgir_WDr8uKtShCQvHJmyBlZz_yBIe_RbVnn5GKB2475RZnuW7qjwUyOIirJPiXuCNgB79g2jdPAzt8PefXHM/s1600/michael4_sm.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="173" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGCV0rQE75GSaOb0OGThCssam9wIW5qeDdvv93DK7q3-MUjOcpGwmuacVgir_WDr8uKtShCQvHJmyBlZz_yBIe_RbVnn5GKB2475RZnuW7qjwUyOIirJPiXuCNgB79g2jdPAzt8PefXHM/s200/michael4_sm.jpg" width="200" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Michael Factor, IBM Fellow</td></tr></tbody></table><div class="MsoNormal"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;">When companies deploy a system, they have specific performance, durability, and cost objectives in mind. So, before physical deployment, they will run models and simulations to get a close approximation of how the system will meet those objectives. But after deployment, the system must be checked. This holds true for even open source systems </span><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;">—</span> which have become more and more popular because of their flexibility in terms of available, interchangeable hardware and software. Our team at IBM Research-Haifa, building on open source tools, has been gaining experience and developing techniques to observe and monitor one of the most-popular options, OpenStack <a href="https://wiki.openstack.org/wiki/Swift" target="_blank">Swift</a>. It's the leading open source object storage system that runs in public and private clouds.</span><br /><a name="more"></a><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;"><br /><b> </b></span><br /><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;"><b>Let’s dive into Swift and what we learned from the data collected during monitoring. </b><br /><br /><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLCTJHAoko05CpmnbCsrio8oYYrwdcY5zDQ5gEJmNCRoZx9uGPrZLELhGI7hAzB0_DLaQfVNJ6Q19wddPJvuqrftewn6swr6WwmwIvF46UtKr2E-PtZ1Jt4yO-C5WgdD7ljwenjdG97hs/s1600/Dima1.jpg" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="196" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgLCTJHAoko05CpmnbCsrio8oYYrwdcY5zDQ5gEJmNCRoZx9uGPrZLELhGI7hAzB0_DLaQfVNJ6Q19wddPJvuqrftewn6swr6WwmwIvF46UtKr2E-PtZ1Jt4yO-C5WgdD7ljwenjdG97hs/s200/Dima1.jpg" width="200" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Dmitry Sotnikov, IBM storage engineer</td></tr></tbody></table>Increasing system complexity means increased monitoring complexity — since huge amounts of data need to be analyzed to find out what’s really going on. That’s where our work comes in. Our methodology  uses an open source toolbox and enables understanding the behavior of Swift clusters by examining the data collected during monitoring.<br /><br />Today, performance monitoring and troubleshooting of a running cloud-based object storage is as much an art as a science. Although there are a plethora of open source monitoring tools to gather system metrics, the real challenge is how to use them to find the root cause of a problem. <br /><br />We developed a general, open-source-based, step-by-step methodology to understand performance bottlenecks in a Swift system. Our solution uses standard tools including <a href="https://www.elastic.co/products/logstash" target="_blank">Logstash</a>, <a href="https://collectd.org/" target="_blank">collectd</a>, <a href="https://codeascraft.com/2011/02/15/measure-anything-measure-everything/" target="_blank">StatsD</a>, <a href="https://www.elastic.co/" target="_blank">Elasticsearch</a>, <a href="https://www.elastic.co/products/kibana" target="_blank">Kibana </a>and <a href="http://graphite.wikidot.com/" target="_blank">Graphite</a>. It also includes an additional simple Swift middleware we developed to gain further insights into the source of system bottlenecks.  </span></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4spDf38YlwBo9umkLg7djk8blFxzKEHXBV-RMjL6Ih6c1wFjVIUMI7V9HzzwhUIn_di9pk2LlslD-c_zndA6dIK6GNLoHwLxPtGcCMEGmM_rkdJk1dX75r6YPiN4fxeun-lifixUGTfM/s1600/swiftflowmonitoring.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="282" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4spDf38YlwBo9umkLg7djk8blFxzKEHXBV-RMjL6Ih6c1wFjVIUMI7V9HzzwhUIn_di9pk2LlslD-c_zndA6dIK6GNLoHwLxPtGcCMEGmM_rkdJk1dX75r6YPiN4fxeun-lifixUGTfM/s320/swiftflowmonitoring.jpg" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Swift monitoring flow</td></tr></tbody></table><div class="MsoNormal"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;">Our methodology helps validate the correctness of a Swift cluster configuration, and identifies which important data should be presented in the visualization of the Swift parameters together with the system’s statistics. This visualization helps users gain a better understanding of the internals of Swift and its behavior; it also enables them to see potential problems and misconfigurations. <br /><br />For example, if validation of Swift’s network configuration is required, for instance to understand unexpectedly low performance, , it can be done by using our methodology and the open source toolkit on which it is based. This can be seen in the following charts, which present the network utilization between the Proxy and the Object servers, as well as the Proxy’s public network utilization for a write only workload. Based on these charts one can easily validate that all the data received by the Proxy is replicated three times and sent to the Object servers. </span></div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3MnD1N_YIcblLbdH9arLJu80vCE0YGdEsdmc6MT6M1IB2l9LuEeXKm_lLqu50OUONSupqE9HeC6ZvafC8BcvhQiVUEPfteqLenwA-Atly4ZTE2Viys6NFGQ-VK_ANmcGUjXHeGxlppUA/s1600/swiftvisualization.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3MnD1N_YIcblLbdH9arLJu80vCE0YGdEsdmc6MT6M1IB2l9LuEeXKm_lLqu50OUONSupqE9HeC6ZvafC8BcvhQiVUEPfteqLenwA-Atly4ZTE2Viys6NFGQ-VK_ANmcGUjXHeGxlppUA/s400/swiftvisualization.jpg" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Data visualizations</td></tr></tbody></table><div class="MsoNormal"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;">   <br />At the <a href="https://www.openstack.org/summit/vancouver-2015/" target="_blank">OpenStack summit on May 20</a> in Vancouver, we will be <a href="https://openstacksummitmay2015vancouver.sched.org/event/ba79147a4be111341bf7a3d91f504a40#.VUtInZMj9JI" target="_blank">demonstrating </a>the results obtained from our approach, used with an internal deployment of Swift. You are welcome to join us at 1:50 pm, room 109. </span><br /><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 107%; mso-bidi-font-family: Helv;"></span></div>

Editor’s note: This blog entry is by IBM Fellow Michael Factor and IBM storage research engineer Dmitry Sotnikov from IBM Research – Haifa...