N1005 IBM Research: IoT

Putting wearables to work for improved safety

Asaf Adi Editor's note: This post was authored by Dr. Asaf Adi, Senior Manager of IoT and Wearables at IBM Research - Haifa. Safety fir...

Watson: What the IoT has Been Waiting For

3:30 AM N1000 0

<br /><div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://c1.staticflickr.com/1/413/19687798544_128b7c7998_m.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://c1.staticflickr.com/1/413/19687798544_128b7c7998_m.jpg" /></a></div><i><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;">Editor's note<span style="font-family: "helvneue light for ibm" , sans-serif;">:</span> This article is by <a href="http://researcher.watson.ibm.com/researcher/view.php?person=zurich-cur">Dr. Alessandro Curioni</a>, Vice President, Europe, and Director, IBM Research - Zurich</span></i><br /><br /><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;">“How much does Watson cost?”</span><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt;"></span><br /><br /><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;">I often get asked this question whenever I give a lecture or speak with clients about cognitive computing.</span><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt;"></span><br /><br /><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;">My response is always the same<span style="font-family: "helvneue light for ibm" , sans-serif;">:</span> “What is the price of not knowing?” And depending on who I am speaking to, the context could be not knowing when the next devastating storm will arrive, what the next maintenance failure on an aircraft will be, or even what’s going to be the next fashion trend.</span><br /><a name="more"></a><br /><br /><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;">Right now, we don’t know a lot. Petabytes of data are being created every second<span style="font-family: "helvneue light for ibm" , sans-serif;"> of</span> every day, in large part by the <a href="http://www.ibm.com/IoT">Internet of Things</a> (IoT). Yet, 90<span style="font-family: "helvneue light for ibm" , sans-serif;"> percent</span> of it, which is often referred to as <i>data on the edge</i>, is never captured, analyzed or acted upon — it essentially can’<span style="font-family: "helvneue light for ibm" , sans-serif;">t</span> be read by computers. For example, only 1 percent of the data collected from an oil rig with 30,000 sensors is examined for anomalies. To me that translates into a lost opportunity, or worse a pending disaster.</span><br /><br /><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;">Today, I’m in Munich, Germany, where IBM is changing all of this as we bet big on a future IoT powered by IBM Watson and cognitive computing. Munich is where this new business unit will be headquartered, bringing together 1,000 IBM developers, consultants, designers and researchers — representing IBM’s largest investment in Europe in more than two decades. </span><span style="font-family: "ms gothic"; font-size: 13.0pt; mso-bidi-font-family: "MS Gothic";"> </span><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;">As the vice president for Research in Europe and director of IBM’s lab in Zurich, I’ll be supporting this new unit globally to drive IoT research across all 12 of our global labs.</span><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt;"></span><br /><br /><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;">Of course we aren’t starting from scratch. In March 2015, we announced a $3B commitment to IoT. <span style="font-family: "helvneue light for ibm" , sans-serif;">M</span>ore recently we announced plans to acquire the Weather Company’s IT assets<span style="font-family: "helvneue light for ibm" , sans-serif;">. It</span> includes a remarkable three billion weather forecast reference points, and a cloud platform supporting 26 billion inquiries per day, mostly from mobile apps. Combined with the dozens of Watson APIs available on IBM’s Watson IoT Cloud, this platform will bring the promise of IoT to both small businesses and large enterprises alike.</span><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; line-height: 107%;"> </span><br /><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; line-height: 107%;"> </span> <br /><a href="https://c2.staticflickr.com/6/5718/23177762070_72c8e2eb97.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="213" src="https://c2.staticflickr.com/6/5718/23177762070_72c8e2eb97.jpg" width="320" /></a><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;">In China, our <a href="http://www-03.ibm.com/press/us/en/pressrelease/48255.wss">Green Horizon Project</a> is using sensor data and analytics to generate high-resolution 1km-by-1km pollution forecasts 72 hours in advance<span style="font-family: "helvneue light for ibm" , sans-serif;">. It can also make </span>pollution trend predictions up to 10 days into the future to warn citizens and to help officials take action. Similar projects are taking place in <a href="http://www-03.ibm.com/press/us/en/pressrelease/48415.wss">South Africa</a>and <a href="http://www-03.ibm.com/press/us/en/pressrelease/48416.wss">India</a>as well.</span><br /><br /><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;">Back home in Switzerland, my colleagues in Zurich are working with IBM researchers in Dublin to bring the IoT to automobiles. Cars make excellent sensor platforms since they have strong batteries and are idle 95<span style="font-family: "helvneue light for ibm" , sans-serif;"> percent</span> of the day. The sensors can be used to find available parking spaces or perhaps even more critically, they can detect gas leaks in a city's<span style="font-family: "helvneue light for ibm" , sans-serif;"> </span>infrastructure. </span><br /><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;"><br />So, what is the price of using cognitive computing and IoT to predict a gas leak before it can do any harm? Priceless.</span><br /><span style="font-family: "helvneue light for ibm" , sans-serif; font-size: 13.0pt; mso-bidi-font-family: Arial;"><br /><i>Follow Alessandro on Twitter <a href="https://twitter.com/Ale_Curioni">@Ale_Curioni</a></i></span><br /><br /><a href="http://www.pinterest.com/pin/create/extension/?url=https%3A%2F%2Fwww.blogger.com%2Fblogger.g%3FblogID%3D3821464279005499761%23editor%2Ftarget%3Dpost%3BpostID%3D5362970009364900284%3BonPublishedMenu%3Dallposts%3BonClosedMenu%3Dallposts%3BpostNum%3D0%3Bsrc%3Dlink&media=https%3A%2F%2Fc2.staticflickr.com%2F6%2F5718%2F23177762070_72c8e2eb97.jpg&xm=h&xv=sa1.37.01&xuid=tYDKQnV6ndWa&description=" style="background-color: transparent; background-image: url(data:image/png; border: none; cursor: pointer; display: none; height: 20px; left: 359px; opacity: 0.85; position: absolute; top: 1236px; width: 40px; z-index: 8675309;"></a><a href="http://www.pinterest.com/pin/create/extension/?url=https%3A%2F%2Fwww.blogger.com%2Fblogger.g%3FblogID%3D3821464279005499761%23editor%2Ftarget%3Dpost%3BpostID%3D5362970009364900284%3BonPublishedMenu%3Dallposts%3BonClosedMenu%3Dallposts%3BpostNum%3D0%3Bsrc%3Dlink&media=https%3A%2F%2Fc2.staticflickr.com%2F6%2F5718%2F23177762070_72c8e2eb97.jpg&xm=h&xv=sa1.37.01&xuid=tYDKQnV6ndWa&description=" style="background-color: transparent; background-image: url(data:image/png; border: none; cursor: pointer; display: none; height: 20px; left: 359px; opacity: 0.85; position: absolute; top: 1236px; width: 40px; z-index: 8675309;"></a>

IBM Research - Zurich IBM Watson Internet-of-Things IoT Munich Watson: What the IoT has Been Waiting For

Editor's note : This article is by Dr. Alessandro Curioni , Vice President, Europe, and Director, IBM Research - Zurich “How much does ...

Reflections en-route from the OpenStack Summit

6:42 AM N1000 0

<br /><br /><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRypNkD9GqFUitQ_faFZ61uAFHGLHmv-pfosxloB3VC9xmddcynWwMhfv5FGE3tBUIvgCHqcoabCjxnG0qhABpTyBxFRwBoDm2q9q2-buKZVbJeSiHvJcTBmnIwrX82ZfnKKB08egpBpc/s1600/michael4_sm.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="173" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRypNkD9GqFUitQ_faFZ61uAFHGLHmv-pfosxloB3VC9xmddcynWwMhfv5FGE3tBUIvgCHqcoabCjxnG0qhABpTyBxFRwBoDm2q9q2-buKZVbJeSiHvJcTBmnIwrX82ZfnKKB08egpBpc/s200/michael4_sm.jpg" width="200" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Michael Factor, IBM Research</td></tr></tbody></table><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"></span><i><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">Editor’s note: This post was authored by Michael Factor, IBM Fellow </span></i> </div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">As I sat on the plane during the return trip from the </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="https://www.openstack.org/summit/tokyo-2015/"><span style="mso-bidi-font-family: Helv;">OpenStack Summit in Tokyo</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">, I couldn’t help but reflect on the tremendous value of open source. One might ask: Why is IBM so interested in open source?</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">It’s not just about OpenStack and its various projects such as <a href="https://www-01.ibm.com/support/knowledgecenter/SST55W_4.3.0/liacc/liaccnova.html">Nova</a>, <a href="http://www.ibm.com/developerworks/cloud/library/cl-openstack-swift-cinder/">Cinder, or Swift</a>. There are so many benefits in the broad world of open source projects; some of which are being integrated with and used in IBM’s cloud, including OpenStack and others. Examples of these projects include <a href="http://www.ibm.com/analytics/us/en/technology/spark/">Spark</a>, <a href="https://developer.ibm.com/bluemix/2015/10/23/ibm-and-docker-bring-production-ready-containers-to-europe/">Docker</a>, <a href="https://www-01.ibm.com/support/knowledgecenter/SSPT3X_4.1.0/com.ibm.swg.im.infosphere.biginsights.product.doc/doc/bi_kafka.html">Kafka</a>, <a href="https://console.ng.bluemix.net/catalog/elasticsearch-by-compose/">ElasticSearch</a>, <a href="http://www.ibm.com/developerworks/data/library/techarticle/dm-1510-parquet-big-sql/index.html">Parquet</a>, among others. Each of these projects brings value on its own – not the least of which is enabling consistency and choice as to where the workload runs since these projects can all be deployed in local, dedicated, and public clouds. But the really big value comes from combining projects to address real world problems. It’s a definite case of the whole is greater than the sum of its parts. In short, 1+1=3.</span><br /><a name="more"></a></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">We can look at all of these services as puzzle pieces that when put together solve real world problems. In an ideal world, the different open source projects we need would all be available as services on public, private, and dedicated clouds. And it would be trivial to integrate the services by a point and a click. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">For example, we should be able to take a message bus like Kafka, and use a simple configuration command with no coding to have it archive the messages in an object store service. Once in that service, another point-and-click could pull the data into an analytics engine (like Apache Spark).<span style="mso-spacerun: yes;">  </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><b style="mso-bidi-font-weight: normal;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">The best (and worst) of open source times</span></b></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">It is the best of times for developers because of the wide array of open source puzzle pieces out there, which can be used to build a solution. It is also the worst of times since there is still work needed to make sure these pieces snap together easily, and too often this burdens the developer.</span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpUg7xgbtyI_dcZVnqN5oxAdukfJg89UITYrCfadVsXoCewnANca8mA22zGwNkRVcRsQO7Ti6LqahLBInEPLT0sOvsqdn0JSOpnmLoOoijKRWbpTBhMllopBp8NINdi-jhtcu311zw-1w/s1600/Putting+Pieces+Together+Nov+2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="97" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpUg7xgbtyI_dcZVnqN5oxAdukfJg89UITYrCfadVsXoCewnANca8mA22zGwNkRVcRsQO7Ti6LqahLBInEPLT0sOvsqdn0JSOpnmLoOoijKRWbpTBhMllopBp8NINdi-jhtcu311zw-1w/s400/Putting+Pieces+Together+Nov+2.jpg" width="400" /></a></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">But my colleagues at IBM’s Research lab in Haifa and I are working on integrating several of these projects – so we really can snap together those puzzle pieces!<span style="mso-spacerun: yes;">   </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">For example, my colleague <a href="http://researcher.watson.ibm.com/researcher/view.php?person=il-GILV">Gil Vernik</a>, a cloud storage, security and analytics expert in the lab, is </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="http://www.spark.tc/revving-up-performance-for-the-tachyon-file-system/"><span style="mso-bidi-font-family: Helv;">enabling Tachyon to use Swift</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"> as its persistent underlying storage system. <a href="http://ibmresearchnews.blogspot.co.il/2015/08/tachyon-for-ultra-fast-big-data.html">Tachyon</a>is a very active, but relatively new, open source project that provides an in-memory file system with automated tiering. While Tachyon is general purpose, one of its most known use cases is improving Spark’s performance over conventional stores. While Tachyon is in-memory, it needs some place to persist data when it is no longer being used, or if it runs out of memory in higher performance tiers. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">When Gil first started this work, the main choices for persisting data were <a href="http://www-01.ibm.com/software/data/infosphere/hadoop/hdfs/">HDFS</a> or Amazon S3 (other options exist today). Both are good solutions but each has its limitations: HDFS was not designed as a long term, multi-tenant store; Amazon’s S3 is part of a public cloud. By adding support for OpenStack Swift, there is now a multi-tenant, object store under Tachyon that can provide long term, cost-effective persistent store. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">And <a href="https://www.linkedin.com/profile/view?id=ACgAAAa-VpYBQF56yrrdDwbVLrKRiBeLm84evZs&authType=name&authToken=Of6l">Guy Hadash</a>, another colleague on our cloud team, is developing a solution to aggregate and store messages from the Kafka message bus in OpenStack Swift. Visual bookmark social media site Pinterest had a small project called </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="https://github.com/pinterest/secor"><span style="mso-bidi-font-family: Helv;">Secor</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"> which knows how to subscribe to Kafka, aggregate messages, and then put them in Amazon's S3. These messages are stored sequentially in the object. The object can later be retrieved for batch processing of the messages. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">Guy extended Secor so it could store messages via the Swift API. This gives Secor users the choice of using Kafka-Secor-Swift in all three deployment models: local, dedicated, and public. He then stored the data in </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="https://parquet.apache.org/"><span style="mso-bidi-font-family: Helv;">Parquet</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"> format, not just as a list of messages. Parquet is designed for objects or files that contain tabular data (like CSV). This data is stored in a columnar format, enabling efficient compression and the retrieval of only selected columns from the store. Lastly, Guy annotated the created objects with metadata, such as the minimum or maximum values of various columns in the table. We have already implemented prototypes of all three steps. A patch for the first step is already part of the community Secor code, and we are starting to work with the community on the second. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQ0NyvP4zJdJJqR8L1QjwDbOXVwTDIehq4s68WM5-PaWB0vTdZCYAYiEWXSHXKief1f85Yv3TcOiil0gDvim1QltqI_jCygCqyNFo4O1dqDpEUafSOIrIEbG0btpNaq7h7uIBOdQrvnSU/s1600/twitter_paula.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQ0NyvP4zJdJJqR8L1QjwDbOXVwTDIehq4s68WM5-PaWB0vTdZCYAYiEWXSHXKief1f85Yv3TcOiil0gDvim1QltqI_jCygCqyNFo4O1dqDpEUafSOIrIEbG0btpNaq7h7uIBOdQrvnSU/s320/twitter_paula.jpg" width="256" /></a></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">Then there’s what we are doing with our partners in the context of the EU </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="http://iot-cosmos.eu/"><span style="mso-bidi-font-family: Helv;">COSMOS</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"> project. <a href="https://spark-summit.org/eu-2015/speakers/paula-ta-shma/">Paula Ta-Shma</a>, a cloud security and analytics expert on our team, presented </span><span style="font-family: "verdana" , "sans-serif"; font-size: 10.0pt;"><a href="https://spark-summit.org/eu-2015/events/how-spark-enables-the-internet-of-things-efficient-integration-of-multiple-spark-components-for-smart-city-use-cases/"><span style="mso-bidi-font-family: Helv;">this work</span></a></span><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"> at the recent Spark Summit in Amsterdam. The goal of this work is to help improve the timeliness of Madrid’s city buses by automating their reactions to changes in traffic. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">Data is available from thousands of static sensors. Using NodeRed (an open source project for defining data flows), the data is retrieved from the sensors and put on Kafka. As described above, Secor aggregates, formats, and annotates this data, before storing it in Swift. Spark is then used to analyze this data. Using standard Spark machine learning, the solution defines the expected values for speed and traffic density on different days and times. A complex event processing engine, which also subscribes to the messages from the sensors, takes the threshold values and can issue warnings to trigger corrective action, such as rerouting a bus or changing the traffic light pattern, when needed. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="color: black; font-family: "verdana" , "sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;">These examples clearly show the tremendous value in putting puzzle pieces together from different open source projects to build something even better. We are continuing to work on integrating services and look forward to describing some of our other efforts in future blogs.<span style="mso-spacerun: yes;">  </span></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal"><br /></div>

Reflections en-route from the OpenStack Summit

COSMOS Gil Vernik Guy Hadash IBM Research - Haifa IoT Kafka Michael Factor open stack Parquet Paula Ta-shma Secor Spark storage Swift Reflections en-route from the OpenStack Summit

Michael Factor, IBM Research Editor’s note: This post was authored by Michael Factor, IBM Fellow As I sat on the plane during the return tr...