N1005 IBM Research: Zvi Kons

<div class="MsoNormal" style="line-height: 150%;"><i><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;"><i><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">IBM develops analytics and classification technology to provide data for a new kind of multimedia-based search engine</span></i></span></i></div><br /><div class="MsoNormal" style="line-height: 150%;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvZuNwW6oeIz0HZ_9vrmjy3ldcCRtdGO762gZeskZg9DeqrfP9Jvn5zKiPRKR4i-lepXdLNSqOCT6Wmg7IM4ZL7vcWGKvbvCW925x-C7ZKAIVLVRmt3dT6laCV54n4cSDlRBOR1CnVMQ4Z/s1600/zvi_kons.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvZuNwW6oeIz0HZ_9vrmjy3ldcCRtdGO762gZeskZg9DeqrfP9Jvn5zKiPRKR4i-lepXdLNSqOCT6Wmg7IM4ZL7vcWGKvbvCW925x-C7ZKAIVLVRmt3dT6laCV54n4cSDlRBOR1CnVMQ4Z/s1600/zvi_kons.jpg" /></a></div><i><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">Editor’s note: This posting was authored by Zvi Kons, researcher in the Speech Technologies group at IBM Research - Haifa</span></i></div><div class="MsoNormal" style="line-height: 150%;"><br /></div><div class="MsoNormal" style="line-height: 150%;"><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">When you walk down a busy street, do you ever notice the sounds that surround you? People, traffic, music; city sounds are often like the foreign language the couple next to you at the café is speaking—background chatter. That city buzz, though, together with related visual images, has the potential to generate a continuous stream of information that can indicate real-time dynamics of the city.</span><br /><br /><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">To gather, process, analyze and ultimately separate useful sound from white noise, my team at IBM Research-Haifa is working on new technology for searchable audio analysis as part of the EU-funded project called </span><a href="http://www.smartfp7.eu/"><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">SMART</span></a><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;"> (<b>S</b>earch Engine for <b>M</b>ultimedi<b>a</b> Envi<b>r</b>onment Generated Conten<b>t</b>).</span><br /><a name="more"></a></div><div class="MsoNormal" style="line-height: 150%;"><br /><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">We’re developing algorithms and an engine to analyze those city sounds, extracting information that can be cross referenced with video images to generate real-time content. Our research on audio classification is an integral aspect of a new kind of internet search engine that could provide locally oriented, readily available and informative content with practical applications.</span><b><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;"> </span></b><br /><br /><b><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">Capturing the sights and sounds of city streets to gain insight</span></b><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;"> </span><br /><br /><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">Our team collected data from two locations in Santander, Spain. Because the municipality is a partner in the SMART project, they offered to support the technical aspects of the infrastructure needed and are helping test the technology. Cameras and microphones set up in the town square and market area provided continuous audio and visual data of normal daily activity for one month, collecting more than 1,000 hours of data. We analyzed the sounds to note various types of activities, and to identify patterns and anomalies, like peak hours for busy crowds in the market square, traffic, and special events. </span><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;"> </span><br /><br /><iframe allowfullscreen="" frameborder="0" height="315" src="//www.youtube.com/embed/m829xQcChuw" width="420"></iframe> <br /><i>Santander city square</i><br /><br /><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSztpxCinEdoH2kkPwMQtKPatA0KzwQNKojq_1gfZKtooxqesa4U38G3klJgu6_PsMgcb2K5NJBDZqBa66Ji4uaPfuTHofMYwBfUZom3bRbHXj1WoAh64cL9lKvgLpi2a_dNkBYsm1bVY/s1600/SMARTsounds.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSztpxCinEdoH2kkPwMQtKPatA0KzwQNKojq_1gfZKtooxqesa4U38G3klJgu6_PsMgcb2K5NJBDZqBa66Ji4uaPfuTHofMYwBfUZom3bRbHXj1WoAh64cL9lKvgLpi2a_dNkBYsm1bVY/s1600/SMARTsounds.jpg" width="400" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Visual representation of weekly audio from the city square</td></tr></tbody></table><br /><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">The audio from the video above and others produced this diagram that shows a visual representation of the weekly crowd activity level; blue for low activity, red for high activity.</span><br /><br /><br /><br /><br /><br /><br /><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">Another sample detected a day with unusual crowd noise, music, and applause. By cross-referencing with video footage from nearby street cameras, it turned out to be from a protest rally on a nearby street, which could be important information for analyzing any immediate security risk, or the need to send a news team to report on a developing story.</span></div><div class="MsoNormal" style="line-height: 150%;"><br /><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">Listen to the mid-day rally as it passed on the top right corner of the frame:</span></div><div class="MsoNormal" style="line-height: 150%;"><br /><b><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;"></span></b><iframe allowfullscreen="" frameborder="0" height="315" src="//www.youtube.com/embed/6hIjKuw6N5k" width="420"></iframe><br /><br /><b><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">Applying audio analytics</span></b></div><div class="MsoNormal" style="line-height: 150%;"></div><br /><div align="right"><table align="right" border="1" cellpadding="0" cellspacing="0" class="MsoTableGrid" style="border-collapse: collapse; border: none; margin-left: 6.75pt; margin-right: 6.75pt; mso-border-alt: solid white .5pt; mso-border-insideh-themecolor: background1; mso-border-insideh: .5pt solid white; mso-border-insidev-themecolor: background1; mso-border-insidev: .5pt solid white; mso-border-themecolor: background1; mso-padding-alt: 0in 5.4pt 0in 5.4pt; mso-table-anchor-horizontal: column; mso-table-anchor-vertical: paragraph; mso-table-left: right; mso-table-lspace: 9.0pt; mso-table-overlap: never; mso-table-rspace: 9.0pt; mso-table-top: .05pt; mso-yfti-tbllook: 1184;"> <tbody><tr style="mso-yfti-firstrow: yes; mso-yfti-irow: 0; mso-yfti-lastrow: yes;">  <td style="border: solid white 1.0pt; mso-border-alt: solid white .5pt; mso-border-themecolor: background1; mso-border-themecolor: background1; padding: 0in 5.4pt 0in 5.4pt; width: 2.25in;" valign="top" width="216"><div class="MsoNormal" style="line-height: 150%; margin-bottom: .0001pt; margin-bottom: 0in; mso-element-anchor-horizontal: column; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: right; mso-element-top: .05pt; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;"><b><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">The   sounds of privacy </span></b></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0in; mso-element-anchor-horizontal: column; mso-element-anchor-vertical: paragraph; mso-element-frame-hspace: 9.0pt; mso-element-left: right; mso-element-top: .05pt; mso-element-wrap: around; mso-element: frame; mso-height-rule: exactly;"><span style="color: #444444;"><i><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt;">To address   potential privacy and legal issues, the SMART team used wide angles and low   resolution for the video cameras. The microphones were placed at a distance   to pick up crowd noise rather than intelligible speech or individual   conversations.</span></i></span></div></td> </tr></tbody></table></div><div class="MsoNormal" style="line-height: 150%;"><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">The idea behind SMART’s new multi-media-based search engine is the incorporation of information gleaned from the environment. We can use data from city sounds and video images, as well as social media like tweets, to identify events and situations in real-time and make that information available online. The sounds of the city can help identify a drunken brawl, a spontaneous demonstration, a musical event, or an accident during rush hour. This kind of readily available information could be valuable for security systems, municipal and media use, and helpful knowledge for city residents.</span><br /><br /><span style="font-family: "Verdana",sans-serif; font-size: 10.0pt; line-height: 150%;">Our research highlights the enormous potential of easily accessible information in our physical surroundings. The technology to use that information has exciting and practical applications for smart cities, with innovative ways to interpret sounds and images.</span></div><div class="MsoNormal" style="line-height: 150%;"><br /></div>

audio EU IBM Research - Haifa multimedia Santander search SMART speech analytics Zvi Kons What are the sounds of the city saying?

IBM develops analytics and classification technology to provide data for a new kind of multimedia-based search engine Editor’s note: This po...