N1005 IBM Research: data fabrication platform

<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRhUKMbLhz984VDLAXmwkFzHdVKLLKwfN_hNbUU8r1-aHYIDJ_fQwPb7nyOzCqdTcFur7fr7DLWzo3NOWRpXDT59j72KYcSy5osV_6fvAHmK0pbgZQLyMdcU0iwlx8XxaaemFQWSv6xd4/s1600/tamer.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRhUKMbLhz984VDLAXmwkFzHdVKLLKwfN_hNbUU8r1-aHYIDJ_fQwPb7nyOzCqdTcFur7fr7DLWzo3NOWRpXDT59j72KYcSy5osV_6fvAHmK0pbgZQLyMdcU0iwlx8XxaaemFQWSv6xd4/s1600/tamer.png" width="149" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Tamer Salman</td></tr></tbody></table><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><i><span style="color: #004023; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Default Sans\000A Serif";">Editor’s note: This article is authored by Tamer Salman, senior researcher of <a href="https://www.research.ibm.com/haifa/dept/vst/simulation.shtml" target="_blank">Security & Quality Technologies</a> at IBM Research – Haifa.</span></i></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Tms Rmn";">How does an insurance company, financial institution, healthcare organization, or government body get personal and confidential data to develop and test the performance of new software? </span><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Default Sans\000A Serif";">The challenges associated with managing personal and confidential data are huge, especially with the increasingly stringent data privacy regulations. </span><span style="font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Tms Rmn";">Some data is private and confidential, other data may have been redesigned and transformed, and some may not exist at all. Typically, project leaders or database administrators will set up an additional environment for development and testing. The big challenge is how to populate it with data.</span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; tab-stops: -36.0pt 0cm 36.0pt 72.0pt 108.0pt 144.0pt 180.0pt 216.0pt; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Default Sans\000A Serif";">With expertise in constraint satisfaction and automatic test generation, IBM researchers in Haifa developed the Data Fabrication Platform (DFP). It’s a solution that efficiently creates high quality test data, while eliminating any potential data security or privacy concerns. The platform is already helping a large insurance company revamp their current processes around test data. <br style="mso-special-character: line-break;" /></span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; tab-stops: -36.0pt 0cm 36.0pt 72.0pt 108.0pt 144.0pt 180.0pt 216.0pt; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><b><span style="font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Tms Rmn";">Generating masses of personal (but fabricated) data</span></b></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><span style="font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Tms Rmn";">For most situations, generating the mass of data needed involves in-house scripting, simple data generation techniques, manual insertions and updates, and a lot of masking and data scrubbing. Even after the test data is ready, the requirements can change during development, rendering the current data useless and necessitating a repeat of some processes. The result is a tedious, costly, and time consuming process that doesn’t necessarily deliver results. </span></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;"><br /></div><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; tab-stops: -36.0pt 0cm 36.0pt 72.0pt 108.0pt 144.0pt 180.0pt 216.0pt; text-autospace: none;"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Default Sans\000A Serif";">In order to accommodate distributed and outsourced development and testing, our client needed test data that would not be susceptible to leaks or breaches in security and privacy. They also need the ability to transform and evolve the data as the business needs changed or were updated. DFP does this by allowing for rule sharing and migration. It also minimizes test-data generation efforts by eliminating security and privacy concerns, and offering support for early development and regression tests. </span></div><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Default Sans\000A Serif";"><br /></span><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: Helv;"><iframe allowfullscreen="" frameborder="0" height="360" src="//www.youtube.com/embed/U68OKwUFHF8?feature=player_detailpage" width="640"></iframe></span><br /><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; tab-stops: -36.0pt 0cm 36.0pt 72.0pt 108.0pt 144.0pt 180.0pt 216.0pt; text-autospace: none;"><br /></div><div class="MsoNormal"><b><span style="font-family: "Verdana","sans-serif"; font-size: 10.0pt; line-height: 115%;">Data rules</span></b></div><br /><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Default Sans\000A Serif";">The  logic of what’s needed in these secure, confidential instances can be  described using various rules that define the relationships between  different columns in your databases, resources for populating new data  columns, or transformations from archived data. DFP lets companies put these rules into the  system, and get the data needed as output. The platform consumes the  provided rules and generates the requested data, which can be  automatically inserted into the target databases, or any of a variety of  formats, such as XML, CSV, and DML files.</span><br /><br /><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Default Sans\000A Serif";">At the heart of the DFP lies a powerful </span><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Default Sans\000A Serif";"><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Default Sans\000A Serif";"><a href="http://en.wikipedia.org/wiki/Constraint_satisfaction_problem" target="_blank">Constraint Satisfaction Problem</a></span> (CSP) solver, also developed in Haifa. A CSP typically involves numerous possibilities that can’t be solved automatically by straightforward algorithms within an acceptable amount of time. A form of artificial intelligence, the CSP solver from IBM solves these unique complex problems using it's ability to arrive at many more buildable solutions than traditional optimization approaches. The CSP solver provides accelerated solutions and helps eliminate errors by generating only data that is valid for the specific requirements.</span><br /><br /><span style="color: black; font-family: "Verdana","sans-serif"; font-size: 10.0pt; mso-bidi-font-family: "Default Sans\000A Serif";">In summary , the IBM Data Fabrication Platform is an easy to use technology  that allows for rule sharing and migration, minimizes test-data  generation efforts, eliminates security and privacy concerns, and makes it easier for companies to outsource development and testing. </span>

Tamer Salman Editor’s note: This article is authored by Tamer Salman, senior researcher of Security & Quality Technologies at IBM Resea...