Welcome!

XML Authors: Esmeralda Swartz, Lori MacVittie, Carmen Gonzalez, Pat Romanski, Peter Silva

Related Topics: SOA & WOA, XML

SOA & WOA: Article

$3 Trillion Problem: Three Best Practices for Today's Dirty Data Pandemic

Maybe your software is healthy, but is your data terminally ill?

In survey after survey, about half of IT executives consistently agree that data quality and data consistency is one of the biggest roadblocks to them getting full value from their data.

This has been consistently true all since the Chinese invented the abacus. I suspect it will be true long after quantum computing has solved every other problem that humanity faces.

 

Incorrect, inconsistent, fraudulent and redundant data cost the U.S. economy over $3 Trillion a year - an astounding figure that is over twice the amount of the 2011 Federal Deficit.

Similarly, many experts estimate that HALF the money spent on developers goes towards "software repair". So we're living in a world of sick software and dirty data. And the cost of all this is staggering.

 

I've long been a proponent of healthy software - but healthy software can only function properly in the presence of healthy data. Does quality software even matter if the underlying data are defective? Agreed - that's pushing the point to the extreme.

The rapid, iterative, continuous testing model has measurably improved the quality of software development. Evangelists such as Kent Beck have had a huge impact on this. I recently posted a freely downloadable white paper on this topic. But where are the evangelists for data quality? Where is an open source "JUnit for Data" and if it's out there, why isn't everyone using it?

The Cost of Bad Data
Anyone care to make a guess at how much money is wasted every year due to dirty or duplicate / redundant data? I'll start by presenting one common user story - one you probably have also recently experienced. And then expand on it.

Recently, I went to my mailbox and waiting for me was yet another invitation from a major bank to join their credit card program.

This shouldn't come as a surprise, as people everywhere are deluged by credit card offers. Except that I already have the particular card in question. Not only that, but because the particular bank in question has managed to acquire a number of other banks and credit card lines of business, between my personal and my corporation, I believe I now have five Visa cards from this particular bank.

I also occasionally get mail from them offering me cash bonuses to open up a checking account at their bank. Probably wasted postage, as I already have two checking accounts there. I suppose I could open up a third, just to get the $100.

Every month, I get a significant number of expensive looking direct mail offers from this bank, often with slightly different variations on my name, which I promptly throw away. Aside from the impact on the environment and the wasted direct mail expense, it's a bit irritating to me. I hate junk mail, and I feel compelled to shred things like credit card offers. So they've burdened me (an existing customer) with yet another "thing to do". So they've spent money, hurt the environment, irritated an existing customer, and now I get to make fun of them online. Bad investment on their part.

QAS (an Experian company) estimates that the average company wastes $180,000 per year simply on direct mail that does not reach the intended recipient because of inaccurate data. But this is just one miniscule slice of the data quality issue. In fact it's only one small part of the "direct mail" data quality issue. A lot more money is wasted in "inappropriate offers" and "duplicate offers" such as the ones my bank sends. I also get offers from several companies that are convinced that I'm married to the previous owner of my house. Those offers reach me, yet are immediately shredded. No sense opening them. So the "big picture" just for direct mail is much larger than what QAS shows.

None of this accounts for the "irritation" factor - what is the cost of annoying existing customers (or potential customers) with badly targeted offers?

Yet direct mail and all other forms of advertising together add up to a tiny slice of the bad-data pie.

Fraud Is a Bad Data Problem
Some time back, the US Attorney General's office stated that they believed that 14 percent of health care dollars are wasted in fraud or inaccurate billing.

Why do I lump fraud in with "bad data"? Bad data comes in two forms - accidentally created bad data and intentionally created bad data (for example, fraudulent billing). Either way, it's bad data. It doesn't matter how it got there, it's defective. And a lot of it could be detected and remediated "at the point of entry".

Healthcare accounts for over 16% of the U.S. GDP (Canada is 10%, Australia is 9% as a comparison). The U.S. GDP is currently approximately $14 Trillion - therefore healthcare spending in the U.S. amounts to $2.25 trillion. And the cost of bad data in Healthcare- $314 Billion.

That's just for fraud or inaccurate billing. What about other areas in healthcare (e.g. lost data, "bad patient outcomes", duplicate patient testing, manual rework, etc.)?  Even if we round down, we're still taking about $500 Billion for one industry alone.  If I extrapolate that out to the entire U.S. economy, we're talking about a $3.1 Trillion problem.  No matter how far off my estimate is (on the high side or the low side), it's a problem of astonishing proportions.

Cost of Bad Data to Business and IT
A classic but very worthwhile book from information governance expert Larry English posits that the business cost of nonquality data may be as high as 10-25% of an organization's revenue, and that as much as 50% of the typical IT budget may be spent in "information scrap and rework".  If that is the case, then my $3.1 estimate is not out of line.

In the introduction to his book, English states "With this proliferation of information, the challenge of managing data and providing quality information has never been more important or complex."

That was in 1999. With so much more data today, and a surprising lack of attention to the data quality issue, I can only imagine the total economic impact of things today. I do not doubt that the cost of bad data has risen.

Dealing with bad data at the I.T. level is expensive. But if I.T. doesn't deal with the bad data problem, then the cost gets pushed downstream to the "business", where the business costs are geometrically higher. The model is not that different from that of "healthy software", where it costs $1 to uncover a defect during developer/unit testing, but $100 to fix that defect if the software is released to the end-users.

"Low Hanging Fruit" - Best Practices for Bad Data Avoidance
I am not saying that there are any easy fixes to the bad data problem. Even something as relatively simple as cleaning, standardizing and de-duping a mailing list with 10,000,000 entries is essentially impossible to get completely right no matter how much effort is put into it. Yet there are some relatively easy things that can be done to substantially improve the quality of our data.  As with so many other problems in life, the some version of the 80/20 rule applies to this as well.

Best Practice #1: When integrating data, fix the quality problem during integration
As data are added or integrated, data should be tested. Profiling is a simple, fast, relatively easily implemented and highly effective way for eliminating significant volumes of defective data.

When developers write a new application for the input of some new data, it's normal for input fields to be "validated" - a simple "hard coded" form of profiling. Month number needs to be between 1-12. 13 is never correct.  Not rocket science. And it's universally done.

Yet people have far fewer reservations about integrating data from here, there and everywhere - often not checking for even the most egregious data errors, and thereby polluting the organizational drinking water (i.e. all the data and applications downstream).

I strongly suspect that's why I get so many offers from my current mega-bank. Since the banking implosion, this particular bank has purchased every other bank around. And their credit card businesses. And their marketing databases. And (apparently) smashed them together. So I get offers for Hollis Tibbetts, Hollis W. Tibbetts, Hollis Winslow Tibbetts, Hollis Tibbets, Hollis Tibbitts and so on.

Integration of data isn't necessarily just a "big bang" event - like when one company acquires another and smashes all the data together, or when two divisional customer applications get merged. It can be more insidious and more when you have "trickle" integration - the slow feed of new data from one system into another (either within the organization or from customers/suppliers/partners).  This is the class of integration that is causing a lot of the problems previously discussed with healthcare fraud.

Either way, FIX IT before integrating it. Once the poison enters the corporate drinking water, it's a lot harder to get out (not just technically, but especially politically/organizationally).

Best Practice #2: When migrating data, fix the data problem as PART of the migration project
Spending $1 billion to upgrade your Seibel system like the US Government is doing? Sounds like a great time to fix your data quality problem.

If you're doing something like migrating your customer data from Seibel to Netsuite or Salesforce.com, data quality should be a major element in your project plan (and budget). Fixing the problems during the migration are easier than fixing them later:

  1. You probably already possess a lot of knowledge about the existing legacy systems, the types of problems in the data. But your new system is relatively unknown to you. So it's likely to be easier to fix data issues from a technical perspective BEFORE they get loaded into the new system.
  2. As part of the data migration process, you can export the data to a staging platform (On Prem or Cloud), leverage any number of data quality tools/engines, and then import the data into the the application platform.  This approach may partially pay for itself in an easier/smoother upgrade to the new application, but that's a rounding error in the overall scheme of things.
  3. Organizationally and politically, companies are much more likely to spend money to clean data if it's part of a project like "upgrade the CRM system". I'd hate to be the CIO that spends a mountain of money to upgrade the CRM system and then goes back to the board asking for another mountain of money to fix all the bad data that just got loaded into the CRM system. That's how CIO's become ex-CIOs.

Best Practice #3: Data profiling and data de-duplication engines
Data profiling engines are a great technology for quickly improving the quality of data as it is integrated from one system into another. At the highest level, they are an engine that scans data, and applies certain easily definable rules to data elements, such as formats, ranges, allowable values and can evaluate relationships between different fields.

Furthermore, these engines can also be used to analyze existing data stores very rapidly and generate "exceptions files" for manual, or semi-automated remediation (if anyone can find a totally automated data remediation system, I'd love to know about it). So they can be used in "continuous testing" or "batch testing" mode.  In batch mode, they're ideal for application migrations or big-bang integrations, as they're easiest to use them if you have your data in something like a staging database.  But they can also be used to test data as it is "trickle integrated" into production systems.

De-duping engines generally fit into the same category. I haven't seen them be as effective as data profiling engines, yet I believe they're essential. The technology for de-duping is considerably more sophisticated - with a large number of different algorithms and tunable thresholds and such. It's a harder class of technology to implement. More manual effort is involved. And, unlike profiling (where there is NEVER a month "13"), de-duping can "get it wrong", so the technology needs to be applied more selectively.

Conclusion
I've never understood why these engines haven't been more popular. There is no "JUnit for data" as far as I know. But commercial solutions are available - they're not terribly expensive and rapidly pay for themselves.

On the other hand, I've never understood why organizations are so tolerant of bad, dirty data. They waste millions and millions directly because of it (and untold quantities of money in "wasted opportunities"), but are reluctant to spend $15,000 on a data quality engine to help fix a significant portion of the problem.

More Stories By Hollis Tibbetts

Hollis Tibbetts, or @SoftwareHollis as his 50,000+ followers know him on Twitter, is listed on various “top 100 expert lists” for a variety of topics – ranging from Cloud to Technology Marketing, Hollis is by day Evangelist & Software Technology Director at Dell Software. By night and weekends he is a commentator, speaker and all-round communicator about Software, Data and Cloud in their myriad aspects. You can also reach Hollis on LinkedIn – linkedin.com/in/SoftwareHollis. His latest online venture is OnlineBackupNews - a free reference site to help organizations protect their data, applications and systems from threats. Every year IT Downtime Costs $26.5 Billion In Lost Revenue. Even with such high costs, 56% of enterprises in North America and 30% in Europe don’t have a good disaster recovery plan. Online Backup News aims to make sure you all have the news and tips needed to keep your IT Costs down and your information safe by providing best practices, technology insights, strategies, real-world examples and various tips and techniques from a variety of industry experts.

Hollis is a regularly featured blogger at ebizQ, a venue focused on enterprise technologies, with over 100,000 subscribers. He is also an author on Social Media Today "The World's Best Thinkers on Social Media", and maintains a blog focused on protecting data: Online Backup News.
He tweets actively as @SoftwareHollis

Additional information is available at HollisTibbetts.com

All opinions expressed in the author's articles are his own personal opinions vs. those of his employer.

@ThingsExpo Stories
The Internet of Things (IoT) is going to require a new way of thinking and of developing software for speed, security and innovation. This requires IT leaders to balance business as usual while anticipating for the next market and technology trends. Cloud provides the right IT asset portfolio to help today’s IT leaders manage the old and prepare for the new. Today the cloud conversation is evolving from private and public to hybrid. This session will provide use cases and insights to reinforce the value of the network in helping organizations to maximize their company’s cloud experience.
The Internet of Things (IoT) is making everything it touches smarter – smart devices, smart cars and smart cities. And lucky us, we’re just beginning to reap the benefits as we work toward a networked society. However, this technology-driven innovation is impacting more than just individuals. The IoT has an environmental impact as well, which brings us to the theme of this month’s #IoTuesday Twitter chat. The ability to remove inefficiencies through connected objects is driving change throughout every sector, including waste management. BigBelly Solar, located just outside of Boston, is trans...

SUNNYVALE, Calif., Oct. 20, 2014 /PRNewswire/ -- Spansion Inc. (NYSE: CODE), a global leader in embedded systems, today added 96 new products to the Spansion® FM4 Family of flexible microcontrollers (MCUs). Based on the ARM® Cortex®-M4F core, the new MCUs boast a 200 MHz operating frequency and support a diverse set of on-chip peripherals for enhanced human machine interfaces (HMIs) and machine-to-machine (M2M) communications. The rich set of periphera...

SYS-CON Events announced today that Matrix.org has been named “Silver Sponsor” of Internet of @ThingsExpo, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Matrix is an ambitious new open standard for open, distributed, real-time communication over IP. It defines a new approach for interoperable Instant Messaging and VoIP based on pragmatic HTTP APIs and WebRTC, and provides open source reference implementations to showcase and bootstrap the new standard. Our focus is on simplicity, security, and supporting the fullest feature set.
Predicted by Gartner to add $1.9 trillion to the global economy by 2020, the Internet of Everything (IoE) is based on the idea that devices, systems and services will connect in simple, transparent ways, enabling seamless interactions among devices across brands and sectors. As this vision unfolds, it is clear that no single company can accomplish the level of interoperability required to support the horizontal aspects of the IoE. The AllSeen Alliance, announced in December 2013, was formed with the goal to advance IoE adoption and innovation in the connected home, healthcare, education, aut...
SYS-CON Events announced today that Red Hat, the world's leading provider of open source solutions, will exhibit at Internet of @ThingsExpo, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Red Hat is the world's leading provider of open source software solutions, using a community-powered approach to reliable and high-performing cloud, Linux, middleware, storage and virtualization technologies. Red Hat also offers award-winning support, training, and consulting services. As the connective hub in a global network of enterprises, partners, a...
The only place to be June 9-11 is Cloud Expo & @ThingsExpo 2015 East at the Javits Center in New York City. Join us there as delegates from all over the world come to listen to and engage with speakers & sponsors from the leading Cloud Computing, IoT & Big Data companies. Cloud Expo & @ThingsExpo are the leading events covering the booming market of Cloud Computing, IoT & Big Data for the enterprise. Speakers from all over the world will be hand-picked for their ability to explore the economic strategies that utility/cloud computing provides. Whether public, private, or in a hybrid form, clo...
Software AG helps organizations transform into Digital Enterprises, so they can differentiate from competitors and better engage customers, partners and employees. Using the Software AG Suite, companies can close the gap between business and IT to create digital systems of differentiation that drive front-line agility. We offer four on-ramps to the Digital Enterprise: alignment through collaborative process analysis; transformation through portfolio management; agility through process automation and integration; and visibility through intelligent business operations and big data.
The Transparent Cloud-computing Consortium (abbreviation: T-Cloud Consortium) will conduct research activities into changes in the computing model as a result of collaboration between "device" and "cloud" and the creation of new value and markets through organic data processing High speed and high quality networks, and dramatic improvements in computer processing capabilities, have greatly changed the nature of applications and made the storing and processing of data on the network commonplace.
Be Among the First 100 to Attend & Receive a Smart Beacon. The Physical Web is an open web project within the Chrome team at Google. Scott Jenson leads a team that is working to leverage the scalability and openness of the web to talk to smart devices. The Physical Web uses bluetooth low energy beacons to broadcast an URL wirelessly using an open protocol. Nearby devices can find all URLs in the room, rank them and let the user pick one from a list. Each device is, in effect, a gateway to a web page. This unlocks entirely new use cases so devices can offer tiny bits of information or simple i...
The Internet of Things (IoT) is going to require a new way of thinking and of developing software for speed, security and innovation. This requires IT leaders to balance business as usual while anticipating for the next market and technology trends. Cloud provides the right IT asset portfolio to help today’s IT leaders manage the old and prepare for the new. Today the cloud conversation is evolving from private and public to hybrid. This session will provide use cases and insights to reinforce the value of the network in helping organizations to maximize their company’s cloud experience.
Things are being built upon cloud foundations to transform organizations. This CEO Power Panel at 15th Cloud Expo, moderated by Roger Strukhoff, Cloud Expo and @ThingsExpo conference chair, will address the big issues involving these technologies and, more important, the results they will achieve. How important are public, private, and hybrid cloud to the enterprise? How does one define Big Data? And how is the IoT tying all this together?
TechCrunch reported that "Berlin-based relayr, maker of the WunderBar, an Internet of Things (IoT) hardware dev kit which resembles a chunky chocolate bar, has closed a $2.3 million seed round, from unnamed U.S. and Switzerland-based investors. The startup had previously raised a €250,000 friend and family round, and had been on track to close a €500,000 seed earlier this year — but received a higher funding offer from a different set of investors, which is the $2.3M round it’s reporting."
The Industrial Internet revolution is now underway, enabled by connected machines and billions of devices that communicate and collaborate. The massive amounts of Big Data requiring real-time analysis is flooding legacy IT systems and giving way to cloud environments that can handle the unpredictable workloads. Yet many barriers remain until we can fully realize the opportunities and benefits from the convergence of machines and devices with Big Data and the cloud, including interoperability, data security and privacy.
All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades. Over the summer Gartner released its much anticipated annual Hype Cycle report and the big news is that Internet of Things has now replaced Big Data as the most hyped technology. Indeed, we're hearing more and more about this fascinating new technological paradigm. Every other IT news item seems to be about IoT and its implications on the future of digital busines...
Cultural, regulatory, environmental, political and economic (CREPE) conditions over the past decade are creating cross-industry solution spaces that require processes and technologies from both the Internet of Things (IoT), and Data Management and Analytics (DMA). These solution spaces are evolving into Sensor Analytics Ecosystems (SAE) that represent significant new opportunities for organizations of all types. Public Utilities throughout the world, providing electricity, natural gas and water, are pursuing SmartGrid initiatives that represent one of the more mature examples of SAE. We have s...
The Internet of Things needs an entirely new security model, or does it? Can we save some old and tested controls for the latest emerging and different technology environments? In his session at Internet of @ThingsExpo, Davi Ottenheimer, EMC Senior Director of Trust, will review hands-on lessons with IoT devices and reveal privacy options and a new risk balance you might not expect.
IoT is still a vague buzzword for many people. In his session at Internet of @ThingsExpo, Mike Kavis, Vice President & Principal Cloud Architect at Cloud Technology Partners, will discuss the business value of IoT that goes far beyond the general public's perception that IoT is all about wearables and home consumer services. The presentation will also discuss how IoT is perceived by investors and how venture capitalist access this space. Other topics to discuss are barriers to success, what is new, what is old, and what the future may hold.
Swiss innovators dizmo Inc. launches its ground-breaking software, which turns any digital surface into an immersive platform. The dizmo platform seamlessly connects digital and physical objects in the home and at the workplace. Dizmo breaks down traditional boundaries between device, operating systems, apps and software, transforming the way users work, play and live. It supports orchestration and collaboration in an unparalleled way enabling any data to instantaneously be accessed on any surface, anywhere and made interactive. Dizmo brings fantasies as seen in Sci-fi movies such as Iro...
There’s Big Data, then there’s really Big Data from the Internet of Things. IoT is evolving to include many data possibilities like new types of event, log and network data. The volumes are enormous, generating tens of billions of logs per day, which raise data challenges. Early IoT deployments are relying heavily on both the cloud and managed service providers to navigate these challenges. In her session at 6th Big Data Expo®, Hannah Smalltree, Director at Treasure Data, to discuss how IoT, Big Data and deployments are processing massive data volumes from wearables, utilities and other mach...