Welcome!

XML Authors: Carmen Gonzalez, Liz McMillan, David Dossot, Yeshim Deniz, Elizabeth White

Related Topics: SOA & WOA, XML

SOA & WOA: Article

$3 Trillion Problem: Three Best Practices for Today's Dirty Data Pandemic

Maybe your software is healthy, but is your data terminally ill?

In survey after survey, about half of IT executives consistently agree that data quality and data consistency is one of the biggest roadblocks to them getting full value from their data.

This has been consistently true all since the Chinese invented the abacus. I suspect it will be true long after quantum computing has solved every other problem that humanity faces.

 

Incorrect, inconsistent, fraudulent and redundant data cost the U.S. economy over $3 Trillion a year - an astounding figure that is over twice the amount of the 2011 Federal Deficit.

Similarly, many experts estimate that HALF the money spent on developers goes towards "software repair". So we're living in a world of sick software and dirty data. And the cost of all this is staggering.

 

I've long been a proponent of healthy software - but healthy software can only function properly in the presence of healthy data. Does quality software even matter if the underlying data are defective? Agreed - that's pushing the point to the extreme.

The rapid, iterative, continuous testing model has measurably improved the quality of software development. Evangelists such as Kent Beck have had a huge impact on this. I recently posted a freely downloadable white paper on this topic. But where are the evangelists for data quality? Where is an open source "JUnit for Data" and if it's out there, why isn't everyone using it?

The Cost of Bad Data
Anyone care to make a guess at how much money is wasted every year due to dirty or duplicate / redundant data? I'll start by presenting one common user story - one you probably have also recently experienced. And then expand on it.

Recently, I went to my mailbox and waiting for me was yet another invitation from a major bank to join their credit card program.

This shouldn't come as a surprise, as people everywhere are deluged by credit card offers. Except that I already have the particular card in question. Not only that, but because the particular bank in question has managed to acquire a number of other banks and credit card lines of business, between my personal and my corporation, I believe I now have five Visa cards from this particular bank.

I also occasionally get mail from them offering me cash bonuses to open up a checking account at their bank. Probably wasted postage, as I already have two checking accounts there. I suppose I could open up a third, just to get the $100.

Every month, I get a significant number of expensive looking direct mail offers from this bank, often with slightly different variations on my name, which I promptly throw away. Aside from the impact on the environment and the wasted direct mail expense, it's a bit irritating to me. I hate junk mail, and I feel compelled to shred things like credit card offers. So they've burdened me (an existing customer) with yet another "thing to do". So they've spent money, hurt the environment, irritated an existing customer, and now I get to make fun of them online. Bad investment on their part.

QAS (an Experian company) estimates that the average company wastes $180,000 per year simply on direct mail that does not reach the intended recipient because of inaccurate data. But this is just one miniscule slice of the data quality issue. In fact it's only one small part of the "direct mail" data quality issue. A lot more money is wasted in "inappropriate offers" and "duplicate offers" such as the ones my bank sends. I also get offers from several companies that are convinced that I'm married to the previous owner of my house. Those offers reach me, yet are immediately shredded. No sense opening them. So the "big picture" just for direct mail is much larger than what QAS shows.

None of this accounts for the "irritation" factor - what is the cost of annoying existing customers (or potential customers) with badly targeted offers?

Yet direct mail and all other forms of advertising together add up to a tiny slice of the bad-data pie.

Fraud Is a Bad Data Problem
Some time back, the US Attorney General's office stated that they believed that 14 percent of health care dollars are wasted in fraud or inaccurate billing.

Why do I lump fraud in with "bad data"? Bad data comes in two forms - accidentally created bad data and intentionally created bad data (for example, fraudulent billing). Either way, it's bad data. It doesn't matter how it got there, it's defective. And a lot of it could be detected and remediated "at the point of entry".

Healthcare accounts for over 16% of the U.S. GDP (Canada is 10%, Australia is 9% as a comparison). The U.S. GDP is currently approximately $14 Trillion - therefore healthcare spending in the U.S. amounts to $2.25 trillion. And the cost of bad data in Healthcare- $314 Billion.

That's just for fraud or inaccurate billing. What about other areas in healthcare (e.g. lost data, "bad patient outcomes", duplicate patient testing, manual rework, etc.)?  Even if we round down, we're still taking about $500 Billion for one industry alone.  If I extrapolate that out to the entire U.S. economy, we're talking about a $3.1 Trillion problem.  No matter how far off my estimate is (on the high side or the low side), it's a problem of astonishing proportions.

Cost of Bad Data to Business and IT
A classic but very worthwhile book from information governance expert Larry English posits that the business cost of nonquality data may be as high as 10-25% of an organization's revenue, and that as much as 50% of the typical IT budget may be spent in "information scrap and rework".  If that is the case, then my $3.1 estimate is not out of line.

In the introduction to his book, English states "With this proliferation of information, the challenge of managing data and providing quality information has never been more important or complex."

That was in 1999. With so much more data today, and a surprising lack of attention to the data quality issue, I can only imagine the total economic impact of things today. I do not doubt that the cost of bad data has risen.

Dealing with bad data at the I.T. level is expensive. But if I.T. doesn't deal with the bad data problem, then the cost gets pushed downstream to the "business", where the business costs are geometrically higher. The model is not that different from that of "healthy software", where it costs $1 to uncover a defect during developer/unit testing, but $100 to fix that defect if the software is released to the end-users.

"Low Hanging Fruit" - Best Practices for Bad Data Avoidance
I am not saying that there are any easy fixes to the bad data problem. Even something as relatively simple as cleaning, standardizing and de-duping a mailing list with 10,000,000 entries is essentially impossible to get completely right no matter how much effort is put into it. Yet there are some relatively easy things that can be done to substantially improve the quality of our data.  As with so many other problems in life, the some version of the 80/20 rule applies to this as well.

Best Practice #1: When integrating data, fix the quality problem during integration
As data are added or integrated, data should be tested. Profiling is a simple, fast, relatively easily implemented and highly effective way for eliminating significant volumes of defective data.

When developers write a new application for the input of some new data, it's normal for input fields to be "validated" - a simple "hard coded" form of profiling. Month number needs to be between 1-12. 13 is never correct.  Not rocket science. And it's universally done.

Yet people have far fewer reservations about integrating data from here, there and everywhere - often not checking for even the most egregious data errors, and thereby polluting the organizational drinking water (i.e. all the data and applications downstream).

I strongly suspect that's why I get so many offers from my current mega-bank. Since the banking implosion, this particular bank has purchased every other bank around. And their credit card businesses. And their marketing databases. And (apparently) smashed them together. So I get offers for Hollis Tibbetts, Hollis W. Tibbetts, Hollis Winslow Tibbetts, Hollis Tibbets, Hollis Tibbitts and so on.

Integration of data isn't necessarily just a "big bang" event - like when one company acquires another and smashes all the data together, or when two divisional customer applications get merged. It can be more insidious and more when you have "trickle" integration - the slow feed of new data from one system into another (either within the organization or from customers/suppliers/partners).  This is the class of integration that is causing a lot of the problems previously discussed with healthcare fraud.

Either way, FIX IT before integrating it. Once the poison enters the corporate drinking water, it's a lot harder to get out (not just technically, but especially politically/organizationally).

Best Practice #2: When migrating data, fix the data problem as PART of the migration project
Spending $1 billion to upgrade your Seibel system like the US Government is doing? Sounds like a great time to fix your data quality problem.

If you're doing something like migrating your customer data from Seibel to Netsuite or Salesforce.com, data quality should be a major element in your project plan (and budget). Fixing the problems during the migration are easier than fixing them later:

  1. You probably already possess a lot of knowledge about the existing legacy systems, the types of problems in the data. But your new system is relatively unknown to you. So it's likely to be easier to fix data issues from a technical perspective BEFORE they get loaded into the new system.
  2. As part of the data migration process, you can export the data to a staging platform (On Prem or Cloud), leverage any number of data quality tools/engines, and then import the data into the the application platform.  This approach may partially pay for itself in an easier/smoother upgrade to the new application, but that's a rounding error in the overall scheme of things.
  3. Organizationally and politically, companies are much more likely to spend money to clean data if it's part of a project like "upgrade the CRM system". I'd hate to be the CIO that spends a mountain of money to upgrade the CRM system and then goes back to the board asking for another mountain of money to fix all the bad data that just got loaded into the CRM system. That's how CIO's become ex-CIOs.

Best Practice #3: Data profiling and data de-duplication engines
Data profiling engines are a great technology for quickly improving the quality of data as it is integrated from one system into another. At the highest level, they are an engine that scans data, and applies certain easily definable rules to data elements, such as formats, ranges, allowable values and can evaluate relationships between different fields.

Furthermore, these engines can also be used to analyze existing data stores very rapidly and generate "exceptions files" for manual, or semi-automated remediation (if anyone can find a totally automated data remediation system, I'd love to know about it). So they can be used in "continuous testing" or "batch testing" mode.  In batch mode, they're ideal for application migrations or big-bang integrations, as they're easiest to use them if you have your data in something like a staging database.  But they can also be used to test data as it is "trickle integrated" into production systems.

De-duping engines generally fit into the same category. I haven't seen them be as effective as data profiling engines, yet I believe they're essential. The technology for de-duping is considerably more sophisticated - with a large number of different algorithms and tunable thresholds and such. It's a harder class of technology to implement. More manual effort is involved. And, unlike profiling (where there is NEVER a month "13"), de-duping can "get it wrong", so the technology needs to be applied more selectively.

Conclusion
I've never understood why these engines haven't been more popular. There is no "JUnit for data" as far as I know. But commercial solutions are available - they're not terribly expensive and rapidly pay for themselves.

On the other hand, I've never understood why organizations are so tolerant of bad, dirty data. They waste millions and millions directly because of it (and untold quantities of money in "wasted opportunities"), but are reluctant to spend $15,000 on a data quality engine to help fix a significant portion of the problem.

More Stories By Hollis Tibbetts

Hollis Tibbetts, or @SoftwareHollis as his 50,000+ followers know him on Twitter, is listed on various “top 100 expert lists” for a variety of topics – ranging from Cloud to Technology Marketing, Hollis is by day Evangelist & Software Technology Director at Dell Software. By night and weekends he is a commentator, speaker and all-round communicator about Software, Data and Cloud in their myriad aspects. You can also reach Hollis on LinkedIn – linkedin.com/in/SoftwareHollis. His latest online venture is OnlineBackupNews - a free reference site to help organizations protect their data, applications and systems from threats. Every year IT Downtime Costs $26.5 Billion In Lost Revenue. Even with such high costs, 56% of enterprises in North America and 30% in Europe don’t have a good disaster recovery plan. Online Backup News aims to make sure you all have the news and tips needed to keep your IT Costs down and your information safe by providing best practices, technology insights, strategies, real-world examples and various tips and techniques from a variety of industry experts.

Hollis is a regularly featured blogger at ebizQ, a venue focused on enterprise technologies, with over 100,000 subscribers. He is also an author on Social Media Today "The World's Best Thinkers on Social Media", and maintains a blog focused on protecting data: Online Backup News.
He tweets actively as @SoftwareHollis

Additional information is available at HollisTibbetts.com

All opinions expressed in the author's articles are his own personal opinions vs. those of his employer.

@ThingsExpo Stories
Explosive growth in connected devices. Enormous amounts of data for collection and analysis. Critical use of data for split-second decision making and actionable information. All three are factors in making the Internet of Things a reality. Yet, any one factor would have an IT organization pondering its infrastructure strategy. How should your organization enhance its IT framework to enable an Internet of Things implementation? In his session at Internet of @ThingsExpo, James Kirkland, Chief Architect for the Internet of Things and Intelligent Systems at Red Hat, described how to revolutioniz...
The major cloud platforms defy a simple, side-by-side analysis. Each of the major IaaS public-cloud platforms offers their own unique strengths and functionality. Options for on-site private cloud are diverse as well, and must be designed and deployed while taking existing legacy architecture and infrastructure into account. Then the reality is that most enterprises are embarking on a hybrid cloud strategy and programs. In this Power Panel at 15th Cloud Expo (http://www.CloudComputingExpo.com), moderated by Ashar Baig, Research Director, Cloud, at Gigaom Research, Nate Gordon, Director of T...
Technology is enabling a new approach to collecting and using data. This approach, commonly referred to as the "Internet of Things" (IoT), enables businesses to use real-time data from all sorts of things including machines, devices and sensors to make better decisions, improve customer service, and lower the risk in the creation of new revenue opportunities. In his General Session at Internet of @ThingsExpo, Dave Wagstaff, Vice President and Chief Architect at BSQUARE Corporation, discuss the real benefits to focus on, how to understand the requirements of a successful solution, the flow of ...
The security devil is always in the details of the attack: the ones you've endured, the ones you prepare yourself to fend off, and the ones that, you fear, will catch you completely unaware and defenseless. The Internet of Things (IoT) is nothing if not an endless proliferation of details. It's the vision of a world in which continuous Internet connectivity and addressability is embedded into a growing range of human artifacts, into the natural world, and even into our smartphones, appliances, and physical persons. In the IoT vision, every new "thing" - sensor, actuator, data source, data con...
"BSQUARE is in the business of selling software solutions for smart connected devices. It's obvious that IoT has moved from being a technology to being a fundamental part of business, and in the last 18 months people have said let's figure out how to do it and let's put some focus on it, " explained Dave Wagstaff, VP & Chief Architect, at BSQUARE Corporation, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4-6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.

ARMONK, N.Y., Nov. 20, 2014 /PRNewswire/ --  IBM (NYSE: IBM) today announced that it is bringing a greater level of control, security and flexibility to cloud-based application development and delivery with a single-tenant version of Bluemix, IBM's platform-as-a-service. The new platform enables developers to build ap...

Focused on this fast-growing market’s needs, Vitesse Semiconductor Corporation (Nasdaq: VTSS), a leading provider of IC solutions to advance "Ethernet Everywhere" in Carrier, Enterprise and Internet of Things (IoT) networks, introduced its IStaX™ software (VSC6815SDK), a robust protocol stack to simplify deployment and management of Industrial-IoT network applications such as Industrial Ethernet switching, surveillance, video distribution, LCD signage, intelligent sensors, and metering equipment. Leveraging technologies proven in the Carrier and Enterprise markets, IStaX is designed to work ac...
"There is a natural synchronization between the business models, the IoT is there to support ,” explained Brendan O'Brien, Co-founder and Chief Architect of Aria Systems, in this SYS-CON.tv interview at the 15th International Cloud Expo®, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
C-Labs LLC, a leading provider of remote and mobile access for the Internet of Things (IoT), announced the appointment of John Traynor to the position of chief operating officer. Previously a strategic advisor to the firm, Mr. Traynor will now oversee sales, marketing, finance, and operations. Mr. Traynor is based out of the C-Labs office in Redmond, Washington. He reports to Chris Muench, Chief Executive Officer. Mr. Traynor brings valuable business leadership and technology industry expertise to C-Labs. With over 30 years' experience in the high-tech sector, John Traynor has held numerous...
Bit6 today issued a challenge to the technology community implementing Web Real Time Communication (WebRTC). To leap beyond WebRTC’s significant limitations and fully leverage its underlying value to accelerate innovation, application developers need to consider the entire communications ecosystem.
The 3rd International @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devices - computers, smartphones, tablets, and sensors - connected to the Internet by 2020. This number will continue to grow at a rapid pace for the next several decades.
The Internet of Things is not new. Historically, smart businesses have used its basic concept of leveraging data to drive better decision making and have capitalized on those insights to realize additional revenue opportunities. So, what has changed to make the Internet of Things one of the hottest topics in tech? In his session at @ThingsExpo, Chris Gray, Director, Embedded and Internet of Things, discussed the underlying factors that are driving the economics of intelligent systems. Discover how hardware commoditization, the ubiquitous nature of connectivity, and the emergence of Big Data a...
Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity.
SYS-CON Events announced today that Windstream, a leading provider of advanced network and cloud communications, has been named “Silver Sponsor” of SYS-CON's 16th International Cloud Expo®, which will take place on June 9–11, 2015, at the Javits Center in New York, NY. Windstream (Nasdaq: WIN), a FORTUNE 500 and S&P 500 company, is a leading provider of advanced network communications, including cloud computing and managed services, to businesses nationwide. The company also offers broadband, phone and digital TV services to consumers primarily in rural areas.
SYS-CON Events announced today that IDenticard will exhibit at SYS-CON's 16th International Cloud Expo®, which will take place on June 9-11, 2015, at the Javits Center in New York City, NY. IDenticard™ is the security division of Brady Corp (NYSE: BRC), a $1.5 billion manufacturer of identification products. We have small-company values with the strength and stability of a major corporation. IDenticard offers local sales, support and service to our customers across the United States and Canada. Our partner network encompasses some 300 of the world's leading systems integrators and security s...
IoT is still a vague buzzword for many people. In his session at @ThingsExpo, Mike Kavis, Vice President & Principal Cloud Architect at Cloud Technology Partners, discussed the business value of IoT that goes far beyond the general public's perception that IoT is all about wearables and home consumer services. He also discussed how IoT is perceived by investors and how venture capitalist access this space. Other topics discussed were barriers to success, what is new, what is old, and what the future may hold. Mike Kavis is Vice President & Principal Cloud Architect at Cloud Technology Pa...
Cloud Expo 2014 TV commercials will feature @ThingsExpo, which was launched in June, 2014 at New York City's Javits Center as the largest 'Internet of Things' event in the world. The next @ThingsExpo will take place November 4-6, 2014, at the Santa Clara Convention Center, in Santa Clara, California. Since its launch in 2008, Cloud Expo TV commercials have been aired and CNBC, Fox News Network, and Bloomberg TV. Please enjoy our 2014 commercial.
From a software development perspective IoT is about programming "things," about connecting them with each other or integrating them with existing applications. In his session at @ThingsExpo, Yakov Fain, co-founder of Farata Systems and SuranceBay, will show you how small IoT-enabled devices from multiple manufacturers can be integrated into the workflow of an enterprise application. This is a practical demo of building a framework and components in HTML/Java/Mobile technologies to serve as a platform that can integrate new devices as they become available on the market.
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
Located in booth #314, the Bsquare team will present DataV demos and discuss how DataV will help customers put their data to work to improve business outcomes. DataV is unlocking new initiatives across a wide landscape of customers in industries such as industrial manufacturing, transportation, retail and mobile. The solution is designed to complement a new project start or help to enrich an existing machine investment.