Welcome!

Industrial IoT Authors: Yeshim Deniz, Liz McMillan, William Schmarzo, Allwyn Sequeira, Pat Romanski

Related Topics: Microservices Expo, Industrial IoT

Microservices Expo: Article

$3 Trillion Problem: Three Best Practices for Today's Dirty Data Pandemic

Maybe your software is healthy, but is your data terminally ill?

In survey after survey, about half of IT executives consistently agree that data quality and data consistency is one of the biggest roadblocks to them getting full value from their data.

This has been consistently true all since the Chinese invented the abacus. I suspect it will be true long after quantum computing has solved every other problem that humanity faces.

 

Incorrect, inconsistent, fraudulent and redundant data cost the U.S. economy over $3 Trillion a year - an astounding figure that is over twice the amount of the 2011 Federal Deficit.

Similarly, many experts estimate that HALF the money spent on developers goes towards "software repair". So we're living in a world of sick software and dirty data. And the cost of all this is staggering.

 

I've long been a proponent of healthy software - but healthy software can only function properly in the presence of healthy data. Does quality software even matter if the underlying data are defective? Agreed - that's pushing the point to the extreme.

The rapid, iterative, continuous testing model has measurably improved the quality of software development. Evangelists such as Kent Beck have had a huge impact on this. I recently posted a freely downloadable white paper on this topic. But where are the evangelists for data quality? Where is an open source "JUnit for Data" and if it's out there, why isn't everyone using it?

The Cost of Bad Data
Anyone care to make a guess at how much money is wasted every year due to dirty or duplicate / redundant data? I'll start by presenting one common user story - one you probably have also recently experienced. And then expand on it.

Recently, I went to my mailbox and waiting for me was yet another invitation from a major bank to join their credit card program.

This shouldn't come as a surprise, as people everywhere are deluged by credit card offers. Except that I already have the particular card in question. Not only that, but because the particular bank in question has managed to acquire a number of other banks and credit card lines of business, between my personal and my corporation, I believe I now have five Visa cards from this particular bank.

I also occasionally get mail from them offering me cash bonuses to open up a checking account at their bank. Probably wasted postage, as I already have two checking accounts there. I suppose I could open up a third, just to get the $100.

Every month, I get a significant number of expensive looking direct mail offers from this bank, often with slightly different variations on my name, which I promptly throw away. Aside from the impact on the environment and the wasted direct mail expense, it's a bit irritating to me. I hate junk mail, and I feel compelled to shred things like credit card offers. So they've burdened me (an existing customer) with yet another "thing to do". So they've spent money, hurt the environment, irritated an existing customer, and now I get to make fun of them online. Bad investment on their part.

QAS (an Experian company) estimates that the average company wastes $180,000 per year simply on direct mail that does not reach the intended recipient because of inaccurate data. But this is just one miniscule slice of the data quality issue. In fact it's only one small part of the "direct mail" data quality issue. A lot more money is wasted in "inappropriate offers" and "duplicate offers" such as the ones my bank sends. I also get offers from several companies that are convinced that I'm married to the previous owner of my house. Those offers reach me, yet are immediately shredded. No sense opening them. So the "big picture" just for direct mail is much larger than what QAS shows.

None of this accounts for the "irritation" factor - what is the cost of annoying existing customers (or potential customers) with badly targeted offers?

Yet direct mail and all other forms of advertising together add up to a tiny slice of the bad-data pie.

Fraud Is a Bad Data Problem
Some time back, the US Attorney General's office stated that they believed that 14 percent of health care dollars are wasted in fraud or inaccurate billing.

Why do I lump fraud in with "bad data"? Bad data comes in two forms - accidentally created bad data and intentionally created bad data (for example, fraudulent billing). Either way, it's bad data. It doesn't matter how it got there, it's defective. And a lot of it could be detected and remediated "at the point of entry".

Healthcare accounts for over 16% of the U.S. GDP (Canada is 10%, Australia is 9% as a comparison). The U.S. GDP is currently approximately $14 Trillion - therefore healthcare spending in the U.S. amounts to $2.25 trillion. And the cost of bad data in Healthcare- $314 Billion.

That's just for fraud or inaccurate billing. What about other areas in healthcare (e.g. lost data, "bad patient outcomes", duplicate patient testing, manual rework, etc.)?  Even if we round down, we're still taking about $500 Billion for one industry alone.  If I extrapolate that out to the entire U.S. economy, we're talking about a $3.1 Trillion problem.  No matter how far off my estimate is (on the high side or the low side), it's a problem of astonishing proportions.

Cost of Bad Data to Business and IT
A classic but very worthwhile book from information governance expert Larry English posits that the business cost of nonquality data may be as high as 10-25% of an organization's revenue, and that as much as 50% of the typical IT budget may be spent in "information scrap and rework".  If that is the case, then my $3.1 estimate is not out of line.

In the introduction to his book, English states "With this proliferation of information, the challenge of managing data and providing quality information has never been more important or complex."

That was in 1999. With so much more data today, and a surprising lack of attention to the data quality issue, I can only imagine the total economic impact of things today. I do not doubt that the cost of bad data has risen.

Dealing with bad data at the I.T. level is expensive. But if I.T. doesn't deal with the bad data problem, then the cost gets pushed downstream to the "business", where the business costs are geometrically higher. The model is not that different from that of "healthy software", where it costs $1 to uncover a defect during developer/unit testing, but $100 to fix that defect if the software is released to the end-users.

"Low Hanging Fruit" - Best Practices for Bad Data Avoidance
I am not saying that there are any easy fixes to the bad data problem. Even something as relatively simple as cleaning, standardizing and de-duping a mailing list with 10,000,000 entries is essentially impossible to get completely right no matter how much effort is put into it. Yet there are some relatively easy things that can be done to substantially improve the quality of our data.  As with so many other problems in life, the some version of the 80/20 rule applies to this as well.

Best Practice #1: When integrating data, fix the quality problem during integration
As data are added or integrated, data should be tested. Profiling is a simple, fast, relatively easily implemented and highly effective way for eliminating significant volumes of defective data.

When developers write a new application for the input of some new data, it's normal for input fields to be "validated" - a simple "hard coded" form of profiling. Month number needs to be between 1-12. 13 is never correct.  Not rocket science. And it's universally done.

Yet people have far fewer reservations about integrating data from here, there and everywhere - often not checking for even the most egregious data errors, and thereby polluting the organizational drinking water (i.e. all the data and applications downstream).

I strongly suspect that's why I get so many offers from my current mega-bank. Since the banking implosion, this particular bank has purchased every other bank around. And their credit card businesses. And their marketing databases. And (apparently) smashed them together. So I get offers for Hollis Tibbetts, Hollis W. Tibbetts, Hollis Winslow Tibbetts, Hollis Tibbets, Hollis Tibbitts and so on.

Integration of data isn't necessarily just a "big bang" event - like when one company acquires another and smashes all the data together, or when two divisional customer applications get merged. It can be more insidious and more when you have "trickle" integration - the slow feed of new data from one system into another (either within the organization or from customers/suppliers/partners).  This is the class of integration that is causing a lot of the problems previously discussed with healthcare fraud.

Either way, FIX IT before integrating it. Once the poison enters the corporate drinking water, it's a lot harder to get out (not just technically, but especially politically/organizationally).

Best Practice #2: When migrating data, fix the data problem as PART of the migration project
Spending $1 billion to upgrade your Seibel system like the US Government is doing? Sounds like a great time to fix your data quality problem.

If you're doing something like migrating your customer data from Seibel to Netsuite or Salesforce.com, data quality should be a major element in your project plan (and budget). Fixing the problems during the migration are easier than fixing them later:

  1. You probably already possess a lot of knowledge about the existing legacy systems, the types of problems in the data. But your new system is relatively unknown to you. So it's likely to be easier to fix data issues from a technical perspective BEFORE they get loaded into the new system.
  2. As part of the data migration process, you can export the data to a staging platform (On Prem or Cloud), leverage any number of data quality tools/engines, and then import the data into the the application platform.  This approach may partially pay for itself in an easier/smoother upgrade to the new application, but that's a rounding error in the overall scheme of things.
  3. Organizationally and politically, companies are much more likely to spend money to clean data if it's part of a project like "upgrade the CRM system". I'd hate to be the CIO that spends a mountain of money to upgrade the CRM system and then goes back to the board asking for another mountain of money to fix all the bad data that just got loaded into the CRM system. That's how CIO's become ex-CIOs.

Best Practice #3: Data profiling and data de-duplication engines
Data profiling engines are a great technology for quickly improving the quality of data as it is integrated from one system into another. At the highest level, they are an engine that scans data, and applies certain easily definable rules to data elements, such as formats, ranges, allowable values and can evaluate relationships between different fields.

Furthermore, these engines can also be used to analyze existing data stores very rapidly and generate "exceptions files" for manual, or semi-automated remediation (if anyone can find a totally automated data remediation system, I'd love to know about it). So they can be used in "continuous testing" or "batch testing" mode.  In batch mode, they're ideal for application migrations or big-bang integrations, as they're easiest to use them if you have your data in something like a staging database.  But they can also be used to test data as it is "trickle integrated" into production systems.

De-duping engines generally fit into the same category. I haven't seen them be as effective as data profiling engines, yet I believe they're essential. The technology for de-duping is considerably more sophisticated - with a large number of different algorithms and tunable thresholds and such. It's a harder class of technology to implement. More manual effort is involved. And, unlike profiling (where there is NEVER a month "13"), de-duping can "get it wrong", so the technology needs to be applied more selectively.

Conclusion
I've never understood why these engines haven't been more popular. There is no "JUnit for data" as far as I know. But commercial solutions are available - they're not terribly expensive and rapidly pay for themselves.

On the other hand, I've never understood why organizations are so tolerant of bad, dirty data. They waste millions and millions directly because of it (and untold quantities of money in "wasted opportunities"), but are reluctant to spend $15,000 on a data quality engine to help fix a significant portion of the problem.

More Stories By Hollis Tibbetts

Hollis Tibbetts, or @SoftwareHollis as his 50,000+ followers know him on Twitter, is listed on various “top 100 expert lists” for a variety of topics – ranging from Cloud to Technology Marketing, Hollis is by day Evangelist & Software Technology Director at Dell Software. By night and weekends he is a commentator, speaker and all-round communicator about Software, Data and Cloud in their myriad aspects. You can also reach Hollis on LinkedIn – linkedin.com/in/SoftwareHollis. His latest online venture is OnlineBackupNews - a free reference site to help organizations protect their data, applications and systems from threats. Every year IT Downtime Costs $26.5 Billion In Lost Revenue. Even with such high costs, 56% of enterprises in North America and 30% in Europe don’t have a good disaster recovery plan. Online Backup News aims to make sure you all have the news and tips needed to keep your IT Costs down and your information safe by providing best practices, technology insights, strategies, real-world examples and various tips and techniques from a variety of industry experts.

Hollis is a regularly featured blogger at ebizQ, a venue focused on enterprise technologies, with over 100,000 subscribers. He is also an author on Social Media Today "The World's Best Thinkers on Social Media", and maintains a blog focused on protecting data: Online Backup News.
He tweets actively as @SoftwareHollis

Additional information is available at HollisTibbetts.com

All opinions expressed in the author's articles are his own personal opinions vs. those of his employer.

@ThingsExpo Stories
Microservices are a very exciting architectural approach that many organizations are looking to as a way to accelerate innovation. Microservices promise to allow teams to move away from monolithic "ball of mud" systems, but the reality is that, in the vast majority of organizations, different projects and technologies will continue to be developed at different speeds. How to handle the dependencies between these disparate systems with different iteration cycles? Consider the "canoncial problem" ...
Why do your mobile transformations need to happen today? Mobile is the strategy that enterprise transformation centers on to drive customer engagement. In his general session at @ThingsExpo, Roger Woods, Director, Mobile Product & Strategy – Adobe Marketing Cloud, covered key IoT and mobile trends that are forcing mobile transformation, key components of a solid mobile strategy and explored how brands are effectively driving mobile change throughout the enterprise.
As businesses adopt functionalities in cloud computing, it’s imperative that IT operations consistently ensure cloud systems work correctly – all of the time, and to their best capabilities. In his session at @BigDataExpo, Bernd Harzog, CEO and founder of OpsDataStore, will present an industry answer to the common question, “Are you running IT operations as efficiently and as cost effectively as you need to?” He will expound on the industry issues he frequently came up against as an analyst, and...
Keeping pace with advancements in software delivery processes and tooling is taxing even for the most proficient organizations. Point tools, platforms, open source and the increasing adoption of private and public cloud services requires strong engineering rigor - all in the face of developer demands to use the tools of choice. As Agile has settled in as a mainstream practice, now DevOps has emerged as the next wave to improve software delivery speed and output. To make DevOps work, organization...
My team embarked on building a data lake for our sales and marketing data to better understand customer journeys. This required building a hybrid data pipeline to connect our cloud CRM with the new Hadoop Data Lake. One challenge is that IT was not in a position to provide support until we proved value and marketing did not have the experience, so we embarked on the journey ourselves within the product marketing team for our line of business within Progress. In his session at @BigDataExpo, Sum...
SYS-CON Events announced today that MobiDev, a client-oriented software development company, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MobiDev is a software company that develops and delivers turn-key mobile apps, websites, web services, and complex softw...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
What sort of WebRTC based applications can we expect to see over the next year and beyond? One way to predict development trends is to see what sorts of applications startups are building. In his session at @ThingsExpo, Arin Sime, founder of WebRTC.ventures, will discuss the current and likely future trends in WebRTC application development based on real requests for custom applications from real customers, as well as other public sources of information,
"My role is working with customers, helping them go through this digital transformation. I spend a lot of time talking to banks, big industries, manufacturers working through how they are integrating and transforming their IT platforms and moving them forward," explained William Morrish, General Manager Product Sales at Interoute, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
Apache Hadoop is emerging as a distributed platform for handling large and fast incoming streams of data. Predictive maintenance, supply chain optimization, and Internet-of-Things analysis are examples where Hadoop provides the scalable storage, processing, and analytics platform to gain meaningful insights from granular data that is typically only valuable from a large-scale, aggregate view. One architecture useful for capturing and analyzing streaming data is the Lambda Architecture, represent...
SYS-CON Events announced today that Ocean9will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Ocean9 provides cloud services for Backup, Disaster Recovery (DRaaS) and instant Innovation, and redefines enterprise infrastructure with its cloud native subscription offerings for mission critical SAP workloads.
With billions of sensors deployed worldwide, the amount of machine-generated data will soon exceed what our networks can handle. But consumers and businesses will expect seamless experiences and real-time responsiveness. What does this mean for IoT devices and the infrastructure that supports them? More of the data will need to be handled at - or closer to - the devices themselves.
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
SYS-CON Events announced today that Linux Academy, the foremost online Linux and cloud training platform and community, will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Linux Academy was founded on the belief that providing high-quality, in-depth training should be available at an affordable price. Industry leaders in quality training, provided services, and student certification passes, its goal is to c...
SYS-CON Events announced today that CA Technologies has been named “Platinum Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY, and the 21st International Cloud Expo®, which will take place October 31-November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. CA Technologies helps customers succeed in a future where every business – from apparel to energy – is being rewritten by software. From ...
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), will provide an overview of various initiatives to certifiy the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldw...
SYS-CON Events announced today that Loom Systems will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Founded in 2015, Loom Systems delivers an advanced AI solution to predict and prevent problems in the digital business. Loom stands alone in the industry as an AI analysis platform requiring no prior math knowledge from operators, leveraging the existing staff to succeed in the digital era. With offices in S...
SYS-CON Events announced today that Interoute, owner-operator of one of Europe's largest networks and a global cloud services platform, has been named “Bronze Sponsor” of SYS-CON's 20th Cloud Expo, which will take place on June 6-8, 2017 at the Javits Center in New York, New York. Interoute is the owner-operator of one of Europe's largest networks and a global cloud services platform which encompasses 12 data centers, 14 virtual data centers and 31 colocation centers, with connections to 195 add...
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...
SYS-CON Events announced today that HTBase will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. HTBase (Gartner 2016 Cool Vendor) delivers a Composable IT infrastructure solution architected for agility and increased efficiency. It turns compute, storage, and fabric into fluid pools of resources that are easily composed and re-composed to meet each application’s needs. With HTBase, companies can quickly prov...