{"id":4375423,"date":"2020-01-22T10:01:00","date_gmt":"2020-01-22T08:01:00","guid":{"rendered":"https:\/\/www.deptagency.com\/insight\/four-kinds-of-failure-in-data-management\/"},"modified":"2025-12-08T13:44:32","modified_gmt":"2025-12-08T11:44:32","slug":"four-kinds-of-failure-in-data-management","status":"publish","type":"article","link":"https:\/\/www.deptagency.com\/en-in\/insight\/four-kinds-of-failure-in-data-management\/","title":{"rendered":"Four kinds of failure in data management"},"content":{"rendered":"
\n\t\n\t<\/svg>\n\t\t\t\n\t\t\tBack to all articles\t\t<\/span>\n\t\n<\/button>\n\t\tFour kinds of failure in data management\t<\/h1>\n\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t\t\n\t\t<\/picture>\n\t\n\t<\/div>
\n\t\t\t\n\t\n\t
\n\t\tDavid Grover\t\t
\n\t\tData Architect\n\t\t\t<\/div>\n<\/div>\n\t
\n\t\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\tDate\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t\t22 January 2020\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t<\/div>\n<\/div>\n\n
\n\t
\n\t\t\n\t\t\t\t\t
\n\t\t\t\t\n
\n\t\tIn 2018 I started work on an analytical framework to help evaluate the quality of data management systems. What I was looking for was a way to say to potential clients \u201cyour system has these problems, and it will cost $X to fix it, and the end result will be measurably<\/em> better.\u201d It\u2019s the measurably<\/em> better<\/em> claim that\u2019s the real hard part.\u00a0 I\u2019ve got a lot of experience in data management, but an analytical framework beats your life story.

The first step to a measurement framework was defining failure points. If we can sort out how a system\u2019s components fail, and then count actual failures, which are likely to fail and which won\u2019t, then we can start to assess a fix and assign a dollar value. What I concluded was that when we do data management – the kind of thing we do when we build a data warehouse or even a transaction database behind an application – there\u2019s only four operations we ever do, period.\u00a0 That is, there\u2019s only four operations in data management, and those operations can fail, so those are our failure points. I\u2019m going to explain what those four are below.

That claim should provoke some skepticism. Why only four? Surely in the rich history of data management, there’s more than four ways to screw up. It\u2019s lucky there\u2019s only four, though, because one of the problems we face explaining the cost of poor-quality data management is task scope<\/em>.\u00a0 It\u2019s better that there are four instead of 38 because it\u2019s easier to sort out what to do with four problems.

If I’m managing the construction of widgets on an assembly line and I want to make widgets faster, cheaper, or slightly different, I need to be able to divide the construction processes into component parts and abstract those processes so I can figure out how to do more with the same amount of effort. This is obvious, no?

So to improve the overall output produced by data management teams we need to identify some discrete and low-finite number of things those data management teams do. “ETL,” “BI” and “database administration” are architectural options and not frameworks.\u00a0 That is, where we choose to transform data into something useful for our client depends on their preference and our skillsets, but we can do pretty much the same things in the ETL, BI or persistence layers of our system. If we know there’s only four operations happening across all of those layers, we can start to apply some intelligence to particular local problems. And we can start to calculate the cost of doing things the right way versus the wrong way, the cost of improving a single operation so it does its work better, and the cost of the failure of that operation to do its expected job.

This is important. I have former coworkers looking at multiple petabyte-size file servers stuffed with test data who believe there’s value in those systems. The only framework they have for evaluating the utility of that vast catalog of data is subject matter: They know roughly what a particular set of directories is about<\/em>, but the only quality scale most of us have for such environments is “messy-to-pretty clean.” This is not much more useful than using your thumbs to measure out the lumber you need to build a house.

I also have colleagues who’ve inherited systems built by move-fast-and-break-stuff<\/em>-oriented co-workers. These people have to turn these systems into reliable, useful reporting tools. But there’s no way to convey the difficulty of that conversion to hopeful managers handing off their latest prototype, including where the prototype\u2019s failure points will be, or of providing an operating-cost analysis. Now I’ve built a few of those rube-goldberg systems myself, so let’s not condemn the enthusiastic-but-inexperienced out-of-hand. But if you need to explain why it costs so much to maintain a half-assed data lake with a swampy layer of assumptions where the semantic layer ought to be, in comparison to something that’s had some thought put into it, all you’ve got again is thumbs.

There’s no way, in short, to explain why good data management practices cost less than bad ones.\t<\/p><\/div><\/div>\n\t\t\t<\/div>\n\t\t\t<\/div>\n\n\t\n\t