Welcome!

Industrial IoT Authors: Pat Romanski, William Schmarzo, Elizabeth White, Liz McMillan, Jeev Trika

Related Topics: Industrial IoT

Industrial IoT: Article

Use a Native XML Database for Your XML Data

Deciding when an XQuery-based native XML database is better than an SQL database

Developers tend to use the most familiar technologies. For data storage, that is the relational database. During design it's easy to see tables of data everywhere; however, not everything is relational in nature. When dealing with XML data or data easily expressed as XML, XQuery-based native XML databases (NXDs) present a viable and cost-effective alternative to relational databases, file system storage, or custom developed storage implementations. So, when is it time to consider an NXD?

Can native XML databases really provide a better answer for your data storage needs? In this article we'll examine some guidelines to help answer that question.

When to Consider a Native XML Database

  1. Do you have thousands of XML files?
  2. Is your XML data larger than 200MB?
  3. Are you trying to build a hierarchy into tables?
  4. Could your data change over time?
  5. Have you spent more that $100 on books explaining SQLXML?
The File System Isn't a Database
The first two questions are practical in nature. If estimates indicate more than 1000 XML files or 200MB of XML data exist, the file system isn't the right tool for the job. File systems are not built to manage large numbers of files in a single directory or huge directory hierarchies. Managing concurrency, out of disk space conditions, and other common problems will plague your application unless you use a database.

Naturally Hierarchical
The third reason to consider an NXD really has to do with the impedance mismatch between relational databases and XML data. Fundamentally, XML data is hierarchical and is a poor match for a relational database's rows and columns. Relational databases have always had a hard time modeling hierarchies. You'll find dozens of workarounds for this, but no real simple and efficient solution. Any XML-to-relational mapping tool will have to pick one of these techniques and manage this common case as best as possible. Regardless of the solution, performance will suffer. The end result will also be more brittle over time. Changing the structure of the data - which is easy, natural, and useful to do in XML - forces a redesign of the relational database and changes to the mapping layer. A single mapping mistake could completely skew an entire data set. As an XML document structure changes over time, it's possible for attributes to become elements, and vice versa. Over time, incremental changes to the logical structure of the XML data can force physical changes to the database, and wholesale dump and reload may be required of the relational system to keep pace.

Structured Yet Flexible Data
The fourth question suggests that requirements change over time, something true of most real world business systems. Once a relational database schema has been set in stone, only the database administrator (DBA) is qualified to change it without disrupting services. The contract between a relational database and the program that use it becomes the weakest link in the system. As requirements change, the DBA will spend endless hours mitigating the issues that arise. Contrast that with XML databases. XML is both structured and flexible. Even XML documents conforming to a DTD or XML Schema maintain a high degree of flexibility when compared to relational schemas. Most NXDs will optionally validate document structure. Even when document validation is not enforced, XML documents maintain a high degree of implicit structure. Therefore in either case, XML documents are flexible and structured and as a result the contract between the database and the programs using it is not brittle. It can withstand change without requiring a DBA.

Data Mapping Is Wasted Time, Money, and Effort
The last question is really a reality check. Take a second to consider the amount of time and money you've spent trying to make a solution workable. The best thing you can do when digging a hole is to stop digging and get out of the hole. With that in mind, let's look at SQLXML. If you need to mix and match SQL data and XML data, it's not a horrible way to go. However, if you view it as a way to squeeze XML into a relational system in which you've already invested, you might want to reconsider. You are going to pay a performance penalty for every document stored in terms of CPU and memory. Your ability to query, index, and optimize will be impacted as well. Executing XQuery against XML data mapped into relational tables will be hindered by the non-native storage format. An optimized native XML database won't have the same penalty.

Let's take a look at how to use an NXD to learn more about its advantages over a relational database.

Examples and Code
Berkeley DB XML is an open source XQuery implementation built atop the Berkeley DB transactional database system. It supports optimized XML storage, XQuery query planning, massive scale and concurrency, and is available for download as source code for multiple platforms and as a Windows installer from Sleepycat Software (www.sleepycat.com). Berkeley DB XML has also been integrated with Stylus Studio if you're more comfortable using an IDE for development. Berkeley DB XML is readily available to anyone, so we'll use it for the following examples.

First let's create an in-memory container for XML documents. Follow along using the "dbxml" command line provided with Berkeley DB XML.


dbxml> createContainer ""
dbxml> putDocument myDoc <names><name>joe</name><name>fred</
name><name>jane</name></names>
dbxml> query collection('')/names/name[.='joe']
dbxml> print
<name>joe</name>
Line 1 creates the container, while line 2 places a simple document into the container. The third line performs a query on the container returning each document that matches the XPath query /names/name[.=joe]. Finally it displays what we returned by the query statement.

That's all quite useful, but let's say that you need to access your XML storage programmatically. Because Berkeley DB XML is a library, and as such is linked into your application just as any other library would be, it does not incur the overhead of client/server communication. You interact with Berkeley DB XML using one of the supported language APIs. The primary one is C++, as the product is written in C++. Java, Python, Perl, PHP, and TCL are all supported API languages. Many other languages are supported by third parties and are readily available on the Internet.

With that in mind, let's next try some simple C++ code that calls for the same thing as the dbxml commands used in the previous example (see Listing 1).

The underlined sections of the code relate back to the first example. The first underlined section creates a container. This container is called 'test.dbxml,' and is on disk, and rather than in memory. The next underlined section places the same simple XML document into the new container. The last underlined section issues the query, and the while loop equates to the print statement. Put it all together and you have essentially the same result as before. In Java the code looks much the same; again the underlined areas are the common key sections of code (see Listing 2).

As you can see, the API is fairly straightforward and similar across languages. You'll note that the C++ and Java examples create a database on disk rather than in memory simply by giving the container a name.

Let's move back to the dbxml shell and try a more complicated example. This time let's add a few documents, so we can explore the performance of the system.

First let's populate an imaginary parts database.


dbxml> createContainer parts
Creating document storage container
That created an empty container and opened it as the default container in the shell. We can now use the putDocument command to run our XQuery and insert the sample data.

More Stories By Gregory Burd

Gregory Burd is the Product Manager for Sleepycat Software, now a part of Oracle. Prior to Sleepycat, he was on the business team at KnowNow, a Kleiner Perkins startup in the San Francisco Bay Area. He has many years of software development and product leadership within companies such as JavaSoft, a division of Sun Microsystems, Marble Associates, a consulting company, and NeXT Computer, now part of Apple Computer.

More Stories By Kimbro Staken

Kimbro Staken is an independent consultant, author, and open source developer specializing in technologies for XML data management. He is one of the primary developers of the dbXML Core Open Source native XML database and a cofounder of the XML:DB Initiative.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@ThingsExpo Stories
You think you know what’s in your data. But do you? Most organizations are now aware of the business intelligence represented by their data. Data science stands to take this to a level you never thought of – literally. The techniques of data science, when used with the capabilities of Big Data technologies, can make connections you had not yet imagined, helping you discover new insights and ask new questions of your data. In his session at @ThingsExpo, Sarbjit Sarkaria, data science team lead ...
Extracting business value from Internet of Things (IoT) data doesn’t happen overnight. There are several requirements that must be satisfied, including IoT device enablement, data analysis, real-time detection of complex events and automated orchestration of actions. Unfortunately, too many companies fall short in achieving their business goals by implementing incomplete solutions or not focusing on tangible use cases. In his general session at @ThingsExpo, Dave McCarthy, Director of Products...
"delaPlex is a software development company. We do team-based outsourcing development," explained Mark Rivers, COO and Co-founder of delaPlex Software, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
WebRTC is bringing significant change to the communications landscape that will bridge the worlds of web and telephony, making the Internet the new standard for communications. Cloud9 took the road less traveled and used WebRTC to create a downloadable enterprise-grade communications platform that is changing the communication dynamic in the financial sector. In his session at @ThingsExpo, Leo Papadopoulos, CTO of Cloud9, discussed the importance of WebRTC and how it enables companies to focus...
Is your aging software platform suffering from technical debt while the market changes and demands new solutions at a faster clip? It’s a bold move, but you might consider walking away from your core platform and starting fresh. ReadyTalk did exactly that. In his General Session at 19th Cloud Expo, Michael Chambliss, Head of Engineering at ReadyTalk, will discuss why and how ReadyTalk diverted from healthy revenue and over a decade of audio conferencing product development to start an innovati...
Early adopters of IoT viewed it mainly as a different term for machine-to-machine connectivity or M2M. This is understandable since a prerequisite for any IoT solution is the ability to collect and aggregate device data, which is most often presented in a dashboard. The problem is that viewing data in a dashboard requires a human to interpret the results and take manual action, which doesn’t scale to the needs of IoT.
SYS-CON Events announced today that 910Telecom will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Housed in the classic Denver Gas & Electric Building, 910 15th St., 910Telecom is a carrier-neutral telecom hotel located in the heart of Denver. Adjacent to CenturyLink, AT&T, and Denver Main, 910Telecom offers connectivity to all major carriers, Internet service providers, Internet backbones and ...
CenturyLink has announced that application server solutions from GENBAND are now available as part of CenturyLink’s Networx contracts. The General Services Administration (GSA)’s Networx program includes the largest telecommunications contract vehicles ever awarded by the federal government. CenturyLink recently secured an extension through spring 2020 of its offerings available to federal government agencies via GSA’s Networx Universal and Enterprise contracts. GENBAND’s EXPERiUS™ Application...
IoT generates lots of temporal data. But how do you unlock its value? You need to discover patterns that are repeatable in vast quantities of data, understand their meaning, and implement scalable monitoring across multiple data streams in order to monetize the discoveries and insights. Motif discovery and deep learning platforms are emerging to visualize sensor data, to search for patterns and to build application that can monitor real time streams efficiently. In his session at @ThingsExpo, ...
Verizon Communications Inc. (NYSE, Nasdaq: VZ) and Yahoo! Inc. (Nasdaq: YHOO) have entered into a definitive agreement under which Verizon will acquire Yahoo's operating business for approximately $4.83 billion in cash, subject to customary closing adjustments. Yahoo informs, connects and entertains a global audience of more than 1 billion monthly active users** -- including 600 million monthly active mobile users*** through its search, communications and digital content products. Yahoo also co...
"There's a growing demand from users for things to be faster. When you think about all the transactions or interactions users will have with your product and everything that is between those transactions and interactions - what drives us at Catchpoint Systems is the idea to measure that and to analyze it," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York Ci...
"Tintri was started in 2008 with the express purpose of building a storage appliance that is ideal for virtualized environments. We support a lot of different hypervisor platforms from VMware to OpenStack to Hyper-V," explained Dan Florea, Director of Product Management at Tintri, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
The best-practices for building IoT applications with Go Code that attendees can use to build their own IoT applications. In his session at @ThingsExpo, Indraneel Mitra, Senior Solutions Architect & Technology Evangelist at Cognizant, provided valuable information and resources for both novice and experienced developers on how to get started with IoT and Golang in a day. He also provided information on how to use Intel Arduino Kit, Go Robotics API and AWS IoT stack to build an application tha...
SYS-CON Events announced today that LeaseWeb USA, a cloud Infrastructure-as-a-Service (IaaS) provider, will exhibit at the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. LeaseWeb is one of the world's largest hosting brands. The company helps customers define, develop and deploy IT infrastructure tailored to their exact business needs, by combining various kinds cloud solutions.
Whether your IoT service is connecting cars, homes, appliances, wearable, cameras or other devices, one question hangs in the balance – how do you actually make money from this service? The ability to turn your IoT service into profit requires the ability to create a monetization strategy that is flexible, scalable and working for you in real-time. It must be a transparent, smoothly implemented strategy that all stakeholders – from customers to the board – will be able to understand and comprehe...
The cloud market growth today is largely in public clouds. While there is a lot of spend in IT departments in virtualization, these aren’t yet translating into a true “cloud” experience within the enterprise. What is stopping the growth of the “private cloud” market? In his general session at 18th Cloud Expo, Nara Rajagopalan, CEO of Accelerite, explored the challenges in deploying, managing, and getting adoption for a private cloud within an enterprise. What are the key differences between wh...
SYS-CON Events announced today that Venafi, the Immune System for the Internet™ and the leading provider of Next Generation Trust Protection, will exhibit at @DevOpsSummit at 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Venafi is the Immune System for the Internet™ that protects the foundation of all cybersecurity – cryptographic keys and digital certificates – so they can’t be misused by bad guys in attacks...
Large scale deployments present unique planning challenges, system commissioning hurdles between IT and OT and demand careful system hand-off orchestration. In his session at @ThingsExpo, Jeff Smith, Senior Director and a founding member of Incenergy, will discuss some of the key tactics to ensure delivery success based on his experience of the last two years deploying Industrial IoT systems across four continents.
There will be new vendors providing applications, middleware, and connected devices to support the thriving IoT ecosystem. This essentially means that electronic device manufacturers will also be in the software business. Many will be new to building embedded software or robust software. This creates an increased importance on software quality, particularly within the Industrial Internet of Things where business-critical applications are becoming dependent on products controlled by software. Qua...
SYS-CON Events has announced today that Roger Strukhoff has been named conference chair of Cloud Expo and @ThingsExpo 2016 Silicon Valley. The 19th Cloud Expo and 6th @ThingsExpo will take place on November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. "The Internet of Things brings trillions of dollars of opportunity to developers and enterprise IT, no matter how you measure it," stated Roger Strukhoff. "More importantly, it leverages the power of devices and the Interne...