Table of Links
Abstract and 1 Introduction
2 Background
3 Approach and 3.1 Differential Testing for XML Processors
3.2 XPath Expression Generation
3.3 XML Generation
4 Evaluation
4.1 Effectiveness
4.2 Efficiency
4.3 Comparison to the State of the Art
4.4 Analysis of BaseX Historical Bug Reports
5 Related Work
6 Conclusion, Acknowledgments, and References
4 EVALUATION
In the evaluation, we sought to investigate whether our technique is effective and efficient in finding bugs for XPath expression processors. Specifically, we were interested in the following questions:
Q1. Is XPress effective in finding new XPath-related bugs in established XML processors (see Section 4.1)?
Q2. Does the query generation approach described in Section 3.2 improve the bug-finding efficiency of XPress with respect to real-world baselines and a random generation approach (see Section 4.2)?
Q3. How does the differential testing test oracle compare to the state-of-the-art oracle (see Section 4.3)?
Q4. What kind of XPath-related bugs might be overlooked by XPress (see Section 4.4)?
Tested XML Processors. We tested our method on six mature, well-known, and actively maintained XPath processors: BaseX, exist-DB, Saxon-HE, PostgreSQL, libXML2, and a commercial DBMS, whose name we have omitted due to its “DeWitt clause” [24]. We started testing on BaseX 10.4, eXist-DB 6.2.0, Saxon Home Edition 12.2, PostgreSQL version 15, and libXML2 commit version 106153. As bugs were resolved, we constantly updated to the latest available version. We selected BaseX, eXist-DB, and Saxon to be our main testing targets, because they all implement the more recent XPath 3.0 standard. BaseX ranks as the most popular Native XML DBMS on the DB-Engines Ranking [9]. eXist-DB is widely applied in data centers, systems, and platforms, as referenced on the eXist-DB reference page [11]. Saxon is an in-memory processor and therefore is not included in the DB-Engines rankings. However, the official website of Saxon [16] states: “More than 170 software vendors have built Saxon into their own applications” and “6 of the world’s top 10 software vendors are Saxonica clients”, demonstrating that Saxon is a widely-used and popular XML processor. For XPath 1.0 standard implementations, we tested PostgreSQL, libXML2, and the commercial DBMS. PostgreSQL is a popular open-source DBMS, which ranks 4 on the DB-Engines ranking and has 12.8k stars on GitHub. libXML2 is a software library developed for the GNOME project. The commercial DBMS is often considered the most popular and important DBMS overall, as also reflected in various rankings. All XML processors have been actively maintained for over 15 years.
Experimental setup. We implemented the tool, XPress, in around 8,000 LOC in Java. In our experiments, we configured it to generate XML documents that contain 1 to 50 nodes. We create half as many node templates as element nodes. For each XML document, we generated 200 XPath expressions. Each XPath expression had an equal possibility to hold 1 to 7 sections. We set one predicate to hold at most 10 subjects (see Section 3.2) and the depth of the predicate tree to be at most 10. We used the default settings of each XML processor. We conducted all our experiments using a personal computer with a 64-Core AMD EPYC 7763 CPU at 2.45GHz and 512GB memory running Ubuntu 22.04.
Authors:
(1) Shuxin Li, Southern University of Science and Technology China and Work done during an internship at the National University of Singapore ([email protected]);
(2) Manuel Rigger, National University of Singapore Singapore ([email protected]).
