Annotation of embedaddon/libxml2/doc/xmlreader.html, revision 1.1.1.1
1.1 misho 1: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
2: "http://www.w3.org/TR/html4/loose.dtd">
3: <html>
4: <head>
5: <meta http-equiv="Content-Type" content="text/html">
6: <style type="text/css"></style>
7: <!--
8: TD {font-family: Verdana,Arial,Helvetica}
9: BODY {font-family: Verdana,Arial,Helvetica; margin-top: 2em; margin-left: 0em; margin-right: 0em}
10: H1 {font-family: Verdana,Arial,Helvetica}
11: H2 {font-family: Verdana,Arial,Helvetica}
12: H3 {font-family: Verdana,Arial,Helvetica}
13: A:link, A:visited, A:active { text-decoration: underline }
14: </style>
15: -->
16: <title>Libxml2 XmlTextReader Interface tutorial</title>
17: </head>
18:
19: <body bgcolor="#fffacd" text="#000000">
20: <h1 align="center">Libxml2 XmlTextReader Interface tutorial</h1>
21:
22: <p></p>
23:
24: <p>This document describes the use of the XmlTextReader streaming API added
25: to libxml2 in version 2.5.0 . This API is closely modeled after the <a
26: href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">XmlTextReader</a>
27: and <a
28: href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlReader.html">XmlReader</a>
29: classes of the C# language.</p>
30:
31: <p>This tutorial will present the key points of this API, and working
32: examples using both C and the Python bindings:</p>
33:
34: <p>Table of content:</p>
35: <ul>
36: <li><a href="#Introducti">Introduction: why a new API</a></li>
37: <li><a href="#Walking">Walking a simple tree</a></li>
38: <li><a href="#Extracting">Extracting informations for the current
39: node</a></li>
40: <li><a href="#Extracting1">Extracting informations for the
41: attributes</a></li>
42: <li><a href="#Validating">Validating a document</a></li>
43: <li><a href="#Entities">Entities substitution</a></li>
44: <li><a href="#L1142">Relax-NG Validation</a></li>
45: <li><a href="#Mixing">Mixing the reader and tree or XPath
46: operations</a></li>
47: </ul>
48:
49: <p></p>
50:
51: <h2><a name="Introducti">Introduction: why a new API</a></h2>
52:
53: <p>Libxml2 <a href="http://xmlsoft.org/html/libxml-tree.html">main API is
54: tree based</a>, where the parsing operation results in a document loaded
55: completely in memory, and expose it as a tree of nodes all availble at the
56: same time. This is very simple and quite powerful, but has the major
57: limitation that the size of the document that can be hamdled is limited by
58: the size of the memory available. Libxml2 also provide a <a
59: href="http://www.saxproject.org/">SAX</a> based API, but that version was
60: designed upon one of the early <a
61: href="http://www.jclark.com/xml/expat.html">expat</a> version of SAX, SAX is
62: also not formally defined for C. SAX basically work by registering callbacks
63: which are called directly by the parser as it progresses through the document
64: streams. The problem is that this programming model is relatively complex,
65: not well standardized, cannot provide validation directly, makes entity,
66: namespace and base processing relatively hard.</p>
67:
68: <p>The <a
69: href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">XmlTextReader
70: API from C#</a> provides a far simpler programming model. The API acts as a
71: cursor going forward on the document stream and stopping at each node in the
72: way. The user's code keeps control of the progress and simply calls a
73: Read() function repeatedly to progress to each node in sequence in document
74: order. There is direct support for namespaces, xml:base, entity handling and
75: adding DTD validation on top of it was relatively simple. This API is really
76: close to the <a href="http://www.w3.org/TR/DOM-Level-2-Core/">DOM Core
77: specification</a> This provides a far more standard, easy to use and powerful
78: API than the existing SAX. Moreover integrating extension features based on
79: the tree seems relatively easy.</p>
80:
81: <p>In a nutshell the XmlTextReader API provides a simpler, more standard and
82: more extensible interface to handle large documents than the existing SAX
83: version.</p>
84:
85: <h2><a name="Walking">Walking a simple tree</a></h2>
86:
87: <p>Basically the XmlTextReader API is a forward only tree walking interface.
88: The basic steps are:</p>
89: <ol>
90: <li>prepare a reader context operating on some input</li>
91: <li>run a loop iterating over all nodes in the document</li>
92: <li>free up the reader context</li>
93: </ol>
94:
95: <p>Here is a basic C sample doing this:</p>
96: <pre>#include <libxml/xmlreader.h>
97:
98: void processNode(xmlTextReaderPtr reader) {
99: /* handling of a node in the tree */
100: }
101:
102: int streamFile(char *filename) {
103: xmlTextReaderPtr reader;
104: int ret;
105:
106: reader = xmlNewTextReaderFilename(filename);
107: if (reader != NULL) {
108: ret = xmlTextReaderRead(reader);
109: while (ret == 1) {
110: processNode(reader);
111: ret = xmlTextReaderRead(reader);
112: }
113: xmlFreeTextReader(reader);
114: if (ret != 0) {
115: printf("%s : failed to parse\n", filename);
116: }
117: } else {
118: printf("Unable to open %s\n", filename);
119: }
120: }</pre>
121:
122: <p>A few things to notice:</p>
123: <ul>
124: <li>the include file needed : <code>libxml/xmlreader.h</code></li>
125: <li>the creation of the reader using a filename</li>
126: <li>the repeated call to xmlTextReaderRead() and how any return value
127: different from 1 should stop the loop</li>
128: <li>that a negative return means a parsing error</li>
129: <li>how xmlFreeTextReader() should be used to free up the resources used by
130: the reader.</li>
131: </ul>
132:
133: <p>Here is similar code in python for exactly the same processing:</p>
134: <pre>import libxml2
135:
136: def processNode(reader):
137: pass
138:
139: def streamFile(filename):
140: try:
141: reader = libxml2.newTextReaderFilename(filename)
142: except:
143: print "unable to open %s" % (filename)
144: return
145:
146: ret = reader.Read()
147: while ret == 1:
148: processNode(reader)
149: ret = reader.Read()
150:
151: if ret != 0:
152: print "%s : failed to parse" % (filename)</pre>
153:
154: <p>The only things worth adding are that the <a
155: href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">xmlTextReader
156: is abstracted as a class like in C#</a> with the same method names (but the
157: properties are currently accessed with methods) and that one doesn't need to
158: free the reader at the end of the processing. It will get garbage collected
159: once all references have disapeared.</p>
160:
161: <h2><a name="Extracting">Extracting information for the current node</a></h2>
162:
163: <p>So far the example code did not indicate how information was extracted
164: from the reader. It was abstrated as a call to the processNode() routine,
165: with the reader as the argument. At each invocation, the parser is stopped on
166: a given node and the reader can be used to query those node properties. Each
167: <em>Property</em> is available at the C level as a function taking a single
168: xmlTextReaderPtr argument whose name is
169: <code>xmlTextReader</code><em>Property</em> , if the return type is an
170: <code>xmlChar *</code> string then it must be deallocated with
171: <code>xmlFree()</code> to avoid leaks. For the Python interface, there is a
172: <em>Property</em> method to the reader class that can be called on the
173: instance. The list of the properties is based on the <a
174: href="http://dotgnu.org/pnetlib-doc/System/Xml/XmlTextReader.html">C#
175: XmlTextReader class</a> set of properties and methods:</p>
176: <ul>
177: <li><em>NodeType</em>: The node type, 1 for start element, 15 for end of
178: element, 2 for attributes, 3 for text nodes, 4 for CData sections, 5 for
179: entity references, 6 for entity declarations, 7 for PIs, 8 for comments,
180: 9 for the document nodes, 10 for DTD/Doctype nodes, 11 for document
181: fragment and 12 for notation nodes.</li>
182: <li><em>Name</em>: the <a
183: href="http://www.w3.org/TR/REC-xml-names/#ns-qualnames">qualified
184: name</a> of the node, equal to (<em>Prefix</em>:)<em>LocalName</em>.</li>
185: <li><em>LocalName</em>: the <a
186: href="http://www.w3.org/TR/REC-xml-names/#NT-LocalPart">local name</a> of
187: the node.</li>
188: <li><em>Prefix</em>: a shorthand reference to the <a
189: href="http://www.w3.org/TR/REC-xml-names/">namespace</a> associated with
190: the node.</li>
191: <li><em>NamespaceUri</em>: the URI defining the <a
192: href="http://www.w3.org/TR/REC-xml-names/">namespace</a> associated with
193: the node.</li>
194: <li><em>BaseUri:</em> the base URI of the node. See the <a
195: href="http://www.w3.org/TR/xmlbase/">XML Base W3C specification</a>.</li>
196: <li><em>Depth:</em> the depth of the node in the tree, starts at 0 for the
197: root node.</li>
198: <li><em>HasAttributes</em>: whether the node has attributes.</li>
199: <li><em>HasValue</em>: whether the node can have a text value.</li>
200: <li><em>Value</em>: provides the text value of the node if present.</li>
201: <li><em>IsDefault</em>: whether an Attribute node was generated from the
202: default value defined in the DTD or schema (<em>unsupported
203: yet</em>).</li>
204: <li><em>XmlLang</em>: the <a
205: href="http://www.w3.org/TR/REC-xml#sec-lang-tag">xml:lang</a> scope
206: within which the node resides.</li>
207: <li><em>IsEmptyElement</em>: check if the current node is empty, this is a
208: bit bizarre in the sense that <code><a/></code> will be considered
209: empty while <code><a></a></code> will not.</li>
210: <li><em>AttributeCount</em>: provides the number of attributes of the
211: current node.</li>
212: </ul>
213:
214: <p>Let's look first at a small example to get this in practice by redefining
215: the processNode() function in the Python example:</p>
216: <pre>def processNode(reader):
217: print "%d %d %s %d" % (reader.Depth(), reader.NodeType(),
218: reader.Name(), reader.IsEmptyElement())</pre>
219:
220: <p>and look at the result of calling streamFile("tst.xml") for various
221: content of the XML test file.</p>
222:
223: <p>For the minimal document "<code><doc/></code>" we get:</p>
224: <pre>0 1 doc 1</pre>
225:
226: <p>Only one node is found, its depth is 0, type 1 indicate an element start,
227: of name "doc" and it is empty. Trying now with
228: "<code><doc></doc></code>" instead leads to:</p>
229: <pre>0 1 doc 0
230: 0 15 doc 0</pre>
231:
232: <p>The document root node is not flagged as empty anymore and both a start
233: and an end of element are detected. The following document shows how
234: character data are reported:</p>
235: <pre><doc><a/><b>some text</b>
236: <c/></doc></pre>
237:
238: <p>We modifying the processNode() function to also report the node Value:</p>
239: <pre>def processNode(reader):
240: print "%d %d %s %d %s" % (reader.Depth(), reader.NodeType(),
241: reader.Name(), reader.IsEmptyElement(),
242: reader.Value())</pre>
243:
244: <p>The result of the test is:</p>
245: <pre>0 1 doc 0 None
246: 1 1 a 1 None
247: 1 1 b 0 None
248: 2 3 #text 0 some text
249: 1 15 b 0 None
250: 1 3 #text 0
251:
252: 1 1 c 1 None
253: 0 15 doc 0 None</pre>
254:
255: <p>There are a few things to note:</p>
256: <ul>
257: <li>the increase of the depth value (first row) as children nodes are
258: explored</li>
259: <li>the text node child of the b element, of type 3 and its content</li>
260: <li>the text node containing the line return between elements b and c</li>
261: <li>that elements have the Value None (or NULL in C)</li>
262: </ul>
263:
264: <p>The equivalent routine for <code>processNode()</code> as used by
265: <code>xmllint --stream --debug</code> is the following and can be found in
266: the xmllint.c module in the source distribution:</p>
267: <pre>static void processNode(xmlTextReaderPtr reader) {
268: xmlChar *name, *value;
269:
270: name = xmlTextReaderName(reader);
271: if (name == NULL)
272: name = xmlStrdup(BAD_CAST "--");
273: value = xmlTextReaderValue(reader);
274:
275: printf("%d %d %s %d",
276: xmlTextReaderDepth(reader),
277: xmlTextReaderNodeType(reader),
278: name,
279: xmlTextReaderIsEmptyElement(reader));
280: xmlFree(name);
281: if (value == NULL)
282: printf("\n");
283: else {
284: printf(" %s\n", value);
285: xmlFree(value);
286: }
287: }</pre>
288:
289: <h2><a name="Extracting1">Extracting information for the attributes</a></h2>
290:
291: <p>The previous examples don't indicate how attributes are processed. The
292: simple test "<code><doc a="b"/></code>" provides the following
293: result:</p>
294: <pre>0 1 doc 1 None</pre>
295:
296: <p>This proves that attribute nodes are not traversed by default. The
297: <em>HasAttributes</em> property allow to detect their presence. To check
298: their content the API has special instructions. Basically two kinds of operations
299: are possible:</p>
300: <ol>
301: <li>to move the reader to the attribute nodes of the current element, in
302: that case the cursor is positionned on the attribute node</li>
303: <li>to directly query the element node for the attribute value</li>
304: </ol>
305:
306: <p>In both case the attribute can be designed either by its position in the
307: list of attribute (<em>MoveToAttributeNo</em> or <em>GetAttributeNo</em>) or
308: by their name (and namespace):</p>
309: <ul>
310: <li><em>GetAttributeNo</em>(no): provides the value of the attribute with
311: the specified index no relative to the containing element.</li>
312: <li><em>GetAttribute</em>(name): provides the value of the attribute with
313: the specified qualified name.</li>
314: <li>GetAttributeNs(localName, namespaceURI): provides the value of the
315: attribute with the specified local name and namespace URI.</li>
316: <li><em>MoveToAttributeNo</em>(no): moves the position of the current
317: instance to the attribute with the specified index relative to the
318: containing element.</li>
319: <li><em>MoveToAttribute</em>(name): moves the position of the current
320: instance to the attribute with the specified qualified name.</li>
321: <li><em>MoveToAttributeNs</em>(localName, namespaceURI): moves the position
322: of the current instance to the attribute with the specified local name
323: and namespace URI.</li>
324: <li><em>MoveToFirstAttribute</em>: moves the position of the current
325: instance to the first attribute associated with the current node.</li>
326: <li><em>MoveToNextAttribute</em>: moves the position of the current
327: instance to the next attribute associated with the current node.</li>
328: <li><em>MoveToElement</em>: moves the position of the current instance to
329: the node that contains the current Attribute node.</li>
330: </ul>
331:
332: <p>After modifying the processNode() function to show attributes:</p>
333: <pre>def processNode(reader):
334: print "%d %d %s %d %s" % (reader.Depth(), reader.NodeType(),
335: reader.Name(), reader.IsEmptyElement(),
336: reader.Value())
337: if reader.NodeType() == 1: # Element
338: while reader.MoveToNextAttribute():
339: print "-- %d %d (%s) [%s]" % (reader.Depth(), reader.NodeType(),
340: reader.Name(),reader.Value())</pre>
341:
342: <p>The output for the same input document reflects the attribute:</p>
343: <pre>0 1 doc 1 None
344: -- 1 2 (a) [b]</pre>
345:
346: <p>There are a couple of things to note on the attribute processing:</p>
347: <ul>
348: <li>Their depth is the one of the carrying element plus one.</li>
349: <li>Namespace declarations are seen as attributes, as in DOM.</li>
350: </ul>
351:
352: <h2><a name="Validating">Validating a document</a></h2>
353:
354: <p>Libxml2 implementation adds some extra features on top of the XmlTextReader
355: API. The main one is the ability to DTD validate the parsed document
356: progressively. This is simply the activation of the associated feature of the
357: parser used by the reader structure. There are a few options available
358: defined as the enum xmlParserProperties in the libxml/xmlreader.h header
359: file:</p>
360: <ul>
361: <li>XML_PARSER_LOADDTD: force loading the DTD (without validating)</li>
362: <li>XML_PARSER_DEFAULTATTRS: force attribute defaulting (this also imply
363: loading the DTD)</li>
364: <li>XML_PARSER_VALIDATE: activate DTD validation (this also imply loading
365: the DTD)</li>
366: <li>XML_PARSER_SUBST_ENTITIES: substitute entities on the fly, entity
367: reference nodes are not generated and are replaced by their expanded
368: content.</li>
369: <li>more settings might be added, those were the one available at the 2.5.0
370: release...</li>
371: </ul>
372:
373: <p>The GetParserProp() and SetParserProp() methods can then be used to get
374: and set the values of those parser properties of the reader. For example</p>
375: <pre>def parseAndValidate(file):
376: reader = libxml2.newTextReaderFilename(file)
377: reader.SetParserProp(libxml2.PARSER_VALIDATE, 1)
378: ret = reader.Read()
379: while ret == 1:
380: ret = reader.Read()
381: if ret != 0:
382: print "Error parsing and validating %s" % (file)</pre>
383:
384: <p>This routine will parse and validate the file. Error messages can be
385: captured by registering an error handler. See python/tests/reader2.py for
386: more complete Python examples. At the C level the equivalent call to cativate
387: the validation feature is just:</p>
388: <pre>ret = xmlTextReaderSetParserProp(reader, XML_PARSER_VALIDATE, 1)</pre>
389:
390: <p>and a return value of 0 indicates success.</p>
391:
392: <h2><a name="Entities">Entities substitution</a></h2>
393:
394: <p>By default the xmlReader will report entities as such and not replace them
395: with their content. This default behaviour can however be overriden using:</p>
396:
397: <p><code>reader.SetParserProp(libxml2.PARSER_SUBST_ENTITIES,1)</code></p>
398:
399: <h2><a name="L1142">Relax-NG Validation</a></h2>
400:
401: <p style="font-size: 10pt">Introduced in version 2.5.7</p>
402:
403: <p>Libxml2 can now validate the document being read using the xmlReader using
404: Relax-NG schemas. While the Relax NG validator can't always work in a
405: streamable mode, only subsets which cannot be reduced to regular expressions
406: need to have their subtree expanded for validation. In practice it means
407: that, unless the schemas for the top level element content is not expressable
408: as a regexp, only chunk of the document needs to be parsed while
409: validating.</p>
410:
411: <p>The steps to do so are:</p>
412: <ul>
413: <li>create a reader working on a document as usual</li>
414: <li>before any call to read associate it to a Relax NG schemas, either the
415: preparsed schemas or the URL to the schemas to use</li>
416: <li>errors will be reported the usual way, and the validity status can be
417: obtained using the IsValid() interface of the reader like for DTDs.</li>
418: </ul>
419:
420: <p>Example, assuming the reader has already being created and that the schema
421: string contains the Relax-NG schemas:</p>
422: <pre><code>rngp = libxml2.relaxNGNewMemParserCtxt(schema, len(schema))<br>
423: rngs = rngp.relaxNGParse()<br>
424: reader.RelaxNGSetSchema(rngs)<br>
425: ret = reader.Read()<br>
426: while ret == 1:<br>
427: ret = reader.Read()<br>
428: if ret != 0:<br>
429: print "Error parsing the document"<br>
430: if reader.IsValid() != 1:<br>
431: print "Document failed to validate"</code><br>
432: </pre>
433:
434: <p>See <code>reader6.py</code> in the sources or documentation for a complete
435: example.</p>
436:
437: <h2><a name="Mixing">Mixing the reader and tree or XPath operations</a></h2>
438:
439: <p style="font-size: 10pt">Introduced in version 2.5.7</p>
440:
441: <p>While the reader is a streaming interface, its underlying implementation
442: is based on the DOM builder of libxml2. As a result it is relatively simple
443: to mix operations based on both models under some constraints. To do so the
444: reader has an Expand() operation allowing to grow the subtree under the
445: current node. It returns a pointer to a standard node which can be
446: manipulated in the usual ways. The node will get all its ancestors and the
447: full subtree available. Usual operations like XPath queries can be used on
448: that reduced view of the document. Here is an example extracted from
449: reader5.py in the sources which extract and prints the bibliography for the
450: "Dragon" compiler book from the XML 1.0 recommendation:</p>
451: <pre>f = open('../../test/valid/REC-xml-19980210.xml')
452: input = libxml2.inputBuffer(f)
453: reader = input.newTextReader("REC")
454: res=""
455: while reader.Read():
456: while reader.Name() == 'bibl':
457: node = reader.Expand() # expand the subtree
458: if node.xpathEval("@id = 'Aho'"): # use XPath on it
459: res = res + node.serialize()
460: if reader.Next() != 1: # skip the subtree
461: break;</pre>
462:
463: <p>Note, however that the node instance returned by the Expand() call is only
464: valid until the next Read() operation. The Expand() operation does not
465: affects the Read() ones, however usually once processed the full subtree is
466: not useful anymore, and the Next() operation allows to skip it completely and
467: process to the successor or return 0 if the document end is reached.</p>
468:
469: <p><a href="mailto:xml@gnome.org">Daniel Veillard</a></p>
470:
471: <p>$Id$</p>
472:
473: <p></p>
474: </body>
475: </html>
FreeBSD-CVSweb <freebsd-cvsweb@FreeBSD.org>
![[BACK]](/icons/cvsweb/back.gif){kind=icon}
Return to xmlreader.html CVS log ![[TXT]](/icons/cvsweb/text.gif){kind=icon}
![[DIR]](/icons/cvsweb/dir.gif){kind=icon}
Up to [ELWIX - Embedded LightWeight unIX -] / embedaddon / libxml2 / doc