Enable NCR if XML declaration uses version 1.1

[RedMser]

• Are you running the latest version?
• Have you included sample input, output, error, and expected output?
• Have you checked if you are using correct configuration?
• Did you try online tool?
• Have you checked the docs for helpful APIs and examples?

Description

Currently the <?xml ... ?> prolog declaration is not used by the parser (see #774).
This is a problem because certain control characters can only be escaped in XML 1.1 by using numeric character references (NCR). In version 1.0 this is invalid XML.

I can set htmlEntities: true to allow parsing NCR, but that also automatically parses other HTML entities which is non-conformant and undesired.

I'd also need to manually extract the version first, before parsing the XML string, which I believe is something that the library should do for me.

My proposal:

• A new config flag parseDeclaration (default to true), which reads the XML declaration if it exists, and uses it to automatically set some parser options.
• New possible value for htmlEntities: 'ncr' which only enables the NCR parsing. This is what'll be used by parseDeclaration when it encounters version 1.1 XML.

Input

<?xml version="1.1" encoding="UTF-8" ?>
<root>&#x1B;&#27;</root>

Code

const xml = `...`;
const fxp = require('fast-xml-parser');
const parser = new fxp.XMLParser({
    ignoreAttributes: false,
});
const result = parser.parse(xml);
console.log(result);

Output

{
  '?xml': { '@_version': '1.1', '@_encoding': 'UTF-8' },
  root: ''
}

expected data

{
  '?xml': { '@_version': '1.1', '@_encoding': 'UTF-8' },
  root: '\x1B\x1B'
}

Example in Java, change version="1.1" to "1.0" and you'll see a parser error.

Would you like to work on this issue?

• Yes
• No

[amitguptagwl]

This would be handled in v5.7.0. I would push the code to Github today and will try to release tomorrow

[amitguptagwl]

changes are available on Github

[RedMser]

Thanks @amitguptagwl for the quick update! I tried the latest master locally and noticed some problems with the implementation:

• fast-xml-parser/src/xmlparser/OrderedObjParser.js

  Line 343 in 869ec8b

  if (attsMap && attsMap["version"]) {

  This branch is not taken if you are using a attributeNamePrefix because this code checks only for "version" and not "@_version".
• NCR control codes are parsed, but are not returned in the result. Also when I set XML version 1.0, it still silently removes the NCR despite \x1B being marked as unsupported in XML 1.0.

const xml = `
<?xml version="1.1" encoding="UTF-8" ?>
<root>abc&#25;def</root>
`;

const fxp = require('./fast-xml-parser-master/src/fxp.js');
const parser = new fxp.XMLParser({
    ignoreAttributes: false,
    attributeNamePrefix: "",
});
const result = parser.parse(xml);
console.log(result);
console.log(result.root.length);

// console.log output:
{ '?xml': { version: '1.1', encoding: 'UTF-8' }, root: 'abcdef' }
6

// I expect \x1B in the string, and length to be 7

Kind regards.

[amitguptagwl]

Thanks @RedMser for testing it before release.

FXP provides basic features of @nodable/entities. You would have to set EntityDecoder for advance features.

https://github.com/NaturalIntelligence/fast-xml-parser/blob/master/docs/v4%2C%20v5/2.XMLparseOptions.md#entitydecoder

Let me know if it helps and solves the issue. YOU basically need to set ncr config.

const entityDecoder = new EntityDecoder({
            namedEntities: COMMON_HTML,
            ncr: { onNCR: 'leave' }
        });
        const xmlData = `<?xml version="1.0" encoding="UTF-8"?>
        <note>&unknown;'&#60;'</note> `;
        const expected = {
            "?xml": {
                "version": "1.0",
                "encoding": "UTF-8"
            },
            "note": "&unknown;'&#60;'"
        }
        const options = {
            attributeNamePrefix: "",
            ignoreAttributes: false,
            processEntities: true,
            htmlEntities: false, //doesn't matter when entity decoder is set
            entityDecoder: entityDecoder,
        };
        const parser = new XMLParser(options);
        let result = parser.parse(xmlData);

[RedMser]

Sorry but I'm a bit confused still. If I don't specify a custom entityDecoder, I understand that FXP uses @nodable/entities by default.
And the code here
https://github.com/nodable/val-parsers/blob/05d5c84488533916132a9f82a594d8d96ef5f509/Entity/src/EntityDecoder.js#L166
says that ncr should default to "allow". So why don't they get converted in my code snippet, but are removed? Does FXP pass other default values? Or do I misunderstand this config flag? Maybe it needs more documentation on the FXP side, what the default behavior is, and link to the entities lib docs on how to reconfigure it.

Also can you confirm that if I don't specify attributeNamePrefix (so it uses @_ as default), the XML declaration "version" detection code is currently broken? If you like, I can create a PR to fix this + add tests.

[amitguptagwl]

@RedMser Yes I feel necessary documentation is needed. and there are some issues in setting version too. Let me fix that. I'll let you know. appreciate you tested it before release.

[amitguptagwl]

Issue was in setting xml version. Here are the tests. Please check if we both have common understanding.

fit("should remove '#x1B' ncr as per XML version 1.0 with default config", function () {

        const xmlData = `<?xml version="1.0" ?><root>&#x1B;&#27;</root>`;
        const options = {
            //ignoreAttributes: false, //should work without this
            processEntities: true,
            // htmlEntities: true
        };
        const parser = new XMLParser(options);
        let result = parser.parse(xmlData);
        // console.log(JSON.stringify(result, null, 4));

        expect(result.root.length).toEqual(0);
    })
    fit("should leave '#x1B' ncr as per XML version 1.1 with default config", function () {

        const xmlData = `<?xml version="1.1" ?><root>&#x1B;&#27;</root>`;
        //const expected = {
        //    "root": '&#x1B;&#27;'
        //}
        const options = {
            //ignoreAttributes: false, //should work without this
            processEntities: true
        };
        const parser = new XMLParser(options);
        let result = parser.parse(xmlData);
        // console.log(JSON.stringify(result, null, 4));

        expect(result.root.length).toEqual(11);
    })

I've also made setting ignoreAttributesoptional to read xml version. docs are added. Please note that if you set EntityDecoder in config explicitly then you'll get 10% better performance. You can also choose the set of entities you need to improve the performance further.

[amitguptagwl]

feelfree to reopen or further discuss on this

[RedMser]

Thank you very much, I tried the latest version and it works just like you describe. I'll be sure to configure this accordingly in my projects.

I also made a PR on @nodable/entities to fix some type errors I got trying to import the EntityDecoder class.