JRuby fails newline tests on Windows

[headius]

The JRuby builds on Windows have been disabled in CI since #521. The extension appears to build fine now on Windows, but fails a few tests that check for Windows newlines. The failures are in tests test_generate_custom, test_generate_pretty, and test_generate_pretty and look generally like this:

Failure: test_generate_custom(JSONGeneratorTest)
D:/a/json/json/test/json/json_generator_test.rb:144:in 'test_generate_custom'
     141:   def test_generate_custom
     142:     state = State.new(:space_before => " ", :space => "   ", :indent => "<i>", :object_nl => "\n", :array_nl => "<a_nl>")
     143:     json = generate({1=>{2=>3,4=>[5,6]}}, state)
  => 144:     assert_equal(<<~'JSON'.chomp, json)
     145:       {
     146:       <i>"1" :   {
     147:       <i><i>"2" :   3,
org/jruby/RubyKernel.java:1397:in 'catch'
org/jruby/RubyKernel.java:1392:in 'catch'
org/jruby/RubyArray.java:2088:in 'each'
org/jruby/RubyArray.java:2088:in 'each'
org/jruby/RubyKernel.java:1397:in 'catch'
org/jruby/RubyKernel.java:1392:in 'catch'
<"{\r\n" +
"<i>\"1\" :   {\r\n" +
"<i><i>\"2\" :   3,\r\n" +
"<i><i>\"4\" :   [<a_nl><i><i><i>5,<a_nl><i><i><i>6<a_nl><i><i>]\r\n" +
"<i>}\r\n" +
"}"> expected but was
<"{\n" +
"<i>\"1\" :   {\n" +
"<i><i>\"2\" :   3,\n" +
"<i><i>\"4\" :   [<a_nl><i><i><i>5,<a_nl><i><i><i>6<a_nl><i><i>]\n" +
"<i>}\n" +
"}">

We should fix the dumping to emit proper line terminators.

[headius]

Short repro script based on test_generate_custom:

require 'json'
state = JSON::State.new(:space_before => " ", :space => "   ", :indent => "<i>", :object_nl => "\n", :array_nl => "<a_nl>")
p state.generate({1=>{2=>3,4=>[5,6]}})

I'm looking into it a little bit but it's hard to figure out at the moment without a Windows box for testing.

[danini-the-panini]

Ok, it looks like this is happening if you clone on windows with the option for git to replace \n with \r\n, the heredoc gets \r\n newlines and thus fails.

CRuby appears to convert \r\n to \n when reading a file, hence why it passes.

[headius]

@danini-the-panini Good sleuthing! So where are these \r coming from exactly? The reproduction I provided should only be dumping code... how are \r characters getting into that?

[danini-the-panini]

Your repro doesn't output any \r

"{\n<i>\"1\" :   {\n<i><i>\"2\" :   3,\n<i><i>\"4\" :   [<a_nl><i><i><i>5,<a_nl><i><i><i>6<a_nl><i><i>]\n<i>}\n}"

the issue is that the test file has \r\n in it. the generate method is correctly outputting only \n but the HEREDOC has \r\n on JRuby on Window, hence the failure. (You can see that the expected output is the one with the CRLF)

[danini-the-panini]

There seems to be an issue logged for this already: jruby/jruby#3742

[headius]

There seems to be an issue logged for this already: jruby/jruby#3742

Yeah that's very likely about the same problem. Perhaps it's time to fix it.

[headius]

Your repro doesn't output any \r

So basically my repro does show the problem... we're not emitting Windows line terminators in some case used by the json dumping logic.

I'm assuming that the CRuby CI is checking out the code exactly the same way (with Windows line terminators) so that's not really the bug. The bug is that we don't end up matching those line terminators.

[headius]

I tried looking into this but I was unable to get ragel to work properly on Windows to generate the JRuby parsers. Confusingly, the CI jobs don't appear to run ragel at all.

[headius]

I managed to force ragel to run sufficiently to get the build to work, and I've narrowed down the problem.

I believe it's an issue in JRuby's parser, whereby we do not convert \r\n to \n. The outputting of \n for newlines in json is correct in the JRuby extension.

Source, and this file has \r\n line terminators:

#!/usr/bin/env ruby
# frozen_string_literal: true

expected = <<~'JSON'.chomp
  {
  <i>"1" :   {
  <i><i>"2" :   3,
  <i><i>"4" :   [<a_nl><i><i><i>5,<a_nl><i><i><i>6<a_nl><i><i>]
  <i>}
  }
JSON
p expected

Contents of the file as an inspected Ruby string (read as binary to avoid newline conversion):

"#!/usr/bin/env ruby\r\n# frozen_string_literal: true\r\n\r\nexpected = <<~'JSON'.chomp\r\n  {\r\n  <i>\"1\" :   {\r\n  <i><i>\"2\" :   3,\r\n  <i><i>\"4\" :   [<a_nl><i><i><i>5,<a_nl><i><i><i>6<a_nl><i><i>]\r\n  <i>}\r\n  }\r\nJSON\r\np expected\r\n"

Output with Ruby 3.4.7 on Windows:

C:\Users\headius\work\json>ruby test\json\json_generator_test.rb
"{\n<i>\"1\" :   {\n<i><i>\"2\" :   3,\n<i><i>\"4\" :   [<a_nl><i><i><i>5,<a_nl><i><i><i>6<a_nl><i><i>]\n<i>}\n}"

Output with JRuby 10-head on Windows:

C:\Users\headius\work\json>jruby test\json\json_generator_test.rb
"{\r\n<i>\"1\" :   {\r\n<i><i>\"2\" :   3,\r\n<i><i>\"4\" :   [<a_nl><i><i><i>5,<a_nl><i><i><i>6<a_nl><i><i>]\r\n<i>}\r\n}"

I will re-file this as a JRuby issue.

[headius]

This was after all a JRuby issue (jruby/jruby#9018) and was fixed there (jruby/jruby#9019). I've already updated the json CI matrix to include windows-latest with jruby-head (#868).