LinearOpaqueConstantFoldingTransformer fail when processing a specific edge case

[luiox]

recaf version: 4.0.0

Describe the bug

The LinearOpaqueConstantFoldingTransformer fails when processing a specific edge case. Instead of handling it gracefully, it attempts to process and occur an error.

To Reproduce

Steps to reproduce the behavior:

1. Open sample 'sample.class', this file is in the 'sample.zip'
2. Open the deobfuscate window
3. Use LinearOpaqueConstantFoldingTransformer
4. the decompiler will repot that this transformer make the bytecode error

Screenshots

the origin class

after transform

sample.zip

[luiox]

I have output the transformed bytecode, and below is the output result.

    .method private static bridge synthetic kQ9eiigBOq (Ljava/lang/String;)Ljava/lang/String; {
        parameters: { v0 },
        code: {
        A: 
            new java/lang/StringBuilder
            dup 
            invokespecial java/lang/StringBuilder.<init> ()V
            astore v1
            ldc -50240075
            ldc -1500519697
            dup2 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            swap 
            dup_x1 
            iconst_m1 
            ixor 
            ior 
            swap 
            iconst_m1 
            ixor 
            isub 
            iadd 
            dup_x2 
            pop 
            iconst_1 
            isub 
            iconst_m1 
            ixor 
            iconst_m1 
            iadd 
            swap 
            iconst_1 
            isub 
            iconst_m1 
            ixor 
            iconst_m1 
            iadd 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            swap 
            dup_x1 
            iconst_m1 
            ixor 
            ior 
            swap 
            iconst_m1 
            ixor 
            isub 
            iadd 
            swap 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            dup_x1 
            iconst_m1 
            ixor 
            iand 
            iadd 
            swap 
            ineg 
            iconst_m1 
            iadd 
            isub 
            i2l 
            ldc 1725601193
            ldc 1028920051
            dup2 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            swap 
            dup_x1 
            iconst_m1 
            ixor 
            ior 
            swap 
            iconst_m1 
            ixor 
            isub 
            iadd 
            dup_x2 
            pop 
            swap 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            dup_x1 
            iconst_m1 
            ixor 
            iand 
            iadd 
            swap 
            ineg 
            iconst_m1 
            iadd 
            isub 
            isub 
            i2l 
            lxor 
            l2i 
            istore i2
            goto C
        B: 
            aload v1
            aload v0
            iload i2
            invokevirtual java/lang/String.charAt (I)C
            ldc -70822210L
            ldc 633062966
            ldc -806523127
            ldc 806523126
            swap 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            dup_x1 
            iconst_m1 
            ixor 
            iand 
            iadd 
            swap 
            ineg 
            iconst_m1 
            iadd 
            isub 
            dup_x2 
            pop 
            swap 
            iconst_m1 
            ixor 
            iconst_1 
            iadd 
            iconst_m1 
            iadd 
            swap 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            dup_x1 
            iconst_m1 
            ixor 
            iand 
            iadd 
            swap 
            ineg 
            iconst_m1 
            iadd 
            isub 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            swap 
            dup_x1 
            iconst_m1 
            ixor 
            ior 
            swap 
            iconst_m1 
            ixor 
            isub 
            iadd 
            i2l 
            ldc 1862785500
            ldc 2123768023
            dup2 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            swap 
            dup_x1 
            iconst_m1 
            ixor 
            ior 
            swap 
            iconst_m1 
            ixor 
            isub 
            iadd 
            dup_x2 
            pop 
            swap 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            dup_x1 
            iconst_m1 
            ixor 
            iand 
            iadd 
            swap 
            ineg 
            iconst_m1 
            iadd 
            isub 
            isub 
            i2l 
            lxor 
            l2i 
            i2l 
            lxor 
            l2i 
            dup2 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            swap 
            dup_x1 
            iconst_m1 
            ixor 
            ior 
            swap 
            iconst_m1 
            ixor 
            isub 
            iadd 
            dup_x2 
            pop 
            swap 
            dup_x1 
            ineg 
            iconst_m1 
            iadd 
            dup_x1 
            iconst_m1 
            ixor 
            iand 
            iadd 
            swap 
            ineg 
            iconst_m1 
            iadd 
            isub 
            isub 
            i2c 
            invokevirtual java/lang/StringBuilder.append (C)Ljava/lang/StringBuilder;
            pop 
            iinc i2 1
        C: 
            iload i2
            aload v0
            invokevirtual java/lang/String.length ()I
            if_icmplt B
            aload v1
            invokevirtual java/lang/StringBuilder.toString ()Ljava/lang/String;
            areturn 
        D: 
        }
    }

[luiox]

I discovered the erroneous sequence of instructions. I remove the nop remove, so it will be clear.

the orginal insnlist

        ldc 633062966
        ldc -806523127
        dup2 
        iconst_m1
        ixor 
        istore t1
        istore t2
        istore t3
        istore t4

the transform result

      ldc 633062966
      ldc -806523127
      nop 
      nop 
      ldc 806523126
      istore t1
      istore t2
      istore t3
      istore t4

then we see the decompiler

        int t1 = ~-806523127;
        int t2 = 633062966;
        int t3 = -806523127;
        int t4 = 633062966;

so the right result should be here

  ldc 633062966
  ldc -806523127
  ldc 633062966
  ldc 806523126

and if I replace the dup2 with the ldc, it's right. so maybe dup2 can not be handled correctly.

I analyzed it and found that during ixor processing, another value generated by dup2 would be lost. Therefore, there should be a problem with the current algorithm.

[luiox]

Obviously, under the linear algorithm, constant folding is almost impossible to be implemented at one time.

If it is linear, the stack can only be balanced by filling the pop first, and then the constants can be loaded. In addition, other converters should be used to clean up this pop; otherwise, implementing constant folding in the case of a single linear scan would be very complicated.

[luiox]

I used a strange way to fix the bug. Refer to #947

[Col-E]

I think it would be best to solve this by checking the stack size of expected arguments of the operations (like iadd taking two ints, vs lmul taking two longs) and then checking if one of the argument instructions fills both. So for the example with dup2 that pushes either one wide type, or two single slot types. When we see a iadd and the previous argument instruction is dup2 then it counts as both argument instructions.

I'm also writing up unit tests to cover the other dup like instructions I didn't cover in the original work on the transformer.

[Col-E]

Fixed with a series of changes to the linear opaque constant folding transformer.