UMassCTF 2023: java_jitters & java_jitters_2 Writeups

Posted 2023/03/26

Over the weekend, I competed in UMassCTF 2023, the capture-the-flag contest hosted by UMass Amherst’s Cybersecurity Club. I competed in the CTF with team CVE2K9, a group of CTF loving friends I found on the fediverse.

This is my writeup for the two Java reverse engineering problems in the CTF: java_jitters and java_jitters_2 (this writeup has also been crossposted to the team website here)

java_jitters

Description

sips coffee o vault of secrets, teller of wisdom and UMastery, tell me the secret phrase and I shall share my wisdom

File: javajitters.jar

The program provided is run in the command line and asks the user to supply a password as an argument, if the password is correct, the program prints out the flag.

Unpacking the JAR

This is pretty simple, JAR files are just fancy ZIP files with a set directory structure so once you unzip it you get this:

unpacked jar

The first thing that you should look at is the META-INF/MANIFEST.MF file which defines what class is run by Java when executing the JAR:

Manifest-Version: 1.0
Main-Class: edu.umass.javajitters.Main

This tells us to look at the main function within the Main class in edu/umass/javajitters

Decompiling the `Main` class

It turns out that the Java code for this application has been obfuscated using Skidfuscator, so most Java decompilers fail to decompile the main function within this class. For example, the version of Fernflower packaged with IntelliJ decompiled every other function but did this on the main function:

public static void main(String[] var0) throws NoSuchAlgorithmException {
        // $FF: Couldn't be decompiled
    }

This isn’t great as reading through the Java bytecode, you can tell most of the program logic falls within this main function. All is not lost though, after trying many other decompilers like Procyon, CFR, and JADX, I found that the version of Fernflower packaged with Recaf gives us enough code for us to figure out the rest¹. The code Recaf gave us isn’t perfect and failed to run no matter how many tweaks I made but we can still figure out the code’s logic from it.

I also decompiled skid/Factory.class and skid/nerzotvnwdsdpsre.class using IntelliJ’s version of Fernflower².

Refactoring the code for my sanity

After painstakingly reading through the whole code, I made a refactored/annotated version of the source code.

The first problem when refactoring is that a lot of the Java bytecode uses the invokedynamic instruction for many function calls which Fernflower hates, decompiling the instruction to something like this:

var175 = var173.opotarbpahmyugec<invokedynamic>(var173, var174);

You can sometimes guess what function is being used based on the context and datatypes of the variables but you can look at the bytecode (using IntelliJ or Recaf) to find what function is being called:

ALOAD 173
ILOAD 174
INVOKEDYNAMIC opotarbpahmyugec(Ljava/lang/Object;I)C [
      // handle kind 0x6 : INVOKESTATIC
skid/Ref.dispatch(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
      // arguments:
      182, 
      "java.lang.StringBuilder", 
      "charAt", 
      "(I)C"
]
ISTORE 175

This can by translated by hand into this: (After consulting Java docs to figure out how StringBuilder worked)

var175 = var173.charAt(var174);

After tediously doing this for every one of the 66 invokedynamic calls you get proper Java code!!!

I also renamed the 0o$Oo$K, K0o$KOo$KK, and Oo0o$OoOo$OoK functions into crypto_func1, crypto_func2, and xor respectively.

Tracing `crypto_func1` and `crypto_func2`

I looked at crypto_func1 first after realizing it was called very early on in the main function. From reading the code and running through it line by line in JShell³ with the args crypto_func1("password", 58777307);⁴, I figured out that the function would SHA-256 hash whatever string was passed to it and then call crypto_func2. There is one common code pattern to look at here:

StringBuilder var12 = new StringBuilder("\uB236\u8436\u9636\u4E36\u7036\u7E36\u7836");
int var13 = 0;
int var14 = 0;

for (var9 = 0 ^ var9; var13 < var12.length(); var13 += 1974077748 ^ var9) {
    var14 = var12.codePointAt(var13);
    var14 = ((var14 & (1974138570 ^ var9)) >> (1974077756 ^ var9) | var14 << (1974077746 ^ var9)) & (1974138570 ^ var9);
    var14 ^= 1974076994 ^ var9;
    var14 ^= 1974140119 ^ var9;
    var14 ^= 1974134954 ^ var9;
    var12.setCharAt(var13, (char) var14);
}

String var2 = var12.toString();

A similar code block to this with different variables pops up once within crypto_func1 and very often in main. Running through it line by line you will see that it initializes a StringBuilder object with ciphertext, does some sort of decryption character by character with var9 being used as a key, and then converts the decrypted StringBuilder to a String. This is Skidfuscator’s string encryption, obfuscation which we can bypass by keeping track of the key variable’s value as the program runs.

Tracing through crypto_func2 in a similar manner, you will find that it just takes the hash that crypto_func1 generated in byte[] form and converts it to a hexadecimal string.

One thing to note is that as another obfuscation method, you won’t find return statements within these two functions. Instead, you will find nerzotvnwdsdpsre exceptions being thrown. This is a custom exception created by the obfuscator that looks like this:

public class nerzotvnwdsdpsre extends RuntimeException {
    private String var;

    public nerzotvnwdsdpsre(String var1) {
        this.var = var1;
    }

    public String get() {
        return this.var;
    }
}

This exception gets used like this to simulate return statements:

try {
    while (true) {
        crypto_func2(var5, 503562056);
    }
} catch (nerzotvnwdsdpsre var17) {
    String var6 = var17.get(); // returned value from crypto_func2
    throw new nerzotvnwdsdpsre(var6);  // throwing new exception for crypto_func1's return
}

Analyzing `main`

With those two functions out of the way, let’s start analyzing the main program code. First we have to find the key which I’ve called int1:

int int1 = 1800875868 ^ 2076430062 ^ gjMAozlvk2;
int1 = 841098174 ^ int1;

This key is initialized with a seeded random number generator:

int var3 = (new Random(-8261172285046822504L)).nextInt();  // = -535977286
gjMAozlvk2 = -266593489 ^ var3;  // = 269597077

This means our key starts with the value 849836953.

The program starts by checking that a password is supplied, then over a series of calculations (traced through JShell), the key changes to 2024113895. Throughout these calculations, functions in the Factory helper class are called which appear to be a checksum function to make sure that the key is correct throughout the program’s execution. The program then calls crypto_func1(str2, 58777307) to SHA-256 hash our password.

The rest of the program is a series of nested if-else blocks which do the following:

Performs a XOR calculation against some integer literal to calculate a new key
Decrypts some ciphertext to a known SHA-256 hash (using the method outlined above)
Compares that hash against the user’s password hash
If the hashes match:
1. Performs a XOR calculation against some integer literal to calculate a new key again
2. Decrypts a message to output to the user (using the method outlined above)
3. Prints out the message
4. Exits the program
If the hashes don’t match, do more XOR calculations and Factory checksums to change up the key and repeat

There are many known passwords as some output easter egg messages like Congratulations, you've unlocked the secret to a perfect cup of Java!. If none of the passwords match, the last else block prints a password not found message.

Tracing the program in JShell, halfway into the code we get to a block where the key is 1140202191 and has a SHA-256 hash of 8900fbb69012f45062aa6802718ad464eaea0854b66fe8916b3b38e775c296a8.

StringBuilder var168 = new StringBuilder("\uA849\uB849\u2849\u2849\u884C\u484C\u484C\u8849\uB849\u2849\u3849\u4849\u884C\u6849\u7849\u2849\u8849\u4849\u384C\u384C\u8849\uA849\u2849\u4849\u9849\u3849\uA849\u384C\u684C\u6849\u8849\u6849\u784C\u384C\u784C\u384C\u2849\uA849\u7849\u6849\u484C\u8849\u8849\u884C\u784C\uA849\uB849\u3849\u8849\u484C\u5849\u484C\u5849\uA849\u784C\u9849\u9849\u7849\u584C\u4849\uB849\u8849\u384C\uA849");
int var169 = 0;
int var170 = 0;

for (int1 = 0 ^ int1; var169 < var168.length(); var169 += 1140202190 ^ int1) {
    var170 = var168.charAt(var169);
    var170 += 1140203535 ^ int1;
    var170 = ((var170 & (1140254000 ^ int1)) << (1140202186 ^ int1) | var170 >> (1140202180 ^ int1)) & (1140254000 ^ int1);
    var170 = ((var170 & (1140254000 ^ int1)) << (1140202176 ^ int1) | var170 >> (1140202190 ^ int1)) & (1140254000 ^ int1);
    var170 -= 1140197327 ^ int1;
    var170 -= 1140197801 ^ int1;
    var168.setCharAt(var169, (char) var170);
}

String var46 = var168.toString();  // 8900fbb69012f45062aa6802718ad464eaea0854b66fe8916b3b38e775c296a8
byte var14 = (byte) (var6.equals(var46) ? 1 : 0);  // var6.equals(var46);
if (var14 != (1140202191 ^ int1)) {
    int1 = 850230892 ^ int1;  // 1901814947
    PrintStream var15 = System.out;
    StringBuilder var188 = new StringBuilder("\u1b2a\u1bea\u1b8a\u9b0a\u9b0a\u9a4a\u9b8a\u580a\udaaa\udaaa\u980a\u980a\u9b6a\u1aea\u5b8a\u9aca\u980a\u9a0a\u9b6a\u1aea\u980a\u9b6a\udbca\u5b8a\uda2a\u5b8a\u9b6a\u9a8a\uda0a\u1a8a\uda4a\u1a4a\u1a6a");
    int var189 = 0;
    int var190 = 0;

    for (int1 = 0 ^ int1; var189 < var188.length(); var189 += 1901814946 ^ int1) {
        var190 = var188.charAt(var189);
        var190 = ((var190 & (1901828956 ^ int1)) << (1901814952 ^ int1) | var190 >> (1901814950 ^ int1)) & (1901828956 ^ int1);
        var190 ^= 1901840784 ^ int1;
        var190 = ((var190 & (1901828956 ^ int1)) >> (1901814959 ^ int1) | var190 << (1901814951 ^ int1)) & (1901828956 ^ int1);
        var190 = (var190 ^ -1901814948 ^ int1) & ((1901814946 ^ int1) << (1901814963 ^ int1)) - (1901814946 ^ int1);
        var190 ^= ((var190 >> (1901814952 ^ int1) ^ var190 >> (1901814947 ^ int1)) & ((1901814946 ^ int1) << (1901814951 ^ int1)) - (1901814946 ^ int1)) << (1901814952 ^ int1) | ((var190 >> (1901814952 ^ int1) ^ var190 >> (1901814947 ^ int1)) & ((1901814946 ^ int1) << (1901814951 ^ int1)) - (1901814946 ^ int1)) << (1901814947 ^ int1);
        var190 = ((var190 & (1901828956 ^ int1)) >> (1901814945 ^ int1) | var190 << (1901814957 ^ int1)) & (1901828956 ^ int1);
        var188.setCharAt(var189, (char) var190);
    }

    String var47 = var188.toString();
    println15(var15, var47, 862983464);
    int1 = 1019968462 ^ int1;

    try {
        if (Factory.method1(int1) != 105345395) {  // int1.mcaysezmbpdkwegq<invokedynamic>(int1)
            throw null;
        }

        throw new IllegalAccessException();
    } catch (IllegalAccessException var221) {
        switch (Factory.method2(int1)) {
            case 1603759402:
                int1 = xor(int1, 1207713573);  // int1.mnumuswfopdwpjwp<invokedynamic>(int1, 1207713573);
                break;
            case 1820392298:
                int1 = 1100926445 ^ int1;
                break;
            default:
                throw new RuntimeException("Error in hash");
        }
    }

    while (true) {
        switch (Factory.method1(int1)) {
            case 37686904:
            default:
                throw new RuntimeException();
            case 123517910:
                int1 = 982632165 ^ int1;
                return;
            case 691634510:
                break;
            case 1995906324:
                return;
        }
    }
} else {
    ...

Decrypting the message of this block, we get our flag: UMASS{C0ff33_m@k3s_m3_J@v@_crazy}

java_jitters_2

Please please please don’t look down here until you’ve understood the java_jitters solution above. It builds on the previous solution significantly.

Description

sips coffee if only i remembered the password to my amazing app… maybe i could get those java beans coins…

File: javajitters_v2.jar

Assumptions from java_jitters

Like the original java_jitters challenge, we can unpack the JAR and decompile it with Recaf’s version of Fernflower. The code is also obfuscated using the same Skidfuscator tool, so the string literals encryption algorithm is the same (just with a new key that we need to trace). Like the original problem, there are a lot of invokedynamic instructions that you will need to convert to Java code by looking at the bytecode.

I also assumed that the 0o$Oo$K and K0o$KOo$KK functions were the same SHA-256 password hashing functions from the original challenge and did not analyze those.

Key differences from java_jitters outside of main program logic

Like the original challenge, java_jitters_2 also uses exceptions to replace return statements, but this time there are two exception classes. xpbyayedzpfnsdwh replaces string return statements and edyxsdbugbromxsl replaces byte array return statements.

This byte array return exception gets used in a new function called Oo0o$OoOo$OoK(String var0, int var1, int var2) which I renamed to decode_data(String ciphertext, int len, int seed_arg) for reasons I will discuss later.

Somewhat annotated/refactored version of the code available here

Analyzing the `main` function

Like the original problem, we can trace the program flow and the encryption key using JShell. The key is initialized in the same way as the last problem:

int var3 = (new Random(187796278769191316L)).nextInt();
seed = 331542956 ^ var3;  // 694828334
int key = 232217141 ^ 2023401522 ^ seed;  // 1546112809
key = 1678878592 ^ key;  // 943089833

The program then proceeds to check the number of arguments supplied and then perform XOR and Factory checksum operations to change the key and then hashes the user’s password.

The big difference between this program and the last is that instead of constantly checking hashes against known hashes in if-else blocks, we start by building a dictionary of known hashes and their respective message data (encoded in Base64):

HashMap hash_to_base64 = new HashMap();  // dictionary of hashes and message data
key = 1550490048 ^ key;  // 1099337665
StringBuilder var177 = new StringBuilder("맳룺돵돳럺맷럹룺럺맴당맴맷맴룹돳돵돳럺맶럺맶맸돵돴럺룺돳돳럺럹맳룹럹맸룹돵맳돴럹맷맵당맵돸돵돸돵룹맴룺룹돵럹돴룺맷맵돳돴럺돵돵맳");  // encrypted version of known hash
int var178 = 0;
int var179 = 0;

// decrypting hash
for(key = 0 ^ key; var178 < var177.length(); var178 += 1099337664 ^ key) {
    var179 = var177.charAt(var178);
    var179 = (var179 ^ -1099337666 ^ key) & ((1099337664 ^ key) << (1099337681 ^ key)) - (1099337664 ^ key);
    var179 += 1099340170 ^ key;
    var179 ^= ((var179 >> (1099337668 ^ key) ^ var179 >> (1099337665 ^ key)) & ((1099337664 ^ key) << (1099337666 ^ key)) - (1099337664 ^ key)) << (1099337668 ^ key) | ((var179 >> (1099337668 ^ key) ^ var179 >> (1099337665 ^ key)) & ((1099337664 ^ key) << (1099337666 ^ key)) - (1099337664 ^ key)) << (1099337665 ^ key);
    var179 -= 1099335157 ^ key;
    var179 = ((var179 & (1099329598 ^ key)) << (1099337674 ^ key) | var179 >> (1099337668 ^ key)) & (1099329598 ^ key);
    var179 ^= 1099332967 ^ key;
    var177.setCharAt(var178, (char)var179);
}

String var36 = var177.toString();  // 30ec879082a2721cec84846eb80cc8931961e3b975a5fefe1201e9b075cb8ee3
StringBuilder var182 = new StringBuilder("怇帧帧慧捇捇执悧揧捇执愧捧枧捧慧悧帧座搇文座敧控悧崇扇掇揇敧揇掇揧敧座愧掇捧拧愧捧敧拇揧揇帧帧挧敇捇捧捇捧捇执挧揧揧惧敧掇揧敧悧敇捇惧捇揧敧愧挧捇崇拇揧捧捇揧文敇捧帧悧敧幧弧悧揧柇怇揇");  // encrypted version of message for hash
int var183 = 0;
int var184 = 0;

// decrypting the message to b64
for(key = 0 ^ key; var183 < var182.length(); var183 += 1099337664 ^ key) {
    var184 = var182.charAt(var183);
    var184 = ((var184 & (1099329598 ^ key)) >> (1099337676 ^ key) | var184 << (1099337666 ^ key)) & (1099329598 ^ key);
    var184 = ((var184 & (1099329598 ^ key)) >> (1099337673 ^ key) | var184 << (1099337673 ^ key)) & (1099329598 ^ key);
    var184 ^= 1099308569 ^ key;
    var184 += 1099349694 ^ key;
    var184 ^= 1099333357 ^ key;
    var184 -= 1099342553 ^ key;
    var182.setCharAt(var183, (char)var184);
}

String var2 = var182.toString();  // "cllfUUNXRUNdV0VfXlhCGhFPXkMWQFQWRFhdWVJdVFIRQllTEUVUVUNTRRZFWRFXEUZURFdTUkIRVURGEVlXFntXR1cQ"
hash_to_base64.put(var36, var2);  // storing the message data to the dictionary with hash as key

The program does this with 11 different sets of hashes and messages, changing the key every time. This gives us a dictionary that looks like this:

{
    "8e09b341e9f7788d5563b587d6eb87dfa90069d668982dcbd19660bd08594564": "aFtEFFxBQkARXFBCVBRZVVUUUBRFRlhEXVEcR1lbRRRUR0FGVEdCWxFAXhRSRlBXWhRFXFhHEURQR0JDXkZVFQ==",
    "beae9a6258a7559ca2f8628763bcef3b44b126a093291af0532f217743bf2d52": "e1VHVRF+WEBFUUNHEVxQRxFZVEARXUVHEVlQQFJcEUNYQFkUSFtERhFEUEdCQ15GVRRSRlBXWl1fUxFHWl1dWEIV",
    "c7755b10c70d123e2082e7b21ef8efe3f400c5253fce315554726c5084c755e7": "ZVxUFHtVR1URQ15GXVARVl5DQhRVW0ZaEUBeFEhbRBRQWlUUSFtERhFEUEdCQ15GVRRSRlBXWl1fUxFHWl1dWEIV",
    "a76fb4e34b67a1fd1055c5bc78674ff627ec96763f4894b35c9a947ae0a6bc61": "e1VHVRF+WEBFUUNHEVVYWhZAEVNeQBFaXkBZXV8TEVtfFEhbRBRQWlUUSFtERhFXQ1VSX1haVhRCX1hYXUcQ",
    "a2c5009dbdd1a2a9935d34e8812257f1a965aa259f0efd9f5d2ecc5e3b809b03": "fVteX0IUXV1aURFNXkEWQlQUVltFFEVcVBR7VUdVEX5YQEVRQ0cRQV9QVEYRV15aRUZeWBFDWEBZFEVcWEcRRFBHQkNeRlUV",
    "30ec879082a2721cec84846eb80cc8931961e3b975a5fefe1201e9b075cb8ee3": "cllfUUNXRUNdV0VfXlhCGhFPXkMWQFQWRFhdWVJdVFIRQllTEUVUVUNTRRZFWRFXEUZURFdTUkIRVURGEVlXFntXR1cQ",
    "9affb91bb2f4f36e847b6bdcbd990d66035d7c0bede187a59ac56d98a21d4899": "aFtERhF+UEJQFFpaXkNdUVVTVBRYRxFWQ1FGXV9TEUBeFEFRQ1JUV0VdXloRQ1hAWRRFXFhHEURQR0JDXkZVFQ==",
    "eecd928fbae7909ec54cae3efc510470cb190f7c74ff0e3d87d00b26c5e76777": "aFtEE0dREVlQUFQUe1VHVRF+WEBFUUNHEVheW1oUXV1aURFQVFdQUhFDWEBZFEhbREYRRFBHQkNeRlUUUkZQV1pdX1MRRENbRlFCRxA=",
    "b0f8b56898e123c658a566bdd6d55e26a0b6cb0b5039fd53ef2f772c5ce2e3d5": "G0dYREIUUltXUlRRGw==",
    "8900fbb69012f45062aa6802718ad464eaea0854b66fe8916b3b38e775c296a8": "ZHlwZ2JPQwdHB0NHWFpWa1sARwBuBUJrBWtbBUVAAkZIa1sEU0k=",
    "6ceb89b10244f1d54471b0b6d595e78802252b7f269304ddb367440b966d988d": "aFtEE0dREUFfWF5XWlFVFEVcVBR7VUdVEUBDUVBHREZUFEZdRVwRQFldQhRBVUJHRltDUBA="
}

The program then checks if the hash from the user’s password is found in the dictionary (which we can ignore when tracing in JShell) and then moves on to a message decoding and printing code block:

PrintStream var33 = System.out;
String var54 = new String();
key = 1211708954 ^ key; // 1269109784
Decoder var4 = Base64.getDecoder();
String var75 = var10.toString();  // hash of input
Object var71 = hash_to_base64.get(var75);  // gets data matching hash
String var72 = (String)var71;  // b64 data
Charset var76 = StandardCharsets.UTF_8;
byte[] var73 = var72.getBytes(var76);
byte[] var69 = var4.decode(var73);
Charset charsetUTF = StandardCharsets.UTF_8;
String var3 = new String(var69, charsetUTF);  // b64 decoded data
byte var70 = (byte)(1269109784 ^ key);  // 0
String var66 = args[var70];  // password (unhashed)
key = 569683477 ^ key;  // 1783740941
int var67 = var66.length();  // password length
key = 1651564803 ^ key;  // 136403726

try {
    while(true) {
        decode_data(var3, var67, 133764025);
    }
} catch (edyxsdbugbromxsl var224) {
    byte[] var65 = var224.get();
    Charset var68 = StandardCharsets.UTF_8;
    var54.<init>(var65, var68);
    key = 858574774 ^ key;
    ymvjazxcysollnvc(var33, var54, 231835709);
    key = 1346937356 ^ key;
    return;
}

It starts by grabbing the message data for the hash provided, decoding the data with Base64, and then runs the decode_data function on the decoded data with the user’s password length and 133764025 as arguments. From this, we can deduce that decode_data is a decryption function for the message that takes password length as a key.

This is a pretty simple key to brute force, which we can do for every message in JShell:

jshell> str = "ZHlwZ2JPQwdHB0NHWFpWa1sARwBuBUJrBWtbBUVAAkZIa1sEU0k="
str ==> "ZHlwZ2JPQwdHB0NHWFpWa1sARwBuBUJrBWtbBUVAAkZIa1sEU0k="

jshell> for(int i = 1; i <= 256; i++) {
    ..>     try {
    ..>         decode_data(new String(Base64.getDecoder().decode(str.getBytes(StandardCharsets.UTF_8), StandardCharsets.UTF_8), i, 133764025);
    ..>     } catch (edyxsdbugbromxsl e) {
    ..>         System.out.println(new String(e.get()));
    ..>     }
    ..> }

This gives us all the messages in the program like You must have had a triple-shot espresso to crack this password! and *sips coffee*, but most importantly it also gives us the message that contains our flag: UMASS{r3v3rsing_j4v4_1s_4_j1tt3ry_j0b}

All the source code that was relevant for both java_jitters and java_jitters_v2 in decompiled and annotated/refactored forms can be found here

Recaf being able to do it but not IntelliJ is interesting since JetBrains is the creator of Fernflower so you would think the IntelliJ version is the best/latest version ↩
Turns out these two classes and everything under skid/ is Skidfuscator’s helper classes ↩
JShell for the uninitiated, is Java’s secret REPL shell which allows you to test Java code in an interpreter like Python’s interpreter ↩
password was just a test, 58777307 was a constant from how it was called in main ↩

UMassCTF 2023: java_jitters & java_jitters_2 Writeups

java_jitters

Description

Unpacking the JAR

Decompiling the Main class

Refactoring the code for my sanity

Tracing crypto_func1 and crypto_func2

Analyzing main

java_jitters_2

Description

Assumptions from java_jitters

Key differences from java_jitters outside of main program logic

Analyzing the main function

Decompiling the `Main` class

Tracing `crypto_func1` and `crypto_func2`

Analyzing `main`

Analyzing the `main` function