I don't have the time at the moment to formulate a proper solution but I'll give some hints that will hopefully help you.

I would honestly break this down to a simpler problem, switch the integers down to fewer bits. For instance you could take the algorithm and modify it such that it works with 8-bit nonces, then switch your rotation amount down to 2 bits (one fourth of the nonce size). Then you switch your masks from 8 bits to 2 bits as well and your cipher xor into 2 bits. This way you're working with a more manageable problem. You could even reduce it into 4-bit nonces and 1 bit cipher.

From this simplified problem you formulate equations for each bit from the plaintext to the cipher text. Plugging it into something like Z3 might be the easiest way of getting an answer out. Here's an article using Z3 to attack a simple hash function.

Glancing at it it looks like if you have the plaintext and the ciphertext you can solve for the key (nonce1 and nonce2). The addition of nonce2 might be an issue due to it actually being an addition modulus 2^32. As it's an addition there's really only two values due to the constraints on nonce2 and x that can map to the same value so the it's not impossible to invert.

If you're not sure whether you can solve the equations yourself I would bring the problem to math.stackexchange.com once you've got it figured out.

I don't have the time at the moment to formulate a proper solution but I'll give some hints that will hopefully help you.

I would honestly break this down to a simpler problem, switch the integers down to fewer bits. For instance you could take the algorithm and modify it such that it works with 8-bit nonces, then switch your rotation amount down to 2 bits (one fourth of the nonce size). Then you switch your masks from 8 bits to 2 bits as well and your cipher xor into 2 bits. This way you're working with a more manageable problem. You could even reduce it into 4-bit nonces and 1 bit cipher.

From this simplified problem you formulate equations for each bit from the plaintext to the cipher text. Plugging it into something like Z3 might be the easiest way of getting an answer out. Here's an article using Z3 to attack a simple hash function.

Glancing at it it looks like if you have the plaintext and the ciphertext you can solve for the key (nonce1 and nonce2). The addition of nonce2 might be an issue due to it actually being an addition modulus 2^32. As it's an addition there's really only two values due to the constraints on nonce2 and x that can map to the same value so the it's not impossible to invert.

If you're not sure whether you can solve the equations yourself I would bring the problem to math.stackexchange.com once you've got it figured out.

I don't have the time at the moment to formulate a proper solution but I'll give some hints that will hopefully help you.

I would honestly break this down to a simpler problem, switch the integers down to fewer bits. For instance you could take the algorithm and modify it such that it works with 8-bit nonces, then switch your rotation amount down to 2 bits (one fourth of the nonce size). Then you switch your masks from 8 bits to 2 bits as well and your cipher xor into 2 bits. This way you're working with a more manageable problem. You could even reduce it into 4-bit nonces and 1 bit cipher.

From this simplified problem you formulate equations for each bit from the plaintext to the cipher text. Plugging it into something like Z3 might be the easiest way of getting an answer out.

Glancing at it it looks like if you have the plaintext and the ciphertext you can solve for the key (nonce1 and nonce2). The addition of nonce2 might be an issue due to it actually being an addition modulus 2^32. As it's an addition there's really only two values due to the constraints on nonce2 and x that can map to the same value so the it's not impossible to invert.

If you're not sure whether you can solve the equations yourself I would bring the problem to math.stackexchange.com once you've got it figured out.

I don't have the time at the moment to formulate a proper solution but I'll give some hints that will hopefully help you.

I would honestly break this down to a simpler problem, switch the integers down to fewer bits. For instance you could take the algorithm and modify it such that it works with 8-bit nonces, then switch your rotation amount down to 2 bits (one fourth of the nonce size). Then you switch your masks from 8 bits to 2 bits as well and your cipher xor into 2 bits. This way you're working with a more manageable problem. You could even reduce it into 4-bit nonces and 1 bit cipher.

From this simplified problem you formulate equations for each bit from the plaintext to the cipher text. Plugging it into something like Z3 might be the easiest way of getting an answer out. Here's an article using Z3 to attack a simple hash function.

Glancing at it it looks like if you have the plaintext and the ciphertext you can solve for the key (nonce1 and nonce2). The addition of nonce2 might be an issue due to it actually being an addition modulus 2^32. As it's an addition there's really only two values due to the constraints on nonce2 and x that can map to the same value so the it's not impossible to invert.

If you're not sure whether you can solve the equations yourself I would bring the problem to math.stackexchange.com once you've got it figured out.