Since you've already mostly decoded the code, there are two things left: 1) understand what the code is doing and 2) understand how to compute the appropriate input.
The original code
First, here's a slightly modified bit of code based on what you already had:
int check_password(const char *s) {
char fixed[] __attribute__ ((aligned (16))) = { 79, 8, 29, 58, 81, 21, 49, 123, 114 };
if (strlen(s) != 9)
return -1;
for (int i=0; i < 3; ++i) {
for (int j=0; j < 3; ++j) {
int sum = 0;
for (int k=0; k < 3; ++k) {
sum = (fixed[3 * i + k] * s[3 * k + j] + sum) % 127;
}
if (i == j) {
if (sum != 1) {
return -2;
}
} else if (sum) {
return -2;
}
}
}
return 0;
}
The __attribute__ ((aligned (16))) can be ignored for our purposes. (I had included that to get gcc to use the same stack offset as your sample code.) What the code is doing is multiplying two 3x3 matrices, mod 127, and checking for the identity matrix.
The mathematics
Unfortunately, ReverseEngineering does not support MathJax, or I'd be able to write out pretty equations. Since I can't we'll do it the hard way. Let's say s = "ABCDEFGHI" is the input password and we treat it as a 3x3 matrix. For now, if we just assign uppercase letters to each letter in the password, expressed as a matrix, this is:
A B C 79 8 29 1 0 0
D E F x 58 81 21 = 0 1 0 (mod 127)
G H I 49 123 114 0 0 1
If you're familiar with matrix manipulation already, you may already recognize that this means that s must be the inverse of the fixed matrix. There are multiple ways of calculating this, such as by Gauss-Jordan elimination. Another way is to multiply the reciprocal of the determinant of the matrix by the transpose of its cofactor matrix. Note that all of the mathematics is done mod 127, as per the original code.
Worked example
To make things a bit more concrete, I'll use the latter method and show the step-by-step worked solution. First, we calculate the determinant using the Leibniz formula.
D = 79*81*114 + 8*21*49 + 29*58*123 - 29*81*49 - 8*58*114 - 79*21*123 (mod 127)
D = 572550 (mod 127)
D = 34 (mod 127)
To compute the reciprocal, of this we can't just use 1/34 because we're working with modular mathematics. So just in the way that we'd expect that 34 * 1/34 = 1 for regular mathematics, in modular mathematics, we're looking for something that satifies the equation 34 * x = 1 (mod 127). One way to do this is to use the extended Euclidean algorithm. I won't go through that algorithm here, but in this case, the answer is 71 (which we can verify by noting that 71 * 34 = 2414 = 1 (mod 127).
Next we calculate the cofactor matrix. Here again, we won't go through all of the steps, but the cofactor matrix of fixed is:
6651 -5583 3165
2655 7585 -9325 (mod 127)
-2181 23 5935
Since we're working with modular mathematics, we can reduce this:
47 5 117
115 92 73 (mod 127)
105 23 93
Now all that's left is to multiply them together:
47 5 117
71 * 115 92 73 (mod 127)
105 23 93
3337 355 8307
8165 6532 5183 (mod 127)
7455 1633 6603
35 101 52
37 55 103 (mod 127)
89 109 126
Finally, we take the transpose:
35 37 89
101 55 109
52 103 126
If we then translate this matrix back into an ASCII string, we get "#%Ye7m4g~" which is the inverse of the given matrix and the solution to this reverse engineering problem.
Why this might matter
While it may be an interesting enough puzzle by itself, these kinds of transformations using discrete mathematics and modular arithmetic are fundamental to many areas of modern cryptography. Since you've already started looking into reverse engineering, you may find it useful and interesting to study these topics as well.
int8_tlimits. I would write a proper answer once I have verified that in IDA.