I took a little Perl program that i'd written a while ago when i had a similar problem, and changed it to match your data. This is supposed to generate all permutations of "test:admin", which any character as the separator, and including all abbreviations (to handle the case that only a few bytes from the username are taken).
I got around 50 results beginning in 3785a9, but none with 3785a9c6. So, this seems to rule out the case that (a part of) the username is concatenated to the password, possibly with a separator character, and the MD5 of the result is computed. But of course, they might use MD4 or something else that produces 16 bytes. They might xor the user with the password first. They might read the MD5 backwards. Lots of possibilities to make it harder to find out the algorithm.
If you can find out which binary creates the password in the first place, try to run signsrch on it, this might help in finding the algorithm it uses.
If you want to try a few more things with my perl program, here it is. Took about 1 hour on 1 core on a Xeon E5. There's a few things that you can optimize, but i supposed it doesn't make much sense to spend 30 minutes to make the program run 5 minutes faster.
#!/usr/bin/perl
use Digest::MD5 qw(md5_hex);
use List::Util qw(shuffle);
use strict;
use warnings;
$|=1;
my @admintest=(split("", "admintest"), ":", chr(127));
# my @admintest=(split("", "abc"), ":", chr(127));
my $n=0;
permute("", @admintest);
sub permute {
my $str=shift;
my @letters=@_;
my $str2;
for (my $i=0; $i<=$#letters; $i++) {
if ($letters[$i] eq chr(127)) {
tryseparatorsonstring($str);
return;
}
$str2=$str.$letters[$i];
permute($str2, @letters[0..$i-1], @letters[$i+1..$#letters]);
}
}
print "\n";
sub tryseparatorsonstring {
my $str=shift;
if ($str =~ /:/) {
for (my $sep=0; $sep<128; $sep++) {
my $str2=$str;
$str2=~s/:/chr($sep)/ge;
trystring($str2, $sep);
}
} else {
trystring($str, "");
}
}
sub trystring {
my $str=shift;
my $sep=shift;
my $pass=md5_hex($str);
print "$n\t$str\t$sep\t$pass\n" if $pass =~ /^3785a9/;
if (++$n%1_000_000==0) {
printf "%dM\r", $n/1_000_000;
}
# print "$str\n";
}
exit 0;
