The boot log seems similar to the one from this page, so there's a high possibility this device is using an AllWinner SoC. You can try the tools from the linked repository.
Another option could be to dump the flash and analyze it.
A couple of years of experience later, I revisited this project and figured it out!
The compression is in fact LZSS, and the QuickBMS script from my question is able to decompress it correctly. The second section I mentioned was problematic just because there are uncompressed blocks in between compressed blocks in the chip dump, so, obviously, they need to ...
Don’t know what swap is about, but spare area, or OOB area is an additional area at the end of each page storing metadata such as:
bad block markers
ECC (error checking and correcting) code
custom data which may be used by the system
It seems the spare option enables dumping it which is why the dump is slightly bigger.
For more info about NAND flash ...