Based on those images, this appears to be a fairly straight-forward run-length encoded 32-bits-per-pixel bitmap format.
The general structure seems to consist of number of records each with a single 'tag' byte followed by a multiple of 4 bytes of RGBA data. These records appear to encapsulate pixel data only, with no dimensions or other metadata.
I read about similar issue with ZTE modem: ZTE encrypted backup config file
It was ZLIB-compressed and encrypted with AES in ECB mode with a 16-byte key. You can read this topic, maybe it can work for your modem too.
I'll post this as an answer to techniques to reverse engineer an unknown file. Reverse engineering is not about just looking at binary data. It is a detective mystery. You need to gather clues and try to solve the mystery. Sometimes the clues are outside the binary data.
Look at the program generating the data. Look for places where you know the inputs and ...
I'm working on some tooling for automatic reverse engineering.
Having messages of varying length makes it much easier to determine which fields are related to overall message lengths. It also makes it much easier to identify where the 'header' portion is, as it will have a consistent format and precede the variable length portion.
The more data and the more ...