Extracting a NAND flash dump with OOB data

Question

I'm trying to extract a NAND flash dump of an old Walkman player. The dump was done by a friend and unfortunately cannot be redone because the chip was destroyed. The dump was supposed to be just user data without OOB but it seems something went wrong and OOB is still there. I tried various ways of removing OOB and I can get some reasonably-looking data but many things are still off and I can't extract proper files.

Here's what I tried.

Just by looking visually into hex dump, there are a 16-byte lines beginning with "01 00 00 00" every 0x200 bytes which look out of place. I made a short script to remove 16 bytes after every 0x200:

import sys
inf = open(sys.argv[1] ,"rb")
of = open(sys.argv[2],"wb")
clen = 0x200
c = inf.read(clen)
off = 0
while c:
  off = inf.tell()
  oob = inf.read(0x10)
  if oob[:4]!="\x01\x00\x00\x00":
   print " bad OOB at %08X?" % off, oob.encode('hex')
   break
  of.write(c)
  if (off&0x1000) ==0:
   print "%08X" % off
  c = inf.read(clen)


inf.close()
of.close()
print "done."

The added sanity check triggers pretty quickly:

00000200
00000410
00000620
00000830
00000A40
00000C50
00000E60
 bad OOB at 00001070? 042080e4fbffffea10009fe530ff2fe1
done.

And indeed, the next line starting with 01 00 00 00 is at 1240, not 1070. I tried to account for it and restart, but I ran into similar issues later. So I wonder if I'm missing something. The full file is 4GB which is a bit heavy so here are few cut out chunks:

1. beginning of the dump (initial bootloader?). offset 0
2. start of U-boot bootloader. offset 0x104000
3. start of the kernel image.offset 0x186000

if you'd like to see the whole dump I've shared a link in our chat.

Hardware details: Device is Sony Walkman NWZ-A829. The flash chip is most likely TH58NVG6D1DTG20. The CPU is a NEC MP201 (ARMv5le).

GPL sources (U-Boot/Linux kernel) are available here: http://oss.sony.net/Products/Linux/Audio/NWZ-S715.html

The final goal is to figure out the firmware update encryption and produce custom firmware for the device.

Answer 1

So, I have figured it out myself in the end. I'll try to describe the process.

First, a bit of background on NAND: it is organized in pages which are grouped into blocks. You can read or write a single page at a time but erasing (which turns all bits to 1s (so bytes to FFs)) can be only done one block at a time (writing can only change bits from 1 to 0 but not the other way around, so to write new data the block usually has to be erased first). Usually the chip also has some extra storage per page to store ECC (error-correction code) and/or arbitrary "spare" data (also called OOB: out-of-band) which is not considered to be part of the useful storage of the chip but can still be read and written using low-level functions.

While browsing the hex dump, I came across this region after a bunch of FFs:

0000081FE0: FF FF FF FF FF FF FF FF │ FF FF FF FF FF FF FF FF          
0000081FF0: FF FF FF FF FF FF FF FF │ FF FF FF FF FF FF FF FF          
0000082000: 6C 4F 41 64 00 00 00 00 │ 00 00 00 00 00 00 00 00  lOAd
0000082010: 00 00 00 00 00 00 00 00 │ 00 00 00 00 00 00 00 00
0000082020: 00 00 00 00 00 00 00 00 │ 00 00 00 00 00 00 00 00
0000082030: 00 00 00 00 00 00 00 00 │ 00 00 00 00 00 00 00 00
0000082040: 00 00 00 00 00 00 00 00 │ 00 00 00 00 00 00 00 00
0000082050: 00 00 00 00 00 00 00 00 │ 00 00 00 00 00 00 00 00

It had the usual OOB-looking 16-byte chunks after each 0x200 bytes but then something different:

83040: 70 41 52 74 18 00 00 00 │ 48 00 00 00 90 00 00 00  pARt↑   H   
83050: A0 00 00 00 00 00 00 00 │ 00 00 00 00 00 00 00 00   
83060: 00 00 00 00 00 00 00 00 │ 00 00 00 00 00 00 00 00
....
84080: 75 42 54 74 20 00 00 00 │ 02 00 00 00 00 00 00 00  uBTt    ☻
84090: 00 00 00 00 00 00 00 00 │ 00 00 00 00 00 00 00 00
840A0: 00 00 00 00 00 00 00 00 │ 00 00 00 00 00 00 00 00
840B0: 00 00 00 00 00 00 00 00 │ 00 00 00 00 00 00 00 00

What's this? Sounds like U-Boot. Let's check U-boot sources from Sony's website... and bingo!

 /* for Loader Management */
 #define MNG_SIG_LOADER         0x64414F6C      /* "lOAd" */
 #define TBL_SIG_PART           0x74524170      /* "pARt" */
 #define TBL_SIG_UBOOT          0x74544275      /* "uBTt" */
 #define TBL_SIG_FUKRNL         0x744B5566      /* "fUKt" */
 #define TBL_SIG_FUINRD         0x74525566      /* "fURt" */
 #define TBL_SIG_BAD                0x74444162      /* "bADt" */

 #define    MNG_SIG_KRNL            0x6C4E526B      /* "kRNl" */
 #define TBL_SIG_KRNL           0x744C4E6B      /* "kNLt" */
 #define TBL_SIG_INRD           0x74445269      /* "iRDt" */

(from icx1087_nand.c) So, apparently these blocks are related to "Loader Management"... looking at the references we come across this code chunk:

/* read management table */
for( rty = 0; rty < NAND_READ_RETRY; rty++ ) {
    if( nand_search_tbl(NAND_BLK_LOADER_START, NAND_BLK_LOADER_LMT,
                        MNG_SIG_LOADER, TBL_SIG_PART, (void *)&ptbl, 0) ) {
            continue;
    }
    /* success to read Kernel information if reach here */
    break;
}

So, it's looking for MNG_SIG_LOADER (lOAd) in the NAND block NAND_BLK_LOADER_START. And the value is...

#define NAND_BLK_LOADER_START   1

So looks like 82000 is the start of NAND block 1. If we check the code further, we can see that it's looking for other signatures (e.g. TBL_SIG_PART or TBL_SIG_FUKRNL) in pages of this block. And let's see:

• MNG_SIG_LOADER: 82000 (block 1 page 0?)

• TBL_SIG_PART: 83040 (block 1 page 1?)

• TBL_SIG_UBOOT: 84080 (block 1 page 2?)

83040-82000=1040 and 84080-83040=1040 too! So looks like the page size in the dump is 1040. and 1040*128 =82000 which means we have 128 pages per block. The actual data is probably 0x1000 bytes and 0x40 is the "spare" data. So, this explains why I had desynchronization after each four 0x200 sectors: there are 8 sectors per page but only 0x40 bytes of OOB data.

I'm not sure why the dump ended up having such weird structure; I think it's because the software used for dumping (IIRC it was DumpFlash by Matt Oh) assumes 512-byte pages in many places and somehow the OOB data ended up being spread in chunks instead of being grouped at the end of each page. Possibly it's also related to this passage in the datasheet for a similar (but not same) Toshiba chip:

8 bit ECC for each 512Byte is required

(but no explanation on how it should be done)

In any case, once I figured out the magic numbers of 0x1040 for page size and 128 pages per block, it was not difficult to fix my script:

NAND_PAGE_SIZE  = 0x1000
NAND_PAGE_BLK   = 128
NAND_SECTOR_PER_PAGE = 8
NAND_SECTOR_SIZE = NAND_PAGE_SIZE/NAND_SECTOR_PER_PAGE
OOBLEN = 16

def page2off(pgno):
  return pgno*0x1040 # (NAND_PAGE_SIZE + NAND_SECTOR_PER_PAGE*OOBLEN)

def read_page(inf, blkno, pgno):
 blklen = page2off(NAND_PAGE_BLK)
 fileoff = blklen*blkno + page2off(pgno)
 print "reading block %d page %d: offset %08X" % (blkno, pgno, fileoff)
 inf.seek(fileoff, 0)
 block = inf.read(blklen)
 s = ""
 soff =0
 for i  in range(NAND_SECTOR_PER_PAGE):
  print "sector %d offset %08X" % (i, soff)
  s+= block[soff:soff+NAND_SECTOR_SIZE]
  soff += NAND_SECTOR_SIZE+OOBLEN
 return s

With this I could dump parts of the flash with the kernel and ramdisk and extract the binaries.

Answer 2

I am a developer of the Rockbox open-source project. We figured out the encryption of the firmware upgrade a look time ago by essentially doing the same thing (dumping the chip).

upgtool can extract the firmware from UPG tools, those can be extracted from the firmware upgrade installer using cabextract and looking at the file Data/Device/NW_WM_FW.UPG It requires an encryption key that can either be brute-forced (upgtool can do so if asked to [3]), or you can ask the device for the key using scsitool. We have a port to a bunch of Sony devices, the NW-A820 will not be supported because it has a very old kernel but most of the information we reverse-engineered probably still applies.

[3] upgtool has the key for the A820 in its database, it was found by brute force

Answer 3

Just a fast answer as future tip for NAND issues.

When make reading of NAND flashes (doesn't care if bga, tsop..), not all programmers make clear dump, normally it includes dummy blocks as you mentioned OOB data.

As NAND dumps should be multiple of 8, like 64, 128, 256MB...if the file you dump is larger than typical size, e.g. 132MB, then is mandatory to analyze the binary for remove dummy chunks and find out the pattern, then you could start playing with the binary.