The structure of the Python's marshaled code object (or .pyc file)

Question

Help me to decipher the Python's marshaled code object. The .pyc files are almost the same: The structure of .pyc files.

I have:

1. The code object compiled from the source.
2. The marshaled representation of this code object.
3. The recursive disassembly of its (code object) code section.
4. All its fields values.

The main purpose:

I want to find out, how the different code objects stored and referenced from each other. That is, how the links to the child code objects are stored? The module should have references to all its functions. The function should have references to all other functions, callable from it. Etc. Does Virtual Machine preserves the code object id, when storing it to the .pyc? I don't thinks so, because can't see ids in the .pyc file.

For example, I have such instruction in the disassembled source:

LOAD_CONST        2 (<code object baz at 0x7f380995e5d0, file "foo.py", line 7>)

Hence:

• How the Virtual Machine will found the baz code object? I can't see all this information: 0x7f380995e5d0, file "foo.py", line 7 in the marshaled string. Does the object id 0x7f380995e5d0 stored in the marshaled code or it is created each time the program is running?
• If it is not stored, how the connection of objects is preserved in the marshaled code objects (.pyc files)?

I think, I will investigate by the gdb further, but maybe this approach (.pyc file deciphering) also will do the job.

Current result:

I used all this information for creating the next file: the first column is the binary representation of marshaled code object, the second is the meaning of each byte sequences, which I have determined already.

b'
\xe3                    <don't know>
\x00\x00\x00\x00        <foo.py: co_argcount: 0>
\x00\x00\x00\x00        <foo.py: co_kwonlyargcount: 0>
\x00\x00\x00\x00        <foo.py: co_nlocals: 0>
\x03\x00\x00\x00        <foo.py: co_stacksize: 3>               
@\x00\x00\x00           <foo.py: co_flags = '@' = 0x40 = 64>
s.\x00\x00\x00          <foo.py: number of bytes for module instructions = '.' = 46>
d\x00                   <foo.py: co_code:  0 LOAD_CONST        0 (1)
Z\x00                   <foo.py: co_code:  2 STORE_NAME        0 (a)
d\x01                   <foo.py: co_code:  4 LOAD_CONST        1 (2)
Z\x01                   <foo.py: co_code:  6 STORE_NAME        1 (b)
e\x00                   <foo.py: co_code:  8 LOAD_NAME         0 (a)
e\x01                   <foo.py: co_code: 10 LOAD_NAME         1 (b)
\x17\x00                <foo.py: co_code: 12 BINARY_ADD
Z\x02                   <foo.py: co_code: 14 STORE_NAME        2 (c)
d\x02                   <foo.py: co_code: 16 LOAD_CONST        2 (<code object baz at 0x7f380995e5d0, file "foo.py", line 7>)
d\x03                   <foo.py: co_code: 18 LOAD_CONST        3 ('baz')
\x84\x00                <foo.py: co_code: 20 MAKE_FUNCTION     0
Z\x03                   <foo.py: co_code: 22 STORE_NAME        3 (baz)
e\x03                   <foo.py: co_code: 24 LOAD_NAME         3 (baz)
e\x00                   <foo.py: co_code: 26 LOAD_NAME         0 (a)
e\x01                   <foo.py: co_code: 28 LOAD_NAME         1 (b)
\x83\x02                <foo.py: co_code: 30 CALL_FUNCTION     2
Z\x04                   <foo.py: co_code: 32 STORE_NAME        4 (multiplication)
e\x04                   <foo.py: co_code: 34 LOAD_NAME         4 (multiplication)
d\x01                   <foo.py: co_code: 36 LOAD_CONST        1 (2)
\x13\x00                <foo.py: co_code: 38 BINARY_POWER
Z\x05                   <foo.py: co_code: 40 STORE_NAME        5 (square)
d\x04                   <foo.py: co_code: 42 LOAD_CONST        4 (None)
S\x00                   <foo.py: co_code: 44 RETURN_VALUE
)\x05                   <foo.py: co_const: size>
\xe9\x01\x00\x00\x00    <foo.py: co_const[0]: 1>
\xe9\x02\x00\x00\x00    <foo.py: co_const[1]: 2>
c                       <TYPE_CODE>
\x02\x00\x00\x00        <baz: co_argcount: 2>
\x00\x00\x00\x00        <baz: co_kwonlyargcount: 0>
\x02\x00\x00\x00        <baz: co_nlocals: 2>
\x02\x00\x00\x00        <baz: co_stacksize: 2>               
C\x00\x00\x00           <baz: co_flags = 'C' = 0x43 = 67>
s\x08\x00\x00\x00       <baz: co_code: size = 8 bytes>
|\x00                   <baz: co_code: 0 LOAD_FAST                0 (x) 
|\x01                   <baz: co_code: 2 LOAD_FAST                1 (y) 
\x14\x00                <baz: co_code: 4 BINARY_MULTIPLY                
S\x00                   <baz: co_code: 6 RETURN_VALUE                   
)\x01                   <baz: co_const: size>
N                       <baz: co_const[0]: None>
\xa9\x00                <don't know> 
)\x02                   <baz: co_varnames: size>
\xda\x01                <baz: number of characters of next item>
x                       <baz: co_varnames[0]: x>
\xda\x01                <baz: number of characters of next item>
y                       <baz: co_varnames[1]: y>
r\x03\x00\x00\x00       <baz: don't know. But the 'r' = 'TYPE_REF'>
r\x03\x00\x00\x00       <baz: don't know. But the 'r' = 'TYPE_REF'>
\xfa\x06                <baz: next item length>
foo.py                  <baz: co_filename>
\xda\x03                <baz: number of characters of next item>
baz                     <baz: co_name: 'baz'>
\x07\x00\x00\x00        <baz: co_firstlineno: 7>
s\x02\x00\x00\x00       <baz: co_lnotab: size = 2 >
\x00\x01                <baz: co_lnotab>
r\x07\x00\x00\x00       <foo.py: co_const[3]: reference to baz>
N                       <foo.py: co_const[4]: None>
)\x06                   <foo.py: co_names: size> 
\xda\x01                <foo.py: number of characters of next item>
a                       <foo.py: co_names[0]: a>
\xda\x01                <foo.py: number of characters of next item>
b                       <foo.py: co_names[1]: b>
\xda\x01                <foo.py: number of characters of next item>
c                       <foo.py: co_names[2]: c>
r\x07\x00\x00\x00       <foo.py: co_names[3]: reference to baz>
Z\x0e                   <foo.py: number of characters of next item>
multiplication          <foo.py: co_names[4]: multiplication>
Z\x06                   <foo.py: number of characters of next item>
square                  <foo.py: co_names[5]: square>
r\x03\x00\x00\x00       <foo.py: don't know>     
r\x03\x00\x00\x00       <foo.py: don't know>     
r\x03\x00\x00\x00       <foo.py: don't know>     
r\x06\x00\x00\x00       <foo.py: don't know>     
\xda\x08                <foo.py: number of characters of next item>
<module>                <foo.py: co_name>
\x03\x00\x00\x00        <foo.py: co_firstlineno>
s\n\x00\x00\x00         <foo.py: co_lnotab: size = '\n' = 0A>
\x04\x01                <foo.py: o_lnotab> 
\x04\x01                <foo.py: o_lnotab>
\x08\x02                <foo.py: o_lnotab>
\x08\x07                <foo.py: o_lnotab>
\n\x01'                 <foo.py: o_lnotab>

---

Code snippets required for replication:

1) The source code: foo.py

a = 1 
b = 2 
c = a + b 

def baz(x,y):
    return x * y

multiplication = baz(a,b)
square = multiplication ** 2

2) The marshaled representation of foo.py.

source_py = "foo.py"

with open(source_py) as f_source:
    source_code = f_source.read()

code_obj_compile = compile(source_code, source_py, "exec")

data = marshal.dumps(code_obj_compile)

print(data)

3) The full (recursive) disassembly of the code object.

import types

dis.dis(code_obj_compile)

for x in code_obj_compile.co_consts:
    if isinstance(x, types.CodeType):
        sub_byte_code = x
        func_name = sub_byte_code.co_name
        print('\nDisassembly of %s:' % func_name)
        dis.dis(sub_byte_code)

4) All code object's field values.

def print_co_obj_fields(code_obj):
    # Iterating through all instance attributes
    # and calling all having the 'co_' prefix
    for name in dir(code_obj):
        if name.startswith('co_'):
            co_field = getattr(code_obj, name)
            print(f'{name:<20} = {co_field}')

print_co_obj_fields(code_obj_compile)

Answer 1

The answer below is with reference to Python 2.7

How the Virtual Machine will found the baz code object? I can't see all this information: 0x7f380995e5d0, file "foo.py", line 7 in the marshaled string. Does the object id 0x7f380995e5d0 stored in the marshaled code or it is created each time the program is running?

The baz code object is located within the co_consts member. Going by your example.

>>> import marshal
>>> import dis
>>> 
>>> source_py = "foo.py"
>>> 
>>> with open(source_py) as f_source:
...     source_code = f_source.read()
>>> 

>>> code_obj_compile = compile(source_code, source_py, "exec")

If you disassemble, the newly generated code object you can find references to baz

>>> dis.dis(code_obj_compile)
  1           0 LOAD_CONST               0 (7)
              3 STORE_NAME               0 (a)

  2           6 LOAD_CONST               1 (5)
              9 STORE_NAME               1 (b)

  3          12 LOAD_NAME                0 (a)
             15 LOAD_NAME                1 (b)
             18 BINARY_ADD
             19 STORE_NAME               2 (c)

  5          22 LOAD_CONST               2 (<code object baz at 0x7f1dcdb06bb0, file "foo.py", line 5>)
             25 MAKE_FUNCTION            0
... snip...

The baz code object is located within the co_consts array of the parent code object as shown below.

>>> code_obj_compile.co_consts[2]
<code object baz at 0x7f1dcdb06bb0, file "foo.py", line 5>

And you can disassemble that as well.

>>> dis.dis(code_obj_compile.co_consts[2])
  6           0 LOAD_FAST                0 (x)
              3 LOAD_FAST                1 (y)
              6 BINARY_MULTIPLY
              7 RETURN_VALUE

The objects will be created each time the program is run. Hence the addresses will change correspondingly.

If it is not stored, how the connection of objects is preserved in the marshaled code objects (.pyc files)?

Just explained. If you look at the instruction carefully you can notice the LOAD_CONST instruction takes a offset as a parameter - the operand.

  5          22 LOAD_CONST               2 (<code object baz at 0x7f1dcdb06bb0, file "foo.py", line 5>)

Here the offset is 2 which instructs the Python Virtual Machine to load the third (zero based) item in the co_consts array on to the evaluation stack. So the "connection" is preserved using the offsets into the other metadata members.

Answer 2

The purpose of a code object marshaling is the storing and restoring the program to/from the file. Therefore, it should have the coding scheme for all Python's features: objects, bytecode, names, etc, otherwise it can't restore a program from the file.

So, it uses multiple type identifiers, which can be divided into four groups:

• single TYPE: {the type identifier} the size is 1 byte.

   Example: TYPE_NONE = 'N'`, `TYPE_TRUE = 'T'.
  

• short TYPE: {the type identifier} + 1 byte value

   Example: TYPE_SHORT_ASCII_INTERNED = 'Z'.
  

• long TYPE: {the type identifier} + 4 bytes value

   Example: TYPE_STRING = 's'.
  

• object TYPE: {the type identifier} + the combination of all different types, including the object TYPE itself. That is, it has the recursive structure.

   Example: TYPE_CODE = 'c'
  

All types can be seen here: cpython/Python/marshal.c

Also, the code object has multiple int fields. They doesn't have identifier in the marshaled string, just the sequence of four bytes values.

    int co_argcount;            /* #arguments, except *args */
    int co_kwonlyargcount;      /* #keyword only arguments */
    int co_nlocals;             /* #local variables */
    int co_stacksize;           /* #entries needed for evaluation stack */
    int co_flags;               /* CO_..., see below */
    int co_firstlineno;         /* first source line number */
    

The full code object structure is here: cpython/Include/code.h

It is useful to know the order, in which the code object have been dumped, because then we can calculate the each field offset in the resulting string, like - the first four byte is co_argcount, the second is co_kwonlyargcount, etc.

The order of code object dumping:

    # PyCodeObject *co - pointer to the code object
    # p                - pointer to the file object,
    that accumulating marshaled code object before
    writing to the file.
    
    W_TYPE(TYPE_CODE, p);
    w_long(co->co_argcount, p);
    w_long(co->co_kwonlyargcount, p);
    w_long(co->co_nlocals, p);
    w_long(co->co_stacksize, p);
    w_long(co->co_flags, p);
    w_object(co->co_code, p);
    w_object(co->co_consts, p);
    w_object(co->co_names, p);
    w_object(co->co_varnames, p);
    w_object(co->co_freevars, p);
    w_object(co->co_cellvars, p);
    w_object(co->co_filename, p);
    w_object(co->co_name, p);
    w_long(co->co_firstlineno, p);
    w_object(co->co_lnotab, p);

The result: foo.py marshaled string fully deciphered:

b'
\xe3                    <foo.py: '\xe3' & 0x80 (FLAG_REF)  = 'c' (TYPE_CODE)>
\x00\x00\x00\x00        <foo.py: co_argcount: 0>
\x00\x00\x00\x00        <foo.py: co_kwonlyargcount: 0>
\x00\x00\x00\x00        <foo.py: co_nlocals: 0>
\x03\x00\x00\x00        <foo.py: co_stacksize: 3>               
@\x00\x00\x00           <foo.py: co_flags = '@' = 0x40 = 64>
s.\x00\x00\x00          <foo.py: number of bytes for module instructions = '.' = 46>
d\x00                   <foo.py: co_code:  0 LOAD_CONST        0 (1)
Z\x00                   <foo.py: co_code:  2 STORE_NAME        0 (a)
d\x01                   <foo.py: co_code:  4 LOAD_CONST        1 (2)
Z\x01                   <foo.py: co_code:  6 STORE_NAME        1 (b)
e\x00                   <foo.py: co_code:  8 LOAD_NAME         0 (a)
e\x01                   <foo.py: co_code: 10 LOAD_NAME         1 (b)
\x17\x00                <foo.py: co_code: 12 BINARY_ADD
Z\x02                   <foo.py: co_code: 14 STORE_NAME        2 (c)
d\x02                   <foo.py: co_code: 16 LOAD_CONST        2 (<code object baz at 0x7f380995e5d0, file "foo.py", line 7>)
d\x03                   <foo.py: co_code: 18 LOAD_CONST        3 ('baz')
\x84\x00                <foo.py: co_code: 20 MAKE_FUNCTION     0
Z\x03                   <foo.py: co_code: 22 STORE_NAME        3 (baz)
e\x03                   <foo.py: co_code: 24 LOAD_NAME         3 (baz)
e\x00                   <foo.py: co_code: 26 LOAD_NAME         0 (a)
e\x01                   <foo.py: co_code: 28 LOAD_NAME         1 (b)
\x83\x02                <foo.py: co_code: 30 CALL_FUNCTION     2
Z\x04                   <foo.py: co_code: 32 STORE_NAME        4 (multiplication)
e\x04                   <foo.py: co_code: 34 LOAD_NAME         4 (multiplication)
d\x01                   <foo.py: co_code: 36 LOAD_CONST        1 (2)
\x13\x00                <foo.py: co_code: 38 BINARY_POWER
Z\x05                   <foo.py: co_code: 40 STORE_NAME        5 (square)
d\x04                   <foo.py: co_code: 42 LOAD_CONST        4 (None)
S\x00                   <foo.py: co_code: 44 RETURN_VALUE
)\x05                   <foo.py: co_const: size>
\xe9\x01\x00\x00\x00    <foo.py: co_const[0]: 1; '\xe9' & 0x80 (FLAG_REF) = 'i' (TYPE_INT)>
\xe9\x02\x00\x00\x00    <foo.py: co_const[1]: 2; '\xe9' & 0x80 (FLAG_REF) = 'i' (TYPE_INT)>
c                       <foo.py: co_const[2]: 'c' = TYPE_CODE>
\x02\x00\x00\x00        <baz: co_argcount: 2>
\x00\x00\x00\x00        <baz: co_kwonlyargcount: 0>
\x02\x00\x00\x00        <baz: co_nlocals: 2>
\x02\x00\x00\x00        <baz: co_stacksize: 2>               
C\x00\x00\x00           <baz: co_flags = 'C' = 0x43 = 67>
s\x08\x00\x00\x00       <baz: co_code: size = 8 bytes>
|\x00                   <baz: co_code: 0 LOAD_FAST                0 (x) 
|\x01                   <baz: co_code: 2 LOAD_FAST                1 (y) 
\x14\x00                <baz: co_code: 4 BINARY_MULTIPLY                
S\x00                   <baz: co_code: 6 RETURN_VALUE                   
)\x01                   <baz: co_const: size>
N                       <baz: co_const[0]: None>
\xa9\x00                <baz: co_names: size = 0  '\xa9' & 0x80 (FLAG_REF)  = ')'> 
)\x02                   <baz: co_varnames: size>
\xda\x01                <baz: number of characters of next item; '\xda' & 0x80 (FLAG_REF)  = 'Z'>
x                       <baz: co_varnames[0]: x>
\xda\x01                <baz: number of characters of next item; '\xda' & 0x80 (FLAG_REF)  = 'Z'>
y                       <baz: co_varnames[1]: y>
r\x03\x00\x00\x00       <baz: co_freevars: reference to empty tuple '()'>     
r\x03\x00\x00\x00       <baz: co_cellvars: reference to empty tuple '()'>
\xfa\x06                <baz: next item length>
foo.py                  <baz: co_filename>
\xda\x03                <baz: number of characters of next item>
baz                     <baz: co_name: 'baz'>
\x07\x00\x00\x00        <baz: co_firstlineno: 7>
s\x02\x00\x00\x00       <baz: co_lnotab: size = 2 >
\x00\x01                <baz: co_lnotab>
r\x07\x00\x00\x00       <foo.py: co_const[3]: reference to 'baz'>
N                       <foo.py: co_const[4]: None>
)\x06                   <foo.py: co_names: size> 
\xda\x01                <foo.py: number of characters of next item>
a                       <foo.py: co_names[0]: a>
\xda\x01                <foo.py: number of characters of next item>
b                       <foo.py: co_names[1]: b>
\xda\x01                <foo.py: number of characters of next item>
c                       <foo.py: co_names[2]: c>
r\x07\x00\x00\x00       <foo.py: co_names[3]: reference to 'baz'>
Z\x0e                   <foo.py: number of characters of next item>
multiplication          <foo.py: co_names[4]: multiplication>
Z\x06                   <foo.py: number of characters of next item>
square                  <foo.py: co_names[5]: square>
r\x03\x00\x00\x00       <foo.py: co_varnames: reference to empty tuple '()'>     
r\x03\x00\x00\x00       <foo.py: co_freevars: reference to emtpy tuple '()'>     
r\x03\x00\x00\x00       <foo.py: co_cellvars: reference to empty tuple '()'>
r\x06\x00\x00\x00       <foo.py: co_filename: reference to 'foo.py'>     
\xda\x08                <foo.py: number of characters of next item>
<module>                <foo.py: co_name>
\x03\x00\x00\x00        <foo.py: co_firstlineno>
s\n\x00\x00\x00         <foo.py: co_lnotab: size = '\n' = 0A>
\x04\x01                <foo.py: o_lnotab> 
\x04\x01                <foo.py: o_lnotab>
\x08\x02                <foo.py: o_lnotab>
\x08\x07                <foo.py: o_lnotab>
\n\x01'                 <foo.py: o_lnotab>

The useful information:

How to create a code object in python?