In this post I will take what we learned in the last post and use it to write some telemetry into a video.

Writing telemetry samples as binary (for `mdat` box)

As you now now, each telemetry samples need to be turned into binary for injection into the `mdat` box.

The Python3 struct library can be used to perform conversions between Python values and C structs represented as Python bytes objects which can be written into `mdat`.

Lets use this CAMM case 6 sample to demonstrate;

Sample 0

``````{"time_gps_epoch": "1553343930000.0", "gps_fix_type": 3, "latitude": 51.26006666666667, "longitude": -0.9531388888888889, "altitude": 127.7, "horizontal_accuracy": 1.0, "vertical_accuracy": 1.0, "velocity_east": -0.42552019866750024, "velocity_north": 0.2466133528597531, "velocity_up": 0.0, "speed_accuracy": 0.0}
``````

Here’s how we can convert the values to Python bytes using struct;

``````python3
``````
``````import struct
time_gps_epoch_0 = struct.pack('<d', 1553343930000.0)
gps_fix_type_0 = struct.pack('<i', 3)
latitude_0 = struct.pack('<d', 51.26006666666667)
longitude_0 = struct.pack('<d', -0.9531388888888889)
altitude_0 = struct.pack('<f', 127.7)
horizontal_accuracy_0 = struct.pack('<f', 1.0)
vertical_accuracy_0 = struct.pack('<f', 1.0)
velocity_east_0 = struct.pack('<f', -0.42552019866750024)
velocity_north_0 = struct.pack('<f', 0.2466133528597531)
velocity_up_0 = struct.pack('<f', 0.0)
speed_accuracy_0 = struct.pack('<f', 0.0)
``````

The `<` defines the string as little endian, this is defined in the CAMM specification;

All fields are little-endian (least significant byte first), and the 32-bit floating points are of IEEE 754-1985 format.

Following the `<` character is either `d`, `i`, or `f`, this defines the data type; either double, int32 or float. The datatype required for each property is also defined in the CAMM spec.

You’ll notice don’t need to write the property name (e.g. `latitude`) into the binary. The structure of the telemetry (CAMM) will be defined later in the telemetry `trak` box where someone (or something) can understand what these value represent. You only need to write the actual values of these properties into binary.

If we print one of these fields to examine it, we get a response in Python bytes, for example;

``````print(time_gps_epoch_0)
b'\x00\x00\xe95\xa8\x9avB'
``````

We can also decode it to check;

``````struct.unpack('<d', time_gps_epoch_0)[0]
1553343930000.0
``````

To create the telemetry data we can concatenate these together into a single sample;

``````sample_0_no_head = (time_gps_epoch_0+gps_fix_type_0+latitude_0+longitude_0+altitude_0+horizontal_accuracy_0+vertical_accuracy_0+velocity_east_0+velocity_north_0+velocity_up_0+speed_accuracy_0)
``````

Which looks like this;

``````print(sample_0_no_head)
b'\x00\x00\xe95\xa8\x9avB\x03\x00\x00\x00y\x0eR\xddI\xa1[email protected]\x91Z\x8a \x1d\x80\xee\xbfff\xffB\x00\x00\x80?\x00\x00\x80?\xc9\xdd\xd9\xbe6\x88|>\x00\x00\x00\x00\x00\x00\x00\x00'
``````

Now we also need to account for the header to define the CAMM case as per the specification (in this example, case 6).

The header consists of 2 parts: the first 2 bytes contains `0` as defined by the specification and next 2 bytes contains CAMM case number. We can write this like so;

``````reserved = struct.pack('<H', 0)
camm_case = struct.pack('<H', 6)

``````

Which looks like this;

``````print(header)
b'\x00\x00\x06\x00'
``````

Therefore to create a valid CAMM case 6 sample;

``````sample_0=(header+sample_0_no_head)
``````

Which encoded looks like this;

``````print(sample_0)
b'\x00\x00\x06\x00\x00\x00\xe95\xa8\x9avB\x03\x00\x00\x00y\x0eR\xddI\xa1[email protected]\x91Z\x8a \x1d\x80\xee\xbfff\xffB\x00\x00\x80?\x00\x00\x80?\xc9\xdd\xd9\xbe6\x88|>\x00\x00\x00\x00\x00\x00\x00\x00'
``````

I know from the CAMM spec that CAMM case 6 samples are always 60 bytes (3 `double` values of 8 bytes + 1 `int32` value of 4 bytes + 7 `float` values of 4 bytes + 4 byte header). We can be sure of the byte sizes of each CAMM 6 case sample will be the same as shown above, as all field values must be reported in the payload.

Now let’s continue the example. Imagine there are 5 more samples reported by the GPS sensor as follows;

Sample 1

Input values:

``````{"time_gps_epoch": "1553343930200.0", "gps_fix_type": 3, "latitude": 51.26007777777778, "longitude": -0.9531694444444444, "altitude": 126.2, "horizontal_accuracy": 1.0, "vertical_accuracy": 1.0, "velocity_east": -0.2960776004545653, "velocity_north": 0.18875339246390743, "velocity_up": 0, "speed_accuracy": 0}
``````

Binary output:

``````print(sample_1)
b'\x00\x00\x06\x00\x00\x80\xf55\xa8\x9avB\x03\x00\x00\x00h\xfc\x86:J\xa1[email protected]\x9d\xee\xed4]\x80\xee\xbfff\xfcB\x00\x00\x80?\x00\x00\x80?|\x97\x97\xbe\x92HA>\x00\x00\x00\x00\x00\x00\x00\x00'
``````

Sample 2

Input values:

``````{"time_gps_epoch": "1553343930400.0", "gps_fix_type": 3, "latitude": 51.26009444444445, "longitude": -0.9532111111111111, "altitude": 124.3, "horizontal_accuracy": 1.0, "vertical_accuracy": 1.0, "velocity_east": -0.30465075085200577, "velocity_north": 0.1477752379350783, "velocity_up": 0, "speed_accuracy": 0}
``````

Binary output:

``````print(sample_2)
b'\x00\x00\x06\x00\x00\x80\xf55\xa8\x9avB\x03\x00\x00\x00h\xfc\x86:J\xa1[email protected]\x9d\xee\xed4]\x80\xee\xbfff\xfcB\x00\x00\x80?\x00\x00\x80?|\x97\x97\xbe\x92HA>\x00\x00\x00\x00\x00\x00\x00\x00'
``````

Sample 3

Input values:

``````{"time_gps_epoch": "1553343930600.0", "gps_fix_type": 3, "latitude": 51.26011388888889, "longitude": -0.953275, "altitude": 122.6, "horizontal_accuracy": 1.0, "vertical_accuracy": 1.0, "velocity_east": -0.20935273424251769, "velocity_north": 0.11122101212837258, "velocity_up": 0, "speed_accuracy": 0}
``````

Binary output:

``````print(sample_3)
b'\x00\x00\x06\x00\x00\x00\x026\xa8\x9avB\x03\x00\x00\x00NaV\xc6J\xa1[email protected]\xfe\x8c\x96\xb4\x80\xee\xbf\x9a\x99\xf8B\x00\x00\x80?\x00\x00\x80?/\xfb\x9b\xbedR\x17>\x00\x00\x00\x00\x00\x00\x00\x00'
``````

Sample 4

Input values:

``````{"time_gps_epoch": "1553343930800.0", "gps_fix_type": 3, "latitude": 51.260133333333336, "longitude": -0.9533333333333333, "altitude": 120.8, "horizontal_accuracy": 1.0, "vertical_accuracy": 1.0, "velocity_east": -0.1918253910417178, "velocity_north": 0.11464815023472924, "velocity_up": 0, "speed_accuracy": 0}
``````

Binary output:

``````print(sample_4)
b'\x00\x00\x06\x00\x00\x80\x0e6\xa8\x9avB\x03\x00\x00\x00\xaf\x01siK\xa1[email protected]\x05\xa3\x92:\x81\xee\xbf33\xf5B\x00\x00\x80?\x00\x00\x80?\x90`V\xbe\xd8\xc7\xe3=\x00\x00\x00\x00\x00\x00\x00\x00'
``````

Sample 5

Input values:

``````{"time_gps_epoch": "1553343931000.0", "gps_fix_type": 3, "latitude": 51.26015833333334, "longitude": -0.9533999999999999, "altitude": 120.1, "horizontal_accuracy": 1.0, "vertical_accuracy": 1.0, "velocity_east": -0.18010009841425276, "velocity_north": 0.11694262588071799, "velocity_up": 0, "speed_accuracy": 0}
``````

Binary output:

``````print(sample_5)
b'\x00\x00\x06\x00\x00\x00\x1b6\xa8\x9avB\x03\x00\x00\x00\x11\xa2\x8f\x0cL\xa1[email protected]\x81N\x1b\xe8\xb4\x81\xee\xbf\x9a\x99\xf1B\x00\x00\x80?\x00\x00\x80?\xe0mD\xbe\xa6\xcc\xea=\x00\x00\x00\x00\x00\x00\x00\x00'
``````

To help demonstrate it, try downloading and running this script (`example-camm6-telemetry.py`) showing how I obtain all the values above:

``````curl https://gist.githubusercontent.com/himynamesdave/ca819516d4dad7b96d602d0531945227/raw/87fd6593dab7f8044322c16dd8ba499fc8d45b0c/example-camm6-telemetry.py  > example-camm6-telemetry.py
python3 example-camm6-telemetry.py
``````

Now that we have our binary samples we can add them to the `mdat` box.

Firstly, remember last week I talked about how `mdat` usually contained an interleaved mix of video, audio and telemetry (and more possibly other) byte data. As I explained, this is because on cameras with limited memory, it is essential to write the data being generated from all sensors as quickly as possible.

However, there is no requirement to do this. One of the advantages of post-processing of telemetry and adding it to an mp4 is that we can simply append the entire telemetry binary data to the end of the bytes that already exist in the `mdat` box. For example, if the video only contained audio and video, we could add the telemetry after all those bytes in the `mdat` box.

Lets walk through an example of that.

Take the `GS018421.mp4` video, it has no telemetry. How do I know? Let’s look at the output of Telemetry Injector tools script `print_video_atoms_overview.py`;

``````python3 print_video_atoms_overview.py GS018421.mp4
``````
``````mpeg4 [294220070]
├── b'ftyp' [8, 12]
├── b'wide' [8, 0]
├── b'mdat' [8, 294196360]
├── b'skip' [8, 11602]
└── b'moov' [8, 12056]
├── b'mvhd' [8, 100]
├── b'trak' [8, 6989]
│   ├── b'tkhd' [8, 84]
│   ├── b'tapt' [8, 60]
│   ├── b'edts' [8, 28]
│   ├── b'mdia' [8, 6331]
│   │   ├── b'mdhd' [8, 24]
│   │   ├── b'hdlr' [8, 41]
│   │   └── b'minf' [8, 6242]
│   │       ├── b'vmhd' [8, 12]
│   │       ├── b'hdlr' [8, 48]
│   │       ├── b'dinf' [8, 28]
│   │       └── b'stbl' [8, 6122]
│   │           ├── b'stsd' [8, 177]
│   │           │   └── b'avc1' [8, 161]
│   │           ├── b'stts' [8, 208]
│   │           ├── b'stss' [8, 92]
│   │           ├── b'sdtp' [8, 609]
│   │           ├── b'stsc' [8, 1220]
│   │           ├── b'stsz' [8, 2432]
│   │           └── b'stco' [8, 1328]
│   └── b'uuid' [8, 446]
└── b'trak' [8, 4943]
├── b'tkhd' [8, 84]
├── b'edts' [8, 28]
└── b'mdia' [8, 4807]
├── b'mdhd' [8, 24]
├── b'hdlr' [8, 41]
└── b'minf' [8, 4718]
├── b'smhd' [8, 8]
├── b'hdlr' [8, 48]
├── b'dinf' [8, 28]
└── b'stbl' [8, 4602]
├── b'stsd' [8, 150]
│   └── b'mp4a' [8, 134]
│       └── b'wave' [8, 82]
│           ├── b'frma' [8, 4]
│           ├── b'mp4a' [8, 4]
│           ├── b'esds' [8, 42]
│           └── b'\x00\x00\x00\x00' [8, 0]
├── b'stts' [8, 16]
├── b'stsc' [8, 428]
├── b'stsz' [8, 3800]
└── b'stco' [8, 168]
``````

You can only see an audio (`avc1`) `trak` and video (`mp4a`) `trak` in the metadata box (`moov`).

So lets add some. We know the `mdat` track it `294196360` bytes long and we want to append the telemetry data to the end of this file (first byte of telemetry will be at `294196361` bytes in the `mdat`).

In total we have 6 telemetry samples of 60 bytes (so 360 bytes total of telemetry to add).

``````all_samples = (sample_0+sample_1+sample_2+sample_3+sample_4+sample_5)
``````

Which is:

``````print(all_samples)
\x00\x00\x06\x00\x00\x00\xe95\xa8\x9avB\x03\x00\x00\x00y\x0eR\xddI\[email protected]\x91Z\x8a \x1d\x80\xee\xbfff\xffB\x00\x00\x80?\x00\x00\x80?\xc9\xdd\xd9\xbe6\x88|>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x80\xf55\xa8\x9avB\x03\x00\x00\x00h\xfc\x86:J\[email protected]\x9d\xee\xed4]\x80\xee\xbfff\xfcB\x00\x00\x80?\x00\x00\x80?|\x97\x97\xbe\x92HA>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\x026\xa8\x9avB\x03\x00\x00\x00NaV\xc6J\[email protected]\xfe\x8c\x96\xb4\x80\xee\xbf\x9a\x99\xf8B\x00\x00\x80?\x00\x00\x80?/\xfb\x9b\xbedR\x17>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x80\x0e6\xa8\x9avB\x03\x00\x00\x00\xaf\x01siK\[email protected]\x05\xa3\x92:\x81\xee\xbf33\xf5B\x00\x00\x80?\x00\x00\x80?\x90`V\xbe\xd8\xc7\xe3=\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x00\x1b6\xa8\x9avB\x03\x00\x00\x00\x11\xa2\x8f\x0cL\[email protected]\x81N\x1b\xe8\xb4\x81\xee\xbf\x9a\x99\xf1B\x00\x00\x80?\x00\x00\x80?\xe0mD\xbe\xa6\xcc\xea=\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x06\x00\x00\x80'6\xa8\x9avB\x03\x00\x00\x00j\xb9F\xdeL\[email protected]\x80\[email protected]\x82\xee\xbf33\xf0B\x00\x00\x80?\x00\x00\x80?)l8\xbe\x9e\x7f\xef=\x00\x00\x00\x00\x00\x00\x00\x00
``````

Now all that’s left to do is append `all_samples` to the `mdat` media (starting at `294196361` bytes)

To do this is easy, take the `mdat` binary and add the `all_samples`

TODO - HOW DO WE ACTUALLY DO THIS IN CODE? – TAKE THIS BINARY AND APPEND IT TO PRODUCE THE ABOVE?

Now this is complete, we need to describe the data that’s been added (appended) to the `mdat` binary so that the telemetry is contains can be decoded and processed correctly.

Writing metadata samples as binary (for `moov` box)

We’ve now added CAMM case 5 telemetry to the `mdat` media. Now we need to describe it.

In the last post, using a sample CAMM video we saw how we need to update the boxes nested in the `stbl`.

`````` └── b'moov' [8, 5171]
├── b'mvhd' [8, 100]
├── b'meta' [8, 111]
├── b'trak' [8, 2565]
│   ├── b'tkhd' [8, 84]
│   └── b'mdia' [8, 2465]
│       ├── b'mdhd' [8, 24]
│       ├── b'hdlr' [8, 36]
│       └── b'minf' [8, 2381]
│           ├── b'nmhd' [8, 4]
│           ├── b'dinf' [8, 28]
│           └── b'stbl' [8, 2325]
│               ├── b'stsd' [8, 33]
│               │   └── b'camm' [8, 17]
│               ├── b'stts' [8, 1352]
│               ├── b'stsz' [8, 724]
│               ├── b'stsc' [8, 80]
│               └── b'co64' [8, 96]
``````

To do this we first need to write the `stbl` box; which contains `stsd` (and `camm`), `stts`, `stsz`, `stsc`, and `co64` boxes. Note, it is also required to write to other boxes, but I will cover these in the standard specific posts.

Lets walk through this using the earlier example by first explaining each box. I will then cover how to write these as binary.

`stbl` (and `camm`) box

The sample table atom contains information for converting from media time to sample number to sample location. This atom also indicates how to interpret the sample (for example, whether to decompress the video data and, if so, how). This section describes the format and content of the sample table atom.

Source: Quicktime specification

In short this box has a size (of all nested atoms) and type (`stbl`) but then it simply acts as a containter for the boxes that follow.

For example, when writing as binary, I would fitst convert the size and type like so;

``````TODO
stbl_hearer=x
``````

Then I would simply append the nested boxes described from the next section onwards.

``````stbl_box=(stbl_hearer+stsd_box+stts_box+stsz_box+stsc_box+co64_box)
``````

Here is how to create those variables…

`stsd` (and `camm`) box

In short, for telemetry, the sample table box contains information about the telemetry, including the different types and structure of data that can be found in reported samples (e.g. from different sensors) and how it can be interpreted.

Here’s an example of the data elements contained in the `stsd` box for CAMM telemetry;

• atom size (32-bit integer): the total size in bytes of this box (and all child boxes)
• type (32-bit integer): sets the box type, always `stsd`
• version (1-byte specification): default `0` (meets our requirements), if version is 1 then date and duration values are 8 bytes in length
• flags (3-byte space): always set to `0`
• number of entries (32-bit integer): number of entries in the sample descriptions that follow (1 in my example)
• sample description table: An array of sample descriptions

A basic sample description table (for CAMM) looks as follows;

``````8,camm,0,1
``````

That is, the row is `8` bytes, the data type is `camm`, format reserved data `0`, and the data reference index is `1`.

TODO – CAN YOU SUPPLY CODE TO DEMO CREATING THE STSD BOX

`stts` (time to sample box) box

Let’s assume the timescale in the `mdhd` box is defined as 90000.

As we have one point every 0.2 seconds, each sample covers 18000 (`90000/5`).

Therefore we get the following time-to-sample table

``````6,18000
``````

6 points, each covering 18000.

The `stts` box also requires the following data elements;

• atom size (32-bit integer): the total size in bytes of this atom (no nested atom, so sum of box)
• type (32-bit integer): sets the box type, always `stts`
• version (1-byte specification): default `0` (meets our requirements), if version is 1 then date and duration values are 8 bytes in length
• flags (3-byte space): always set to `0`
• number of entries (32-bit integer): number of entries in the sample descriptions that follow (in our example `1`)
• sample description table: An array of sample descriptions (see above)

So taking all this information, we can write the following binary;

TODO – CAN YOU SUPPLY CODE TO DEMO CREATING THE STTS BOX

`stsz` (sample size box)

As we’re dealing with CAMM case 6 telemetry we know that each sample is exactly 60 bytes, which gives a sample size table of;

``````60
60
60
60
60
60
``````

The `stsz` box also requires the following data elements;

• atom size (32-bit integer): the total size in bytes of this atom (no nested atom, so sum of box)
• type (32-bit integer): sets the box type, always `stsz`
• version (1-byte specification): default `0` (meets our requirements), if version is 1 then date and duration values are 8 bytes in length
• flags (3-byte space): always set to `0`
• sample size: count of samples (in our example `6`)
• number of entries (32-bit integer): number of entries in the sample descriptions that follow (in our example `1`)
• sample size table: An array of sample descriptions (see above)

So taking all this information, we can write the following binary;

TODO – CAN YOU SUPPLY CODE TO DEMO CREATING THE STSZ BOX

`stsc` (sample to chunk box)

To keep things simple, mainly because our telemetry is simple (just one CAMM case), I will assign each sample to a chunk, giving a sample to chunk data element in the `stsc` box as follows;

``````1,1,1
``````

Each table entry corresponds to a set of consecutive chunks, each of which contains the same number of samples (thus we only need one line).

This translates to; Chunk 1 has 1 sample in it, and the sample is described in the `stsd` under sample description table where the data reference index column is 1. Chunk 2 has 1 sample in it and the data reference is one, and so on…

The `stsc` box also requires the following data elements;

• atom size (32-bit integer): the total size in bytes of this atom (no nested atom, so sum of box)
• type (32-bit integer): sets the box type, always `stsc`
• version (1-byte specification): default `0` (meets our requirements), if version is 1 then date and duration values are 8 bytes in length
• flags (3-byte space): always set to `0`
• number of entries: in our example `1`
• number of entries (32-bit integer): number of entries in the sample to chunk table
• sample to chunk table table: An array of sample descriptions (see above)

TODO – CAN YOU SUPPLY CODE TO DEMO CREATING THE STSC BOX

`co64` (chunk offset box)

Firstly we use the `co64` box (not not the `stco`) box, as CAMM can use up to 64 bits.

The chunk offset table is fairly easy to write in our example as we appended the telemetry to the end of the `mdat` file.

Remember earlier in the post;

first byte of telemetry will be at `294196361` bytes in the `mdat`

Thus we get a `co64` chunk offset table as follows;

``````294196361
294196421
294196481
294196541
294196601
294196661
``````

The `co64` box also requires the following data elements;

• atom size (32-bit integer): the total size in bytes of this atom (no nested atom, so sum of box)
• type (32-bit integer): sets the box type, always `co64`
• version (1-byte specification): default `0` (meets our requirements), if version is 1 then date and duration values are 8 bytes in length
• flags (3-byte space): always set to `0`
• number of entries (32-bit integer): number of entries in the sample descriptions that follow (in our example `6`)
• chunk offset table: An array of sample descriptions (see above)

So taking all this information, we can write the following binary;

TODO – CAN YOU SUPPLY CODE TO DEMO CREATING THE CO64 BOX

Writing the final boxes

So far we’ve written some of the required boxes. In order to ensure telemetry is read correctly you will also need to write data into some of the other boxes shown in the structure earlier.

In the coming 2 weeks I will finish off the series of posts by showing you what these boxes should contain and how they should be written for CAMM and GPMF.