Extracting DSM-CC data and object carousel content with TS File Demux

TS File Demux now includes a feature to extract DSM-CC data and object carousel content. This feature is meant to be a lighter alternative to a similar feature in DVBSTreamExplorer which might be overkill in some situations. If you need decoding of DSM-CC sections and BIOP messages, DVBStreamExplorer is the still way to go.

Below is a screenshot of a TS file that contains  DSM-CC data.

 

TS File Demux main window

The DSM-CC filters have been added automatically after applying 'Add all'. Streams of type 0x0b, ISO/IEC 13818-6 type B, are considered as DSM-CC carousel streams. You can add DSM-CC filter for arbitrary PID if it wasn't recognized as DSM-CC carousel and you believe that it is.

Below is screenshot of some extracted object carousel files. The files are part of a MHP application.

Extracted files

The files can now be studied offline.

Follow this link to download TS File Demux here.

Capture HbbTV data with DVBStreamExplorer

In this post I will demonstrate how you can capture HbbTV data using DVBStreamExplorer. You will need DVBStreamExplorer Professional Edition for this exercise.
You can then study the HbbTV application offline. You can find more information about HbbTV here: https://www.hbbtv.org/.
First we need to find a service that contains HbbTV application. In this example I will find a service on Astra 19E.
You can find such service in this list: http://dtvlist.dvbstreamexplorer.com/dvbs.html. Look for 'hbbtv' in feature field.
Or you can do a complete SI scan of Astra 19E with DVBStreamExplorer. After completing SI scan switch to service view and you should see something similar to below.

Scroll to the far right to see features field and look for 'hbbtv'.

It can be seem that first service in the list, ORF SPORT+, features HbbTV.
Open the tuning window and tune on 11244H where the ORF SPORT+ service can be found.
Open Transport Stream Monitor (DVB) and start scanning. After a few seconds of scanning you should be able to select ORF SPORT+ from service drop-down list.

Now go to the DSM-CC tab. You should see list of DSM-CC streams found in service. For ORF SPORT+ there will only be one on PID 0x1c2b. Enable scanning of the PID in list.

It will now scan for DSM-CC data. This may take a short while. Once all DSM-CC data has been acquired the Object Carousel tab will be populated with file/folder structure. The content of files will also be shown as ASCII/Hex. Below can be seen partly content of a CSS file named min_portal_1_0_6.css which is part of HbbTV application.

And below is shown content of min_config_1_0_6.js which is a JavaScript file also part of HbbTV application.

Finally below shows partial content of index.html which is a HTML file also part of HbbTV application.

You can now stop scanning to enable file menu. From here the entire Object Carousel be saved. Directory structure in OC is preserved when saved to file system. You can now open the files in your favourite text editor for further examination.

Discover section table_id with TS File Demux

When processing a TS file with TS File Demux a file named sectionreport.csv is created in output directory. This file contains various statistics about sections found in stream. Starting from TS File Demux V2.2.11 a new field, table_id, has been added to this file. This field contains a list of unique table_id's found in section stream. Sometimes this information can be useful, especially for streams containing multiple types of tables.
Below is an example of sectionreport.csv file after processing a DVB TS file.



It can be seen that PID 17 (0x11) contains SDT actual transport stream (0x42) and SDT other transport stream (0x46) tables.
PID 18 (0x12) contains various types of EIT tables.
PID 20 (0x14) contains TDT and TOT tables.
PID 48 (0x40) is a CA EMM stream containing table_id's 0x82 and 0x84 tables.
PID 102 (0x66) is a CA ECM stream containing table_id's 0x80 and 0x81 tables.
PID 8001 (0x1f41) contains a DSM-CC carousel. Table_id 0x3b is userNetworkMessage and table_id 0x3c is downloadDataMessage.

You can open the sample sheet here https://docs.google.com/spreadsheet/ccc?key=0AlzKD8F--RuTdDNrUFpSY0lMWkpxWG1lTTJZaUotTnc&usp=sharing

Decoding of DSM-CC BIOP messages

DVBStreamExplorer V4.0.3 has just been released. Now with DSMC-CC BIOP message decoding in DVB TS Monitor (professional edition only).
Until now DSM-CC analyser would show raw data of Data Carousel modules in one tab and the Object Carousel file system hierarchy in another tab. Starting from DVBStreamExplorer V4.0.3 decoding of BIOP messages is now shown for each module in Data Carousel. The information in BIOP messages is used to build the file system in Object Carousel.

In screenshot below is shown decoding of Service Gateway.

In screenshot below is shown decoding of a file object.

Analysing DVB MHP application using DSM-CC analyser

I was studying some DVB-T services with DVBStreamExplorer. I noticed that on 546 MHz MUX some services were carrying Multimedia Home Platform (MHP) applications. I decided to study MHP application further with DSM-CC analyser in DVBStreamExplorer. In this exercise I shall only be using DVB Transport Stream Monitor. From that I will acquire both PSI/SI and DSM-CC information.
Image below show stream details for one of the services in MUX, LA7. Besides the common video and audio streams the presence of MHP application can also be seen here. In the stream list references to Application Information Table (AIT) and DSM-CC object carousels can be found. For instance one AIT can be found on PID 0x1ba8 (7080). An DSM-CC Object Carousel can be found on PID 0x1ba9 (7081).

Below can be seen the content of an AIT. That contains various information about the application. That covers such information as application name, transport protocol, initial Java class of application and more.

Next step is to look into the DSM-CC carousel containing the application data. Image below shows that three DSM-CC carousel streams were found. Select the streams so that DVBStreamExplorer DSM-CC analyser will start collecting DSI, DII and DDB sections. These sections will be shown fully decoded on DSM-CC sections tab.

Once all DSI, DII and DDB sections for a carousel have been acquired the data carousels can be decoded. Image below shows raw data of carousel modules found. The object carousel will be decoded from these modules.

Image below shows the decoded object carousel. It shows part of the file directory structure of the application. it also shows the binary content of one of the Java class files that is part of the application.

With DSM-CC analyser in DVBStreamExplorer there is the option to save entire carousel on local file system. After doing so you can then launch your preferred Java decompiler and have closer look at the code behind MHP application. I will leave this as an exercise for the reader.

Analysing DVB STB software update using DSM-CC analyser

I was going through the list of MT DTT services on http://jensvaaben.com/.
Two services caught my attention.
"Handan Software Update" and "Intek Software Update".
The service names strongly suggest that these services are carrying software updates for set-top box'es (STB). That would most likely be the set-top box'es offered by GO.

I decided to examine these services further with DVBStreamExplorer. I started by having a look at PSI/SI information for one of the services. I chose to have a closer look at "Handan Software Update". I launched DVBStreamExplorer and tuned on 754 MHz MUX which is carrying the update services. There are two options to retrieve PSI/SI information with DVBStreamExplorer. Either use the PSI/SI/PSIP -> DVB PSI/SI scanning -> current MUX. This will take a snapshot of PSI/SI information currently applicable for stream. It is also possible to use Transport Stream Monitor. This feature will monitor stream continiously. Any changes in stream will be reflected immidiately in UI.
I decided to use PSI/SI/PSIP -> DVB PSI/SI scanning -> current MUX feature. This gave me stream information like below.

The service contain one stream of type 0x0B ISO/IEC 13818-6 type B. The stream is carried on PID 0x0bb8.
This is a good candidate for a stream type carrying a software update. ISO/IEC 13818-6 type B means that it contains DSM-CC data carousel.
So the next step is to examine the service even further from a DSM-CC perspective.
DVBStreamExplorer has two options to analyse DSM-CC. Either use the standalone 'DSMCC Analyzing  (DVB)' function. Or use the DSM-CC analyser which is part of 'Transport Stream Monitor' function. Using DSMCC analyser in 'Transport Stream Monitor' has the advantage that you can browse other stream information while DSM-CC analyser is working. Simply navigate through the tabs in 'Transport Stream Monitor'.
After starting scan in 'Transport Stream Monitor' wait a few seconds. Then select the update service in service selector in top part of 'Transport Stream Monitor' window. Then select 'DSMCC' tab. By now it should have found one or more PID streams which are good candidates to carry DSMCC data. Select one or more streams to start analysing.
One disadvantage of DSM-CC analyser in  'Transport Stream Monitor' is that if a PID stream is not recognized as one carrying DSM-CC then it's not possible to analyse that PID stream. With standalone DSM-CC analyser any arbitrary PID stream can be scanned for DSM-CC carousel.

Another difference between the two DSMCC analysers in DVBStreamExplorer is that in 'Transport Stream Monitor' you can start examining DSMCC sections (DSI, DII and DDB) as they arrive. In standalone DSMCC analyser you have to wait until all DSMCC sections have been found before you can start examining content.

In screenshot above is shown part of Download Info Indication (DII) message for the Handan update service. It contains one module. The module info node shows various properties for the module. I.e. ModuleID 0x001c, moduleSize 690039 etc. There is also the moduleInfo block. In reality this is a descriptor loop. Unfortunately these descriptors are not decoded in current version of DVBStreamExplorer. Hopefully descriptor decoding will be added as a feature in a future release of DVBStreamExplorer. But after brief decoding by hand it can be seen that descriptor loop contains type descriptor (tag 0x01), CRC32   descriptor (tag 0x05) and compressed_module descriptor (tag 0x09).

We can now go to 'Data Carousel' tab to see the modules found. Another advantage of DSMCC analyser in  'Transport Stream Monitor' is that it provides a binary/ASCII view of the content of the modules found.

Both DSMCC analysers in DVBStreamExplorer have the ability to save the data carousel modules to disk. That can be useful if you want to examine the module further offline with other tools.
This concludes the brief introduction on how to analyse DSM-CC carousels with DVBStreamExplorer.