DVBStreamExplorer service XML files

Introduction

DVBStreamExplorer has the feature to save service information as XML files. This can either be done explicitly from File->Export->XML menu, or it can be saved automatically on completion of a multiple MUX scanning session.
The folder where to save automatically can be set on Options->preferences menu on SI Scan tab.

The XML file contains information about all the services found during SI scan by analysing PAT, PMT, NIT, SDT and NIT tables. Post processing this XML file can have many useful applications.
In this post I will be demonstrate how to read the service XML file and post content to a web service encoded in JSON. For instance this could be used to feed an on-line web application with service data.

Service

Information about one service is stored in one XML element.
An example of a service XML element is shown below.

<service position="Astra 19.2°E" frequency="11303" polarity="H" symbolrate="22000" fec="2/3" modulation_system="1" modulation_type="2" roll_off="0" name="ORF1 HD" provider="ORF" network_type="dvbs" network_name="ASTRA 1" onid="1" nid="1" tsid="1007" sid="4911" pmt="107" pcr="1920" type="25" free_ca_mode="1" lcn="-1">
  <ca_list>
    <ca CA_system_ID="3333" CA_PID="220" private_bytes=""/>
    <ca CA_system_ID="3477" CA_PID="270" private_bytes=""/>
    <ca CA_system_ID="1608" CA_PID="120" private_bytes=""/>
    <ca CA_system_ID="5890" CA_PID="320" private_bytes=""/>
    <ca CA_system_ID="6195" CA_PID="420" private_bytes=""/>
    <ca CA_system_ID="2500" CA_PID="480" private_bytes=""/>
    <ca CA_system_ID="2444" CA_PID="490" private_bytes=""/>
    <ca CA_system_ID="3480" CA_PID="272" private_bytes=""/>
    <ca CA_system_ID="1616" CA_PID="122" private_bytes=""/>
  </ca_list>
  <streams>
    <stream type="27" pid="1920" type2="14496-10">
      <ca_list/>
    </stream>
    <stream type="4" pid="1921" language="ger" type2="13818-3">
      <ca_list/>
    </stream>
    <stream type="4" pid="1922" language="eng" type2="13818-3">
      <ca_list/>
    </stream>
    <stream type="6" pid="1923" language="ger" type2="ac3">
      <ca_list/>
    </stream>
    <stream type="6" pid="1925" type2="teletext">
      <ca_list/>
    </stream>
    <stream type="5" pid="7310" application_name="ORF HbbTV - SAT HD" type2="hbbtv">
      <ca_list/>
    </stream>
    <stream type="11" pid="7311" type2="data">
      <ca_list/>
    </stream>
  </streams>
</service>

As it can be seen there is quite some information. This includes properties like name, provider, tuner parameters, NID/ONID/TSID/SID and more. It also includes CA information, both from first loop of descriptors and the loop for each stream.

Each stream contains a type field which is simply the stream_type field from PMT. It contains a type2 field. The value is decided after analysing of stream_type field and certain descriptors. Besides benefaction of various audio and video streams this will also identify certain features and applications such as teletext, subtitle, mhp, hbbtv and more.

Client application

The client application is a .NET application written in C#. Source code is available from here https://dvbseserviceproc.codeplex.com/
It will allow you to select service XML file; network type; web service URL and method. After applying 'Process' service content will be sent to specified URL encoded in JSON.
You can modify the application for your specific needs if this will not work for you.

Server application

PHP

The server application is a simple PHP script that will consume the JSON data posted by client application. It will validate the presence of  a few expected properties, name and provider. Other than that it won't do anything with the data. In a real world application one might want to do stuff like saving the service data to a database.
The PHP script can be found below and can me modified for your specific needs. It was tested on a basic Apache running on Linux.

Google App Engine

There is is a live test web-service here http://dvbseserviceapi.appspot.com running on Google App Engine. Source code for web application is available here https://code.google.com/p/appengine-dvbseserviceapi/. 

Conclusion

This was just an minimum example to get you started. It doesn't really do anything useful as is. But it does show the potential of post processing service XML file.

H.262 usage reports in CSV format in TS File Demux 2.2.13

The latest version of TS File Demux introduces a new feature whereby extended reports about H.262 usage can be created in CSV format. The new feature is available as an option for all ES filters.
One report, h262report.csv, includes count of all picture_start_code, slice_start_code, user_data_start_code, sequence_header_code, sequence_error_code, extension_start_code, sequence_end_code and  group_start_code packets for all selected ES filters where H.262 report feature is enabled.
An example of such file is shown below. Document can also be opened directly in Google Drive here: https://docs.google.com/spreadsheets/d/1Qb7g-47NG9_kVpstgWGsyaUb0Gp9VeWNGU5-rgVzK64/edit?usp=sharing



Then for each selected ES filter where H.262 report feature is enabled a CSV file will be created listing all H.262 packets in order of occurrence.
An example of such file can open document directly in Google Drive from link below. The file is too large to be shown in-line.
https://docs.google.com/spreadsheets/d/1WaZNhbynCfofZ8AY_f4oUUs1YxBqkyDYHauzh4ALa2s/edit?usp=sharing

Discover section table_id with TS File Demux

When processing a TS file with TS File Demux a file named sectionreport.csv is created in output directory. This file contains various statistics about sections found in stream. Starting from TS File Demux V2.2.11 a new field, table_id, has been added to this file. This field contains a list of unique table_id's found in section stream. Sometimes this information can be useful, especially for streams containing multiple types of tables.
Below is an example of sectionreport.csv file after processing a DVB TS file.



It can be seen that PID 17 (0x11) contains SDT actual transport stream (0x42) and SDT other transport stream (0x46) tables.
PID 18 (0x12) contains various types of EIT tables.
PID 20 (0x14) contains TDT and TOT tables.
PID 48 (0x40) is a CA EMM stream containing table_id's 0x82 and 0x84 tables.
PID 102 (0x66) is a CA ECM stream containing table_id's 0x80 and 0x81 tables.
PID 8001 (0x1f41) contains a DSM-CC carousel. Table_id 0x3b is userNetworkMessage and table_id 0x3c is downloadDataMessage.

You can open the sample sheet here https://docs.google.com/spreadsheet/ccc?key=0AlzKD8F--RuTdDNrUFpSY0lMWkpxWG1lTTJZaUotTnc&usp=sharing

Decoding of DSM-CC BIOP messages

DVBStreamExplorer V4.0.3 has just been released. Now with DSMC-CC BIOP message decoding in DVB TS Monitor (professional edition only).
Until now DSM-CC analyser would show raw data of Data Carousel modules in one tab and the Object Carousel file system hierarchy in another tab. Starting from DVBStreamExplorer V4.0.3 decoding of BIOP messages is now shown for each module in Data Carousel. The information in BIOP messages is used to build the file system in Object Carousel.

In screenshot below is shown decoding of Service Gateway.

In screenshot below is shown decoding of a file object.

DVBStreamExplorer WS export online test service

If you have ever wanted to test DVBStreamExplorer export to WS features but gave up because of the hassle of creating a web service, then check out http://www.jensvaaben.com/dvbsewstest/.
It will allow you to export directly from DVBStreamExplorer to a live web service. Afterwards you can view the data that you have exported. Detailed instructions on http://www.jensvaaben.com/dvbsewstest/.

TS File Demux split PES/ES files on continuity error

TS File Demux V2.1.8 has just been released. A new feature has been introduced that will allow PES/ES files to be split whenever a continuity error is detected in TS packet.

In previous version you could end up with PES/ES files with missed packets due to continuity error in TS packet.
It might be desirable to produce PES/ES files without missed packets. The new feature will split PES/ES files every time continuity error is detected in TS packet for the PID in question. A new file will be created with name derived from  the base name of PES/ES file and .001, .002 etc appended.
Below is shown a screenshot below of output folder where split PES files have been created.

NOTE:

• Only TS packet header is decoded in order to detect continuity errors. Content of PES/ES packet is not decoded and/or analyzed in any way.
• This option can cause an excessive number of files to be created in output folder if there are a lot of continuity errors in TS file.

DVBStreamExplorer V4.0 configuration changes

Introduction

A new feature in DVBStreamExplorer V4.0 is the ability to run multiple instances. This feature is described in following blog post: Running multiple instances of DVBStreamExplorer. Introduction of this feature has some effect on how DVBStreamExplorer handles configuration and logging compared to DVBStreamExplorer V3.0.

Configuration

In DVBStreamExplorer V3.0 all configuration data, excpet for MUX lists, was stored globally while MUX lists were stored in context of the selected device.
With the introductiion of multiple running instances this was not pratical since changes made in one running instance would affect configuration in another running instance.
As a consequence all configuration is now stored in the context of selected device. This has the negative side effect that all custom configuration done must be applied for each device that you are using with DVBStreamExplorer. However that would seem a reasonable price to pay for the ability to run multiple instances of DVBStreamExplorer.

Logging

When options preferences window is opened following will show.

Here the logging folder is shown. In DVBStreamExplorer V3.0 log files are saved in the root of this folder. In DVBStreamExplorer V4.0 log files are saved in subfolders of this folder. Each subfolder is created at run-time with a name derived from process ID of running DVBStreamExplorer instance. A negative side effect is that these subfolders with content are left behind and accumulate used disk space over time. This issue will be addressed in a future release. Until then it is advised to clean up log folder from time to time. Note that log files are not really targeted application user. They are intended to be used by DVBStreamExplorer author to troubleshoot any issues found.