DVB TS Monitor ECM stream list.

ECM

DVB TS Monitor in latest DVBStreamExplorer (v4.0.25) now includes a list of all ECM streams found in TS.

This has been added as a convenience to have all ECM streams in TS listed in one place. Each ECM stream item includes basic information such as CA_system_ID and PID. It also includes information about which program(s) referenced the ECM stream.
Without this list one would have to walk through each service and its elementary streams to get information about ECM stream usage.

Set tuner from DVB TS Monitor

Another new feature is the option to set tuner directly from TS Monitor. No more switching back to main window, set tuner and then open TS Monitor again.

Even better, for certain relevant tree items in TS Monitor it's now possible to right-click on item to show a popup menu from where tuner dialog can be opened. Tuner dialog will be preset with parameters relevant in the context of selected item.

This should make it more convenient to quickly switch to a TS of interest. 

Currently context menu can be activated from following tree items:

• Other MUX Services items.
• Other MUX Network items.
• EIT service items under Present/Following and Schedule.

Note that for all tuner operations TS Monitor scanning must be in stopped state.

DVBStreamExplorer scanning ES headers

DVBStreamExploter EIT scanning

DVBStreamExplorer Service Viever filter feature

Capture HbbTV data with DVBStreamExplorer

In this post I will demonstrate how you can capture HbbTV data using DVBStreamExplorer. You will need DVBStreamExplorer Professional Edition for this exercise.
You can then study the HbbTV application offline. You can find more information about HbbTV here: https://www.hbbtv.org/.
First we need to find a service that contains HbbTV application. In this example I will find a service on Astra 19E.
You can find such service in this list: http://dtvlist.dvbstreamexplorer.com/dvbs.html. Look for 'hbbtv' in feature field.
Or you can do a complete SI scan of Astra 19E with DVBStreamExplorer. After completing SI scan switch to service view and you should see something similar to below.

Scroll to the far right to see features field and look for 'hbbtv'.

It can be seem that first service in the list, ORF SPORT+, features HbbTV.
Open the tuning window and tune on 11244H where the ORF SPORT+ service can be found.
Open Transport Stream Monitor (DVB) and start scanning. After a few seconds of scanning you should be able to select ORF SPORT+ from service drop-down list.

Now go to the DSM-CC tab. You should see list of DSM-CC streams found in service. For ORF SPORT+ there will only be one on PID 0x1c2b. Enable scanning of the PID in list.

It will now scan for DSM-CC data. This may take a short while. Once all DSM-CC data has been acquired the Object Carousel tab will be populated with file/folder structure. The content of files will also be shown as ASCII/Hex. Below can be seen partly content of a CSS file named min_portal_1_0_6.css which is part of HbbTV application.

And below is shown content of min_config_1_0_6.js which is a JavaScript file also part of HbbTV application.

Finally below shows partial content of index.html which is a HTML file also part of HbbTV application.

You can now stop scanning to enable file menu. From here the entire Object Carousel be saved. Directory structure in OC is preserved when saved to file system. You can now open the files in your favourite text editor for further examination.

DVBStreamExplorer service XML files

Introduction

DVBStreamExplorer has the feature to save service information as XML files. This can either be done explicitly from File->Export->XML menu, or it can be saved automatically on completion of a multiple MUX scanning session.
The folder where to save automatically can be set on Options->preferences menu on SI Scan tab.

The XML file contains information about all the services found during SI scan by analysing PAT, PMT, NIT, SDT and NIT tables. Post processing this XML file can have many useful applications.
In this post I will be demonstrate how to read the service XML file and post content to a web service encoded in JSON. For instance this could be used to feed an on-line web application with service data.

Service

Information about one service is stored in one XML element.
An example of a service XML element is shown below.

<service position="Astra 19.2°E" frequency="11303" polarity="H" symbolrate="22000" fec="2/3" modulation_system="1" modulation_type="2" roll_off="0" name="ORF1 HD" provider="ORF" network_type="dvbs" network_name="ASTRA 1" onid="1" nid="1" tsid="1007" sid="4911" pmt="107" pcr="1920" type="25" free_ca_mode="1" lcn="-1">
  <ca_list>
    <ca CA_system_ID="3333" CA_PID="220" private_bytes=""/>
    <ca CA_system_ID="3477" CA_PID="270" private_bytes=""/>
    <ca CA_system_ID="1608" CA_PID="120" private_bytes=""/>
    <ca CA_system_ID="5890" CA_PID="320" private_bytes=""/>
    <ca CA_system_ID="6195" CA_PID="420" private_bytes=""/>
    <ca CA_system_ID="2500" CA_PID="480" private_bytes=""/>
    <ca CA_system_ID="2444" CA_PID="490" private_bytes=""/>
    <ca CA_system_ID="3480" CA_PID="272" private_bytes=""/>
    <ca CA_system_ID="1616" CA_PID="122" private_bytes=""/>
  </ca_list>
  <streams>
    <stream type="27" pid="1920" type2="14496-10">
      <ca_list/>
    </stream>
    <stream type="4" pid="1921" language="ger" type2="13818-3">
      <ca_list/>
    </stream>
    <stream type="4" pid="1922" language="eng" type2="13818-3">
      <ca_list/>
    </stream>
    <stream type="6" pid="1923" language="ger" type2="ac3">
      <ca_list/>
    </stream>
    <stream type="6" pid="1925" type2="teletext">
      <ca_list/>
    </stream>
    <stream type="5" pid="7310" application_name="ORF HbbTV - SAT HD" type2="hbbtv">
      <ca_list/>
    </stream>
    <stream type="11" pid="7311" type2="data">
      <ca_list/>
    </stream>
  </streams>
</service>

As it can be seen there is quite some information. This includes properties like name, provider, tuner parameters, NID/ONID/TSID/SID and more. It also includes CA information, both from first loop of descriptors and the loop for each stream.

Each stream contains a type field which is simply the stream_type field from PMT. It contains a type2 field. The value is decided after analysing of stream_type field and certain descriptors. Besides benefaction of various audio and video streams this will also identify certain features and applications such as teletext, subtitle, mhp, hbbtv and more.

Client application

The client application is a .NET application written in C#. Source code is available from here https://dvbseserviceproc.codeplex.com/
It will allow you to select service XML file; network type; web service URL and method. After applying 'Process' service content will be sent to specified URL encoded in JSON.
You can modify the application for your specific needs if this will not work for you.

Server application

PHP

The server application is a simple PHP script that will consume the JSON data posted by client application. It will validate the presence of  a few expected properties, name and provider. Other than that it won't do anything with the data. In a real world application one might want to do stuff like saving the service data to a database.
The PHP script can be found below and can me modified for your specific needs. It was tested on a basic Apache running on Linux.

Google App Engine

There is is a live test web-service here http://dvbseserviceapi.appspot.com running on Google App Engine. Source code for web application is available here https://code.google.com/p/appengine-dvbseserviceapi/. 

Conclusion

This was just an minimum example to get you started. It doesn't really do anything useful as is. But it does show the potential of post processing service XML file.

Decoding of DSM-CC BIOP messages

DVBStreamExplorer V4.0.3 has just been released. Now with DSMC-CC BIOP message decoding in DVB TS Monitor (professional edition only).
Until now DSM-CC analyser would show raw data of Data Carousel modules in one tab and the Object Carousel file system hierarchy in another tab. Starting from DVBStreamExplorer V4.0.3 decoding of BIOP messages is now shown for each module in Data Carousel. The information in BIOP messages is used to build the file system in Object Carousel.

In screenshot below is shown decoding of Service Gateway.

In screenshot below is shown decoding of a file object.

DVBStreamExplorer WS export online test service

If you have ever wanted to test DVBStreamExplorer export to WS features but gave up because of the hassle of creating a web service, then check out http://www.jensvaaben.com/dvbsewstest/.
It will allow you to export directly from DVBStreamExplorer to a live web service. Afterwards you can view the data that you have exported. Detailed instructions on http://www.jensvaaben.com/dvbsewstest/.

DVBStreamExplorer V4.0 configuration changes

Introduction

A new feature in DVBStreamExplorer V4.0 is the ability to run multiple instances. This feature is described in following blog post: Running multiple instances of DVBStreamExplorer. Introduction of this feature has some effect on how DVBStreamExplorer handles configuration and logging compared to DVBStreamExplorer V3.0.

Configuration

In DVBStreamExplorer V3.0 all configuration data, excpet for MUX lists, was stored globally while MUX lists were stored in context of the selected device.
With the introductiion of multiple running instances this was not pratical since changes made in one running instance would affect configuration in another running instance.
As a consequence all configuration is now stored in the context of selected device. This has the negative side effect that all custom configuration done must be applied for each device that you are using with DVBStreamExplorer. However that would seem a reasonable price to pay for the ability to run multiple instances of DVBStreamExplorer.

Logging

When options preferences window is opened following will show.

Here the logging folder is shown. In DVBStreamExplorer V3.0 log files are saved in the root of this folder. In DVBStreamExplorer V4.0 log files are saved in subfolders of this folder. Each subfolder is created at run-time with a name derived from process ID of running DVBStreamExplorer instance. A negative side effect is that these subfolders with content are left behind and accumulate used disk space over time. This issue will be addressed in a future release. Until then it is advised to clean up log folder from time to time. Note that log files are not really targeted application user. They are intended to be used by DVBStreamExplorer author to troubleshoot any issues found.

Running multiple instances of DVBStreamExplorer

When DVBStreamExplorer V3 is launched and another instance is already running one is  faced with message below.

This limitation has been removed in DVBStreamExplorer V4 which has just been released.
Below is a screenshot from a Windows desktop where two instances of DVBStreamExplorer are running. One instance is scanning DVB-T data from a Cinergy HTC USB XS DVB-T Tuner and the other is scanning DVB-S data from TBS5925 DVB-S2 TV Tuner USB.

As a convenience DVBStreamExplorer application window title bar now includes name of capture device.

Decoding user defined Nordig, DTG (D-Book) and IEC 62216 (E-Book) descriptors

Next version of DVBStreamExplorer will include the option to select which syntax to be used for decoding of user defined (0x80 - 0xFE) descriptors.

• DVB (ETSI EN 300 468)
• Nordig
• DTG (D-Book)
• IEC 62216 (E-Book)

This is of special interest with regards to Logical Channel Number descriptors which are defined in each of the above standards except for EN 300 468. While the syntax is very similar in the standards, there are minor differences regarding the allocation of bits for each field.
The standard to be used for decoding can be selected in DVB PSI/SI / Multiple DVB-S/T/C MUX scanning window before starting scan operation. See screenshot below.

On SI scan completion you can switch to service view. A new 'lcn' column has been added. A value of -1 means that there was no LCN descriptor applicable for service. See screen shot below.

LCN information is also shown for each service found in MUX in DVB TS Monitor. Like for for SI scanning the syntax to be used for decoding has to be selected before starting scan operation. See screenshot below.

Finally LCN information has also been added as a new field to the CVS export of service information.

While LCN descriptor is probably the one of most interest, other user defined (0x80 - 0xFE) descriptors will also be decoded. You will find these in the various descriptor loops of decoded sections.

Follow +DVBStreamExplorer to keep up to date with DVBStreamExplorer related news.

Export to folder after SI scan in DVBStreamExplorer

If you are using DVBStreamExplorer to scan entire DVB, ATSC or ISDB networks for PSI/SI/PSIP information, you might appreciate a new feature in DVBStreamExplorer Professional Edition V3.0.113 that allows you to have various files exported and saved automatically when PSI/SI/PSIP scan is completed. Note that this feature is only applicable for 'Multiple * MUX scanning' features found in PSI/SI/PSIP menu.

The new feature can be enabled from Options / Preferences / SI scan page as shown below.

Here you can enable/disable the feature, select destination folder and select which file types you want to export.
'Save SI XML' corresponds to File / Save/Save As file, which is a file type that can be loaded by DVBStreamExplorer for offline viewing of PSI/SI/PSIP information.
'Save service XML' and 'Save CSV' corresponds to options found in File / Export / XML and File / Export / CSV.
All files will be saved in specified folder. File name will be automatically generated based on network type, export type and date/time of export. The date/time part of file name is based on, but does not necessarily strictly conform to, ISO 8601. The time is specified in UTC to allow files to be easily exchanged between different time zones.

BDA tuner configuration in DVBStreamExplorer

DVBStreamExplorer V3.0.113 fixes a problem that it would fail to initialize some BDA devices on Windows 7. The old device selection box looked like below.

 In case of device initialization failure you would see a message like below when starting application.

While fixing this issue, it was found that the application was not always able to reliably detect BDA tuner configuration. I.e. it was possible that a DVB-T device could be detected as DVB-S device.

You can verify how your BDA tuner device was detected by application by opening Device / Settings page.
Below is a screenshot showing how a DVB-C  device was detected as DVB-S.

As a workaround a new feature has been added, so that it is now possible to specify network (tuner) type when selecting device during application launch.
The new device selection box is shown below.

You can select between auto detect, DVB-S, DVB-T, DVB-C and ATSC.
It is recommended to first try with auto detect. If then network type is not detected correctly by application, you will need to specify correct network type when launching application.
Note that application can only remember the last network type used. So if you change between different BDA device types, you will have to specify correct network type every time you launch application unless you are using auto detect option.

DVBStreamExplorer export to Web Services

The next release of DVBStreamExplorer Professional edition will introduce new features for exporting data to Web Services.

• Service WS export
• EIT WS export

Service WS export
This feature will export the same data as when exporting to CSV file. The difference being that it will export directly to a server that implements a WS that can be consumed by DVBStreamExplorer.
This could be used by users who are using DVBStreamExplorer to produce data for an on-line DTV service list.
For information on how to develop a WS that can be consumed by DVBStreamExplorer visist http://developer.dvbstreamexplorer.com
The WS export is invoked from export menu.

Endpoint URL of Web Service has to be entered. There is also an ID field. The meaning of this is defined by WS implementation. It is not used internally by DVBStreamExplorer. This could for instance be a unique DTV service list instance, so that multiple DTV service lists could be hosted.

EIT WS export
This feature will export EIT event data to a Web Service. DVBStreamExplorer could then be used to provide event data for an on-line TV guide.
For information on how to develop a WS that can be consumed by DVBStreamExplorer visist http://developer.dvbstreamexplorer.com
This feature is part of the DVB TS Monitor EIT page.

Before starting TS Monitor session, export to WS must be configured. Configuration of EIT WS export can be invoked from Options -> Preferences EIT export tab.

Endpoint URL of Web Service has to be entered. There is also an ID field. The meaning of this is defined by WS implementation. It is not used internally by DVBStreamExplorer.
EIT module will cache events in memory. When a sufficient number of events have been cached they will be exported to Web Service. This will happen either when 'Upload interval (minutes)' has passed or count of events has exceeded 'Event threshold', depending on which of these two events occurs first.
A maximum number of 'Max events per request' events will be sent in one request to Web Service. If number of cached events exceeds this value multiple requests to Web Service will be made.

Using Google Drive with DVBStreamExploerer exported CSV files

Service information files exported from DVBStreamExplorer in CSV format can be uploaded to Google Drive and converted to Google Spreadsheets. Service information can then be shared easily with other users and you can access information from everywhere including handheld and mobile devices.
You will need either a Google Account or a Google Apps Account.

Below is a DVBStreamExplorer scan of local DVB-T channels (Malta) exported to Google Docs spreadsheet.

Direct link to spreadsheet: https://docs.google.com/spreadsheet/ccc?key=0AlzKD8F--RuTdDJpUWlRYXF3WFpiT3pRdUxRWFgxUUE

Below is a DVBStreamExplorer scan of 'Astra 19.2°E' exported to Google Docs spreadsheet.



Direct link to spreadsheet: https://docs.google.com/spreadsheet/ccc?key=0AlzKD8F--RuTdHhqcVJNWmdTRDVoNVdhM1ZESWphLXc

Once service CSV file has been converted to spreadsheet one could apply Google Apps Script to add special features. This could for instance be extraction of certain information which would normally be difficult to spot when looking at a large spreadsheet. Or it could be used to spot service changes over time. For instance if 'Astra 19.2°E' was scanned one day and then scanned again a week after a script could generate a report of service changes. Such a report could include added and removed services, and list of services where one or more properties changed.

Follow +DVBStreamExplorer to keep up to date with DVBStreamExplorer  related news.

NorDig support in DVBStreamExplorer

The next release of DVBStreamExplorer will introduce decoding of NorDig logic_channel_descriptor and content_protection_descriptor private descriptors.
This might be of interest to users in Nordic region (Denmark, Finland, Iceland, Norway, Sweden) and Éire.
For logic_channel_descriptor, both Version 1 (descriptor_tag 0x83) and Version 2 (descriptor_tag 0x87) are decoded.
Below is a screenshot of logic_channel_descriptor (Version 1) decoding in DVB PSI/SI scanning.

And below is screenshot of decoding of same descriptor in DVB Transport Stream Monitor.

How DVBStreamExplorer is keeping old PSI/SI section versions

In this post I will describe an often overlooked, but nice feature in DVBStreamExplorer Pro Edition TS Monitor feature. When scanning for SI/PSI table sections DVBStreamExplorer Pro Edition will like most other analysing software keep a copy of all unique sections. However when a new version of a given section arrives in stream, DVBStreamExplorer Pro Edition will keep the old section when inserting the new 'current' section in the tree. Below is an example of how an EIT section version 6 was first acquired and then later superseded by version 7, and finally by version 8. The section version currently applicable in TS is labelled (current) for easy identification. The section version_number field  is 5 bits long. So DVBStreamExplorer Pro Edition will keep up to 32 versions, including current version, of each unique section.

The curious user can then examine different section versions in the tree to find out exactly what changed. Below is EIT section version 7 expanded in SI/PSI tree.

And  below is EIT section version 8 expanded in SI/PSI tree.

It can be seen that the event ID of first event in loop was changed from 707 (section version 7) to 708 (section version 8).
This approach can also be applied for table sections like PAT, CAT, PMT, NIT, SDT, NIT etc. In this example I used EIT because the section version change can be observed after scanning for a relatively short period of time. For the other table section types one usually have to scan for a significantly longer time to catch a section version change.

Creating DVB-T MUX list in DVBStreamExplorer

If you are going to use DVBStreamExplorer with DVB-T capture hardware, one of the first thing you need to do is to create a DVB-T MUX list. If you are going to do a 'Multiple DVB-T MUX scanning' session, only entries in DVB-T MUX list can be scanned. When tuning manually it's also offers an convenient way select entry from list instead of having to type frequency manually.
The DVB-T MUX list can easily be created by the application performing a full scan of all standard frequencies. Non standard frequencies can be added to the list manually. Band scan can be invoked from scan button on 'Options  -> DVB-T MUX list' menu.
If you are using BDA hardware the very first thing you should do is to set the 'Tuning wait for lock delay (ms)' to an appropriate value, i.e. 1000 ms. Leaving it at default value 0 ms can cause problems with some cards. In those cases operations like 'Multiple DVB-T MUXscanning' would fail each tuning attempt and you will end up with no data found. The wait for lock value can be set in Device -> Settings menu. See screenshot below.

The video below demonstrates how to set  'Tuning wait for lock delay (ms)' value and how to start DVB-T MUX list band scan operation.



The video below shows the conclusion of DVB-T MUX list band scan operation. It also shows that DVB-T MUX list entries are now available for 'Multiple DVB-T MUX scanning' session.



Similar operations can be applied for DVB-C, ATSC and ISDB-T MUX lists. The case of DVB-S MUX list is more complex and will be documented separately in future post.

Analysing DVB MHP application using DSM-CC analyser

I was studying some DVB-T services with DVBStreamExplorer. I noticed that on 546 MHz MUX some services were carrying Multimedia Home Platform (MHP) applications. I decided to study MHP application further with DSM-CC analyser in DVBStreamExplorer. In this exercise I shall only be using DVB Transport Stream Monitor. From that I will acquire both PSI/SI and DSM-CC information.
Image below show stream details for one of the services in MUX, LA7. Besides the common video and audio streams the presence of MHP application can also be seen here. In the stream list references to Application Information Table (AIT) and DSM-CC object carousels can be found. For instance one AIT can be found on PID 0x1ba8 (7080). An DSM-CC Object Carousel can be found on PID 0x1ba9 (7081).

Below can be seen the content of an AIT. That contains various information about the application. That covers such information as application name, transport protocol, initial Java class of application and more.

Next step is to look into the DSM-CC carousel containing the application data. Image below shows that three DSM-CC carousel streams were found. Select the streams so that DVBStreamExplorer DSM-CC analyser will start collecting DSI, DII and DDB sections. These sections will be shown fully decoded on DSM-CC sections tab.

Once all DSI, DII and DDB sections for a carousel have been acquired the data carousels can be decoded. Image below shows raw data of carousel modules found. The object carousel will be decoded from these modules.

Image below shows the decoded object carousel. It shows part of the file directory structure of the application. it also shows the binary content of one of the Java class files that is part of the application.

With DSM-CC analyser in DVBStreamExplorer there is the option to save entire carousel on local file system. After doing so you can then launch your preferred Java decompiler and have closer look at the code behind MHP application. I will leave this as an exercise for the reader.

Analysing DVB STB software update using DSM-CC analyser

I was going through the list of MT DTT services on http://jensvaaben.com/.
Two services caught my attention.
"Handan Software Update" and "Intek Software Update".
The service names strongly suggest that these services are carrying software updates for set-top box'es (STB). That would most likely be the set-top box'es offered by GO.

I decided to examine these services further with DVBStreamExplorer. I started by having a look at PSI/SI information for one of the services. I chose to have a closer look at "Handan Software Update". I launched DVBStreamExplorer and tuned on 754 MHz MUX which is carrying the update services. There are two options to retrieve PSI/SI information with DVBStreamExplorer. Either use the PSI/SI/PSIP -> DVB PSI/SI scanning -> current MUX. This will take a snapshot of PSI/SI information currently applicable for stream. It is also possible to use Transport Stream Monitor. This feature will monitor stream continiously. Any changes in stream will be reflected immidiately in UI.
I decided to use PSI/SI/PSIP -> DVB PSI/SI scanning -> current MUX feature. This gave me stream information like below.

The service contain one stream of type 0x0B ISO/IEC 13818-6 type B. The stream is carried on PID 0x0bb8.
This is a good candidate for a stream type carrying a software update. ISO/IEC 13818-6 type B means that it contains DSM-CC data carousel.
So the next step is to examine the service even further from a DSM-CC perspective.
DVBStreamExplorer has two options to analyse DSM-CC. Either use the standalone 'DSMCC Analyzing  (DVB)' function. Or use the DSM-CC analyser which is part of 'Transport Stream Monitor' function. Using DSMCC analyser in 'Transport Stream Monitor' has the advantage that you can browse other stream information while DSM-CC analyser is working. Simply navigate through the tabs in 'Transport Stream Monitor'.
After starting scan in 'Transport Stream Monitor' wait a few seconds. Then select the update service in service selector in top part of 'Transport Stream Monitor' window. Then select 'DSMCC' tab. By now it should have found one or more PID streams which are good candidates to carry DSMCC data. Select one or more streams to start analysing.
One disadvantage of DSM-CC analyser in  'Transport Stream Monitor' is that if a PID stream is not recognized as one carrying DSM-CC then it's not possible to analyse that PID stream. With standalone DSM-CC analyser any arbitrary PID stream can be scanned for DSM-CC carousel.

Another difference between the two DSMCC analysers in DVBStreamExplorer is that in 'Transport Stream Monitor' you can start examining DSMCC sections (DSI, DII and DDB) as they arrive. In standalone DSMCC analyser you have to wait until all DSMCC sections have been found before you can start examining content.

In screenshot above is shown part of Download Info Indication (DII) message for the Handan update service. It contains one module. The module info node shows various properties for the module. I.e. ModuleID 0x001c, moduleSize 690039 etc. There is also the moduleInfo block. In reality this is a descriptor loop. Unfortunately these descriptors are not decoded in current version of DVBStreamExplorer. Hopefully descriptor decoding will be added as a feature in a future release of DVBStreamExplorer. But after brief decoding by hand it can be seen that descriptor loop contains type descriptor (tag 0x01), CRC32   descriptor (tag 0x05) and compressed_module descriptor (tag 0x09).

We can now go to 'Data Carousel' tab to see the modules found. Another advantage of DSMCC analyser in  'Transport Stream Monitor' is that it provides a binary/ASCII view of the content of the modules found.

Both DSMCC analysers in DVBStreamExplorer have the ability to save the data carousel modules to disk. That can be useful if you want to examine the module further offline with other tools.
This concludes the brief introduction on how to analyse DSM-CC carousels with DVBStreamExplorer.