Supporting Behold TV devices in DVBStreamExplorer

Work is in progress to support Behold TV devices in DVBStreamExplorer.  These devices seem interesting offering features not found in most other BDA compatible devices. Unfortunately Behold TV devices don't work with DVBStreamExplorer out of the box. Interaction with proprietary API is required. This is the work currently in progress.
At this point it should be possible to tune on a MUX manually. And then invoke DVB PSI/SI scanning -> Current MUX.
Still pending are the features involving unattended tuning of multiple frequencies.
This includes features such as DVB-T/C MUX finder to create list of MUXes.
It also includes multiple DVB PSI/SI scanning -> DVB-C/T MUX scanning. This feature itself depends on list of MUXes.

You can download Behold TV versions from here if you want to give it a try http://dvbse.com/grIyM6Xq/beta/dvbstreamexplorer4FreeX64.msi and http://dvbse.com/grIyM6Xq/beta/dvbstreamexplorer4ProX64.msi.
Some Behold TV devices support multiple antenna inputs. Make sure to select the correct one where applicable.

After launching DVBStreamExplorer the first time open Source -> settings.
Set 'Tuning lock delay' value to 1000 (ms).
If your Behold TV device supports multiple antenna inputs select the applicable one here.

Please provide feedback how it's working out for you. Either as comment to this post or mail to info@dvbstreamexplorer.com.

Exporting EIT XML from unattended SI scanning

The option to export EIT (event information table) data to XML from DVB TS Monitor has existed in DVBStreamExplorer for some time.
In DVBStreamExplorer V4.2.0 a new feature has been introduced that allows for a similar export from unattended SI scan.

To enable EIT export follow steps below.

In Options / preferences SI scan tab enable 'Auto export EIT to XML' and specify an existing folder where XML files will be saved.

In SI tables configuration enable the four EIT options.
Note that enabling these will very likely extend total scan time substantially. Up to 30 seconds can be spent scanning for EIT on each MUX.

Invoke multible DVB MUX scan of your choice (satellite, cable, terrestrial).
Once scanning is complete you should find the specified folder populated with XML files containing EIT info.A GUID string will be used for naming each file.
Example of EIT XML file below.

The XML root node has onid (original_network_id), nid (network_id) and tsid (transport_stream_id) attributes referring to the MUX where EIT data was acquired. It also contain timestamp attribute specifying time in UTC when MUX was scanned.
Each child node of root contains data for one event. It contains EIT section key data as attributes. It also contain data from short event and extended event descriptors. That would be the text that i.e. an STB would show in EPG.  

Note that you will likely not get all available EIT information from each MUX.  This is because each MUX will only be scanned for a maximum of 30 seconds which is usually not enough time. If you need more EIT data you should use EIT export feature in DVB TS Monitor.
Also if you scan MUX'es from the same network you will likely get duplicate EIT information.
On the other hand if you are scanning MUX'es from multiple networks (i.e. satellite) you should be able to collect a substantial amount of EIT data in just one scan session.

You can view the exported EIT XML files in DVBStreamExplorer Service View. More information here.
Or you can write your own software that make use of the XML files.

DVB TS Monitor ECM stream list.

ECM

DVB TS Monitor in latest DVBStreamExplorer (v4.0.25) now includes a list of all ECM streams found in TS.

This has been added as a convenience to have all ECM streams in TS listed in one place. Each ECM stream item includes basic information such as CA_system_ID and PID. It also includes information about which program(s) referenced the ECM stream.
Without this list one would have to walk through each service and its elementary streams to get information about ECM stream usage.

Set tuner from DVB TS Monitor

Another new feature is the option to set tuner directly from TS Monitor. No more switching back to main window, set tuner and then open TS Monitor again.

Even better, for certain relevant tree items in TS Monitor it's now possible to right-click on item to show a popup menu from where tuner dialog can be opened. Tuner dialog will be preset with parameters relevant in the context of selected item.

This should make it more convenient to quickly switch to a TS of interest. 

Currently context menu can be activated from following tree items:

• Other MUX Services items.
• Other MUX Network items.
• EIT service items under Present/Following and Schedule.

Note that for all tuner operations TS Monitor scanning must be in stopped state.

DVBStreamExplorer scanning ES headers

DVBStreamExploter EIT scanning

DVBStreamExplorer Service Viever filter feature

DVBStreamExplorer Community Preview

A DVBStreamExplorer Community Preview including Python integration has been made public available. This is in a very early stage but will allow you to try out Python integration feature now.

At this point only x64 is available. You need a valid DVBStreamExplorer license or you can try it out for time limited period. You also need to have Python 3 x64 version installed.
Download from here: http://dvbstreamexplorer.com/grIyM6Xq/dvbse/4.0/preview/python/dvbstreamexplorer4ProX64.msi

This version is almost identical to the recent DVBSE 4.0.21 except for the Python integration.
To integrate Python launch DVBStreamExplorer. Open options / preferences. You should notice a new tab 'Python integration'.

Here you can enable Python integration. You can open the the folder where user Python scripts should be stored. The folder path is fixed and cannot be changed.

Currently only Multiple DVB MUX scanning (S/C/T) integrates Python.

Download following script and use it as a starting point, https://drive.google.com/file/d/0B2ETNJgnsmi0NjktRGdSamItZWM/view?usp=sharing. This basically just defines simple stubs for the methods that will be invoked by DVBStreamExplorer.
The print statements do not have any effect when invoked from DVBStreamExplorer since these is no active console for output. They were used when invoking script from a small Python test application. Each method increments a member variable, This is just to demonstrate that state is kept in between calls.
Now you can modify the sample script for your own need. The general rule is if you want DVBStreamExplorer to continue it's current task return 0. If you want DVBStreamExplorer to abort it's current task return 1. In the sample script this is demonstrated for DVB-S scanning that will skip scanning for vertical transponders.

This is work in progress and likely to change before final release. It is strongly recommended not to make any custom solution for production usage. Things can break in furure preview and final releases.

Feel free to provide feedback, either as comment to post or directly to info@dvbstreamexplorer.com

Showing EIT information in DVBStreamExplorer Service View

DVBStreamExplorer V.4.0.20 includes a new feature that will allow EIT information to be continuously saved  to XML files from TS Monitor (DVB). DVBStreamExplorer Service View has been updated so it can now load and display such files. Below is screenshot after loading EIT XML file. Note how service names are initially displayed as triplet of ONID/TSID/SID.

To show real service names, load a DVBStreamExplorer service XML export that contains the necessary information. Below is screenshot after loading such file.

Note that the service tree items are ordered by the underlying ONID/TSID/SID values. This should preferably be changed to order by service names.

How section binary view in DVBStreamExplorer helped locating a bug

A little known feature in DVBStreamExplorer is the ability to view raw section content in main window. This feature recently helped me out locating a bug in DVBStreamExplorer.
Some time time ago I had done some DVB-S scanning of several satellites. SI and SVC XML files had been saved. Recently I tried to open SVC XML file in DVBStreamExplorer Service View which I was running from Visual Studio. I got this error during load.



I opened SVC XML file in wxHexEditor. I was able to locate the 0x07 character.

The problem was related to a service name. I was able to figure which service had the problem. I loaded the SI XML file in DVBStreamExplorer and navigated to that service and got this view.



There appears to be some kind of control character before service name. That's suspicious since control characters are supposed to be filtered out before being displayed. To view raw section content select section on tree view. And select 'value' tab in bottom view.
The first byte in highlighted area is 15 which is hexadecimal for 21 decimal which again is the length of the string. Then there is a sequence of 10 00 07 which constitutes character set selector. From ETSI EN 300 468 table A.4 it can be seen this is Latin/Greek. After this sequence there is the actual service name string. Considering that issue is about an unexpected 0x07 character it's obvious to suspect that 0x07 is considered part of the service name string.
After a code review such a bug was indeed found. This has been fixed in DVBStreamExplorer 4.0.20 just released. SI XML was loaded in new version. Service name was now displayed correctly as seen below.


A new SVC XML file was exported. And this time it could be loaded from DVBStreamExplorer Service View without any issues.

Integrating user defined Python scripts with DVBStreamExplorer

NOTE: this post describes a feature that is not yet included in DVBStreamExplorer public release.

A new feature is planned for a future DVBStreamExplorer release. This feature will allow a user defined Python script to be invoked at certain points during 'Multiple DVB-x MUX scanning'. User script will be called on events like:

• Starting a new scan session.
• Starting scan of new satellite. This is only applicable for DVB-S scanning.
• Starting scan of new MUX. Tuner parameters. 
• Scanning of MUX completed.
• Scanning of satellite completed. This is only applicable for DVB-S scanning.
• Scan session completed.

Depending on event context user script can typically request DVBStreamExplorer to continue scanning, skip scanning of current satellite/MUX or abort the entire scanning session. Events handlers are methods in a class user script must implement. Using a class allows script to maintain state between calls to methods.
In example below user script is handling the start scanning of new DVB-S MUX event.

    def StartScanMuxDvbS(self,frequency, symbolrate, polarisation, fec):

        print(self.count,frequency, symbolrate, polarisation, fec)

        self.count=self.count+1

        if polarisation == "H":

            return 0

        else:

            return 1

This code causes scanning of horisontal transponders only. Scanning of vertical transponders will be skipped.While this might not represent an interesting case for a real production system, it does demonstrate the possibilities. Note that 'self.count=self.count+1' statement is to keep track of the number of times script was called. This is for debugging purposes only.

A more realistic case for a real production system: If you are scanning multiple satellites on a regular basis, you might have certain expectations about hove many transponders should be scanned and how many services should be found. If scan result differs too much from your expectations you could send a notification, i.e. email, to a user to alert about this.

The work on this is still in progress. I now have a working prototype. But there is still some way before it's ready for public release.
Please feel free to provide feedback on this, i.e. if you have any suggestion about information that should be passed on to user script. You can provide feedback either by commenting on this post or send to info@dvbstreamexplorer.com.

PCR, OPCR,PTS and DTS reports in TS File Demux

In TS File Demux v2.3.17 new feature has been added to generate CSV reports containing details about PCR, OPCR, PTS and DTS timestamps.

To create PCR, OPCR CSV report add 'Single PID one file' filter for a PID stream containing such timestamps. That will typically be the PID marked as PCR in program tree. I.e. for PID 0x1001 the binary file ts_pid_0x1001.bin will be created in output folder together with ts_pid_0x1001.bin.csv file.
See embedded sheet below for an example of such CSV file.


To create PTS, DTS report add PES filter for a PES stream containing such timestamps. That would typically be video and audio streams. Note that not all stream types recognized as PES streams may contain timestamps. Refer to ISO 13818-1 for details.  I.e. for PID 0x0fa1 the binary file pes_0x0fa1.bin will be created in output folder together with pes_0x0fa1.bin.csv file.
See embedded sheet below for an example of such CSV file.

Extracting DSM-CC data and object carousel content with TS File Demux

TS File Demux now includes a feature to extract DSM-CC data and object carousel content. This feature is meant to be a lighter alternative to a similar feature in DVBSTreamExplorer which might be overkill in some situations. If you need decoding of DSM-CC sections and BIOP messages, DVBStreamExplorer is the still way to go.

Below is a screenshot of a TS file that contains  DSM-CC data.

 

TS File Demux main window

The DSM-CC filters have been added automatically after applying 'Add all'. Streams of type 0x0b, ISO/IEC 13818-6 type B, are considered as DSM-CC carousel streams. You can add DSM-CC filter for arbitrary PID if it wasn't recognized as DSM-CC carousel and you believe that it is.

Below is screenshot of some extracted object carousel files. The files are part of a MHP application.

Extracted files

The files can now be studied offline.

Follow this link to download TS File Demux here.

Decrypting service in DVBStreamExplorer using CI capable hardware

A little known feature in DVBStreamExplorer is the ability to decrypt DVB services using hardware that supports common interface (CI). You will need a TechnoTrend tuner card with CI support. This includes devices such as TT-connect CT 3650 CI and TT-connect S2 3650 CI. TT Budget cards with CI add-on card will also work. Cards with CI option from manufactures other than TechnoTrend will not work. Also this will only work with DVBStreamExplorer x86 version. CI and DiSEqC features are implemented in a proprietary DLL from TechnoTrend, ttBdaDrvApi_Dll.dll. Apparently this DLL is only available in 32 bit version.
You will also need a conditional access module (CAM) and a valid subscription smart-card. See picture below. It shows TT-connect CT 3650 CI, Conax CAM and subscription smart-card.

Insert your CAM and do a SI scan in DVBStreamExplorer. When completed switch to service view and DVBStreamExplorer should look similar to below.

Note that in lower right corner it can be seen that no service is currently selected.
To select a service for decryption locate an encrypted service in service view and double-click on it.
Lower right corner will now show that a service has been successfully selected.

Note this does not necessarily mean that service will be decrypted successfully. It just mean that tuning and selection of service was executed successfully.

You can check the DVBStreamExplorer log file for CI related entries. The log file can be found from options / preferences / logging.

Here is some logging from when CAM was inserted and initialized. The CA ID 0xb00 is for Conax.

 213 - Apr 25 09:25:51.189 - (9.842) - <4> - CI_OnSlotStatus(0,1)

 214 - Apr 25 09:25:51.189 - (9.842) - <4> - CI_SLOT_MODULE_INSERTED

 215 - Apr 25 09:25:57. 63 - (15.716) - <4> - CI_OnSlotStatus(0,2)

 216 - Apr 25 09:25:57. 63 - (15.716) - <4> - CI_SLOT_MODULE_OK

 217 - Apr 25 09:25:57. 63 - (15.716) - <4> - CI_SLOT_CA_OK

 218 - Apr 25 09:25:57. 63 - (15.716) - <4> - CA:  219 - Apr 25 09:25:57. 63 - (15.716) - <4> - supported CA ID[0]: 0x0B00

And here is some logging from when a service was selected for decryption.

1222 - Apr 25 12:10:11.524 - (9870.177) - <4> - Selecting service 'Discovery Channel'

1223 - Apr 25 12:10:11.623 - (9870.276) - <4> - CI_OnCAStatus(255,14,0)

1224 - Apr 25 12:10:11.623 - (9870.276) - <4> - CI_SWITCH_PRG_REPLY

1225 - Apr 25 12:10:11.623 - (9870.276) - <4> - ERR_NONE

Again this does not not necessarily mean that service will be decrypted successfully. It just means that connection to CAM is in good state.

To check that service is actually decrypted open DVB TS Monitor. Make sure that "Options / Parse A/V elementary headers" is enabled. Start the scan. In current service tab, services node locate the service selected for decryption. Verify that it looks something like below.

If you see details about A/V streams as tagged above, it means that service is successfully decrypted. That is unless the service was not encrypted at all to begin with. Note that it can take up to a minute or in rare cases even more for the header details to appear. This is because each stream in TS is scanned one by one for some time looking for header information.

Once decryption is verified you can save TS to disk. You can then view decrypted service in VLC. You can also stream service over UDP. Other clients can then pick up the decrypted service.

Capture HbbTV data with DVBStreamExplorer

In this post I will demonstrate how you can capture HbbTV data using DVBStreamExplorer. You will need DVBStreamExplorer Professional Edition for this exercise.
You can then study the HbbTV application offline. You can find more information about HbbTV here: https://www.hbbtv.org/.
First we need to find a service that contains HbbTV application. In this example I will find a service on Astra 19E.
You can find such service in this list: http://dtvlist.dvbstreamexplorer.com/dvbs.html. Look for 'hbbtv' in feature field.
Or you can do a complete SI scan of Astra 19E with DVBStreamExplorer. After completing SI scan switch to service view and you should see something similar to below.

Scroll to the far right to see features field and look for 'hbbtv'.

It can be seem that first service in the list, ORF SPORT+, features HbbTV.
Open the tuning window and tune on 11244H where the ORF SPORT+ service can be found.
Open Transport Stream Monitor (DVB) and start scanning. After a few seconds of scanning you should be able to select ORF SPORT+ from service drop-down list.

Now go to the DSM-CC tab. You should see list of DSM-CC streams found in service. For ORF SPORT+ there will only be one on PID 0x1c2b. Enable scanning of the PID in list.

It will now scan for DSM-CC data. This may take a short while. Once all DSM-CC data has been acquired the Object Carousel tab will be populated with file/folder structure. The content of files will also be shown as ASCII/Hex. Below can be seen partly content of a CSS file named min_portal_1_0_6.css which is part of HbbTV application.

And below is shown content of min_config_1_0_6.js which is a JavaScript file also part of HbbTV application.

Finally below shows partial content of index.html which is a HTML file also part of HbbTV application.

You can now stop scanning to enable file menu. From here the entire Object Carousel be saved. Directory structure in OC is preserved when saved to file system. You can now open the files in your favourite text editor for further examination.

DVBStreamExplorer service XML files

Introduction

DVBStreamExplorer has the feature to save service information as XML files. This can either be done explicitly from File->Export->XML menu, or it can be saved automatically on completion of a multiple MUX scanning session.
The folder where to save automatically can be set on Options->preferences menu on SI Scan tab.

The XML file contains information about all the services found during SI scan by analysing PAT, PMT, NIT, SDT and NIT tables. Post processing this XML file can have many useful applications.
In this post I will be demonstrate how to read the service XML file and post content to a web service encoded in JSON. For instance this could be used to feed an on-line web application with service data.

Service

Information about one service is stored in one XML element.
An example of a service XML element is shown below.

<service position="Astra 19.2°E" frequency="11303" polarity="H" symbolrate="22000" fec="2/3" modulation_system="1" modulation_type="2" roll_off="0" name="ORF1 HD" provider="ORF" network_type="dvbs" network_name="ASTRA 1" onid="1" nid="1" tsid="1007" sid="4911" pmt="107" pcr="1920" type="25" free_ca_mode="1" lcn="-1">
  <ca_list>
    <ca CA_system_ID="3333" CA_PID="220" private_bytes=""/>
    <ca CA_system_ID="3477" CA_PID="270" private_bytes=""/>
    <ca CA_system_ID="1608" CA_PID="120" private_bytes=""/>
    <ca CA_system_ID="5890" CA_PID="320" private_bytes=""/>
    <ca CA_system_ID="6195" CA_PID="420" private_bytes=""/>
    <ca CA_system_ID="2500" CA_PID="480" private_bytes=""/>
    <ca CA_system_ID="2444" CA_PID="490" private_bytes=""/>
    <ca CA_system_ID="3480" CA_PID="272" private_bytes=""/>
    <ca CA_system_ID="1616" CA_PID="122" private_bytes=""/>
  </ca_list>
  <streams>
    <stream type="27" pid="1920" type2="14496-10">
      <ca_list/>
    </stream>
    <stream type="4" pid="1921" language="ger" type2="13818-3">
      <ca_list/>
    </stream>
    <stream type="4" pid="1922" language="eng" type2="13818-3">
      <ca_list/>
    </stream>
    <stream type="6" pid="1923" language="ger" type2="ac3">
      <ca_list/>
    </stream>
    <stream type="6" pid="1925" type2="teletext">
      <ca_list/>
    </stream>
    <stream type="5" pid="7310" application_name="ORF HbbTV - SAT HD" type2="hbbtv">
      <ca_list/>
    </stream>
    <stream type="11" pid="7311" type2="data">
      <ca_list/>
    </stream>
  </streams>
</service>

As it can be seen there is quite some information. This includes properties like name, provider, tuner parameters, NID/ONID/TSID/SID and more. It also includes CA information, both from first loop of descriptors and the loop for each stream.

Each stream contains a type field which is simply the stream_type field from PMT. It contains a type2 field. The value is decided after analysing of stream_type field and certain descriptors. Besides benefaction of various audio and video streams this will also identify certain features and applications such as teletext, subtitle, mhp, hbbtv and more.

Client application

The client application is a .NET application written in C#. Source code is available from here https://dvbseserviceproc.codeplex.com/
It will allow you to select service XML file; network type; web service URL and method. After applying 'Process' service content will be sent to specified URL encoded in JSON.
You can modify the application for your specific needs if this will not work for you.

Server application

PHP

The server application is a simple PHP script that will consume the JSON data posted by client application. It will validate the presence of  a few expected properties, name and provider. Other than that it won't do anything with the data. In a real world application one might want to do stuff like saving the service data to a database.
The PHP script can be found below and can me modified for your specific needs. It was tested on a basic Apache running on Linux.

Google App Engine

There is is a live test web-service here http://dvbseserviceapi.appspot.com running on Google App Engine. Source code for web application is available here https://code.google.com/p/appengine-dvbseserviceapi/. 

Conclusion

This was just an minimum example to get you started. It doesn't really do anything useful as is. But it does show the potential of post processing service XML file.

H.262 usage reports in CSV format in TS File Demux 2.2.13

The latest version of TS File Demux introduces a new feature whereby extended reports about H.262 usage can be created in CSV format. The new feature is available as an option for all ES filters.
One report, h262report.csv, includes count of all picture_start_code, slice_start_code, user_data_start_code, sequence_header_code, sequence_error_code, extension_start_code, sequence_end_code and  group_start_code packets for all selected ES filters where H.262 report feature is enabled.
An example of such file is shown below. Document can also be opened directly in Google Drive here: https://docs.google.com/spreadsheets/d/1Qb7g-47NG9_kVpstgWGsyaUb0Gp9VeWNGU5-rgVzK64/edit?usp=sharing



Then for each selected ES filter where H.262 report feature is enabled a CSV file will be created listing all H.262 packets in order of occurrence.
An example of such file can open document directly in Google Drive from link below. The file is too large to be shown in-line.
https://docs.google.com/spreadsheets/d/1WaZNhbynCfofZ8AY_f4oUUs1YxBqkyDYHauzh4ALa2s/edit?usp=sharing

Discover section table_id with TS File Demux

When processing a TS file with TS File Demux a file named sectionreport.csv is created in output directory. This file contains various statistics about sections found in stream. Starting from TS File Demux V2.2.11 a new field, table_id, has been added to this file. This field contains a list of unique table_id's found in section stream. Sometimes this information can be useful, especially for streams containing multiple types of tables.
Below is an example of sectionreport.csv file after processing a DVB TS file.



It can be seen that PID 17 (0x11) contains SDT actual transport stream (0x42) and SDT other transport stream (0x46) tables.
PID 18 (0x12) contains various types of EIT tables.
PID 20 (0x14) contains TDT and TOT tables.
PID 48 (0x40) is a CA EMM stream containing table_id's 0x82 and 0x84 tables.
PID 102 (0x66) is a CA ECM stream containing table_id's 0x80 and 0x81 tables.
PID 8001 (0x1f41) contains a DSM-CC carousel. Table_id 0x3b is userNetworkMessage and table_id 0x3c is downloadDataMessage.

You can open the sample sheet here https://docs.google.com/spreadsheet/ccc?key=0AlzKD8F--RuTdDNrUFpSY0lMWkpxWG1lTTJZaUotTnc&usp=sharing

Decoding of DSM-CC BIOP messages

DVBStreamExplorer V4.0.3 has just been released. Now with DSMC-CC BIOP message decoding in DVB TS Monitor (professional edition only).
Until now DSM-CC analyser would show raw data of Data Carousel modules in one tab and the Object Carousel file system hierarchy in another tab. Starting from DVBStreamExplorer V4.0.3 decoding of BIOP messages is now shown for each module in Data Carousel. The information in BIOP messages is used to build the file system in Object Carousel.

In screenshot below is shown decoding of Service Gateway.

In screenshot below is shown decoding of a file object.

DVBStreamExplorer WS export online test service

If you have ever wanted to test DVBStreamExplorer export to WS features but gave up because of the hassle of creating a web service, then check out http://www.jensvaaben.com/dvbsewstest/.
It will allow you to export directly from DVBStreamExplorer to a live web service. Afterwards you can view the data that you have exported. Detailed instructions on http://www.jensvaaben.com/dvbsewstest/.

TS File Demux split PES/ES files on continuity error

TS File Demux V2.1.8 has just been released. A new feature has been introduced that will allow PES/ES files to be split whenever a continuity error is detected in TS packet.

In previous version you could end up with PES/ES files with missed packets due to continuity error in TS packet.
It might be desirable to produce PES/ES files without missed packets. The new feature will split PES/ES files every time continuity error is detected in TS packet for the PID in question. A new file will be created with name derived from  the base name of PES/ES file and .001, .002 etc appended.
Below is shown a screenshot below of output folder where split PES files have been created.

NOTE:

• Only TS packet header is decoded in order to detect continuity errors. Content of PES/ES packet is not decoded and/or analyzed in any way.
• This option can cause an excessive number of files to be created in output folder if there are a lot of continuity errors in TS file.