Automated Bluetooth testing and qualification of Mynewt NimBLE with auto-pts

We are happy to announce that auto-pts now supports Mynewt NimBLE stack!

To test and qualify your Bluetooth product you have to use a tool called Profile Tuning Suite (PTS) from Bluetooth SIG organization. It is a Windows program that you can use to manually run test cases using a graphical user interface. The process of testing manually requires a tester to operate both the device and the PTS program. This is a tedious job and can take one man-month. We described this process in one of our blog posts before. There is a tool that can speed that up to a few hours by fully automating the testing process.

auto-pts is a Bluetooth PTS automation framework that coordinates test execution between PTS and the Implementation Under Test (IUT). It automates over 400 test cases from GAP, GATT, L2CAP, SM, and MESH profiles. Auto-pts supports already BlueZ and Zephyr and it is supporting Mynewt NimBLE now!

On the IUT side, you will need to use the bttester app. It communicates with the auto-pts via UART. Bluetooth Testing Protocol (BTP) is used as a communication protocol. You can find the specification in the auto-pts repository. Check out the README on how to configure and run the auto-pts tool.

Happy testing! 🙂

Developing Apache Mynewt in Clion with quick and accurate indexing

It is not always possible to use a modern IDE for embedded software development. Sometimes your project uses a custom build system or a workflow that is hard to integrate with your favorite IDE. Apache Mynewt project has a modular design with features split into packages, dependencies and configuration files. It simplifies the architecture of the whole system and gives a lot of power to the user but also complicates the build process. To manage all of that a Newt tool was created. It is a smart build and package management system, that can handle multiple source repositories, decide what to include in a build and manage dependencies between components of the system. And one of it’s newer features is an integration with CMake.

Clion is a powerful cross-platform IDE for C and C++. It is easy to use, very responsive and supports tons of integrations. And it comes with vi emulation too! Most importantly, it uses CMake as a build system.

So now you can use Clion to develop Apache Mynewt applications!

We already described in this blog how to configure and use Eclipse with Mynewt. It is possible but requires some configuration. On the other hand, working on Mynewt in Clion requires almost no configuration.

How to work on Mynewt project in Clion

newt target cmake <target> command generates a CMakeLists.txt file for a specific target inside project root directory. This file contains all the CMake configuration options and a list of files to include in a build. You can then open the project inside Clion and the IDE should load the file correctly. Indexing in Clion is lightning fast! It takes just a few seconds and you are ready to start working on your project. You almost don’t notice it. If you need to change the target you are using, just run the command again with the new target and Newt will overwrite the CMakeLists.txt file. Clion will detect a change in this file and reindex your project. Just like that!

Indexing of a Mynewt project in Eclipse takes over a minute! So this is a huge improvement to your workflow. Another very important factor is the quality of indexing. Clion recognizes all the definitions and detects which files are included in the build and even which parts of the file are excluded. So you can jump to definitions, search by file or by symbol, refactor code, find usage and much more.

Building your project in Clion also works but you will still need to build and flash using Newt tool since both build processes are not yet equivalent.

Debugging Mynewt in Clion

You can also debug Mynewt in Clion but it requires a bit more configuration. You will need to create a GDB Remote Debug configuration inside Run/Debug Configurations window. Set the following fields:

Unfortunately, you will need to create a separate configuration for each target you want to debug.

Inside terminal run newt debug <target> -n – this will run gdb in server mode and will wait for the client to connect. Then in Clion select your debug configuration and click Debug button. That’s it!

Now you can develop your Apache Mynewt project inside a modern, powerful IDE with fast and accurate indexing.

Bluetooth Mesh on Apache Mynewt

Bluetooth Mesh implementation is the latest addition to Apache Mynewt Nimble stack. This tutorial will should you how to use it. If you want to learn more about Bluetooth Mesh visit this site.

This tutorial will provide you with a simple yet fully functional mesh structure.

For this demo, we will use Bluetooth Mesh by Silicon Labs application for Android, which will act as a provisioner.  You can download it for Android using Google Play. Furthermore, we need to use the blemesh application running on multiple nRF52840 PDK devices acting as nodes.

Bluetooth Mesh provides two bearers in form of GATT and advertising. Unfortunately, Silicon Labs application only supports PB-GATT (GATT Provisioning Bearer), so we have to enable PB-GATT for all nodes – this can be done by setting BLE_MESH_PB_GATT configuration option. One of our targets should also contain GATT Proxy Service so it can relay messages between application and other nodes which do not use GATT bearer. This is done by setting BLE_MESH_GATT_PROXY flag.

Building targets

Let’s start by creating target for a proxy device:

$ newt target create blemesh
$ newt target set blemesh app=@apache-mynewt-core/apps/blemesh
$ newt target set blemesh bsp=@apache-mynewt-core/hw/bsp/nordic_pca10056
$ newt target set blemesh build_profile=optimized
$ newt target set blemesh syscfg=BLE_MESH_PB_GATT=1: \
> BLE_MESH_DEV_UUID='(uint8_t[16]){0x22, 0x20, 0}'

Note that we do not set device public address since Bluetooth Mesh implementation uses non-resolvable private address (NRPA) as they are not really important in mesh. What is important though is a device UUID as set above.

We also need target for bootloader:

$ newt target create nrf52840_boot
$ newt target set nrf52840_boot app=@apache-mynewt-core/apps/boot
$ newt target set nrf52840_boot bsp=@apache-mynewt-core/hw/bsp/nordic_pca10056
$ newt target set nrf52840_boot build_profile=optimized

Now you can build and flash bootloader and application onto device:

$ newt build nrf52840_boot
$ newt load nrf52840_boot
$ newt build blemesh
$ newt create-image blemesh 0
$ newt load blemesh

Afterwards, our devices is all set and ready for action. But one device is not enough to form an interesting mesh network, therefore we will prepare more.  This process is pretty similar to the previous one, so you can just amend your previous target or create another one. The only significant difference is that each other device should get different device UUID. Also they should only have GATT Provisioning Service enabled (i.e. no GATT Proxy Service):

$ newt target set blemesh syscfg=BLE_MESH_PB_GATT=1: \
> BLE_MESH_DEV_UUID='(uint8_t[16]){0x22, 0x21, 0}'

Now you can flash bootloader and application onto other device(s).

Configuring Bluetooth Mesh Silicon Labs application

As soon as your devices are ready, it is time to create your own mesh network using the Android app. We’ve recorded a video that should help you with the whole process:

Note that you also need some serial terminal program to monitor the output of every device, as they will show the authorization code that is required to provision your device.

Then, if you follow the rest of the instructions from the video, everything should work just fine.

Happy meshing!

Szymon Czapracki

Interoperability testing is important!

As we all know, each Bluetooth product shall work according to Bluetooth specification. It is fairly big specification, and that’s why before releasing your Bluetooth product it is very important that it can talk to other Bluetooth devices. Basically, we should check if we understand specification in the same way as others. That is why UnPlugFest (UPF) is organized 3 times per year by Bluetooth SIG, so developers can test their solutions with other products to verify they work together. Codecoup understands that and encourages its customers to be there. Speaking of which, we just came back from UPF56 in Atlanta where we had two teams testing two different IoT platforms.

Łukasz Rymanowski

Bluetooth device qualifiaction

Whenever you create a Bluetooth product you have to go through a process of qualification in order to get Bluetooth SIG stamp 🙂 This process is usually painful for developers and testers, especially when you are doing it for the first time. Fortunately, some of the Open Source stacks provide instructions on how to do this, which makes things bit easier. In this post, we will show you how to start a Bluetooth device qualification testing using an nRF52 board running Mynewt OS.

Mynewt is a Real-time operating system build with IoT in mind and provides open source networking stack including Bluetooth Low Energy. More details can be found on their website.

Mynewt allows to develop and run apps that perform specific tasks. bletiny is one of such applications. It allows controlling the Bluetooth stack from the command line. The application includes a simple shell functionality and allows e.g. start advertising or discovery.

A tutorial describing the necessary steps to run the application is available on their documentation website. To communicate with the application we will need to set up a serial connection. There is a tutorial for that too. To check if everything works properly you can run b help command, which lists all available commands in the bletiny app.

[12:08:39:333] b help
[12:08:39:336]
[12:08:39:336] 831277:Available commands:
[12:08:39:345] 831277: adv
[12:08:39:345] 831278: conn
[12:08:39:345] 831278: chrup
[12:08:39:345] 831278: datalen
[12:08:39:355] 831279: disc
[12:08:39:355] 831279: find
[12:08:39:355] 831280: help
[12:08:39:364] 831280: l2cap
[12:08:39:364] 831280: mtu
[12:08:39:364] 831281: passkey
[12:08:39:374] 831281: read
[12:08:39:374] 831281: rssi
[12:08:39:374] 831282: scan
[12:08:39:374] 831282: show
[12:08:39:374] 831282: sec
[12:08:39:382] 831283: set
[12:08:39:382] 831283: store
[12:08:39:390] 831283: term
[12:08:39:390] 831284: update
[12:08:39:390] 831284: tx
[12:08:39:400] 831285: wl
[12:08:39:400] 831285: write
[12:08:39:400] 831285: svcchg

The instructions for the tests can be found in the net/nimble/host/pts directory in the apache-mynewt-core project tree.

tree net/nimble/host/pts/

net/nimble/host/pts/
├── pts-gap.txt
├── pts-gatt.txt
├── pts-l2cap.txt
├── pts-sm.txt
├── README.txt
└── tpg
    ├── 90359-20161220-172100175.tpg
    └── 90359-20161220-172113981.pts

1 directory, 7 files

The contents of the directory are described in the README.txt file:

This folder contains qualification tests results against BT SIG Profile Test
Suite.

pts-FOO.txt files contain result for specific profiles or protocols. This
includes PTS version, test date, enabled tests, results etc.

In addition to tests results 'tpg' folder contains Test Plan Generator
configuration files that can be imported by PTS for tests configuration.

Fragment of the test instructions file:

PTS test results for GAP

PTS version: 7.0
Tested: 10-Jan-2017

Results:
PASS test passed
FAIL test failed
INC test is inconclusive
N/A test is disabled due to PICS setup

-------------------------------------------------------------------------------
Test Name Result Notes
-------------------------------------------------------------------------------
TC_MOD_NDIS_BV_01_C N/A
TC_MOD_LDIS_BV_01_C N/A
TC_MOD_LDIS_BV_02_C N/A
TC_MOD_LDIS_BV_03_C N/A
TC_MOD_GDIS_BV_01_C N/A
TC_MOD_GDIS_BV_02_C N/A
TC_MOD_NCON_BV_01_C N/A
TC_MOD_CON_BV_01_C N/A
-------------------------------------------------------------------------------

TC_BROB_BCST_BV_01_C PASS b adv conn=non
TC_BROB_BCST_BV_02_C PASS b adv conn=non
TC_BROB_BCST_BV_03_C PASS b set irk= e.g: 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:11

........

This file contains a list of test descriptions. Each description consists of a test name, it’s result and list of instructions needed to execute the test.

After configuring the device, we can move on to setting up the PTS.

Prerequisites

Setting-up PTS environment

Install PTS drivers for the dongle

In device manager select the PTS dongle device, open Properties and update driver. Browse for driver in C:\Program Files (x86)\Bluetooth SIG\Bluetooth PTS\PTS Driver\win32 and select csrbc.sys. If you are using a 64-bit system, then change win32 to win64.

Create new workspace and search for device

Launch PTS and select Create new workspace. Set your device in advertising mode and search for it in PTS. It should appear in the device list. Select it, so that the device address field is filled and move to the next step.

Import Test Plan

Fill in the name and location of the workspace and import the Test Plan. The Test Plan can be found in the net/nimble/host/pts directory in the apache-mynewt-core project tree.

Select test suite(s)

Select the Test Suites you are interested in and move them to the list on the right.

Use Bluetooth Protocol Viewer

It’s a great debugging tool. It allows you to see exactly what was transmitted and received by the PTS in great detail. Each frame is formatted into a very readable tree-like structure, and the frames are categorized by the protocol.

Important definitions

Before we move on to the testing itself, some important definitions that are used in PTS:

For more definitions read Test Strategy and Terminology Overview document.

Example #1

As a fist example we will run test TC_BROB_BCST_BV_01_C. In the PTS right-click on the test in Workspace Tool Window and select Run from context menu. PTS starts the test and displays an information window saying, that a user action is required:

Following the instructions, we execute b adv conn=non in the bletiny app which starts the advertising procedure in non-connectable mode. Here is the output of the test execution in the PTS:

Test case : TC_BROB_BCST_BV_01_C started
- MTC: Successfully received HCI_LE_CLEAR_WHITE_LIST_COMPLETE_EVENT.
- MTC: Successfully received HCI_LE_ADD_DEVICE_TO_WHITE_LIST_COMPLETE_EVENT.
- MTC: Successfully received HCI_LE_SET_SCAN_PARAMETERS_COMPLETE_EVENT.
- MTC: Successfully received HCI_LE_SET_SCAN_ENABLE_COMPLETE_EVENT.
- MTC: Receive Advertising report successfully.
- MTC: Flags type is not present in any scan response data received.
- MTC: Successfully received HCI_LE_SET_SCAN_ENABLE_COMPLETE_EVENT.
- CM_EXIT
- CM_EXIT
- CM_EXIT
- CM_EXIT
- MTC: Test case ended
-Final Verdict: PASS
TC_BROB_BCST_BV_01_C finished

Example #2

The second example will be a little more complex. We will run TC_SEC_AUT_BV_13_C test. It verifies that the IUT properly rejects the service request where there is insufficient authentication and then completes service correctly with Lower Tester as a peripheral role.

The instruction for this test is a bit longer:

TC_SEC_AUT_BV_13_C PASS Note: in PTS confirm that IUT supports GATT Server
                        b set sm_data mitm_flag=1
                        b set sm_data io_capabilities=1
                        b conn peer_addr=
                        b disc svc conn=
                        Note: in PTS enter handle for characteristics
                        value which requires authenticated
                        pairing for read
                        b passkey conn= action=3 key=123456
                        Note: enter '123456' passkey in PTS

First, we run the test and confirm in the PTS that our device supports GATT Server. We then execute b set commands that set some Security Manager related data. Then PTS requires us to send a connection request to the PTS.

We send the request by executing b conn peer_addr= command, where is the address of the PTS dongle. Then we start service discovery by executing the `b disc svc conn=`, where is the connection handle. You can list connections with the b show conn command.

After that, the PTS requires us to enter the handle for characteristic value, which requires authentication for reading. You should check the GATT Server database for this value.

PTS then requires us to enter Secure-ID (aka passkey). Before we do that, we need to set a passkey in the bletiny app, so that the device knows what passkey to use, to respond to the PTS. To do that, we type in b passkey conn= action=3 key=123456. After that, we enter the same passkey in the PTS window.

After following these steps, the test should pass. The output of the test is shown below:

Test case : TC_SEC_AUT_BV_13_C started
- MTC: Successfully received HCI_LE_CLEAR_WHITE_LIST_COMPLETE_EVENT.
- MTC: Successfully received HCI_LE_ADD_DEVICE_TO_WHITE_LIST_COMPLETE_EVENT.
- MTC: Successfully received HCI_LE_SET_ADVERTISING_PARAMETERS_COMPLETE_EVENT.
- MTC: Successfully received HCI_LE_SET_ADVERTISING_DATA_COMPLETE_EVENT.
- MTC: Successfully received HCI_LE_SET_SCAN_RESPONSE_DATA_COMPLETE_EVENT.
- MTC: Successfully received HCI_LE_SET_ADVERTISE_ENABLE_COMMAND_COMPLETE_EVENT.
- MTC: ATT connection established successfully.
- MTC: SEC_LE_PAIRING_COMPLETE_IND received successfully.
- MTC: SEC_LE_ACCESS_CFM received successfully.
- MTC: IUT respond as requested.
- MTC: SEC_LE_PAIR_REQUEST_IND received successfully.
- MTC: SEC_LE_PAIR_REQUEST_IND received successfully.
- MTC: SEC_LE_PAIRING_COMPLETE_IND received successfully.
- MTC: SEC_LE_ACCESS_CFM received successfully.
- MTC: ATT_READ_CFM received successfully.
- MTC: ATT connection disconnected successfully.
- CM_EXIT
- CM_EXIT
- CM_EXIT
- CM_EXIT
- MTC: Test case ended
-Final Verdict: PASS
TC_SEC_AUT_BV_13_C finished

References

ELCE and OpenIoT Summit Europe 2016 videos are online

Video from Szymon’s presentation on OpenIoT is available now

There were a lot of interesting sessions this year on both Embedded Linux Conference Europe and OpenIoT Summit Europe. All videos are available here.