ESP32 BLE threads causing extremely slow TLS computation over WiFi

[arifbalik]

I am working with esp32 (esp32-wroom-32-n4) with BLE and Wifi enabled at the same time, and also connecting to an mqtt broker via secure ports.

A standalone application that just connects to BLE and mqtt broker works fine, but whenever I try to create a thread (with lowest preemtive prio) to pull and process the data from these channels the mqtt connection fails becuase the TLS handshake is extremely slow (~30 seconds on avarage) and even after the connection is made the messages arrives way too late.

When I analyze the thread utilization I dont neccesarily see the BT threads starving other threads but whenever I disable BT subsys, everything works just fine. And BT threads have the highest priority (all coopertive threads), maybe interrupts causing this?

I have the following configuration;

# Network Config
CONFIG_NETWORKING=y
CONFIG_NET_IPV6=n
CONFIG_NET_IPV4=y
CONFIG_NET_SOCKETS=y
CONFIG_NET_TCP=y
CONFIG_NET_CONFIG_SETTINGS=y
CONFIG_NET_SOCKETS_SOCKOPT_TLS=y
CONFIG_NET_HOSTNAME_ENABLE=y
CONFIG_NET_HOSTNAME="enda_iot1_appwifi"

# MQTT Config
CONFIG_MQTT_LIB=y
CONFIG_MQTT_LIB_TLS=y
CONFIG_ENDA_MQTT_KEEPALIVE_SECONDS=30
CONFIG_ENDA_MQTT=y

# DNS Config
CONFIG_DNS_RESOLVER=y
CONFIG_DNS_SERVER_IP_ADDRESSES=y
CONFIG_DNS_SERVER1="8.8.8.8"

# mbedTLS Config
CONFIG_MBEDTLS=y
CONFIG_MBEDTLS_BUILTIN=y
CONFIG_MBEDTLS_SSL_MAX_CONTENT_LEN=2048
CONFIG_MBEDTLS_KEY_EXCHANGE_ECDHE_ECDSA_ENABLED=y
CONFIG_MBEDTLS_ECDH_C=y
CONFIG_MBEDTLS_ECDSA_C=y
CONFIG_MBEDTLS_ECP_C=y
CONFIG_MBEDTLS_ECP_DP_SECP256R1_ENABLED=y
CONFIG_MBEDTLS_HMAC_DRBG_ENABLED=y
# PSA
CONFIG_MBEDTLS_PSA_CRYPTO_C=y
CONFIG_MBEDTLS_USE_PSA_CRYPTO=y
CONFIG_PSA_CRYPTO_ENABLE_ALL=y

# Logging
CONFIG_LOG=y
CONFIG_LOG_MODE_IMMEDIATE=y
CONFIG_NET_LOG=n
CONFIG_EVENTS=y

# Entropy
CONFIG_ENTROPY_GENERATOR=y
CONFIG_MBEDTLS_ZEPHYR_ENTROPY=y

# Memory
CONFIG_MAIN_STACK_SIZE=8192
CONFIG_SYSTEM_WORKQUEUE_STACK_SIZE=8192
CONFIG_NET_SOCKETS_SERVICE_STACK_SIZE=4096
CONFIG_ISR_STACK_SIZE=4096
CONFIG_INIT_STACKS=y

# Nanopb
CONFIG_NANOPB=y
CONFIG_ENDA_NANOPB=y

# Bluetooth
CONFIG_BT=y
CONFIG_BT_PERIPHERAL=y
CONFIG_BT_DEVICE_NAME_DYNAMIC=y
CONFIG_BT_DIS=y
CONFIG_BT_GATT_CLIENT=y
CONFIG_BT_GATT_DYNAMIC_DB=y

# Settings
CONFIG_NVS=y
CONFIG_FLASH=y
CONFIG_FLASH_MAP=y
CONFIG_SETTINGS=y
CONFIG_SETTINGS_NVS=y

CONFIG_REBOOT=y
CONFIG_THREAD_NAME=y

# Modbus
CONFIG_MODBUS=y
CONFIG_MODBUS_LOG_LEVEL_ERR=y

# Flash
CONFIG_FLASH=y
CONFIG_FLASH_PAGE_LAYOUT=y

# Network Config
CONFIG_NET_CONFIG_AUTO_INIT=n
CONFIG_NET_DHCPV4=y
CONFIG_NET_IF_MAX_IPV4_COUNT=2
CONFIG_NET_MGMT=y
CONFIG_NET_MGMT_EVENT=y
CONFIG_NET_L2_ETHERNET=y

# WiFi Config
CONFIG_WIFI=y
CONFIG_WIFI_ESP32=y
CONFIG_ESP32_WIFI_STA_AUTO_DHCPV4=y
CONFIG_ESP32_WIFI_STA_RECONNECT=y

# Misc
CONFIG_REQUIRES_FULL_LIBC=y
CONFIG_POSIX_API=y
CONFIG_POSIX_TIMERS=y

[ldomaigne]

There might be indeed some conflicts between the "cooperative vs. preemptive" threads.

Tracing would be the proper tool to analyze this kind of problem.
Do you have access to Percepio's tracealyzer or Segger's SystemView?

Alternatively, you could capture the trace in RAM and use other techniques, as explained in Eve's blog

[arifbalik]

I've played quite a bit with the threads and it does not seem to be the cause. The problem seems more strange more I debug it. For instance when I change CONFIG_ZVFS_OPEN_MAX the problem goes away, but the file descriptors my app needs is only 7, and I have to set CONFIG_ZVFS_OPEN_MAX=100 to make it work. Once I saw this I went and created just a dummy static variable with sufficiently large size (1037 in my case);

static char dummy[...];

TLS connection again takes forever (sometimes never). zsock_connect() never returns. But I'm still not sure if zsock_connect is the reason the system slows down (its called inside main thread).

Somethings in the .bss section is causing this

Most of my stacks are 8K and I have space so I don't yet think this problem is related to lack of memory.

I'll try to do a simple test setup where I enable BLE and MQTT with TLS only.

[arifbalik]

@ldomaigne I checked the threads and nothing seems to starve anything. When I try to connect without secure connection (MQTT over 1883 with no encryption) everything works fine, but whenever I enable encryption everything slows down and most of the time it freezes, mbedtls debug log does not show anything suspicious. Any idea what that might mean? Changing CONFIG_ZVFS_OPEN_MAX effects this issue, when I check the linker map file, some wifi bluetooth and mbedtls variable memory locations are changing, they are placed e.g. 48 bytes further down the memory and behaviour changes