<style type="text/css">
{css, echo=FALSE}
body { counter-reset: h1counter h2counter h3counter h4counter h5counter h6counter; }

h1 { counter-reset: h2counter; }
h2 { counter-reset: h3counter; }
h3 { counter-reset: h4counter; }
h4 { counter-reset: h5counter; }
h5 { counter-reset: h6counter; }

h1:before {
    counter-increment: h1counter;
    content: counter(h1counter) ".\0000a0\0000a0";
}

h2:before {
    counter-increment: h2counter;
    content: counter(h1counter) "." counter(h2counter) ".\0000a0\0000a0";
}

h3:before {
    counter-increment: h3counter;
    content: counter(h1counter) "." counter(h2counter) "." counter(h3counter) ".\0000a0\0000a0";
}

h4:before {
    counter-increment: h4counter;
    content: counter(h1counter) "." counter(h2counter) "." counter(h3counter) "." counter(h4counter) ".\0000a0\0000a0";
}

h5:before {
    counter-increment: h5counter;
    content: counter(h1counter) "." counter(h2counter) "." counter(h3counter) "." counter(h4counter) "." counter(h5counter) ".\0000a0\0000a0";
}

h6:before {
    counter-increment: h6counter;
    content: counter(h1counter) "." counter(h2counter) "." counter(h3counter) "." counter(h4counter) "." counter(h5counter) "." counter(h6counter) ".\0000a0\0000a0";
}

</style>


# 代码功能

proto_forward 会被编译为一个 ELF 可执行程序，main() 函数位于 src/ems/main.c

链接了其他一些程序库：

```console
root@218830000254:~# ldd /opt/lnxall_app/bin/proto_forward
        linux-vdso.so.1 (0x0000007fb5d99000)
        libdyutils.so => /opt/lnxall_app/lib/libdyutils.so (0x0000007fb5d38000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007fb5cfb000)              # 多线程库。glibc 的一部分
        libmqtt_session.so => /opt/lnxall_app/lib/libmqtt_session.so (0x0000007fb5ce3000)           # 对 libmosquitto 的封装
        liblz4.so.1 => /opt/lnxall_app/lib/liblz4.so.1 (0x0000007fb5ca0000)                         # lz4 压缩算法
        libcrypto.so.1.1 => /opt/lnxall_app/lib/libcrypto.so.1.1 (0x0000007fb5a5f000)
        libcurl.so.4 => /opt/lnxall_app/lib/libcurl.so.4 (0x0000007fb59c6000)
        liblua.so.5.1.5 => /opt/lnxall_app/lib/liblua.so.5.1.5 (0x0000007fb598f000)
        libopen62541.so.1 => /opt/lnxall_app/lib/libopen62541.so.1 (0x0000007fb5880000)             # 开源 OPC UA 库
        libsqlite3.so.0 => /opt/lnxall_app/lib/libsqlite3.so.0 (0x0000007fb577a000)                 # SQLite 数据库
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007fb5609000)                          # C 语言标准库
        /lib/ld-linux-aarch64.so.1 (0x0000007fb5d6b000)                                             # 动态链接器
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007fb55f5000)
        libmosquitto.so.1 => /opt/lnxall_app/lib/libmosquitto.so.1 (0x0000007fb55cd000)             # 开源的 MQTT 库
        libipc_helper.so => /opt/lnxall_app/lib/libipc_helper.so (0x0000007fb55ba000)               # 对 libmosquitto 的封装
        libubox.so => /opt/lnxall_app/lib/libubox.so (0x0000007fb559e000)
        libubuslog.so => /opt/lnxall_app/lib/libubuslog.so (0x0000007fb5589000)
        librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007fb5571000)
        libssl.so.1.1 => /opt/lnxall_app/lib/libssl.so.1.1 (0x0000007fb54e8000)                     # open ssl
        libz.so.1 => /opt/lnxall_app/lib/libz.so.1 (0x0000007fb54ba000)                             # 压缩算法
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007fb53fd000)                          # 数学运算库
        libmbedtls.so.20 => /opt/lnxall_app/lib/libmbedtls.so.20 (0x0000007fb53c0000)
        libmbedx509.so.6 => /opt/lnxall_app/lib/libmbedx509.so.6 (0x0000007fb539f000)
        libmbedcrypto.so.15 => /opt/lnxall_app/lib/libmbedcrypto.so.15 (0x0000007fb5313000)
        libubus.so => /opt/lnxall_app/lib/libubus.so (0x0000007fb52fd000)
```

## OPC UA

OLE (Objecvt Linking and Embedding) Process Control, Unified Architecture

会启动一个 OPC UA server. 使用的是开源的 [open62541](https://github.com/open62541/open62541)

入口：

1. main() 填充了 pcs_ctrl_var.param.ua_base_data 的部分字段。
2. src/proto_forward.c:proto_forward_creat()
3. src/opcua/ua-server.c:ua_server_init()
    1. 调用 libopen62541.so 的 API `UA_Server_new()` 创建一个 UA server。
    2. 配置 server cert/key 和 client cert/key:
    ```c
    #define SERVER_CERT_CERT "/app/cert/server_cert.der"
    #define SERVER_CERT_KEY  "/app/cert/server_key.pem"
    ```
    3. 添加用户名密码方式登录。默认有 2 组用户名/密码：
        - localuser/序列号，若序列号为空，则密码默认为 123456.
        - admin/lnxall123
4. src/proto_forward.c:proto_forward_schedule()
5. src/opcua/ua-server.c:ua_server_thread_start() 调用 `UA_Server_run()` 启动 UA server. 

### 使用客户端连接

可使用免费的 OPC UA 客户端 UaExpert https://www.unified-automation.com/products/development-tools/uaexpert.html

打开 UaExpert, 新加 server:

- Configuration name: 取一个有意义的名字，如 "EMS".
- Discovery: Custom Discovery - Double click to add server - 输入 OPC UA server 地址，如 *opc.tcp://172.18.39.254* 把 IP 地址改为自己设备的 IP 地址。
- Authentication Settings: 使用用户名密码 admin/lnxall123

填写完毕后连接，即可在 UaExpert 左侧的 **Address Space** 小窗口看到设备上的 OPC 对象。

## MQTT

Message Queuing Telemetry Transport 消息队列遥测传输协议

MQTT 是一种轻量级的、基于发布-订阅模式的消息传输协议，专为低带宽和高延迟或不可靠的网络设计。它非常适合物联网应用，因为它占用资源少，通信效率高，并且具有良好的网络适应性。MQTT 基于 TCP/IP 协议栈，但也可运行在其他网络协议上，如 UDP、蓝牙、LoRaWAN 等。

### MQTT Broker

MQTT Broker 是负责处理客户端请求的关键组件，包括建立连接、断开连接、订阅和取消订阅、责消息转发。一个高效强大的 MQTT Broker 能够轻松应对海量连接和百万级消息吞吐量。

发布-订阅模式与客户端-服务器模式的不同之处在于，它将发送消息的客户端（发布者）和接收消息的客户端（订阅者）进行了解耦。发布者和订阅者之间无需建立直接连接，而是通过 MQTT Broker 来负责消息的路由和分发。

下图展示了 MQTT 发布/订阅过程。温度传感器作为客户端连接到 MQTT Broker，并通过发布操作将温度数据发布到一个特定主题（例如 Temperature）。MQTT Broker 接收到该消息后会将其转发给订阅了相应主题（Temperature）的订阅者客户端。

```
              Publish 25 degree              Subscribe Temperature
Temperature  ------------------->  BROKER  <----------------------- Client 1
  sensor                           ^  |     ---------------------->
                                   |  |     Publish Temperature 25
                         Subscribe |  |    
                                   |  |
                                   |  V
                                      Publish
                                  Client 2

```

# 配置文件

## /app/config/device_config.json

部分内容：

```json
{
    "dev_class":[
        {
            "type":"METER",
            "devs":[
                {
                    "no":"PCS_METER",
                    "name":"储能表",
                    "type":"PCS",
                    "channel":"RS485_1",
                    "protocol":"modbus_rtu",
                    "user_def":"ct=40",
                    "addr":"1",
                    "template":"METER_正泰_DTSU666_MODBUS.json"
                },
                {"no":"TRAS_LV_METER","name":"需量表", "type":"TRAS_LV", "channel":"RS485_1","protocol":"modbus_rtu","user_def":"ct=40","addr":"1","template":"METER_正泰_DTSU666_MODBUS.json"},
                {"no":"GRID_METER","name":"关口表","type":"GRID","channel":"RS485_2","protocol":"dlt645_2007","user_def":"ct=240","addr":"230322360764","template":"METER_通用_DLT645.json"}
            
            ]
        },
        {
            "type":"PCS",
            "devs":[
                {"no":"PCS","name":"变流器","channel":"172.18.39.150:502","protocol":"modbus_tcp","user_def":"frame_timeout=6000","addr":"1","template":"PCS_英博_100_MODBUS_SPLIT_PHASE.json"}
            ]
        }
    ]
}
```

解析此配置的代码：src/ems/main.c:load_device_cfg(var->device_layer_ptr, CONFIG_PATH "/" COLLECTOR_DEVICE, 1);

`devs` 数组中的每个元素对应于一个 `channel_t` 结构体，调用 channel_create() 创建：
1. device_create() 不同设备的一个重要不同是其 "protocol" 字段不同。
2. parse_template_cfg() 解析 "template" 字段指定的模板文件，在设备的 /app/template/ 目录下。
3. ua_server_add_dev_node() 把 channel 添加为 OPC UA server 的节点，通过 UaExpert 可以查看。
4. dev_tags_creat_and_add_to_channle()

## /app/config/ems_device_master_config.json

```json
{
    "dev_class":[
        {
            "type":"EMS",
            "devs":[
                {
                    "no":"EMS",
                    "name":"EMS",
                    "channel":"EMS",
                    "protocol":"ems",
                    "template":"ems_template_master_config.json"
                }
            ]
        }
    ]
}
```

# 首次编译
## 准备环境
- 构建环境登录你的账号在192.168.22.104服务器
- 产生公钥 ssh-keygen
- 你账号下的配置git 账号/邮箱
- 上传到http://192.168.22.227 服务器ssh keys

## 拉取代码
```bash
mkdir non-openwrt  # 代码的工作目录,名称随意
cd non-openwrt
repo init -u git@192.168.22.227:lnxall/wlos_staging/wlos_nobsp.git --no-clone-bundle --no-repo-verify
repo sync
./wlos_core/scripts/checkout.sh
```
## 全编译
```bash
cd wlos_core
ln -sv target_rk3568_glibc target  # target_rk3568_glibc是我们的主要产品EMS1000的底层构建文件 target_em1000_glibc # 是我们外购的一款硬件产品 
./build.sh   
```
## 全清除
```bash
./build.sh   clean  # 清除所有编译文件
```

## 单应用编译
```bash
cd non-openwrt/wlos_core/package_repos/proto_forward/src
../../../build.bash  # 单独编译 proto_forward

```

## 单应用清除(一般无需使用)
```bash
cd non-openwrt/wlos_core/package_repos/proto_forward/src
../../../build.bash clean # 单独编译 proto_forward
```

## 参考
参考 http://192.168.22.227/lnxall/wlos_staging/wlos_doc/-/blob/master/%E4%BD%BF%E7%94%A8%E9%9D%9Eopenwrt%E4%B8%BAEM1000%E7%BD%91%E5%85%B3%E7%BC%96%E8%AF%91%E8%BD%AF%E4%BB%B6%E7%9A%84%E8%AF%B4%E6%98%8E.md

# 运行

## 连接到设备

设备上有一个 Type-C 接口，使用的是 CH340 USB-串口转换芯片。可通过串口登录：

1. 给电脑安装 [CH340 驱动程序](https://www.wch.cn/downloads/CH341SER_EXE.html)
2. 用 USB 数据线（仅支持充电的线不行）连接电脑和设备，在 Windows 设备管理器中查看 COM 号。
3. 打开 MobaXterm，连接串口，指定上一步查到的 COM 号。波特率 115200.

Ethernet 直连：

1. 用 Ethernet 网线把电脑连接到设备的 LAN 口。
2. 通过串口确认设备的 IP 地址。
3. 把电脑 Ethernet 口的 IP 地址改为与设备在同一个网段。
4. 在电脑上尝试 ssh 登录到设备。如果可以连接但登录不成功，可以换一个 LAN 口再试。

通过 FRPD 反向代理服务器连接远程设备：

1. 登录 frpd 服务器 http://emu.lnxall.com:9500/ 用户名/密码 dywl, dywlfrp123
2. 点击左侧 Proxies - TCP，搜索设备的 SN 号。每个 SN 号对应 3 个搜索结果，分别对应 OPC UA、SSH、Web UI. 通过 SSH 登录时只关心其 SSH 端口号。
3. 通过 `ssh -p <PORT> root@emu.lnxall.com` 登录到设备，把 PORT 改为上一步搜索到的 SSH 端口号。

## proto_forward 进程

在设备上运行时：

```console
root@218830000254:~# ps -elf
4 S root      1077     1  0  80   0 -  2512 do_wai Sep05 ?        00:00:23 /bin/bash /opt/lnxall_app/bin/emsd_startup.sh
0 S root      1488  1077  3  80   0 - 461227 futex_ Sep05 ?       00:53:13 /opt/lnxall_app/bin/proto_forward -e -n eth0
```

启动关系：systemd - emsd_startup.sh (emsd.service) - proto_forward

```console
root@218830000254:/app/template# systemctl status emsd.service
● emsd.service - ems Service
   Loaded: loaded (/lib/systemd/system/emsd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2024-09-05 15:16:55 CST; 24h ago
 Main PID: 1077 (bash)
   CGroup: /system.slice/emsd.service
           ├─1077 /bin/bash /opt/lnxall_app/bin/emsd_startup.sh
           ├─1488 /opt/lnxall_app/bin/proto_forward -e -n eth0
           └─4381 sleep 10
```

## 运行自己修改的程序

要替换为自己编译的 proto_forward 程序，可以：

1. systemctl stop emsd.service
2. 备份设备上的 /opt/lnxall_app/bin/proto_forward
3. 使用 ssh 把自己的 proto_forward 程序上传到设备
4. systemctl start emsd.service

# 部署
## 单个应用部署

## 整改固件部署

## 屏控部署

# 代码管理
代码使用git管理

## 代码检查
代码在git add 前必须使用 git diff ,确认自己修改内容,确保提交内容中满足以下几条
- 不能含有临时测试代码段
- 不能含有不改临时的log打印
- 不能对无关代码格式做自动格式化

## 代码提交
代码提交必须使用 git commit -m "提交类型: 提交信息";举例 git commit -m "fix: 修复下发策略代码奔溃问题"
- 提交类型: feat, fix, docs, style, refactor, test, chore 以下为说明,注意策略类型末尾有冒号和空格符
- 提交信息: 必须可以大概描述提交内容涉及特性,或者关键bug修复信息,可以使用中文. 禁止出现"代码更新/bug修复等提交"
feat：新功能（feature）                   
fix：修补bug 
docs：文档（documentation）           
style： 格式（不影响代码运行的变动）
refactor：重构（即不是新增功能，也不是修改bug的代码变动）
test：增加测试
chore：其他修改, 比如构建流程, 依赖管理    


# Debug
调试方法可借助 gdb, strace 等工具，也可从日志中寻找线索：
## 查看log
- 日志文件: /var/log/syslog
- journalctl -u emsd
## gdb
每个硬件上都默认安装了gdb，可以用来调试程序。

## 例：多个线程卡住

现象：proto_forward 的多个线程卡住。

执行 `ps -elf | grep proto_forward` 找到 proto_forward 进程的 PID:

```
0 t root     23352     1 49  80   0 - 596476 ptrace Sep05 ?       1-19:07:49 /opt/lnxall_app/bin/proto_forward -e -n eth0
```

执行 gdb -p 23352 附加到进程。

```
(gdb) info thread
  Id   Target Id                                         Frame
  1    Thread 0x7f8d20c010 (LWP 23352) "proto_forward"   0x0000007f8d18c26c in do_futex_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
  2    Thread 0x7f8c77b1e0 (LWP 23353) "ubuslog_thread"  0x0000007f8cb58830 in epoll_pwait () from /lib/aarch64-linux-gnu/libc.so.6
  3    Thread 0x7f8bf7a1e0 (LWP 23354) "proto_forward"   0x0000007f8cb51b08 in pselect () from /lib/aarch64-linux-gnu/libc.so.6
  4    Thread 0x7f8b7791e0 (LWP 23355) "proto_forward"   0x0000007f8d189c3c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
  5    Thread 0x7f8af551e0 (LWP 23356) "discoverTask"    0x0000007f8d18d360 in recv () from /lib/aarch64-linux-gnu/libpthread.so.0
  6    Thread 0x7f8a7541e0 (LWP 23361) "led"             0x0000007f8cb519ec in select () from /lib/aarch64-linux-gnu/libc.so.6
  7    Thread 0x7f89f531e0 (LWP 23362) "netLedTask"      0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  8    Thread 0x7f894ce1e0 (LWP 23482) "HistoryCycle"    0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  9    Thread 0x7f88ccd1e0 (LWP 23486) "get cycle info"  0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  10   Thread 0x7f73ffe1e0 (LWP 23487) "proto_forward"   0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  11   Thread 0x7f737fd1e0 (LWP 23488) "UAServerthread"  0x0000007f8cb519ec in select () from /lib/aarch64-linux-gnu/libc.so.6
  12   Thread 0x7f72ffc1e0 (LWP 23489) "RS485_1"         0x0000007f8d18cc08 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
  13   Thread 0x7f727fb1e0 (LWP 23490) "RS485_2"         0x0000007f8d18cc08 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
  14   Thread 0x7f71ffa1e0 (LWP 23491) "127.0.0.1:1502"  0x0000007f8d18cc08 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
  15   Thread 0x7f717f91e0 (LWP 23492) "opc.tcp...:4841" 0x0000007f8d18cc08 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
  16   Thread 0x7f70ff81e0 (LWP 23493) "opc.tcp...:4841" 0x0000007f8d18cc08 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
  17   Thread 0x7f5fffe1e0 (LWP 23494) "opc.tcp...:4841" 0x0000007f8d18cc08 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
* 18   Thread 0x7f5f7fd1e0 (LWP 23495) "EMS"             0x0000007f8d18cc08 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
  19   Thread 0x7f5effc1e0 (LWP 23496) "modbus tcp ser"  0x0000007f8cb519ec in select () from /lib/aarch64-linux-gnu/libc.so.6
  20   Thread 0x7f5e7fb1e0 (LWP 23497) "Iec104Task"      0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  21   Thread 0x7f5dffa1e0 (LWP 23498) "proto_forward"   0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  22   Thread 0x7f5d7f91e0 (LWP 23499) "proto_forward"   0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  23   Thread 0x7f5cff81e0 (LWP 23500) "proto_forward"   0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  24   Thread 0x7f47ffe1e0 (LWP 23501) "Iec104Task"      0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  25   Thread 0x7f477fd1e0 (LWP 23502) "proto_forward"   0x0000007f8cb51b08 in pselect () from /lib/aarch64-linux-gnu/libc.so.6
  26   Thread 0x7f46ffc1e0 (LWP 23503) "proto_forward"   0x0000007f8d189c3c in pthread_cond_timedwait@@GLIBC_2.17 () from /lib/aarch64-linux-gnu/libpthread.so.0
  27   Thread 0x7f467fb1e0 (LWP 23504) "proto_forward"   0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  28   Thread 0x7f45ffa1e0 (LWP 23505) "proto_forward"   0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  29   Thread 0x7f457f91e0 (LWP 23506) "proto_forward"   0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  30   Thread 0x7f1affc1e0 (LWP 23519) "stratagyPeriod"  0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  31   Thread 0x7f1a7fb1e0 (LWP 23520) "loop_1s_task"    0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  32   Thread 0x7f19ffa1e0 (LWP 23521) "demandStart"     0x0000007f8cb58830 in epoll_pwait () from /lib/aarch64-linux-gnu/libc.so.6
  33   Thread 0x7f197f91e0 (LWP 23522) "lc_thread"       0x0000007f8cb2b5dc in nanosleep () from /lib/aarch64-linux-gnu/libc.so.6
  34   Thread 0x7f1b7fd1e0 (LWP 6509) "opcuaW"           0x0000007f8d18cc08 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
  35   Thread 0x7f44ff81e0 (LWP 23373) "opcuaW"          0x0000007f8d18cc3c in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
```

发现有 8 个线程在同一个地址 0x0000007f8d18cc08 处等待 mutex. 切换到任一等待 mutex 的线程，查看调用栈：

```
(gdb) bt
#0  0x0000007f8d18cc08 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
#1  0x0000007f8d185e20 in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
#2  0x0000007f8cc472cc in ?? () from /opt/lnxall_app/lib/libsqlite3.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
```

是 SQLite 在加锁。在 GDB 里执行 `info files` 找到 libsqlite3.so.0 代码段的基地址（下例中为 0x0000007f8cc095e0），加载带调试符号的 libsqlite3.so.0 之后再次查看栈：

```
(gdb) add-symbol-file /home/libsqlite3.so.0.8.6 0x0000007f8cc095e0
(gdb) bt
#0  0x0000007f8d18cc08 in __lll_lock_wait () from /lib/aarch64-linux-gnu/libpthread.so.0
#1  0x0000007f8d185e20 in pthread_mutex_lock () from /lib/aarch64-linux-gnu/libpthread.so.0
#2  0x0000007f8cc472cc in vdbeUnbind (p=0x25d39578, i=1) at sqlite3.c:83849
#3  0x0000007f8cc47464 in bindText (pStmt=0x25d39578, i=1, zData=0x7f1b7fa888, nData=-1, xDel=0x0, encoding=1 '\001') at sqlite3.c:83899
#4  0x0000000000453148 in updata_persistence_data (db=0x258226b8, tag=tag@entry=0x7f50152c80) at ./opcua/ua-server.c:520
#5  0x0000000000453798 in save_history_data (ua_ser_para=0x25817070, tag=0x7f50152c80) at ./opcua/ua-server.c:927
#6  0x000000000041dd64 in reflush_data_up (tag=tag@entry=0x7f50152c80) at proto_forward.c:422
#7  0x00000000004321fc in opcua_write_thread (args=0x261e9ca0) at port_transf.c:3241
#8  0x0000007f8d1837e4 in start_thread () from /lib/aarch64-linux-gnu/libpthread.so.0
#9  0x0000007f8cb5870c in ?? () from /lib/aarch64-linux-gnu/libc.so.6
(gdb) frame 2
#2  0x0000007f8cc472cc in vdbeUnbind (p=0x25d3ace8, i=1) at sqlite3.c:83849
83849   in sqlite3.c
(gdb) p *p->db->mutex
$6 = {mutex = {__data = {__lock = 2, __count = 1, __owner = 28078, __nusers = 1, __kind = 1, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
    __size = "\002\000\000\000\001\000\000\000\256m\000\000\001\000\000\000\001", '\000' <repeats 30 times>, __align = 4294967298}}
```

`__owner = 28078` 指示 sqlite3.c 第 83849 行的 mutex 被线程 28078 加锁，但找不到此线程。可能它对 mutex 加锁后异常退出了，所以其他所有线程都无法获得 mutex.

执行 `systemctl restart emsd.service` 重新启动 proto_forward 进程，发现有 36 个线程，而上面 gdb 显示有 35 个线程。对比后发现少了一个 "opcuaW" 线程，可能它就是 28078 线程，但因为缺少日志，无法断定。

解决思路：

- 检查 "opcuaw" 线程是否会在对 SQLite 内部的 mutex 加锁后异常退出，如果会，应修复。
- 为 mutex 加上 robust 属性，使持有 mutex 的线程异常退出后，其他线程仍有机会加锁。由于该 mutex 是 SQLite3 `sqlite3_stmt` 里的，所以需要修改 SQLite 源代码。风险是 **可能导致 SQLite 数据库里的数据不一致**。
    ```c
    int pthread_mutexattr_setrobust(const pthread_mutexattr_t *attr, PTHREAD_MUTEX_ROBUST);
    ```