拉起mindIE服务（mindieservice_daemon）出错_MindIE_昇腾论坛

# 拉起mindIE服务（mindieservice_daemon）出错_MindIE_昇腾论坛

## 概述

本文档基于昇腾社区论坛帖子生成的技术教程。

**原始链接**: https://www.hiascend.com/forum/thread-0259190566298521254-1-1.html

**生成时间**: 2025-08-27 10:33:55

---

## 问题描述

host300i Duo 96G 单卡open欧拉 mindIE 镜像版本1.0.0-300I-Duo-py311-openeuler24.03-lts 模型deepseek 14b W8A8S 稀疏化 TP=2进行切分按照DeepSeek-R1-Distill-Qwen-14B-模型库-ModelZoo-昇腾社区这个流程 稀疏化后的模型测试通过 cd $ATB_SPEED_HOME_PATH/tests/modeltest/ #2048 输入2048输出batch=1 2个NPU bash run.sh pa_fp16 performance [[2048,2048]] 1 qwen /models/DeepSeek-R1-Distill-Qwen-14B_s-w8a8_2npu 2 batch: 1, seq_len_in: 2048, seq_len_out: 2048, total_time: 114.747720,first_token_time: 1037.010000, non_first_token_time: 55.500000, non_first_token_throughput: 18.018018, e2e_time: 114.747720, e2e_throughput: 17.847849复制 但是在拉起服务的时候却出错 cd /usr/local/Ascend/mindie/latest/mindie-service/bin ./mindieservice_daemon [2025-08-14 22:18:31,908] [119357] [281466830700896] [llm] [INFO] [logging.py-331] : trans to 29 [2025-08-14 22:18:32,413] [119359] [281467410760032] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16]) [2025-08-14 22:18:32,488] [119357] [281466830700896] [llm] [INFO] [logging....

## 相关代码

### 代码示例 1

```
cd $ATB_SPEED_HOME_PATH/tests/modeltest/
#2048 输入，2048输出，batch=1， 2个NPU
bash run.sh pa_fp16 performance [[2048,2048]] 1 qwen /models/DeepSeek-R1-Distill-Qwen-14B_s-w8a8_2npu 2
batch: 1, seq_len_in: 2048, seq_len_out: 2048, total_time: 114.747720,first_token_time: 1037.010000, non_first_token_time: 55.500000, non_first_token_throughput: 18.018018, e2e_time: 114.747720, e2e_throughput: 17.847849复制
```

### 代码示例 2

```
cd $ATB_SPEED_HOME_PATH/tests/modeltest/
#2048 输入，2048输出，batch=1， 2个NPU
bash run.sh pa_fp16 performance [[2048,2048]] 1 qwen /models/DeepSeek-R1-Distill-Qwen-14B_s-w8a8_2npu 2
batch: 1, seq_len_in: 2048, seq_len_out: 2048, total_time: 114.747720,first_token_time: 1037.010000, non_first_token_time: 55.500000, non_first_token_throughput: 18.018018, e2e_time: 114.747720, e2e_throughput: 17.847849
```

### 代码示例 3

```
但是，在拉起服务的时候却出错：
```

### 代码示例 4

```
cd /usr/local/Ascend/mindie/latest/mindie-service/bin
./mindieservice_daemon

[2025-08-14 22:18:31,908] [119357] [281466830700896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-08-14 22:18:32,413] [119359] [281467410760032] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-08-14 22:18:32,488] [119357] [281466830700896] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-08-14 22:18:32,498] [119357] [281466830700896] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-08-14 22:18:32,499] [119357] [281466830700896] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281473060565648 id of vcache is 281473060565456
2025-08-14 22:18:40,412 [ERROR] model.py:39 - [Model]   >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-08-14-22:18:40 (PID:119357, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 119357] 2025-08-14-22:18:40.204.714 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=0, stream_id=6, report_stream_id=6, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

DEVICE[0] PID[119357]:
EXCEPTION STREAM:
Exception info:TGID=1804624, model id=65535, stream id=6, stream phase=3
Message info[0]:RTS_HWTS: Aicore exception, slot_id=21, stream_id=6
Other info[0]:time=2025-08-14-22:18:40.099.439, function=process_hwts_error_exception, line=646, error code=0x26
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up
npu.synchronize()
File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize
return torch_npu._C._npu_synchronize()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-08-14-22:18:40 (PID:119357, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.复制
```

### 代码示例 5

```
cd /usr/local/Ascend/mindie/latest/mindie-service/bin
./mindieservice_daemon

[2025-08-14 22:18:31,908] [119357] [281466830700896] [llm] [INFO] [logging.py-331] : trans to 29
[2025-08-14 22:18:32,413] [119359] [281467410760032] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-08-14 22:18:32,488] [119357] [281466830700896] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-08-14 22:18:32,498] [119357] [281466830700896] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16])
[2025-08-14 22:18:32,499] [119357] [281466830700896] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281473060565648 id of vcache is 281473060565456
2025-08-14 22:18:40,412 [ERROR] model.py:39 - [Model]   >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-08-14-22:18:40 (PID:119357, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
EZ9999: Inner Error!
EZ9999: [PID: 119357] 2025-08-14-22:18:40.204.714 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356]
TraceBack (most recent call last):
Aicore kernel execute failed, device_id=0, stream_id=6, report_stream_id=6, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124]
[AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124]
rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53]
wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161]

DEVICE[0] PID[119357]:
EXCEPTION STREAM:
Exception info:TGID=1804624, model id=65535, stream id=6, stream phase=3
Message info[0]:RTS_HWTS: Aicore exception, slot_id=21, stream_id=6
Other info[0]:time=2025-08-14-22:18:40.099.439, function=process_hwts_error_exception, line=646, error code=0x26
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize
return self.python_model.initialize(config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize
self.generator = Generator(
^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__
self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode)
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up
npu.synchronize()
File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize
return torch_npu._C._npu_synchronize()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015
[ERROR] 2025-08-14-22:18:40 (PID:119357, Device:0, RankID:-1) ERR00100 PTA call acl api failed
[Error]: The aicore execution is abnormal.
Rectify the fault based on the error information in the ascend log.
```

## 完整内容

host300i Duo 96G 单卡open欧拉 mindIE 镜像版本1.0.0-300I-Duo-py311-openeuler24.03-lts 模型deepseek 14b W8A8S 稀疏化 TP=2进行切分按照DeepSeek-R1-Distill-Qwen-14B-模型库-ModelZoo-昇腾社区这个流程 稀疏化后的模型测试通过 cd $ATB_SPEED_HOME_PATH/tests/modeltest/ #2048 输入2048输出batch=1 2个NPU bash run.sh pa_fp16 performance [[2048,2048]] 1 qwen /models/DeepSeek-R1-Distill-Qwen-14B_s-w8a8_2npu 2 batch: 1, seq_len_in: 2048, seq_len_out: 2048, total_time: 114.747720,first_token_time: 1037.010000, non_first_token_time: 55.500000, non_first_token_throughput: 18.018018, e2e_time: 114.747720, e2e_throughput: 17.847849复制 但是在拉起服务的时候却出错 cd /usr/local/Ascend/mindie/latest/mindie-service/bin ./mindieservice_daemon [2025-08-14 22:18:31,908] [119357] [281466830700896] [llm] [INFO] [logging.py-331] : trans to 29 [2025-08-14 22:18:32,413] [119359] [281467410760032] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16]) [2025-08-14 22:18:32,488] [119357] [281466830700896] [llm] [INFO] [logging.py-331] : <<<<<<< ori k_caches[0].shape=torch.Size([1, 32, 128, 16]) [2025-08-14 22:18:32,498] [119357] [281466830700896] [llm] [INFO] [flash_causal_qwen2.py-394] : <<<<<<<after transdata k_caches[0].shape=torch.Size([1, 32, 128, 16]) [2025-08-14 22:18:32,499] [119357] [281466830700896] [llm] [INFO] [logging.py-331] : >>>>>>id of kcache is 281473060565648 id of vcache is 281473060565456 2025-08-14 22:18:40,412 [ERROR] model.py:39 - [Model] >>> Exception:npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 [ERROR] 2025-08-14-22:18:40 (PID:119357, Device:0, RankID:-1) ERR00100 PTA call acl api failed [Error]: The aicore execution is abnormal. Rectify the fault based on the error information in the ascend log. EZ9999: Inner Error! EZ9999: [PID: 119357] 2025-08-14-22:18:40.204.714 Kernel task happen error, retCode=0x26, [aicore exception].[FUNC:PreCheckTaskErr][FILE:davinci_kernel_task.cc][LINE:1356] TraceBack (most recent call last): Aicore kernel execute failed, device_id=0, stream_id=6, report_stream_id=6, task_id=2525, flip_num=0, fault kernel_name=pp_matmul_int8_nz_compress_0, fault kernel info ext=pp_matmul_int8_nz_compress_0, program id=87, hash=3087916139699476489.[FUNC:GetError][FILE:stream.cc][LINE:1124] [AIC_INFO] after execute:args print end[FUNC:GetError][FILE:stream.cc][LINE:1124] rtDeviceSynchronizeWithTimeout execute failed, reason=[aicore exception][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:53] wait for compute device to finish failed, runtime result = 507015.[FUNC:ReportCallError][FILE:log_inner.cpp][LINE:161] DEVICE[0] PID[119357]: EXCEPTION STREAM: Exception info:TGID=1804624, model id=65535, stream id=6, stream phase=3 Message info[0]:RTS_HWTS: Aicore exception, slot_id=21, stream_id=6 Other info[0]:time=2025-08-14-22:18:40.099.439, function=process_hwts_error_exception, line=646, error code=0x26 Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/model_wrapper/model.py", line 37, in initialize return self.python_model.initialize(config)  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 146, in initialize self.generator = Generator(  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 119, in __init__ self.warm_up(max_prefill_tokens, max_seq_len, max_input_len, max_iter_times, inference_mode) File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 305, in warm_up npu.synchronize() File "/usr/local/lib64/python3.11/site-packages/torch_npu/npu/utils.py", line 35, in synchronize return torch_npu._C._npu_synchronize()  RuntimeError: npuSynchronizeDevice:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:467 NPU function error: AclrtSynchronizeDeviceWithTimeout, error code is 507015 [ERROR] 2025-08-14-22:18:40 (PID:119357, Device:0, RankID:-1) ERR00100 PTA call acl api failed [Error]: The aicore execution is abnormal. Rectify the fault based on the error information in the ascend log.复制 看了半天毫无头绪哪位老大给点建议

---

## 技术要点总结

基于以上内容，主要技术要点包括：

1. **问题类型**: 错误处理
2. **涉及技术**: 昇腾, NPU, AI, MindIE
3. **解决方案**: 请参考完整内容中的解决方案

## 相关资源

- 昇腾社区: https://www.hiascend.com/
- 昇腾论坛: https://www.hiascend.com/forum/

---

*本文档由AI自动生成，仅供参考。如有疑问，请参考原始帖子。*