MindIE 2.1RC1部署qwen2.5-vl-7B-Instruct并发推理20张以上图片报错_MindIE_昇腾论坛

# MindIE 2.1RC1部署qwen2.5-vl-7B-Instruct并发推理20张以上图片报错_MindIE_昇腾论坛

## 概述

本文档基于昇腾社区论坛帖子生成的技术教程。

**原始链接**: https://www.hiascend.com/forum/thread-0297191150546798347-1-1.html

**生成时间**: 2025-08-27 10:33:55

---

## 问题描述

镜像版本2.1.RC1-300I-Duo-py311-openeuler24.03-lts 使用一张Atlas 300I Duo部署qwen2.5-vl-7B-Instruct并发推理20张以上图片有时会报错 设置ASCEND_LAUNCH_BLOCKING=1启用同步调试报错如下 [2025-08-21 17:01:32.927048] [debug] [546] [kernel_base.cpp:324] kernel param: 8 in/out, 0 const in, 0 workspaces, 0 hwsync [2025-08-21 17:01:32.927056] [error] [546] [kernel_base.cpp:37] kernel info args is nullptr, please check first error before. [2025-08-21 17:01:32.927062] [error] [546] [kernel_base.cpp:332] build args failed, abort running [2025-08-21 17:01:32.927067] [error] [546] [mki_node_implement.cpp:232] SelfAttentionEncoderFusionOpsRunner910A_33_4[3] UnpadFlashAttentionNzEncoderKernel run fail, launchParam :specificParam: {"alibiLeftAlign":0,"batchRunStatus":[],"cacheType":0,"compressHead":false,"dataDimOrder":0,"headSize":8,"isAlibiMaskSqrt":0,"isTriuMask":0,"kvHead":8,"kvSeqLen":[64,64,64,......],"scaleType":0,"tor":0.11180339753627777,"type":2005,"windowSize":0} intensors[0]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, ...

## 相关代码

### 代码示例 1

```
[2025-08-21 17:01:32.927048] [debug] [546] [kernel_base.cpp:324] kernel param: 8 in/out, 0 const in, 0 workspaces, 0 hwsync
[2025-08-21 17:01:32.927056] [error] [546] [kernel_base.cpp:37] kernel info args is nullptr, please check first error before.
[2025-08-21 17:01:32.927062] [error] [546] [kernel_base.cpp:332] build args failed, abort running
[2025-08-21 17:01:32.927067] [error] [546] [mki_node_implement.cpp:232] SelfAttentionEncoderFusionOpsRunner910A_33_4[3] UnpadFlashAttentionNzEncoderKernel run fail, launchParam
:specificParam: {"alibiLeftAlign":0,"batchRunStatus":[],"cacheType":0,"compressHead":false,"dataDimOrder":0,"headSize":8,"isAlibiMaskSqrt":0,"isTriuMask":0,"kvHead":8,"kvSeqLen":[64,64,64,......],"scaleType":0,"tor":0.11180339753627777,"type":2005,"windowSize":0}
intensors[0]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1
intensors[1]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1
intensors[2]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1
intensors[3]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
intensors[4]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
intensors[5]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
intensors[6]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
outtensors[0]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1

st: code:1, msg:
[2025-08-21 17:01:32.927416] [error] [546] [ops_runner.cpp:655] RunKernel failed! ret:1
[2025-08-21 17:01:32.927423] [error] [546] [ops_runner.cpp:415] RunAllKernel failed! ret:1
[2025-08-21 17:01:32.927428] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927433] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927438] [error] [546] [runner.cpp:132] SelfAttentionEncoderFusionOpsRunner910A_33_4:4 Execute Failed. st: 1
[2025-08-21 17:01:32.927446] [error] [546] [graph_runner.cpp:976] Qwen25VL_VIT_layer_0_graphRunner_33:3  node[4] execute fail, runner name:SelfAttentionEncoderFusionOpsRunner910A
[2025-08-21 17:01:32.927454] [info] [546] [graph_runner.cpp:396] Qwen25VL_VIT_layer_0_graphRunner_33:3 execute fail
[2025-08-21 17:01:32.927460] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927464] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927467] [error] [546] [runner.cpp:132] Qwen25VL_VIT_layer_0_graphRunner_33:4 Execute Failed. st: 1
[2025-08-21 17:01:32.927473] [error] [546] [operation_base.cpp:1023] Qwen25VL_VIT_layer_0_graph_33 execute Qwen25VL_VIT_layer_0_graphRunner fail
[2025-08-21 17:01:32.927479] [info] [546] [operation_base.cpp:1033] Qwen25VL_VIT_layer_0_graph_33 execute Qwen25VL_VIT_layer_0_graphRunner success
[2025-08-21 17:01:32.927485] [info] [546] [operation_base.cpp:1041] Qwen25VL_VIT_layer_0_graph_33 execute statistic:totalTime:5428, streamSyncTime:0, tillingCopyTime:0, kernelExecuteTime:3325, launchTime:3763, preLaunchTime:1665
[2025-08-21 17:01:32.927495] [error] [546] [operation_base.cpp:1100] Qwen25VL_VIT_layer_0_graph_33 Launch fail, error code: 1
[2025-08-21 17:01:32.927] [264] [281456090214752] [llmmodels] [ERROR] [graph_operation.cpp:445] [MIE05E000002] Qwen25VL_VIT_graph nodes[1] atb operation execute fail, error:1
[2025-08-21 17:01:32,927] [264] [281456090214752] [llm] [ERROR] [logging.py-346] : Unknown exception: Atb operation setup fail
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 115
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 116
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 117
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 118
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 119
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 120
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 121
[2025-08-21 17:01:32.021+0800] [264] [281463433982304] [batchscheduler] [ERROR] [standard_model.py:197] : [Model]       >>> global rank-0 Execute type:1, Exception:Atb operation setup fail
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 185, in execute
return self._prefill(requests)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 430, in _prefill
return self.__generate(requests, is_prefill=True, is_mix=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 578, in __generate
generate_output = self.__handle_requests(requests, is_prefill, is_mix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 939, in __handle_requests
generate_output = self.generator.generate_token(metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 386, in generate_token
raise e
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 367, in generate_token
generation_output = self.plugin.generate_token(input_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 91, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/plugins/plugin_manager.py", line 115, in generate_token
result = self.generator_backend.forward(model_inputs, q_lens=self.plugin_data_param.q_len,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 69, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 230, in forward
logits = self._forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 621, in _forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 125, in forward
result = self.forward_from_model_inputs(model_inputs, npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 197, in forward_from_model_inputs
result = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 237, in forward_tensor
result = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 310, in forward
res = self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 118, in forward
inputs_embeds, image_grid_thw, video_grid_thw, second_per_grid_ts = self.prepare_prefill_token_service(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 197, in prepare_prefill_token_service
image_features = self.vision_tower(image_pixel.to(self.vision_tower.dtype), image_grid_thw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 872, in forward
vision_features = self.encoder(
^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 719, in forward
hidden_states = self.graph.forward(self.graph_inputs, self.graph_outputs, self.graph_param)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Atb operation setup fail复制
```

### 代码示例 2

```
[2025-08-21 17:01:32.927048] [debug] [546] [kernel_base.cpp:324] kernel param: 8 in/out, 0 const in, 0 workspaces, 0 hwsync
[2025-08-21 17:01:32.927056] [error] [546] [kernel_base.cpp:37] kernel info args is nullptr, please check first error before.
[2025-08-21 17:01:32.927062] [error] [546] [kernel_base.cpp:332] build args failed, abort running
[2025-08-21 17:01:32.927067] [error] [546] [mki_node_implement.cpp:232] SelfAttentionEncoderFusionOpsRunner910A_33_4[3] UnpadFlashAttentionNzEncoderKernel run fail, launchParam
:specificParam: {"alibiLeftAlign":0,"batchRunStatus":[],"cacheType":0,"compressHead":false,"dataDimOrder":0,"headSize":8,"isAlibiMaskSqrt":0,"isTriuMask":0,"kvHead":8,"kvSeqLen":[64,64,64,......],"scaleType":0,"tor":0.11180339753627777,"type":2005,"windowSize":0}
intensors[0]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1
intensors[1]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1
intensors[2]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1
intensors[3]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
intensors[4]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
intensors[5]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
intensors[6]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0
outtensors[0]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1

st: code:1, msg:
[2025-08-21 17:01:32.927416] [error] [546] [ops_runner.cpp:655] RunKernel failed! ret:1
[2025-08-21 17:01:32.927423] [error] [546] [ops_runner.cpp:415] RunAllKernel failed! ret:1
[2025-08-21 17:01:32.927428] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927433] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927438] [error] [546] [runner.cpp:132] SelfAttentionEncoderFusionOpsRunner910A_33_4:4 Execute Failed. st: 1
[2025-08-21 17:01:32.927446] [error] [546] [graph_runner.cpp:976] Qwen25VL_VIT_layer_0_graphRunner_33:3  node[4] execute fail, runner name:SelfAttentionEncoderFusionOpsRunner910A
[2025-08-21 17:01:32.927454] [info] [546] [graph_runner.cpp:396] Qwen25VL_VIT_layer_0_graphRunner_33:3 execute fail
[2025-08-21 17:01:32.927460] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927464] [debug] [546] [probe.cpp:47] executeCount:3
[2025-08-21 17:01:32.927467] [error] [546] [runner.cpp:132] Qwen25VL_VIT_layer_0_graphRunner_33:4 Execute Failed. st: 1
[2025-08-21 17:01:32.927473] [error] [546] [operation_base.cpp:1023] Qwen25VL_VIT_layer_0_graph_33 execute Qwen25VL_VIT_layer_0_graphRunner fail
[2025-08-21 17:01:32.927479] [info] [546] [operation_base.cpp:1033] Qwen25VL_VIT_layer_0_graph_33 execute Qwen25VL_VIT_layer_0_graphRunner success
[2025-08-21 17:01:32.927485] [info] [546] [operation_base.cpp:1041] Qwen25VL_VIT_layer_0_graph_33 execute statistic:totalTime:5428, streamSyncTime:0, tillingCopyTime:0, kernelExecuteTime:3325, launchTime:3763, preLaunchTime:1665
[2025-08-21 17:01:32.927495] [error] [546] [operation_base.cpp:1100] Qwen25VL_VIT_layer_0_graph_33 Launch fail, error code: 1
[2025-08-21 17:01:32.927] [264] [281456090214752] [llmmodels] [ERROR] [graph_operation.cpp:445] [MIE05E000002] Qwen25VL_VIT_graph nodes[1] atb operation execute fail, error:1
[2025-08-21 17:01:32,927] [264] [281456090214752] [llm] [ERROR] [logging.py-346] : Unknown exception: Atb operation setup fail
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 115
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 116
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 117
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 118
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 119
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 120
[2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 121
[2025-08-21 17:01:32.021+0800] [264] [281463433982304] [batchscheduler] [ERROR] [standard_model.py:197] : [Model]       >>> global rank-0 Execute type:1, Exception:Atb operation setup fail
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 185, in execute
return self._prefill(requests)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 430, in _prefill
return self.__generate(requests, is_prefill=True, is_mix=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 578, in __generate
generate_output = self.__handle_requests(requests, is_prefill, is_mix)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 939, in __handle_requests
generate_output = self.generator.generate_token(metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 386, in generate_token
raise e
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 367, in generate_token
generation_output = self.plugin.generate_token(input_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 91, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/plugins/plugin_manager.py", line 115, in generate_token
result = self.generator_backend.forward(model_inputs, q_lens=self.plugin_data_param.q_len,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 69, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 230, in forward
logits = self._forward(model_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 621, in _forward
logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 125, in forward
result = self.forward_from_model_inputs(model_inputs, npu_cache, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 197, in forward_from_model_inputs
result = self.forward_tensor(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 237, in forward_tensor
result = self.model_runner.forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 310, in forward
res = self.model.forward(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 118, in forward
inputs_embeds, image_grid_thw, video_grid_thw, second_per_grid_ts = self.prepare_prefill_token_service(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 197, in prepare_prefill_token_service
image_features = self.vision_tower(image_pixel.to(self.vision_tower.dtype), image_grid_thw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 872, in forward
vision_features = self.encoder(
^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 719, in forward
hidden_states = self.graph.forward(self.graph_inputs, self.graph_outputs, self.graph_param)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Atb operation setup fail
```

## 相关图片

### 图片 1

![cke_16039.png](https://fileserver.developer.huaweicloud.com/FileServer/getFile/cmtybbs/b34/646/b75/2e183e775bb34646b755e6b32ff02e8b.20250821092208.45838651485469360210619376598205:20250827013301:2400:05BC45CB3D36FDB62432F0FC7FBAE4E69311EB834F5C9D4C9DE3BFF55E9D5E11.png)

**图片地址**: https://fileserver.developer.huaweicloud.com/FileServer/getFile/cmtybbs/b34/646/b75/2e183e775bb34646b755e6b32ff02e8b.20250821092208.45838651485469360210619376598205:20250827013301:2400:05BC45CB3D36FDB62432F0FC7FBAE4E69311EB834F5C9D4C9DE3BFF55E9D5E11.png

**图片描述**: cke_16039.png

## 完整内容

镜像版本2.1.RC1-300I-Duo-py311-openeuler24.03-lts 使用一张Atlas 300I Duo部署qwen2.5-vl-7B-Instruct并发推理20张以上图片有时会报错 设置ASCEND_LAUNCH_BLOCKING=1启用同步调试报错如下 [2025-08-21 17:01:32.927048] [debug] [546] [kernel_base.cpp:324] kernel param: 8 in/out, 0 const in, 0 workspaces, 0 hwsync [2025-08-21 17:01:32.927056] [error] [546] [kernel_base.cpp:37] kernel info args is nullptr, please check first error before. [2025-08-21 17:01:32.927062] [error] [546] [kernel_base.cpp:332] build args failed, abort running [2025-08-21 17:01:32.927067] [error] [546] [mki_node_implement.cpp:232] SelfAttentionEncoderFusionOpsRunner910A_33_4[3] UnpadFlashAttentionNzEncoderKernel run fail, launchParam :specificParam: {"alibiLeftAlign":0,"batchRunStatus":[],"cacheType":0,"compressHead":false,"dataDimOrder":0,"headSize":8,"isAlibiMaskSqrt":0,"isTriuMask":0,"kvHead":8,"kvSeqLen":[64,64,64,......],"scaleType":0,"tor":0.11180339753627777,"type":2005,"windowSize":0} intensors[0]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1 intensors[1]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1 intensors[2]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1 intensors[3]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0 intensors[4]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0 intensors[5]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0 intensors[6]: dtype:undefined, format:undefined, dims:[], strides:[], offset:0, dataSize:0, data:0 outtensors[0]: dtype:float16, format:fractal_nz, dims:[1, 64, 127024, 16], strides:[], offset:0, dataSize:260145152, data:1 st: code:1, msg: [2025-08-21 17:01:32.927416] [error] [546] [ops_runner.cpp:655] RunKernel failed! ret:1 [2025-08-21 17:01:32.927423] [error] [546] [ops_runner.cpp:415] RunAllKernel failed! ret:1 [2025-08-21 17:01:32.927428] [debug] [546] [probe.cpp:47] executeCount:3 [2025-08-21 17:01:32.927433] [debug] [546] [probe.cpp:47] executeCount:3 [2025-08-21 17:01:32.927438] [error] [546] [runner.cpp:132] SelfAttentionEncoderFusionOpsRunner910A_33_4:4 Execute Failed. st: 1 [2025-08-21 17:01:32.927446] [error] [546] [graph_runner.cpp:976] Qwen25VL_VIT_layer_0_graphRunner_33:3 node[4] execute fail, runner name:SelfAttentionEncoderFusionOpsRunner910A [2025-08-21 17:01:32.927454] [info] [546] [graph_runner.cpp:396] Qwen25VL_VIT_layer_0_graphRunner_33:3 execute fail [2025-08-21 17:01:32.927460] [debug] [546] [probe.cpp:47] executeCount:3 [2025-08-21 17:01:32.927464] [debug] [546] [probe.cpp:47] executeCount:3 [2025-08-21 17:01:32.927467] [error] [546] [runner.cpp:132] Qwen25VL_VIT_layer_0_graphRunner_33:4 Execute Failed. st: 1 [2025-08-21 17:01:32.927473] [error] [546] [operation_base.cpp:1023] Qwen25VL_VIT_layer_0_graph_33 execute Qwen25VL_VIT_layer_0_graphRunner fail [2025-08-21 17:01:32.927479] [info] [546] [operation_base.cpp:1033] Qwen25VL_VIT_layer_0_graph_33 execute Qwen25VL_VIT_layer_0_graphRunner success [2025-08-21 17:01:32.927485] [info] [546] [operation_base.cpp:1041] Qwen25VL_VIT_layer_0_graph_33 execute statistic:totalTime:5428, streamSyncTime:0, tillingCopyTime:0, kernelExecuteTime:3325, launchTime:3763, preLaunchTime:1665 [2025-08-21 17:01:32.927495] [error] [546] [operation_base.cpp:1100] Qwen25VL_VIT_layer_0_graph_33 Launch fail, error code: 1 [2025-08-21 17:01:32.927] [264] [281456090214752] [llmmodels] [ERROR] [graph_operation.cpp:445] [MIE05E000002] Qwen25VL_VIT_graph nodes[1] atb operation execute fail, error:1 [2025-08-21 17:01:32,927] [264] [281456090214752] [llm] [ERROR] [logging.py-346] : Unknown exception: Atb operation setup fail [2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 115 [2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 116 [2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 117 [2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 118 [2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 119 [2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 120 [2025-08-21 17:01:32.928] [264] [281456090214752] [llm] [DEBUG] [post_processing_manager.cpp:293] Delete conf for 121 [2025-08-21 17:01:32.021+0800] [264] [281463433982304] [batchscheduler] [ERROR] [standard_model.py:197] : [Model] >>> global rank-0 Execute type:1, Exception:Atb operation setup fail Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 185, in execute return self._prefill(requests)  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 430, in _prefill return self.__generate(requests, is_prefill=True, is_mix=False)  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 578, in __generate generate_output = self.__handle_requests(requests, is_prefill, is_mix)  File "/usr/local/lib/python3.11/site-packages/model_wrapper/standard_model.py", line 939, in __handle_requests generate_output = self.generator.generate_token(metadata)  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 386, in generate_token raise e File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/generator.py", line 367, in generate_token generation_output = self.plugin.generate_token(input_metadata)  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 91, in wrapper return func(*args, **kwargs)  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/plugins/plugin_manager.py", line 115, in generate_token result = self.generator_backend.forward(model_inputs, q_lens=self.plugin_data_param.q_len,  File "/usr/local/lib/python3.11/site-packages/mindie_llm/utils/decorators/time_decorator.py", line 69, in wrapper return func(*args, **kwargs)  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 230, in forward logits = self._forward(model_inputs, **kwargs)  File "/usr/local/lib/python3.11/site-packages/mindie_llm/text_generator/adapter/generator_torch.py", line 621, in _forward logits = self.model_wrapper.forward(model_inputs, self.cache_pool.npu_cache, **kwargs)  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 125, in forward result = self.forward_from_model_inputs(model_inputs, npu_cache, **kwargs)  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 197, in forward_from_model_inputs result = self.forward_tensor(  File "/usr/local/lib/python3.11/site-packages/mindie_llm/modeling/model_wrapper/atb/atb_model_wrapper.py", line 237, in forward_tensor result = self.model_runner.forward(  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 310, in forward res = self.model.forward(**kwargs)  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 118, in forward inputs_embeds, image_grid_thw, video_grid_thw, second_per_grid_ts = self.prepare_prefill_token_service(  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/flash_causal_qwen2_vl.py", line 197, in prepare_prefill_token_service image_features = self.vision_tower(image_pixel.to(self.vision_tower.dtype), image_grid_thw)  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs)  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs)  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 872, in forward vision_features = self.encoder(  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl return self._call_impl(*args, **kwargs)  File "/usr/local/lib64/python3.11/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, **kwargs)  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2_vl/modeling_qwen2_5_vl_vit_atb.py", line 719, in forward hidden_states = self.graph.forward(self.graph_inputs, self.graph_outputs, self.graph_param)  RuntimeError: Atb operation setup fail复制 你是怎么配置的下面这个链接里的你能用吗 https://www.hiascend.com/developer/ascendhub/detail/9eedc82e0c0644b2a2a9d0821ed5e7ad

---

## 技术要点总结

基于以上内容，主要技术要点包括：

1. **问题类型**: 错误处理
2. **涉及技术**: HTTPS, NPU, Atlas, MindIE, 昇腾, AI
3. **解决方案**: 请参考完整内容中的解决方案

## 相关资源

- 昇腾社区: https://www.hiascend.com/
- 昇腾论坛: https://www.hiascend.com/forum/

---

*本文档由AI自动生成，仅供参考。如有疑问，请参考原始帖子。*