真没招了搞了一天了解决不了，求助求助大佬来！！！EZ9999: [PID: 28402] 2025-08-24-16:31:00.300.443 Parse dynamic kernel config fail. Trace_MindIE_昇腾论坛

# 真没招了搞了一天了解决不了，求助求助大佬来！！！EZ9999: [PID: 28402] 2025-08-24-16:31:00.300.443 Parse dynamic kernel config fail. Trace_MindIE_昇腾论坛

## 概述

本文档基于昇腾社区论坛帖子生成的技术教程。

**原始链接**: https://www.hiascend.com/forum/thread-0297191407598053356-1-1.html

**生成时间**: 2025-08-27 10:33:56

---

## 问题描述

第一次部署模型折腾这么久来个大佬帮帮忙问AI解决不了感觉就是环境配置的问题但是不知道怎么排查了 image:swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:2.0.T3-800I-A2-py311-openeuler24.03-lts torch:2.1.0 torch-npu:2.1.0.post12 CANN:8.0.T63 一 [root@xn-c-npu-074 aarch64-linux]# cd $ATB_SPEED_HOME_PATH [root@xn-c-npu-074 atb-models]# python examples/run_pa.py --model_path /data/model/DeepSeek-R1-Distill-Qwen-7B --input_texts "你好" [2025-08-24 16:30:40,949] [28402] [281473860071712] [llm] [INFO] [cpu_binding.py-212] : rank_id: 0, device_id: 0, numa_id: 6, shard_devices: [0, 4], cpus: [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167] [2025-08-24 16:30:40,953] [28402] [281473860071712] [llm] [INFO] [cpu_binding.py-238] : process 28402, new_affinity is [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155], cpu count 12 [2025-08-24 16:30:42,003] [28402] [281473860071712] [llm] [INFO] [logging.py-331] : model_runner.quantize: None, model_runner.kv_quant_type: None, ...

## 完整内容

第一次部署模型折腾这么久来个大佬帮帮忙问AI解决不了感觉就是环境配置的问题但是不知道怎么排查了 image:swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:2.0.T3-800I-A2-py311-openeuler24.03-lts torch:2.1.0 torch-npu:2.1.0.post12 CANN:8.0.T63 一 [root@xn-c-npu-074 aarch64-linux]# cd $ATB_SPEED_HOME_PATH [root@xn-c-npu-074 atb-models]# python examples/run_pa.py --model_path /data/model/DeepSeek-R1-Distill-Qwen-7B --input_texts "你好" [2025-08-24 16:30:40,949] [28402] [281473860071712] [llm] [INFO] [cpu_binding.py-212] : rank_id: 0, device_id: 0, numa_id: 6, shard_devices: [0, 4], cpus: [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167] [2025-08-24 16:30:40,953] [28402] [281473860071712] [llm] [INFO] [cpu_binding.py-238] : process 28402, new_affinity is [144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155], cpu count 12 [2025-08-24 16:30:42,003] [28402] [281473860071712] [llm] [INFO] [logging.py-331] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.bfloat16 [2025-08-24 16:30:48,124] [28402] [281473860071712] [llm] [INFO] [dist.py-81] : initialize_distributed has been Set [2025-08-24 16:30:48,140] [28402] [281473860071712] [llm] [INFO] [logging.py-331] : init tokenizer done: LlamaTokenizerFast(name_or_path='/data/model/DeepSeek-R1-Distill-Qwen-7B', vocab_size=151643, model_max_length=16384, is_fast=True, padding_side='left', truncation_side='right', special_tokens={'bos_token': '<beginofsentence>', 'eos_token': '<endofsentence>', 'pad_token': '<endofsentence>'}, clean_up_tokenization_spaces=False), added_tokens_decoder={ 151643: AddedToken("<endofsentence>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 151644: AddedToken("<User>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151645: AddedToken("<Assistant>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151646: AddedToken("<beginofsentence>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 151647: AddedToken("<|EOT|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151648: AddedToken("<think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151649: AddedToken("</think>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True), 151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), 151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False), } [2025-08-24 16:30:48,159] [28402] [281473860071712] [llm] [INFO] [logging.py-331] : NPUSocInfo(soc_name='', soc_version=104, need_nz=True, matmul_nd_nz=False) [2025-08-24 16:30:48,614] [28402] [281473860071712] [llm] [INFO] [flash_causal_qwen2.py-137] : >>>> qwen_QwenDecoderModel is called. [2025-08-24 16:30:58,451] [28402] [281473860071712] [llm] [INFO] [logging.py-331] : model: FlashQwen2ForCausalLM( (rotary_embedding): PositionRotaryEmbedding() (attn_mask): AttentionMask() (transformer): FlashQwenModel( (wte): TensorParallelEmbedding() (h): ModuleList( (0-27): 28 x FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): FastLinear() ) (c_proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): FastLinear() ) (c_proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (ln_1): QwenRMSNorm() (ln_2): QwenRMSNorm() ) ) (ln_f): QwenRMSNorm() ) (lm_head): TensorParallelHead( (linear): FastLinear() ) ) [2025-08-24 16:31:00,237] [28402] [281473860071712] [llm] [INFO] [logging.py-331] : hbm_capacity(GB): 14.7822265625, init_memory(GB): 1.4782226560637355 [2025-08-24 16:31:00,237] [28402] [281473860071712] [llm] [INFO] [logging.py-331] : pa_runner: PARunner(model_path=/data/model/DeepSeek-R1-Distill-Qwen-7B, input_text=None, max_position_embeddings=None, max_input_length=1024, max_output_length=20, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=None, max_batch_size=1, dtype=torch.bfloat16, block_size=128, model_config=ModelConfig(num_heads=28, num_kv_heads=4, num_kv_heads_origin=4, head_size=128, k_head_size=128, v_head_size=128, num_layers=28, device=npu:0, dtype=torch.bfloat16, soc_info=NPUSocInfo(soc_name='', soc_version=104, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=1, rank=0, pp_rank=0, pp_groups=[[0]], micro_batch_size=1) , cla_share_factor=1, model_type=qwen2), max_memory=15872294912, [2025-08-24 16:31:00,237] [28402] [281473860071712] [llm] [INFO] [logging.py-331] : ---------------begin warm_up--------------- [2025-08-24 16:31:00,237] [28402] [281473860071712] [llm] [INFO] [cache.py-102] : kv cache will allocate 0.0615234375GB memory [2025-08-24 16:31:00,240] [28402] [281473860071712] [llm] [INFO] [logging.py-331] : ------total req num: 1, infer start-------- Traceback (most recent call last): File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 511, in <module> pa_runner.warm_up() File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 241, in warm_up generate_req(req_list, self.model, self.max_batch_size, self.max_prefill_tokens, self.cache_manager) File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 438, in generate_req req_finished, prefill_time = generate_token_with_clocking(model, cache_manager, batch)  File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 346, in generate_token_with_clocking req_finished = generate_token(model, cache_manager, input_batch)  File "/usr/local/Ascend/atb-models/examples/server/generate.py", line 278, in generate_token logits = model.forward(  File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 220, in forward return self.model.forward(**kwargs)  File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 460, in forward self.init_ascend_weight() File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 172, in init_ascend_weight weight_wrapper = self.get_weights()  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 154, in get_weights weight_wrapper = WeightWrapper(self.soc_info, self.tp_rank, attn_wrapper, mlp_wrapper)  File "/usr/local/Ascend/atb-models/atb_llm/utils/data/weight_wrapper.py", line 49, in __init__ self.placeholder = torch.zeros(1, dtype=torch.float16, device="npu")  RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error! EZ9999: [PID: 28402] 2025-08-24-16:31:00.300.443 Parse dynamic kernel config fail. TraceBack (most recent call last): AclOpKernelInit failed opType ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed. [ERROR] 2025-08-24-16:31:00 (PID:28402, Device:0, RankID:-1) ERR01100 OPS call acl api failed 二 [root@xn-c-npu-074 /]# find . -name *ZerosLike* ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_209e1e3c807895591394a370155f4fc6_high_performance.json ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_1c4543fdfe7149b6b0717bd88f061562_high_performance.o ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_1d2fd313763487259859439d7e9b41cf_high_performance.o ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_5c5e58e043224cccf51a06108010ae13_high_performance.o ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_d2db1a80c523e7e59a032c95969880af_high_performance.o ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_c5fb21ce50d8471d9b8df5001b9dc9a5_high_performance.json ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_2060cf8d69781f9a7462e2f67555909f_high_performance.o ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_8098b79498645e9c0da095babdeaf862_high_performance.o ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_5c5e58e043224cccf51a06108010ae13_high_performance.json ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_1c4543fdfe7149b6b0717bd88f061562_high_performance.json ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_d2db1a80c523e7e59a032c95969880af_high_performance.json ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_c5fb21ce50d8471d9b8df5001b9dc9a5_high_performance.o ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_8098b79498645e9c0da095babdeaf862_high_performance.json ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_2060cf8d69781f9a7462e2f67555909f_high_performance.json ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_1d2fd313763487259859439d7e9b41cf_high_performance.json ./usr/local/Ascend/ascend-toolkit/8.0.T63/opp/built-in/op_impl/ai_core/tbe/kernel/ascend910b/zeros_like/ZerosLike_209e1e3c807895591394a370155f4fc6_high_performance.o 用社区最新的2.1.RC1-800I-A3-py311-openeuler24.03-lts镜像呢 目前在尝试不同的镜像一般来说拉镜像应该是没什么环境的问题还在测试 不行试了四个镜像都是一样的结果 REPOSITORY TAG IMAGE ID CREATED SIZE swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie 2.1.RC1-800I-A3-py311-openeuler24.03-lts 7bce3ed3b1b6 11 days ago 16.1GB swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie 2.0.RC2-800I-A3-py311-openeuler24.03-lts afe2150b49df 3 months ago 14.2GB swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie 2.0.RC2-800I-A2-py311-openeuler24.03-lts 200b267906ed 3 months ago 14.3GB swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie 2.0.T3-800I-A2-py311-openeuler24.03-lts 25ba5f455ae3 6 months ago 13.9GB swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie 1.0.0-800I-A2-py311-openeuler24.03-lts 7e1003752fde 7 months ago 13.1GB 都是一样的报错 Traceback (most recent call last): File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 545, in <module> pa_runner = PARunner(**input_dict)  File "/usr/local/Ascend/atb-models/examples/run_pa.py", line 106, in __init__ self.model.load_weights(**kw_args) File "/usr/local/Ascend/atb-models/atb_llm/runner/model_runner.py", line 203, in load_weights self.model = self.model_cls(self.config,  File "/usr/local/Ascend/atb-models/atb_llm/models/qwen2/flash_causal_qwen2.py", line 29, in __init__ super().__init__(config, weights, **kwargs) File "/usr/local/Ascend/atb-models/atb_llm/models/base/flash_causal_lm.py", line 109, in __init__ self.placeholder = torch.zeros(1, dtype=self.dtype, device="npu")  RuntimeError: call aclnnInplaceZero failed, detail:EZ9999: Inner Error! EZ9999: [PID: 108] 2025-08-25-16:46:25.793.710 Parse dynamic kernel config fail. TraceBack (most recent call last): AclOpKernelInit failed opType ZerosLike ADD_TO_LAUNCHER_LIST_AICORE failed. [ERROR] 2025-08-25-16:46:25 (PID:108, Device:0, RankID:-1) ERR01100 OPS call acl api failed 执行npu-smi info可以看见八卡910b但是驱动是| npu-smi 23.0.rc2 Version: 23.0.rc2  我现在考虑是不是驱动的问题 这个是不是用错了镜像先看看镜像的版本以及型号是否一致

---

## 技术要点总结

基于以上内容，主要技术要点包括：

1. **问题类型**: 错误处理
2. **涉及技术**: NPU, MindIE, 昇腾, CANN, AI
3. **解决方案**: 请参考完整内容中的解决方案

## 相关资源

- 昇腾社区: https://www.hiascend.com/
- 昇腾论坛: https://www.hiascend.com/forum/

---

*本文档由AI自动生成，仅供参考。如有疑问，请参考原始帖子。*