基于ERNIE Bot自定义虚拟数字人生成

## 一、项目设计

### 1.1 项目简介

随着科技的不断发展，虚拟数字人已经成为了一个备受关注的话题。虚拟数字人是指通过数字技术创建出来的、具有人类特征的虚拟形象。它们可以被用于各种领域，如娱乐、教 育、医疗、广告等。在这个项目中，我们将使用ERNIE Bot来创建自定义虚拟数字人，并探究其在特定领域的应用。

### 1.2 项目目的

本项目的目的是制作自定义虚拟数字人，并探究其在特定领域的应用。具体来说，我们将通过以下方式实现这一目标：

1、使用ERNIE Bot创建虚拟数字人，并对它们进行训练，使其具有更高的智能水平；

2、对虚拟数字人进行定制化设计，使其具有更加逼真的外观和声音；

3、探究虚拟数字人在特定领域的应用，如教育、医疗、娱乐等；

4、分析用户反馈，不断优化虚拟数字人的设计和应用。

### 1.3 技术路线

实现虚拟数字人生成总共需要调用文心一言API和三个模型，分别是**ENIRE Bot（文心一言）、First Order Motion（表情迁移）、Text to Speech（文本转语音）和Wav2Lip（唇形合成）**。

具体技术步骤如下：

1、使用enirebot定义好虚拟人物，需要提供虚拟人物的名字、背景、聊天对话就可以生成一个全新的数字分身

2、把图像放入First Order Motion模型进行面部表情迁移，让虚拟人物的表情更加逼近真人

3、通过Text to Speech模型，将输入的文字转换成音频输出。

4、得到面部表情迁移的视频和音频之后，通过Wav2Lip模型，将音频和视频合并，并根据音频内容调整唇形，使得虚拟人更加接近真人效果。

![img](http://yg9538.kmgy.top/202311191453405.png)

### 1.4 成果演示

合成效果（视频中有三个虚拟数字人分别是（妇好、华佗、诸葛））

生成一个虚拟数字人的流程视频

## 二、安装依赖

In [ ]

```
#安装文心一言api
!pip install erniebot
```

In [ ]

```
# 升级PaddleHub
!pip install --upgrade paddlehub
```

In [2]

```
#用于表情迁移
!hub install first_order_motion==1.0.0
Download https://bj.bcebos.com/paddlehub/paddlehub_dev/first_order_motion.tar.gz
[##################################################] 100.00%
Decompress /home/aistudio/.paddlehub/tmp/tmp25b_3oxf/first_order_motion.tar.gz
[##################################################] 100.00%
[2023-10-22 13:03:02,849] [    INFO] - Successfully installed dependent packages.dio/.paddlehub/tmp/tmpan0x5mr2/first_order_motion/requirements.txt: \
[2023-10-22 13:03:03,266] [    INFO] - Successfully installed first_order_motion-1.0.0
```

In [ ]

```
!wget -P data https://paddlespeech.bj.bcebos.com/Parakeet/tools/nltk_data.tar.gz
!tar zxvf data/nltk_data.tar.gz
```

In [ ]

```
# 安装PaddleSpeech
!pip install pytest-runner
!pip install paddlespeech
```

In [9]

```
#用于唇形合成
!hub install wav2lip
Download https://bj.bcebos.com/paddlehub/paddlehub_dev/wav2lip.tar.gz
[##################################################] 100.00%
Decompress /home/aistudio/.paddlehub/tmp/tmpwoyeq7yz/wav2lip.tar.gz
[##################################################] 100.00%
[2023-10-22 13:15:59,439] [    INFO] - Successfully installed dependent packages.dio/.paddlehub/tmp/tmpw0x_2srx/wav2lip/requirements.txt: \
[2023-10-22 13:15:59,600] [    INFO] - Successfully installed wav2lip-1.0.0
```

## 三、主要架构

### 3.1 文心一言api调用

这里选择文心一言的‘aistudio’API、erniebot.acess_token = ''此处需要填如自己的访问令牌（个人中心、访问令牌）

In [14]

```
import erniebot

erniebot.api_type = 'aistudio'
#此处需要将你的token也就是AIstudio主页的访问令牌放到下方
erniebot.access_token = "此处填写自己的访问令牌"
```

In [5]

```
response = erniebot.ChatCompletion.create(
    model='ernie-bot',
    messages=[{'role': 'user', 'content': f"介绍一下自己"}],
)
print(response.result)
您好，我是文心一言，英文名是ERNIE Bot。我能够与人对话互动，回答问题，协助创作，高效便捷地帮助人们获取信息、知识和灵感。
```

### 3.2 文心一言自定义数字分身展示

你可以发挥想象，来定义任何的虚拟人物，需要提供三个信息：

1、dd_name  姓名

2、dd_desc  身份信息、背景、事迹

3、dd_example_chat 和别人的聊天

In [15]

```
#调用文心一言API
def chat_yiyan_api(prompt):
    response = erniebot.ChatCompletion.create(
        model='ernie-bot',
        messages=[{'role': 'user', 'content': prompt}]
    )
    content = response.result
    return content
```

In [16]

```
#定义机器人身份
def chat_yiyan(prompt, his, prompt_his_str='你：{}\n分身：{}', dd_name=''):
    Q_his = '\n'.join([prompt_his_str.format(i[0], i[1]) for i in his])
    if Q_his:
        Q_his += '\n'
    Q_motif = f'{Q_his}用户：{prompt}\n{dd_name}：'
    result = chat_yiyan_api(Q_motif)
    return result
```

In [17]

```
#主要步骤
import re
def dd_chat(dd_his_prompt, dd_name, dd_desc, dd_example_chat, msg, his):
    dd_ifm = dd_prompt.format(name=dd_name, desc=dd_desc, chat=dd_example_chat)
    dd_his_prompt = dd_his_prompt.replace('{name}', dd_name)
    o_his = [[dd_ifm, ''], *his]
    answer = chat_yiyan(msg, o_his, prompt_his_str=dd_his_prompt)
    answer = re.sub(dd_name + '[\:：]', '', answer).strip()
    answer = answer.split('用户:')[0].split('用户：')[0]
    his.append([msg, answer])
    str = ''.join(his[-1][1])
    return his
```

In [18]

```
#定义参数
source_image = "zhuge.jpg"
dd_prompt = '''我需要你模仿{name}，请参考以下信息和用户聊天
----
下面是{name}的介绍：
  {desc}
----
下面是{name}和别人的对话：
  {chat}
----
请针对上文，扮演{name}，回复简洁，和用户聊天'''
dd_his_prompt = '用户：{}\n{name}：{}'
dd_name = '妇好'
dd_desc = '妇好是商朝时期的一位女性军事统帅和政治家，也是商王武丁的妻子。她多次受命征战沙场，为商王朝拓展疆土立下汗马功劳。出土的大量甲骨卜辞表明了妇好的身份和地位，她也是中国历史上第一位有据可查的女英雄。'
dd_example_chat = '你好，我是妇好，中国第一位女英雄！'

msg = '请介绍一下自己'  #你想说的话

his = []
txt = dd_chat(dd_his_prompt, dd_name, dd_desc, dd_example_chat, msg, his)[-1][-1]
print(txt)
妇好，中国第一位女英雄，也是商王武丁的妻子。我曾多次征战沙场，为商王朝拓展疆土立下汗马功劳。
```

### 3.3 表情迁移

通过"first_order_motion"模型，输入图像和驱动视频，让人像动起来！

In [10]

```
import paddlehub as hub
FOM_Module = hub.Module(name="first_order_motion")
FOM_Module.generate(source_image="/home/aistudio/Img/zhuge.jpg", # 输入图像
                    driving_video="/home/aistudio/Img/qudong.mp4", # 输入驱动视频
                    ratio=0.4, 
                    image_size=256, 
                    output_dir='./output/', # 输出文件夹
                    filename='output-zhuge.mp4', # 输出文件名
                    use_gpu=True)
W1022 13:16:37.976037   299 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 12.0, Runtime API Version: 10.1
W1022 13:16:37.981480   299 device_context.cc:465] device: 0, cuDNN Version: 7.6.
[10/22 13:16:43] ppgan INFO: Downloading GPEN-512.pdparams from https://paddlegan.bj.bcebos.com/models/GPEN-512.pdparams to /home/aistudio/.cache/ppgan/GPEN-512.pdparams
100%|██████████| 277380/277380 [00:04<00:00, 58345.21it/s]
100%|██████████| 109119/109119 [00:02<00:00, 49561.59it/s]
1 persons have been detected
100%|██████████| 300/300 [00:33<00:00,  8.97it/s]
IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (500, 362) to (512, 368) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).
[swscaler @ 0x50db340] Warning: data is not aligned! This can lead to a speed loss
```

### 3.4 文字转语音

将ENIRE Bot自定义的虚拟人物输出转换生成一段音频！

In [19]

```
from paddlespeech.cli.tts.infer import TTSExecutor
tts = TTSExecutor()
tts(text=txt, output="output.wav")
100%|██████████| 477172/477172 [00:15<00:00, 30709.65it/s]
100%|██████████| 893777/893777 [00:20<00:00, 44051.61it/s]
100%|██████████| 575056/575056 [00:15<00:00, 37747.93it/s]
[2023-10-22 13:23:39,852] [    INFO] - Downloading bert-base-chinese-vocab.txt from https://paddle-hapi.bj.bcebos.com/models/bert/bert-base-chinese-vocab.txt
100%|██████████| 107/107 [00:00<00:00, 4712.21it/s]
Building prefix dict from the default dictionary ...
[2023-10-22 13:23:45] [DEBUG] [__init__.py:113] Building prefix dict from the default dictionary ...
Dumping model to file cache /tmp/jieba.cache
[2023-10-22 13:23:46] [DEBUG] [__init__.py:147] Dumping model to file cache /tmp/jieba.cache
Loading model cost 0.748 seconds.
[2023-10-22 13:23:46] [DEBUG] [__init__.py:165] Loading model cost 0.748 seconds.
Prefix dict has been built successfully.
[2023-10-22 13:23:46] [DEBUG] [__init__.py:166] Prefix dict has been built successfully.
'/home/aistudio/output.wav'
```

### 3.5 唇形合成

把刚刚得到的动态视频和音频文件输入到Wav2Lip模型中，让唇形根据说话的内容动态改变。

In [ ]

```
import paddlehub as hub
W2F_Module = hub.Module(name="wav2lip")

W2F_Module.wav2lip_transfer(face='output/output-zhuge.mp4', 
                            audio='Audio/audio-zhuge.wav', 
                            output_dir='./zhuge/', 
                            use_gpu=True) 
[10/22 11:17:03] ppgan INFO: Found /home/aistudio/.cache/ppgan/GPEN-512.pdparams
Reading video frames...
Number of frames available for inference: 300
Length of mel chunks: 292
Model loaded
  0%|          | 0/3 [00:00<?, ?it/s]
  0%|          | 0/19 [00:00<?, ?it/s]
  5%|▌         | 1/19 [00:00<00:10,  1.66it/s]
 11%|█         | 2/19 [00:01<00:10,  1.68it/s]
 16%|█▌        | 3/19 [00:01<00:09,  1.68it/s]
 21%|██        | 4/19 [00:02<00:08,  1.69it/s]
 26%|██▋       | 5/19 [00:02<00:08,  1.69it/s]
 32%|███▏      | 6/19 [00:03<00:07,  1.68it/s]
 37%|███▋      | 7/19 [00:04<00:07,  1.69it/s]
 42%|████▏     | 8/19 [00:04<00:06,  1.66it/s]
 47%|████▋     | 9/19 [00:05<00:05,  1.67it/s]
 53%|█████▎    | 10/19 [00:06<00:05,  1.59it/s]
 58%|█████▊    | 11/19 [00:06<00:04,  1.61it/s]
 63%|██████▎   | 12/19 [00:07<00:04,  1.63it/s]
 68%|██████▊   | 13/19 [00:07<00:03,  1.64it/s]
 74%|███████▎  | 14/19 [00:08<00:03,  1.66it/s]
 79%|███████▉  | 15/19 [00:09<00:02,  1.66it/s]
 84%|████████▍ | 16/19 [00:09<00:01,  1.67it/s]
 89%|████████▉ | 17/19 [00:10<00:01,  1.68it/s]
 95%|█████████▍| 18/19 [00:10<00:00,  1.63it/s]
100%|██████████| 19/19 [00:11<00:00,  1.72it/s]
100%|██████████| 3/3 [00:36<00:00, 12.13s/it]
ffmpeg version 2.8.15-0ubuntu0.16.04.1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.10) 20160609
  configuration: --prefix=/usr --extra-version=0ubuntu0.16.04.1 --build-suffix=-ffmpeg --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --cc=cc --cxx=g++ --enable-gpl --enable-shared --disable-stripping --disable-decoder=libopenjpeg --disable-decoder=libschroedinger --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librtmp --enable-libschroedinger --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzvbi --enable-openal --enable-open
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Guessed Channel Layout for  Input Stream #0.0 : mono
Input #0, wav, from 'Audio/audio-zhuge.wav':
  Duration: 00:00:09.86, bitrate: 384 kb/s
    Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 24000 Hz, 1 channels, s16, 384 kb/s
Input #1, avi, from 'temp/result.avi':
  Metadata:
    encoder         : Lavf58.31.101
  Duration: 00:00:09.73, start: 0.000000, bitrate: 1247 kb/s
    Stream #1:0: Video: mpeg4 (Simple Profile) (DIVX / 0x58564944), yuv420p, 512x368 [SAR 1:1 DAR 32:23], 1241 kb/s, 30 fps, 30 tbr, 30 tbn, 30 tbc
[libx264 @ 0x1f22020] -qscale is ignored, -crf is recommended.
[libx264 @ 0x1f22020] using SAR=1/1
[libx264 @ 0x1f22020] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 AVX2 LZCNT BMI2
[libx264 @ 0x1f22020] profile High, level 3.0
[libx264 @ 0x1f22020] 264 - core 148 r2643 5c65704 - H.264/MPEG-4 AVC codec - Copyleft 2003-2015 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=11 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
Output #0, mp4, to './zhuge/result.mp4':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: h264 (libx264) ([33][0][0][0] / 0x0021), yuv420p, 512x368 [SAR 1:1 DAR 32:23], q=-1--1, 30 fps, 15360 tbn, 30 tbc
    Metadata:
      encoder         : Lavc56.60.100 libx264
    Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 24000 Hz, mono, fltp, 128 kb/s
    Metadata:
      encoder         : Lavc56.60.100 aac
Stream mapping:
  Stream #1:0 -> #0:0 (mpeg4 (native) -> h264 (libx264))
  Stream #0:0 -> #0:1 (pcm_s16le (native) -> aac (native))
Press [q] to stop, [?] for help
frame=  292 fps=0.0 q=-1.0 Lsize=     540kB time=00:00:09.89 bitrate= 447.1kbits/s    
video:382kB audio:149kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 1.656511%
[libx264 @ 0x1f22020] frame I:2     Avg QP:18.49  size: 13398
[libx264 @ 0x1f22020] frame P:99    Avg QP:22.79  size:  2472
[libx264 @ 0x1f22020] frame B:191   Avg QP:28.84  size:   625
[libx264 @ 0x1f22020] consecutive B-frames: 12.3%  0.7%  2.1% 84.9%
[libx264 @ 0x1f22020] mb I  I16..4: 12.5% 77.6%  9.9%
[libx264 @ 0x1f22020] mb P  I16..4:  0.5%  1.8%  0.2%  P16..4: 15.2%  9.9%  5.7%  0.0%  0.0%    skip:66.8%
[libx264 @ 0x1f22020] mb B  I16..4:  0.1%  0.2%  0.0%  B16..8: 18.3%  3.6%  0.8%  direct: 0.7%  skip:76.3%  L0:49.3% L1:41.7% BI: 9.0%
[libx264 @ 0x1f22020] 8x8 transform intra:73.8% inter:58.3%
[libx264 @ 0x1f22020] coded y,uvDC,uvAC intra: 40.0% 53.1% 18.1% inter: 6.4% 4.5% 0.3%
[libx264 @ 0x1f22020] i16 v,h,dc,p: 59% 11% 25%  5%
[libx264 @ 0x1f22020] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 45%  8% 33%  2%  2%  4%  1%  3%  1%
[libx264 @ 0x1f22020] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 39% 12% 14%  4%  6%  8%  7%  8%  3%
[libx264 @ 0x1f22020] i8c dc,h,v,p: 41% 15% 40%  4%
[libx264 @ 0x1f22020] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x1f22020] ref P L0: 60.4% 12.6% 17.9%  9.2%
[libx264 @ 0x1f22020] ref B L0: 86.8%  9.7%  3.5%
[libx264 @ 0x1f22020] ref B L1: 95.4%  4.6%
[libx264 @ 0x1f22020] kb/s:321.24
```

## 四、Gradio页面介绍

![img](http://yg9538.kmgy.top/202311191453509.png)

![img](http://yg9538.kmgy.top/202311191453545.png)

1、首先需要用于提供自定义数字分身的信息，如：名字、背景、谈话记录!（左下角有例子可以参考）

2、然后就可以何其进行聊天，然后会使用paddlespeach模型，将机器人输出转换文语音!

3、最后点击生成就可以展示生成的效果图了！