Now you can replace the head.
Example: https://www.youtube.com/watch?v=xr5FHd0AdlQ
Requirements:
Post processing skill in Adobe After Effects or Davinci Resolve.
Usage:
1) Find suitable dst footage with the monotonous background behind head
2) Use “extract head” script
3) Gather rich src headset from only one scene (same color and haircut)
4) Mask whole head for src and dst using XSeg editor
5) Train XSeg
6) Apply trained XSeg mask for src and dst headsets
7) Train SAEHD using ‘head’ face_type as regular deepfake model with DF archi. You can use pretrained model for head. Minimum recommended resolution for head is 224.
8) Extract multiple tracks, using Merger:
a. Raw-rgb
b. XSeg-prd mask
c. XSeg-dst mask
9) Using AAE or DavinciResolve, do:
a. Hide source head using XSeg-prd mask: content-aware-fill, clone-stamp, background retraction, or other technique
b. Overlay new head using XSeg-dst mask
Warning: Head faceset can be used for whole_face or less types of training only with XSeg masking.
XSegEditor: added button ‘view trained XSeg mask’, so you can see which frames should be masked to improve mask quality.
Basic usage instruction: https://i.imgur.com/w7LkId2.jpg
'whole_face' requires skill in Adobe After Effects.
For using whole_face you have to extract whole_face's by using
4) data_src extract whole_face
and
5) data_dst extract whole_face
Images will be extracted in 512 resolution, so they can be used for regular full_face's and half_face's.
'whole_face' covers whole area of face include forehead in training square,
but training mask is still 'full_face'
therefore it requires manual final masking and composing in Adobe After Effects.
added option 'masked_training'
This option is available only for 'whole_face' type.
Default is ON.
Masked training clips training area to full_face mask,
thus network will train the faces properly.
When the face is trained enough, disable this option to train all area of the frame.
Merge with 'raw-rgb' mode, then use Adobe After Effects to manually mask, tune color, and compose whole face include forehead.
Removed the wait at first launch for most graphics cards.
Increased speed of training by 10-20%, but you have to retrain all models from scratch.
SAEHD:
added option 'use float16'
Experimental option. Reduces the model size by half.
Increases the speed of training.
Decreases the accuracy of the model.
The model may collapse or not train.
Model may not learn the mask in large resolutions.
true_face_training option is replaced by
"True face power". 0.0000 .. 1.0
Experimental option. Discriminates the result face to be more like the src face. Higher value - stronger discrimination.
Comparison - https://i.imgur.com/czScS9q.png
removed TrueFace model.
added SAEv2 model. Differences from SAE:
+ default e_ch_dims is now 21
+ new encoder produces more stable face and less scale jitter
before: https://i.imgur.com/4jUcol8.gifv
after: https://i.imgur.com/lyiax49.gifv - scale of the face is less changed within frame size
+ decoder now has only 1 residual block instead of 2, result is same quality with less decoder size
+ added mid-full face, which covers 30% more area than half face.
+ added option " Enable 'true face' training "
Enable it only after 50k iters, when the face is sharp enough.
the result face will be more like src.
The most src-like face with 'true-face-training' you can achieve with DF architecture.
With interactive converter you can change any parameter of any frame and see the result in real time.
Converter: added motion_blur_power param.
Motion blur is applied by precomputed motion vectors.
So the moving face will look more realistic.
RecycleGAN model is removed.
Added experimental AVATAR model. Minimum required VRAM is 6GB (NVIDIA), 12GB (AMD)
Usage:
1) place data_src.mp4 10-20min square resolution video of news reporter sitting at the table with static background,
other faces should not appear in frames.
2) process "extract images from video data_src.bat" with FULL fps
3) place data_dst.mp4 video of face who will control the src face
4) process "extract images from video data_dst FULL FPS.bat"
5) process "data_src mark faces S3FD best GPU.bat"
6) process "data_dst extract unaligned faces S3FD best GPU.bat"
7) train AVATAR.bat stage 1, tune batch size to maximum for your card (32 for 6GB), train to 50k+ iters.
8) train AVATAR.bat stage 2, tune batch size to maximum for your card (4 for 6GB), train to decent sharpness.
9) convert AVATAR.bat
10) converted to mp4.bat
updated versions of modules
