简体中文 | English

Introduction

Video-subtitle-extractor (VSE) is a free, open-source tool which can help you rip the hard-coded subtitles from videos and automatically generate corresponding srt files for each video. It includes the following implementations:

• Detect and extract subtitle frames (using traditional graphic method)
• Detect subtitle areas (i.e., coordinates) (as well as scene text if you want) (using deep learning algorithms)
• Converting graphic text into plain-text (using deep learning algorithms)
• Filter non-subtitle text (e.g., Logo and watermark etc.)
• Remove watermark, logo text and original video hard subtitles, see: video-subtitle-remover (VSR).
• Remove duplicated subtitle line and generate srt file (by calculating text similarity)
• Batch extraction. You can select multiple video files at one time and this tool can generate subtitles for each video.
• Multiple language support. You can extract subtitles in 87 languages such as: Simplified Chinese, English, Japanese, Korean, Arabic, Traditional Chinese, French, German, Russian, Spanish, Portuguese, Italian
• Multi-mode:
  • fast: (Recommended) Uses a lightweight model for quick subtitle extraction, though it might miss a small amount of subtitles and contains a few typos.
  • auto: (Recommended) Automatically selects the model. It uses the lightweight model under the CPU, and the precise model under the GPU. While subtitle extraction speed is slower and might miss a minor amount of subtitles, there are almost no typos.
  • accurate: (Not Recommended) Uses the precise model with frame-by-frame detection under the GPU, ensuring no missed subtitles and almost non-existent typos, but the speed is very slow.

Features：

• You don't need to do any preprocessing (e.g., binarization) and don't need to consider all aspects like subtitle fonts and size etc..
• This is an offline project. There is no online API call and you dont need to connect to the Internet service provider in order to get results.

Usage：

• After clicking "Open", select video file(s), adjust the subtitle area, and then click "Run".

  • Single file extraction: When opening a file, choose a single video.
  • Batch extraction: When opening files, choose multiple videos, ensure that every video's resolution and subtitle area remain consistent.

• Remove watermark text/replace specific text:

If specific text needs to be deleted from generated .srt file, or specific text needs to be replaced, you can edit the backend/configs/typoMap.json file and add the content you want to replace or remove.

{
    "l'm": "I'm",
    "l just": "I just",
    "Let'sqo": "Let's go",
    "Iife": "life",
    "威筋": "threat",
    "性感荷官在线发牌": ""
}

In this way, you can replace all occurrences of "威筋" in the text with "threat" and delete all instances of the text "性感荷官在线发牌".

• Directly download the compressed package, unzip it and run it. If it cannot run, follow the tutorial below and try to install the Conda environment and run it using the source code.

Download：

• Windows executable (might be slow when initial start): vse.exe

• Windows GPU version：vse_windows_gpu_v2.0.0.7z

• Windows CPU version：vse_windows_cpu_v2.0.0.zip

• MacOS：vse_macOS_CPU.dmg

Provide your suggestions to improve this project in ISSUES & DISCUSSION

Demo

• Graphic User Interface (GUI):

• Command Line Interface (CLI):

Running Online

• Google Colab Notebook with free GPU:

PS: can only run CLI version on Google Colab

Getting Started with Source Code

1. Download and Install Miniconda

• Windows: Miniconda3-py38_4.11.0-Windows-x86_64.exe

• MacOS：Miniconda3-py38_4.11.0-MacOSX-x86_64.pkg

• Linux: Miniconda3-py38_4.11.0-Linux-x86_64.sh

2. Activate Vitrual Environment

(1) Switch to working directory

cd <your source code path>

(2) create and activate conda environment

conda create -n videoEnv python=3.8 pip

conda activate videoEnv  

3. Install Dependencies

Before you install dependencies, make sure your python 3.8+ has installed as well as conda virtual environment has created and activated.

• For users who have CPU only (including Mac users):

  • Install dependencies：

    pip install -r requirements.txt
    

• For users who have NVIDIA graphic card： GPU version can achieve better accuracy

  • Install CUDA and cuDNN

    Linux

    (1) Download CUDA 11.7

    wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda_11.7.0_515.43.04_linux.run

    (2) Install CUDA 11.7

    sudo sh cuda_11.7.0_515.43.04_linux.run

    1. Input accept

    

    2. make sure CUDA Toolkit 11.7 is chosen (If you have already installed driver, do not select Driver)

    

    3. Add environment variables

    add the following content in ~/.bashrc

    # CUDA
      export PATH=/usr/local/cuda-11.7/bin${PATH:+:${PATH}}
      export LD_LIBRARY_PATH=/usr/local/cuda-11.7/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

    Make sure it works

    source ~/.bashrc

    (3) Download cuDNN 8.4.1

    cudnn-linux-x86_64-8.4.1.50_cuda11.6-archive.tar.xz

    (4) Install cuDNN 8.4.1

     tar -xf cudnn-linux-x86_64-8.4.1.50_cuda11.6-archive.tar.xz
     mv cudnn-linux-x86_64-8.4.1.50_cuda11.6-archive cuda
     sudo cp ./cuda/include/* /usr/local/cuda-11.7/include/
     sudo cp ./cuda/lib/* /usr/local/cuda-11.7/lib64/
     sudo chmod a+r /usr/local/cuda-11.7/lib64/*
     sudo chmod a+r /usr/local/cuda-11.7/include/*

    Windows

    (1) Download CUDA 11.7

    cuda_11.7.0_516.01_windows.exe

    (2) Install CUDA 11.7

    (3) Download cuDNN 8.2.4

    cudnn-windows-x64-v8.2.4.15.zip

    (4) Install cuDNN 8.2.4

    unzip "cudnn-windows-x64-v8.2.4.15.zip", then move all files in "bin, include, lib" in cuda directory to C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\

  • Install paddlepaddle:

    • windows:

        python -m pip install paddlepaddle-gpu==2.4.2.post117 -f https://www.paddlepaddle.org.cn/whl/windows/mkl/avx/stable.html
      

    • Linux:

        python -m pip install paddlepaddle-gpu==2.4.2.post117 -f https://www.paddlepaddle.org.cn/whl/linux/mkl/avx/stable.html
      

    If you installed cuda 10.2，please install cuDNN 7.6.5 instead of cuDNN v8.x

    If you installed cuda 11.2, please install cuDNN 8.1.1. However, RTX 30xx might be incompatible with cuda 11.2

  • Install other dependencies:

    pip install -r requirements_gpu.txt
    

3. Running the program

• Run GUI version

python gui.py

• Run CLI version

python ./backend/main.py

Q & A

1. Running Failure or Environment Problem

Solution: If you are using a nvidia ampere architecture graphic card such as RTX 3050/3060/3070/3080, please use the latest PaddlePaddle version and CUDA 11.6 with cuDNN 8.2.1. Otherwise, check your which cuda and cudnn works with your GPU and then install them.

2. For Windows users, if you encounter errors related to "geos_c.dll"

    _lgeos = CDLL(os.path.join(sys.prefix, 'Library', 'bin', 'geos_c.dll'))
  File "C:\Users\Flavi\anaconda3\envs\subEnv\lib\ctypes\__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found。

Solution：

1. Uninstall Shapely

pip uninstall Shapely -y

2. Reinstall Shapely via conda (make sure you have anaconda or miniconda installed)

conda install Shapely             

3. How to generate executables

Using Nuitka version 0.6.19, copy all the files of site-packages under the Lib folder of the conda virtual environment to the dependencies folder, and comment all codes relevant to subprocess of image.py under the paddle library dataset, and use the following packaging command:

 python -m nuitka --standalone --mingw64 --include-data-dir=D:\vse\backend=backend --include-data-dir=D:\vse\dependencies=dependencies  --nofollow-imports --windows-icon-from-ico=D:\vse\design\vse.ico --plugin-enable=tk-inter,multiprocessing --output-dir=out .\gui.py

Make a single .exe file, (pip install zstandard can compress the file):

 python -m nuitka --standalone --windows-disable-console --mingw64 --lto no --include-data-dir=D:\vse\backend=backend --include-data-dir=D:\vse\dependencies=dependencies  --nofollow-imports --windows-icon-from-ico=D:\vse\design\vse.ico --plugin-enable=tk-inter,multiprocessing --output-dir=out --onefile .\gui.py

Community Support

Jetbrains All Products Pack

The IDE this project used is supported by Jetbrains