bench get-app https://github.com/theapprenticeproject/Video_Translation

[![Listed on Frappe Gems](https://frappegems.com/api/method/frappe_gems.seo.badge?app=theapprenticeproject%2FVideo_Translation)](https://frappegems.com/gems/apps/theapprenticeproject/Video_Translation)

---

## Table of Contents **1. [Product Overview](#1-product-overview)
2. [Technical Architecture](#2-technical-architecture)
3. [Cost Structure](#3-cost-structure)
4. [Installation and Deployment](#4-installation-and-deployment)
5. [Access & Credentials](#5-access--credentials)
6. [Data Flow Info](#6-data-flow-info)
7. [Roadmap & Future Work](#7-roadmap--future-work)
** --- ## 1. Product Overview ### 1.1 Purpose * This tool helps to translate educational videos into multiple native Indian languages. Primarily, educational videos are made in English, this makes sure it localizes every part of video to make it accessible for students of diverse language groups. * The primary usecase is localization of video in a sequence of steps. The target audience are Educators who can localize their educational content into Indic languages, which then could be served to students. ### 1.2 Key Features * Localizes video content into multiple Indic languages asynchronously. * Supports options for both local file & google drive link uploads. * Automated extraction and translation of on-screen text. * Generates localized subtitles and translates audio tracks. * Human-In-The-Philosophy (HITL) to monitor progress steps throughout the entire localization pipeline. * Versioned REST APIs for pipeline extension and integration. ### 1.3 Known Limitations * Supports video files with `.mp4` format. * Currently suppports only 3 languages - Hindi, Marathi, Punjabi. * Need for manual review within pipeline if translation not appropriate. --- ## 2. Technical Architecture ### 2.1 System Architecture #### 2.1.1 API & Media Transformation Flow This diagram illustrates the external API routing and media transformation pipeline. The system branches into specialized workflows based on the selected target language. * **Phase 1: Input & Routing**
The pipeline evaluates the original video and routes it based on the selected target language i.e., splitting into a direct path for Hindi or a path for other Indic languages (e.g., Marathi, Punjabi). * **Phase 2a: Non-Hindi Translation**
A multi-step pipeline for languages which need more control over text translation & audio. * Audio Extraction: FFmpeg extracts the audio track from the uploaded video or link. * Transcription: ElevenLabs STT (Speech-to-Text) generates the original transcript. * Translation: Bhashini API translates the text segments. * HITL 1 (Human-In-The-Loop): The pipeline pauses, allowing the user to review and edit the translated segments via the UI. * Speech Generation: Upon approval, ElevenLabs TTS (Text-to-Speech) synthesizes the new localized audio track. * **Phase 2b: Hindi Direct Dubbing**
A straightforward workflow that uses the ElevenLabs Dubbing API to handle end-to-end dubbing directly from the source video. * **Phase 3: On-Screen Text Localization**
Once the translated audio track or muxed video is created, the pipeline triggers the onscreen text localization process: * Text Recognition (OCR): Google Video Intelligence API extracts text directly from the video frames with verbose response metadata. * Translation: Bhashini translates the extracted OCR text. * Subtitle Generation: The system automatically formats the translations into a VTT subtitle file. * HITL 2 (Human-In-The-Loop): The pipeline pauses for a final user review of the translated on-screen text. * **Phase 4: Final Media Synthesis**
Triggered by the final HITL review, FFmpeg then collects info from childtable & permanently overlays the translated text (using filter script file) into the video frames and muxes the final video to produce the Final Localized Video. #### 2.1.2 Queue Processing Flow This diagram illustrates the queueing & processing flow of a record created in this video localization pipeline. The following highlights the lifecycle of this flow:
* **Upload Phase**
The process begins when a user uploads a video or link. This creates a `Video Info` record in database. The uploaded video file is stored under `public/files/original/` directory. * **Background Job Queues**
Once the localization process is initiated, a chain of background jobs are triggered sequentially. Managed by Frappe's queues, this allows multiple video records to be processed concurrently as each record waits in a queue with designated functionality. The pipeline is segmented as specialized task queues like audio extraction, speech translation, ocr, etc. Each subsequent function is enqueued only after the successful completion of the preceding step. * **Completion/Processing Phase**
As background workers execute these tasks, they continuously update a single `Processed Video Info` record with status & progress. During this automated flow, it includes HITL phases for review which then trigger to start subsequent functions within queues. As stages conclude, resulting files are stored to `public/files/processed` directory. ### 2.2 Tech Stack * **Frappe** is a low-code web framework which handles server, client-side, database and other configurations altogether. * Frontend: Javascript (custom client scripts) * Backend: Python * Database: MariaDB (MySQL) * AI Model Providers: Bhashini & ElevenLabs * Infrastructure: Google Cloud Platform (GCP) ### 2.3 Hosting environment and deployement setup * **Cloud Environment:** * Machine Type: e2-highcpu-8 (8 vCPUs, 8 GB memory) * Gunicorn runs the Frappe application, nginx receives web traffic. Supervisor is for starting gunicorn processes like workers, scheduler. * This pipeline uses frappe background job queues, thus the workers config is defined in `common_site_config.json`. Background workers can be increased to pick up more jobs lined up in the queues. During setup, values are set automatically, but can be changes can be made as necessary. ```json { "background_workers": "2", "default_site": "[your_site_name]", "developer_mode": true, "file_watcher_port": 6787, "frappe_user": "[your_user_name]", "gunicorn_workers": "[2 x CPU_Cores + 1]", "live_reload": true, "rebase_on_pull": false, "redis_cache": "redis://127.0.0.1:13000", "redis_queue": "redis://127.0.0.1:11000", "redis_socketio": "redis://127.0.0.1:13000", "restart_supervisor_on_update": false, "restart_systemd_on_update": false, "serve_default_site": true, "shallow_clone": true, "socketio_port": 9000, "use_redis_auth": false, "webserver_port": 8000 } ``` ### 2.4 Third-party Integrations and Dependencies * **External Dependencies:** * FFMPEG * Bhashini * ElevenLabs * Gdown * Google Cloud Video Intelligence * Groq (Optional) --- ## 3. Cost Structure * More info regarding **ElevenLabs** requests & usage analytics, etc can be found at [API Activity](https://elevenlabs.io/app/api). * Current ElevenLabs utilisation is Pro plan, and a reference cost analaysis can be found here [Comparative Cost Analysis](https://ringed-mouse-89f.notion.site/Possible-Costs-Per-Services-227d272657fe802c974de366b5948641?source=copy_link) amongst others.
Note: Do refer the [elevenlabs api pricing](https://elevenlabs.io/pricing/api) as it could differ based on subscription model or pay-as-you-go model. * Google Video Intelligence's usage can be monitored under Google Cloud Console -> APIs & Services -> Dashboard. The billing would be visible under Google Cloud Console -> Billing and then can be filtered as per project and SKU. ## 4. Installation and Deployment ### 4.1 Bench Installation * Ensure you have a standard Bench [Frappe Installation Guide](https://docs.frappe.io/framework/user/en/installation) environment installed as this project uses Frappe 15. * Note on python dependency managament: If you're on an externally managed environment, follow the instructions in the above Frappe docs but substitute uv over pip/venv. * Once bench is ready: ```bash bench init [directory-name] ``` * Create a new site for this project locally: ```bash cd $PATH_TO_YOUR_BENCH bench new-site [your-site-name] bench use [your-site-name] ``` ### 4.2 Installation #### 4.2.1 App Installation Now go to the bench directory ( if not already ), and install/clone the frappe video translation app: ```bash cd $PATH_TO_YOUR_BENCH bench get-app [github-repo-url-or-ssh] --branch main bench --site [your-site-name] install-app my_app ``` **Note: Run the folowing command from Bench directory to manually create `/original` and `/processed` folders under site's public directory**:
```bash cd $PATH_TO_YOUR_BENCH mkdir sites/[your_site_name]/files/public/{original,processed} ``` (replace `your_site_name` with your appropriate site name created) #### 4.2.2 Video Preview App (Optional) This frappe app provides a better preview for video uploads upon saving a record for a doctype, improving user experience. ```bash bench get-app git@github.com:Z4nzu/frappe-preview-attachment.git bench --site your-site-name install-app preview_attachment ``` Reference: [Frappe Video Preview github](https://github.com/Z4nzu/frappe-preview-attachment) ### 4.3 Dependencies Setup #### 4.3.1 Python Dependency Management Install **`uv`** if it's the choice. (Official site: [`uv` Installation Reference](https://docs.astral.sh/uv/getting-started/installation/)) ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` #### 4.3.2 Installing Dependencies This is valid for both local development & deployement server. * Activate virtual environment if not already: ```bash source env/bin/activate ``` * Go back to the Bench directory & run the following so that dependencies within `pyproject.toml` are installed (editable mode) in central bench frappe environment: ```bash cd $PATH_TO_YOUR_BENCH uv pip install -e apps/my_app ``` * While developing if there is a need to add a new dependency to the project, follow: * Navigate to `apps/my_app` & run: ```bash uv add --no-sync ``` * Return to the *bench* directory & run the following to install updated dependencies into bench environment: ```bash cd $PATH_TO_YOUR_BENCH uv pip install -e apps/my_app ```