S3 Store

bench get-app https://github.com/Centura-AG/s3_store

[![Listed on Frappe Gems](https://frappegems.com/api/method/frappe_gems.seo.badge?app=Centura-AG%2Fs3_store)](https://frappegems.com/gems/apps/Centura-AG/s3_store)

# s3_store S3-backed file storage for Frappe with **native `bench backup` / `bench restore` integration**. Unlike other S3 attachment apps, files uploaded via `s3_store` are pulled into the standard `bench backup --with-files` tar (so it's self-contained) and pushed back to S3 after `bench restore` with a single command — or automatically. No separate scheduled DB pushes, no parallel backup infrastructure. Compatible with **AWS S3, MinIO, and any S3-compatible service**. --- ## Table of contents 1. [Installation](#installation) 2. [Configuration](#configuration) 3. [How uploads work](#how-uploads-work) 4. [How serving works](#how-serving-works) 5. [Backup with files](#backup-with-files) 6. [Restore](#restore) 7. [Bulk migration of pre-existing local files](#bulk-migration-of-pre-existing-local-files) 8. [Settings reference](#settings-reference) 9. [Architecture notes](#architecture-notes) 10. [Testing](#testing) 11. [Troubleshooting](#troubleshooting) 12. [Limitations](#limitations) --- ## Installation ```bash bench get-app https://github.com/Centura-AG/s3_store bench --site install-app s3_store ``` This pulls in a single Python dependency: `boto3`. No node modules, no native extensions. --- ## Configuration Open **S3 Store Settings** (singleton, System Manager only) in Desk. **Minimum required for AWS S3**: | Field | Example | | --------------------- | ---------------------- | | Enabled | ✓ | | Region | `eu-central-1` | | AWS Access Key ID | `AKIAIOSFODNN7EXAMPLE` | | AWS Secret Access Key | `wJalrXUt…` | | Bucket | `my-frappe-uploads` | **For MinIO / S3-compatible providers**, also set: | Field | Example | | ------------ | --------------------------- | | Endpoint URL | `https://minio.example.com` | Saving the settings runs `head_bucket` against the configured S3 endpoint — if credentials, region, or bucket name are wrong, the save fails with a clear error. --- ## How uploads work When `Enabled = 1`, every new File doc routes through the `write_file` hook: 1. The file's bytes are uploaded to S3 under a generated key: `{key_prefix}/{YYYY/MM/DD}/{attached_to_doctype}/{token8}_{sanitized_name}` 2. The File doc's `file_url` is rewritten to the backend serve endpoint: `/api/method/s3_store.s3_store.api.serve?key=` 3. No local copy is kept. The `token8` is a random 4-byte hex prefix — uploads of the same `file.pdf` from different forms get distinct keys, so they can't overwrite each other in S3 or on disk during backup staging. **Ignoring specific doctypes** — list one DocType per line under _Advanced → Ignored Doctypes_ to keep their attachments on the local filesystem (useful for `User` profile pictures, for example). --- ## How serving works `/api/method/s3_store.s3_store.api.serve?key=` resolves the key to a File doc via an exact-match lookup on `file_url`, enforces `check_permission("read")` on that doc, then **streams the object's bytes through the backend** as a download response (`display_content_as: inline`). The S3 object is fetched server-side with `get_object` and proxied to the client — the bucket stays **fully private** and credentials never leave the server. Browsers never talk to S3 directly. > Because every read is proxied through a web worker, large files occupy a worker for the duration of the download and are buffered in memory. For very large public assets you may prefer fronting the bucket with a CDN; that is out of scope for this app. --- ## Backup with files ```bash bench --site backup --with-files ``` Behind the scenes, `s3_store` monkey-patches `BackupGenerator.backup_files` so that — **only when this site has `S3 Store Settings.enabled = 1` and `include_in_native_backup = 1`** — every S3-hosted file is downloaded into `public/files/` or `private/files/` _before_ `tar` runs, and removed _after_. The patch is a no-op on sites without the app installed. The resulting tar is self-contained: you can move it to a different host, a developer laptop, or a fresh bucket and restore without losing any uploads. ### Disk-space pre-flight Before staging, `s3_store` totals `file_size` across all S3-backed File records and requires ~110% free on the disk hosting the site. If insufficient, the backup aborts with: ``` Insufficient disk space to stage S3 files for backup: need bytes, have bytes free under . Free up space or disable include_in_native_backup in S3 Store Settings. ``` ### Encrypted backups `bench backup --with-files --backup-encryption-key ` works unchanged. Encryption runs _after_ `backup_files()` in Frappe core, so the staged S3 contents end up inside the encrypted blob. ### Concurrency A file lock at `sites//.s3_store_backup.lock` prevents two backups from staging on top of each other. The second attempt fails fast with: `"Another s3_store backup is already running for this site."` ### Stale-tar shortcut `BackupGenerator.get_recent_backup` normally returns recent `*-files.tar` if one exists within the `--older-than` window. That would silently skip new S3 uploads — so `s3_store` neutralises the **files-portion only** when S3 is enabled, forcing a fresh stage. The DB shortcut is preserved. --- ## Restore ```bash bench --site restore \ --with-public-files \ --with-private-files ``` After this, the DB has File records whose `file_url` already points at S3 (`?key=…`), and the bytes sit on disk under their staging names (`{token8}_{sanitized_name}`). To push them back to S3: ```bash bench --site s3-store-push-local ``` For each staged file, this uploads to the **exact existing key** referenced by the File doc — no URL rewriting, no fresh keys. Local copies are deleted after a successful push. ### Auto-pilot Set **Auto-Push Local Files after Migrate** in S3 Store Settings, and the push runs automatically as an `after_migrate` hook — i.e. at the end of every `bench migrate`, which conventionally follows `bench restore`. Cautious users leave this off and run the command explicitly. --- ## Bulk migration of pre-existing local files Use this once, after first installing on a site that already has local attachments: **Via the UI**: open **S3 Migration Log → New**, save. The migration runs as a background job; progress is published to the realtime channel and reflected on the log doc (`migrated`, `failed`, `errors`). **Via API**: ```bash bench --site execute s3_store.s3_store.api.start_migration ``` For every File row with a `/files/...` or `/private/files/...` URL, `s3_store`: 1. Mints a fresh S3 key 2. Uploads the bytes 3. Rewrites `file_url` to the backend serve endpoint 4. Deletes the local copy (subject to `delete_local_after_push`) Errors are collected per-file — one bad file doesn't abort the run. The log doc shows status, counts, and the per-file error list. --- ## Settings reference | Field | Default | Description | | -------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------- | | `enabled` | `0` | Master switch. When off, files go to the local filesystem as usual. | | `endpoint_url` | — | S3 endpoint. Blank for AWS; set for MinIO etc. | | `region` | `us-east-1` | AWS region. | | `aws_access_key_id` | — | Required when enabled. | | `aws_secret_access_key` | — | Required when enabled (stored encrypted via Frappe's password field). | | `bucket` | — | Required when enabled. | | `key_prefix` | — | Optional path prefix (e.g. site name). Objects are stored at `{prefix}/{YYYY/MM/DD}/{doctype}/{token}_{name}`. | | `signed_url_expiry` | `3600` | Reserved for future use. Currently unused — all serving goes through the backend proxy endpoint. | | `delete_from_s3` | `1` | Delete the S3 object when a File doc is trashed. | | `include_in_native_backup` | `1` | Stage S3 files locally during `bench backup --with-files`. | | `auto_push_after_migrate` | `0` | After `bench migrate` (typically right after `bench restore`), push local files back to S3. | | `delete_local_after_push` | `1` | Remove the local copy after uploading to S3. | | `ignored_doctypes` | — | Newline-separated DocType names whose attachments stay on the local filesystem. | --- ## Testing ```bash bench --site run-tests --app s3_store ``` The suite covers: - Settings validation (creds + connection check) - `write_file` and delete-on-trash hook - `serve` content proxying, missing-key, permission-error paths - Bulk migration: rewrite URL, delete local, error collection, log doc lifecycle - Post-restore push: re-upload to existing key, error when staged file is missing - Backup staging: cleanup on success, cleanup on exception, mixed-mode skip - Patched `BackupGenerator` is a no-op when S3 disabled; `get_recent_backup` neutralises file tars - URL parsing across virtual-host AWS, path-style MinIO, and the backend serve endpoint All boto3 calls are mocked — the suite never hits the network. ### End-to-end smoke test against MinIO ```bash # Remove any previous attempt before re-creating docker rm -f s3store-minio 2>/dev/null || true # Start MinIO (change ports if 9200/9201 are taken) docker run -d --name s3store-minio \ -p 9200:9000 -p 9201:9001 \ -e MINIO_ROOT_USER=admin \ -e MINIO_ROOT_PASSWORD=adminadmin \ minio/minio server /data --console-address ":9001" # Create the bucket docker run --rm --network container:s3store-minio --entrypoint sh minio/mc \ -c "mc alias set local http://localhost:9000 admin adminadmin --api s3v4 \ && mc mb --ignore-existing local/s3store-bucket" ``` Open **S3 Store Settings** in Desk: | Field | Value | | --------------------- | ----------------------- | | Enabled | ✓ | | Endpoint URL | `http://localhost:9200` | | Region | `us-east-1` | | AWS Access Key ID | `admin` | | AWS Secret Access Key | `adminadmin` | | Bucket | `s3store-bucket` | Console is at `http://localhost:9201` (`admin` / `adminadmin`). ```bash bench --site dev.s3.test install-app s3_store # upload some files via the UI, then: bench --site dev.s3.test backup --with-files # Wipe and restore bench drop-site dev.s3.test bench --site dev.s3.test restore \ --with-public-files --with-private-files bench --site dev.s3.test s3-store-push-local ``` After the push, the local `public/files/` and `private/files/` directories should contain only Frappe's own non-S3 files; the bucket should hold every uploaded attachment under its original key. --- ## Troubleshooting **`S3 connection check failed: An error occurred (403) when calling the HeadBucket operation`** Wrong credentials, wrong region, or the IAM user/policy doesn't grant `s3:ListBucket` on the configured bucket. The same policy ne