xdrop/README.md

<p align="center">
  <img
    src="./apps/web/public/brand-lockup-horizontal.png"
    alt="Xdrop"
    height="200"
  />
</p>

<p align="center">
  <a href="https://codecov.io/github/xixu-me/xdrop">
    <img
      alt="Codecov coverage"
      src="https://codecov.io/github/xixu-me/xdrop/graph/badge.svg?token=qGmRgpmf62"
    />
  </a>
  <a href="https://github.com/xixu-me/xdrop/actions/workflows/ci.yml">
    <img
      alt="GitHub Actions CI status"
      src="https://github.com/xixu-me/xdrop/actions/workflows/ci.yml/badge.svg?branch=main"
    />
  </a>
  <a href="https://github.com/xixu-me/xdrop/actions/workflows/github-code-scanning/codeql">
    <img
      alt="CodeQL code scanning status"
      src="https://github.com/xixu-me/xdrop/actions/workflows/github-code-scanning/codeql/badge.svg?branch=main"
    />
  </a>
  <a href="https://ghcr.io/xixu-me/xdrop">
    <img
      alt="Container image publish status"
      src="https://github.com/xixu-me/xdrop/actions/workflows/publish-images.yml/badge.svg?branch=main"
    />
  </a>
</p>

<p align="center">
  English | <a href="./README.zh.md">汉语</a>
</p>

Xdrop is an open source encrypted file transfer app for browsers and agent-driven terminal
workflows, keeping plaintext file names, contents, and keys off the server.

## Highlights

- End-to-end encryption in the browser before upload.
- Single-file and folder transfers, including local ZIP downloads for received folders.
- Resumable uploads with browser-local state for interrupted transfers.
- Expiring links, sender-side management, and optional privacy mode after upload.
- S3-compatible object storage support with PostgreSQL and Redis on the backend.

## Use Via Agents

You can use Xdrop through an agent by installing the bundled skill:

```bash
bunx skills add https://github.com/xixu-me/xdrop/tree/main/skills/xdrop
```

After that, the agent can use Xdrop from the terminal to:

- Upload local files or directories and return an encrypted share link.
- Download a full Xdrop share link, including `#k=...`, and decrypt it locally.
- Automate repeatable handoff flows without switching to the browser UI.

Useful cases:

- On a cloud server, ask the agent to upload build artifacts, logs, or backups to your Xdrop
  instance and send back a temporary link.
- In a remote dev container or CI-like environment, ask the agent to package a directory and move
  it through Xdrop instead of setting up ad hoc SCP or public object storage access.
- On your local machine, hand the agent a full Xdrop link and ask it to download the files into a
  specific directory.

Example prompts:

- `Upload ./dist to https://xdrop.example.com and give me a 1-hour Xdrop link.`
- `On this VM, send /var/log/myapp through Xdrop so I can inspect it locally.`
- `Download this Xdrop link into ~/downloads and keep the original folder structure.`

## How It Works

1. A sender creates a transfer in the browser. Xdrop generates a random transfer root key and a
   separate link key, optionally strips removable image metadata, and prepares resumable local
   state before upload begins.
2. The API creates the transfer record and returns a manage token plus upload limits. The browser
   registers encrypted file metadata, then requests presigned chunk upload URLs in batches.
   PostgreSQL stores transfer/file/chunk metadata, Redis enforces rate limits, and S3-compatible
   storage keeps only encrypted blobs.
3. The sender shares a full link such as `/t/:transferId#k=...`. The `#k=...` fragment stays in
   the browser and is used to unwrap the transfer root key locally.
4. During upload, file chunks are encrypted in a dedicated Web Worker and streamed to storage.
   After every chunk is uploaded, the browser encrypts the manifest, uploads it, and finalizes the
   transfer with the wrapped root key.
5. A recipient opens the link, fetches the encrypted manifest and chunk URLs, and decrypts the
   transfer entirely in the browser. Folder downloads can be re-packed into a ZIP locally.
6. Background cleanup periodically removes expired or deleted transfer objects from storage.

Xdrop keeps plaintext file names, paths, contents, and decryption keys off the server. The server
still sees operational metadata such as transfer timestamps, file counts, chunk counts, file
sizes, and rate-limit identifiers.

Key technical details:

- **Crypto model:** The browser generates 32-byte random secrets for the transfer root key and the
  share-link key. HKDF-SHA-256 derives separate AES-256-GCM keys for the manifest and for each
  file, and chunk encryption binds `transferId`, `fileId`, `chunkIndex`, size, and protocol
  version as authenticated data.
- **Chunked uploads:** The server advertises chunk size, file-count, and transfer-size limits to
  the browser. This repo defaults to 8 MiB chunks, up to 100 files, and a 256 MiB encrypted
  transfer size cap.
- **Resume behavior:** Xdrop persists source files locally in OPFS when available and falls back
  to IndexedDB-backed blobs when the staged data is still within the fallback storage limit.
  Resume requests ask the API which chunks already exist so the browser only uploads missing work
  after a refresh or reopen.
- **Sender controls:** The manage token is returned once on creation and stored as a SHA-256 hash
  on the server. Privacy mode can scrub sender-side local controls after upload.
- **Backend responsibilities:** The API never decrypts payloads. It validates transfer state,
  rate-limits endpoints, issues presigned URLs, stores metadata, and cleans up expired or deleted
  objects from storage.

## System Architecture

```mermaid
flowchart LR
  subgraph Sender["Sender browser"]
    Select["Choose files or a folder"]
    Worker["Crypto worker<br/>AES-256-GCM + HKDF-SHA-256"]
    Local["OPFS / IndexedDB<br/>resume state and local controls"]
    Browser["Browser app<br/>React + upload/download runtime"]
    Select --> Worker
    Worker <--> Local
    Browser <--> Worker
    Browser <--> Local
  end

  subgraph Edge["Default Xdrop deployment"]
    nginx["nginx<br/>serves SPA and proxies /api + /xdrop"]
    API["Go API<br/>transfer lifecycle, presigning, cleanup"]
    nginx --> API
  end

  Postgres["PostgreSQL<br/>transfers, files, chunks, hashed manage tokens"]
  Redis["Redis<br/>rate limiting"]
  Storage["S3-compatible storage<br/>encrypted manifest and chunk objects"]
  Receiver["Receiver browser<br/>opens /t/:id#k=..."]

  Browser -->|create/register/finalize| nginx
  Browser -->|presigned PUT uploads| nginx
  API --> Postgres
  API --> Redis
  API -->|presigned PUT/GET URLs| Storage
  nginx -->|/xdrop proxy| Storage
  nginx -->|web app + public API| Receiver
  Receiver -->|presigned GET downloads| nginx
  Receiver -->|decrypts locally with #k fragment| Receiver
```

In the default Docker deployment, nginx serves the built frontend and proxies both `/api` and
`/xdrop`. If `S3_PUBLIC_ENDPOINT` points at a different public object-storage endpoint, presigned
upload and download requests can bypass the nginx proxy while the rest of the architecture stays
the same.

## Deployment

### Recommended Production Topology

For a public deployment, run Xdrop behind a reverse proxy such as Caddy or nginx:

- The reverse proxy terminates HTTPS for your public domain.
- The `xdrop` container listens on a loopback-only host port such as `127.0.0.1:8080`.
- MinIO should not be exposed publicly. Bind MinIO ports to `127.0.0.1` only unless you have
  a specific reason to expose them.
- Set `S3_PUBLIC_ENDPOINT` and `ALLOWED_ORIGINS` to your public site URL, for example
  `https://xdrop.example.com`.

### Step 1: Get the Files

If you only want to run the published image, you do not need to clone the whole repository on the
server.

Download the required deployment files:

```bash
mkdir -p xdrop/infra/minio
cd xdrop
curl -fsSL -o docker-compose.yml \
  https://github.com/xixu-me/xdrop/raw/refs/heads/main/docker-compose.yml
curl -fsSL -o infra/minio/init.sh \
  https://github.com/xixu-me/xdrop/raw/refs/heads/main/infra/minio/init.sh
chmod +x infra/minio/init.sh
```

Optionally, download [`.env.example`](./.env.example) as a reference for supported settings:

```bash
curl -fsSL -o .env.example \
  https://github.com/xixu-me/xdrop/raw/refs/heads/main/.env.example
```

If you want to build your own image, clone the repository instead so Docker can use the full build
context. In most cases, it is better to build in CI or on a separate machine and only pull the
final image onto the server.

### Step 2: Review Configuration

Install Docker and Docker Compose on the server, then review the `xdrop` service environment in
`docker-compose.yml`.

At minimum, update these values for your real deployment:

- `S3_PUBLIC_ENDPOINT`
- `ALLOWED_ORIGINS`

Typical production values look like this:

```yaml
services:
  minio:
    ports:
      - '127.0.0.1:9000:9000'
      - '127.0.0.1:9001:9001'

  xdrop:
    ports:
      - '127.0.0.1:8080:80'
    environment:
      S3_PUBLIC_ENDPOINT: https://xdrop.example.com
      ALLOWED_ORIGINS: https://xdrop.example.com
```

Treat `.env.example` as the reference list of supported settings. Changing `.env.example` alone
does not affect the running stack because the provided Compose file uses inline environment values.

### Step 3: Use the Published Image

```bash
docker compose up -d
```

This uses [`ghcr.io/xixu-me/xdrop:latest`](https://ghcr.io/xixu-me/xdrop).

This is enough when the published image already matches the frontend settings you want.

Important caveats:

- Frontend build-time values such as `VITE_SITE_URL` are baked into the image.
- If your deployment uses a different public domain and you care about canonical URLs, Open Graph
  metadata, JSON-LD, or sitemap generation, use your own rebuilt image instead of the published
  one.

### Step 4: Optional: Use Your Own Prebuilt Image

```bash
XDROP_IMAGE=ghcr.io/your-org/xdrop:latest docker compose up -d
```

### Step 5: Optional: Build Your Own Image

Build your own image when you need different frontend build-time settings:

```bash
git clone https://github.com/xixu-me/xdrop.git
cd xdrop
docker compose -f docker-compose.yml -f docker-compose.build.yml up -d --build
```

Edit the build args in `docker-compose.build.yml` before you run that command.

Example:

```yaml
services:
  xdrop:
    build:
      args:
        VITE_SITE_URL: https://xdrop.example.com
        VITE_API_BASE_URL: /api/v1
```

On low-memory servers, building directly on the host may be slow or fail. In that case, build
elsewhere, push the image to a registry, and deploy it with `XDROP_IMAGE`.

### Step 6: Put Xdrop Behind a Reverse Proxy

Example `Caddyfile`:

```caddyfile
xdrop.example.com {
  encode gzip zstd
  reverse_proxy 127.0.0.1:8080
}
```

Then reload Caddy:

```bash
systemctl reload caddy
```

After the stack starts, open <https://xdrop.example.com>.

### Production Notes

- The final container serves the built frontend with nginx and runs the Go API in the same
  container.
- The stack includes `xdrop`, `postgres`, `redis`, `minio`, and the bucket bootstrap container.
- MinIO is intended to be private in the default single-host deployment.
- Public traffic should normally hit only the reverse proxy on ports `80` and `443`.

## Development

### Prerequisites

- Bun
- Go 1.26+
- Docker / Docker Compose

### Step 1: Install Dependencies

```bash
bun install --frozen-lockfile
```

### Step 2: Start Backing Services

For local development, start PostgreSQL, Redis, and MinIO with Docker:

```bash
docker compose up -d postgres redis minio minio-setup
```

### Step 3: Run the API

```bash
cd apps/api
go run ./cmd/api
```

### Step 4: Run the Web App

From the repo root in a second terminal:

```bash
bun run dev:web
```

Open [http://localhost:5173](http://localhost:5173). During local development, the Vite dev
server proxies:

- `/api` to `http://localhost:8080`
- `/xdrop` to `http://localhost:9000`

This keeps frontend hot reload while talking to the local Go API and MinIO.

## Testing

### Web

```bash
bun run lint:web
bun run typecheck:web
bun run test:web
bun run test:web:coverage
bun run build:web
```

### End-to-End

Install Playwright browsers once if needed:

```bash
bun run test:e2e:install
```

The E2E suite expects Xdrop at `http://localhost:8080` by default and uses the local `postgres`
and `redis` Compose services during the tests. Start the full stack first:

```bash
docker compose -f docker-compose.yml -f docker-compose.build.yml up -d --build
```

Then run the suite:

```bash
bun run test:e2e
```

Set `E2E_BASE_URL` and `E2E_API_URL` if you want to target a different environment.

### API

From `apps/api`:

```bash
go test ./... -coverprofile=coverage.out -covermode=atomic
```

Some API integration tests use Docker-backed testcontainers. If Docker is unavailable, those tests
are skipped and coverage will be lower than CI.

### Formatting

```bash
bun run format
bun run format:check
```

## Project Structure

```text
apps/
  api/        Go API
    cmd/api/  API entrypoint
    internal/ Domain packages
  web/        React frontend
    public/   Static assets
    src/      App, components, features, and utilities
packages/
  shared/     Shared TypeScript constants and helpers
    src/      Shared source files
tests/
  e2e/        Playwright end-to-end tests
infra/        Deployment and container configuration
scripts/      Repository automation and helper scripts
```

## Environment Variables

See [`.env.example`](./.env.example) for the full list. The most important settings are:

- `API_ADDR`
- `DATABASE_URL`
- `REDIS_ADDR`
- `S3_ENDPOINT`
- `S3_PUBLIC_ENDPOINT`
- `S3_BUCKET`
- `ALLOWED_ORIGINS`
- `VITE_API_BASE_URL`
- `VITE_SITE_URL`

## License

AGPL-3.0-only. See [`LICENSE`](./LICENSE).