diff --git a/.github/workflows/depoly.yml b/.github/workflows/depoly.yml new file mode 100644 index 0000000..0ea311c --- /dev/null +++ b/.github/workflows/depoly.yml @@ -0,0 +1,47 @@ +name: Deploy Worker + +on: + push: + branches: + - main + paths-ignore: + - '**.md' + - 'LICENSE' + - '.gitignore' + - '.editorconfig' + - '.vscode/**' + - 'docs/**' + workflow_dispatch: + +jobs: + deploy: + runs-on: ubuntu-latest + timeout-minutes: 60 + name: Deploy to Cloudflare Workers + steps: + - name: Checkout code + uses: actions/checkout@v4 + + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '20' + cache: 'npm' + + - name: Install dependencies + run: npm ci + + - name: Run linter + run: npm run lint + + - name: Run type check + run: npm run type-check + + - name: Run tests + run: npm test + + - name: Build & Deploy Worker + uses: cloudflare/wrangler-action@v3 + with: + apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }} + accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..e6068cb --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,145 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +Xget is a high-performance, secure acceleration engine for developer resources, built on Cloudflare Workers. It provides a unified proxy for accessing code repositories, package managers, AI inference APIs, container registries, and more with enhanced performance through Cloudflare's global edge network. + +**Key Features:** + +- Multi-platform support (50+ platforms including GitHub, npm, PyPI, Docker registries, AI APIs) +- Intelligent caching with 30-minute default TTL +- Automatic retry mechanism (3 retries with linear backoff) +- Enterprise-grade security headers +- HTTP/3 and multi-compression support +- Special handling for Git, Docker, and AI inference protocols + +## Development Commands + +### Testing + +- `npm test` - Run all tests with Vitest (uses Cloudflare Workers test environment) +- `npm run test:run` - Run tests once without watch mode +- `npm run test:watch` - Run tests in watch mode +- `npm run test:ui` - Run tests with Vitest UI +- `npm run test:coverage` - Generate test coverage report (80% threshold for branches/functions/lines/statements) + +### Code Quality + +- `npm run lint` - Run ESLint on src/ and test/ directories +- `npm run lint:fix` - Auto-fix ESLint issues +- `npm run format` - Format code with Prettier +- `npm run format:check` - Check code formatting without making changes +- `npm run type-check` - Run TypeScript type checking (noEmit mode) + +### Development & Deployment + +- `npm run dev` or `npm start` - Start local development server with Wrangler +- `npm run deploy` - Deploy to Cloudflare Workers + +## Architecture + +### Request Flow + +1. **Request Reception** ([src/index.js](src/index.js):275-806) - `handleRequest()` is the main entry point +2. **Validation** ([src/index.js](src/index.js):155-175) - Security validation via `validateRequest()` +3. **Platform Detection** ([src/index.js](src/index.js):320-338) - Identifies platform from URL prefix +4. **Path Transformation** ([src/config/platforms.js](src/config/platforms.js):99-191) - Converts proxy path to target platform path +5. **Cache Check** ([src/index.js](src/index.js):379-409) - Checks Cloudflare cache (skipped for Git/Docker/AI) +6. **Upstream Fetch** ([src/index.js](src/index.js):519-662) - Fetches from origin with retry logic +7. **Response Processing** ([src/index.js](src/index.js):689-724) - URL rewriting for npm/PyPI responses +8. **Cache Storage** ([src/index.js](src/index.js):762-795) - Stores successful 200 responses in cache + +### Core Components + +**[src/index.js](src/index.js)** - Main worker logic + +- `handleRequest()` - Primary request handler with caching, retries, security +- `isGitRequest()`, `isDockerRequest()`, `isAIInferenceRequest()` - Protocol detection functions +- `validateRequest()` - Security validation (method whitelist, path length limits) +- `PerformanceMonitor` - Tracks request performance metrics +- Docker authentication handling ([src/index.js](src/index.js):219-266, 354-371, 579-634) + +**[src/config/platforms.js](src/config/platforms.js)** - Platform definitions and URL transformations + +- `PLATFORMS` object - Maps platform keys to base URLs (50+ platforms) +- `transformPath()` - Unified path transformation logic for all platforms +- Special transformations for crates.io, Homebrew, Jenkins + +**[src/config/index.js](src/config/index.js)** - Configuration management + +- `createConfig()` - Merges environment variables with defaults +- Default values: 30s timeout, 3 retries, 1800s cache, 2048 char max path length + +### Platform Categories + +1. **Code Repositories**: gh (GitHub), gl (GitLab), gitea, codeberg, sf (SourceForge), aosp +2. **Package Managers**: npm, pypi, conda, maven, gradle, nuget, crates, etc. +3. **Container Registries**: cr-ghcr, cr-gcr, cr-mcr, cr-quay, etc. (prefixed with `cr-`) +4. **AI Inference Providers**: ip-openai, ip-anthropic, ip-gemini, etc. (prefixed with `ip-`) +5. **Model/Dataset Platforms**: hf (Hugging Face), civitai +6. **Linux Distributions**: debian, ubuntu, fedora, arch, etc. + +### Special Handling + +**Git Operations** ([src/index.js](src/index.js):73-102) + +- Detected via User-Agent, endpoints (`/info/refs`, `/git-upload-pack`, `/git-receive-pack`), or query params +- Allows POST method, sets Git-specific headers +- Skips caching to ensure real-time data + +**Docker/Container Registries** ([src/index.js](src/index.js):42-65) + +- Detected via `/v2/` paths, User-Agent, or Accept headers +- Handles Docker authentication flow with token fetching +- Supports anonymous access to public repositories +- All requests must use `/cr/` prefix (e.g., `/cr/ghcr/owner/repo`) + +**AI Inference APIs** ([src/index.js](src/index.js):110-146) + +- Detected via `/ip/` paths or common AI endpoints +- Allows POST/PUT/PATCH methods +- Sets JSON content-type and preserves all request headers +- Skips caching for real-time inference + +**URL Rewriting** ([src/index.js](src/index.js):689-724) + +- PyPI: Rewrites `files.pythonhosted.org` URLs to go through `/pypi/files` +- npm: Rewrites `registry.npmjs.org` tarball URLs to go through `/npm/` + +## Testing Practices + +- Tests use Cloudflare Workers test environment via `@cloudflare/vitest-pool-workers` +- Use `SELF.fetch()` to make requests to the worker in tests +- Test files are organized by functionality: platforms, security, integration, performance, range-cache +- Fixtures in [test/fixtures/responses.js](test/fixtures/responses.js) +- Test utilities in [test/helpers/test-utils.js](test/helpers/test-utils.js) + +## Configuration + +Runtime configuration can be overridden via Cloudflare Workers environment variables: + +- `TIMEOUT_SECONDS` - Request timeout (default: 30) +- `MAX_RETRIES` - Max retry attempts (default: 3) +- `RETRY_DELAY_MS` - Delay between retries (default: 1000) +- `CACHE_DURATION` - Cache TTL in seconds (default: 1800) +- `ALLOWED_METHODS` - Comma-separated HTTP methods (default: GET,HEAD) +- `MAX_PATH_LENGTH` - Maximum URL path length (default: 2048) + +## Adding New Platforms + +To add a new platform: + +1. Add entry to `PLATFORMS` object in [src/config/platforms.js](src/config/platforms.js) +2. Add path transformation logic in `transformPath()` if needed (most platforms don't need special handling) +3. Update tests in [test/platforms.test.js](test/platforms.test.js) +4. For platforms requiring special protocol handling (like Git/Docker), add detection function in [src/index.js](src/index.js) + +## Security Considerations + +- All responses include strict security headers (HSTS, X-Frame-Options, CSP, etc.) +- HTTP method whitelist enforced (except for Git/Docker/AI operations) +- Path length validation to prevent excessively long URLs +- 30-second request timeout to prevent resource exhaustion +- Input sanitization in URL transformations diff --git a/README.en.md b/README.en.md index 6c952fe..f841fb7 100644 --- a/README.en.md +++ b/README.en.md @@ -2337,37 +2337,26 @@ sudo systemctl restart containerd #### Deployment Steps -1. **Register Cloudflare Account**: Visit [Cloudflare Workers](https://workers.cloudflare.com/) and register an account +1. **Fork this repository**: + Click the Fork button in the upper right corner of the GitHub page -2. **Installing Wrangler CLI**: +2. **Get Cloudflare Credentials**: + - Visit [Cloudflare Dashboard](https://dash.cloudflare.com/) → My Profile → API Tokens + - Create an API Token with "Edit Cloudflare Workers" permission + - Note your Account ID (visible on the right side of the Workers page) - ```bash - npm install -g wrangler - wrangler login - ``` +3. **Configure GitHub Secrets**: + - Go to your GitHub repository → Settings → Secrets and variables → Actions + - Add the following Secrets: + - `CLOUDFLARE_API_TOKEN`: Your API Token + - `CLOUDFLARE_ACCOUNT_ID`: Your Account ID -3. **Clone Repository**: +4. **Trigger Deployment**: + - Pushing code to the `main` branch will automatically trigger deployment + - Changes only to documentation files (`.md`), `LICENSE`, `.gitignore`, etc. won't trigger deployment + - You can also manually trigger deployment from the GitHub Actions page - ```bash - git clone https://github.com/xixu-me/Xget.git - cd Xget - npm install - ``` - -4. **Configuration Project**: - Edit the `wrangler.toml` file and modify the `name` field to your Worker name: - - ```toml - name = "your-xget-worker" - ``` - -5. **Deploy to Cloudflare Workers**: - - ```bash - npm run deploy - ``` - -6. **Bind custom domain name** (optional): +5. **Bind custom domain name** (optional): Bind your custom domain name in the Cloudflare Workers console Once the deployment is complete, your Xget service will be available at the following address: diff --git a/README.md b/README.md index e1ba626..30cba61 100644 --- a/README.md +++ b/README.md @@ -2347,37 +2347,26 @@ sudo systemctl restart containerd #### 部署步骤 -1. **注册 Cloudflare 账户**：访问 [Cloudflare Workers](https://workers.cloudflare.com/) 并注册账户 +1. **fork 本存储库**： + 点击 GitHub 页面右上角的 Fork 按钮 -2. **安装 Wrangler CLI**： +2. **获取 Cloudflare 凭证**： + - 访问 [Cloudflare Dashboard](https://dash.cloudflare.com/) → My Profile → API Tokens + - 创建一个具有 "Edit Cloudflare Workers" 权限的 API Token + - 记录你的 Account ID（在 Workers 页面右侧可见） - ```bash - npm install -g wrangler - wrangler login - ``` +3. **配置 GitHub Secrets**： + - 进入你的 GitHub 存储库 → Settings → Secrets and variables → Actions + - 添加以下 Secrets： + - `CLOUDFLARE_API_TOKEN`：你的 API Token + - `CLOUDFLARE_ACCOUNT_ID`：你的 Account ID -3. **克隆存储库**： +4. **触发部署**： + - 推送代码到 `main` 分支会自动触发部署 + - 仅修改文档文件（`.md`）、`LICENSE`、`.gitignore` 等不会触发部署 + - 也可以在 GitHub Actions 页面手动触发部署 - ```bash - git clone https://github.com/xixu-me/Xget.git - cd Xget - npm install - ``` - -4. **配置项目**： - 编辑 `wrangler.toml` 文件，修改 `name` 字段为你的 Worker 名称： - - ```toml - name = "your-xget-worker" - ``` - -5. **部署到 Cloudflare Workers**： - - ```bash - npm run deploy - ``` - -6. **绑定自定义域名**（可选）： +5. **绑定自定义域名**（可选）： 在 Cloudflare Workers 控制台中绑定你的自定义域名 部署完成后，你的 Xget 服务将在以下地址可用：