diff --git a/CLAUDE.md b/CLAUDE.md index 76dbdab..f5a094b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,12 +1,19 @@ # CLAUDE.md -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. +This file provides guidance to Claude Code (claude.ai/code) when working with +code in this repository. ## Project Overview -Xget is a high-performance, Cloudflare Workers-based acceleration engine for developer resources. It provides unified acceleration for code repositories (GitHub, GitLab, etc.), package registries (npm, PyPI, Maven, etc.), container registries (Docker Hub, GHCR, etc.), and AI inference APIs (OpenAI, Anthropic, etc.). +Xget is a high-performance, Cloudflare Workers-based acceleration engine for +developer resources. It provides unified acceleration for code repositories +(GitHub, GitLab, etc.), package registries (npm, PyPI, Maven, etc.), container +registries (Docker Hub, GHCR, etc.), and AI inference APIs (OpenAI, Anthropic, +etc.). -The project operates as a reverse proxy that transforms incoming requests to match various platform APIs while adding security headers, caching, retry logic, and performance monitoring. +The project operates as a reverse proxy that transforms incoming requests to +match various platform APIs while adding security headers, caching, retry logic, +and performance monitoring. ## Development Commands @@ -36,21 +43,27 @@ npm run commitlint # Validate the latest commit message ## Commit Messages -- Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) for every commit +- Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) for + every commit - Preferred format: `type(scope): description` - Common types: `feat`, `fix`, `docs`, `refactor`, `perf`, `test`, `chore` -- The repository installs a `commit-msg` hook via `npm install`; do not bypass it unless explicitly required +- The repository installs a `commit-msg` hook via `npm install`; do not bypass + it unless explicitly required ## Pre-Commit Requirements -- Before every commit, run the local CI-equivalent checks from `.github/workflows/ci.yml` -- Required commands: `npm run lint`, `npm run format:check`, `npm run test:run`, and `npm run type-check` +- Before every commit, run the local CI-equivalent checks from + `.github/workflows/ci.yml` +- Required commands: `npm run lint`, `npm run format:check`, `npm run test:run`, + and `npm run type-check` - If any required check fails, do not commit until the failure is resolved -- Apply this rule to every commit, including documentation-only changes, unless the user explicitly asks for a different workflow +- Apply this rule to every commit, including documentation-only changes, unless + the user explicitly asks for a different workflow ### Testing Workflow -- Tests use Vitest with `@cloudflare/vitest-pool-workers` for Workers-specific testing +- Tests use Vitest with `@cloudflare/vitest-pool-workers` for Workers-specific + testing - Run `npm run test:run` before committing to ensure all tests pass - Coverage reports are generated in `coverage/` directory @@ -58,14 +71,23 @@ npm run commitlint # Validate the latest commit message ### Request Flow -1. **Entry Point**: `src/index.js` - Exports default Worker with `fetch()` handler -2. **Validation**: `src/utils/validation.js` - Validates HTTP methods, path length, detects protocol types -3. **Platform Detection**: URL path is parsed to identify platform (e.g., `/gh/` → GitHub) -4. **Path Transformation**: `src/config/platforms.js#transformPath()` converts request paths to upstream URLs -5. **Protocol Handling**: Different handlers for Git, Docker, AI inference requests -6. **Upstream Fetch**: Request forwarded with appropriate headers and retry logic -7. **Response Processing**: URL rewriting for certain platforms (npm, PyPI), cache storage -8. **Security Headers**: Added via `src/utils/security.js` before returning to client +1. **Entry Point**: `src/index.js` - Exports default Worker with `fetch()` + handler +2. **Validation**: `src/utils/validation.js` - Validates HTTP methods, path + length, detects protocol types +3. **Platform Detection**: URL path is parsed to identify platform (e.g., `/gh/` + → GitHub) +4. **Path Transformation**: + `src/routing/platform-transformers.js#transformPath()` converts request paths + to upstream URLs +5. **Protocol Handling**: Different handlers for Git, Docker, AI inference + requests +6. **Upstream Fetch**: Request forwarded with appropriate headers and retry + logic +7. **Response Processing**: URL rewriting for certain platforms (npm, PyPI), + cache storage +8. **Security Headers**: Added via `src/utils/security.js` before returning to + client ### Key Components @@ -77,16 +99,22 @@ npm run commitlint # Validate the latest commit message - `CACHE_DURATION`: Cache TTL (default: 1800s = 30 minutes) - `SECURITY.ALLOWED_METHODS`: HTTP methods (default: GET, HEAD) -- **`platforms.js`**: Platform definitions and path transformations - - `PLATFORMS`: Object mapping platform keys to base URLs - - `SORTED_PLATFORMS`: Pre-sorted keys for efficient matching +- **`platform-catalog.js`**: Platform base URL definitions + - `PLATFORM_CATALOG`: Object mapping platform keys to base URLs + +- **`routing/platform-index.js`**: Pre-sorted keys for efficient matching + - `SORTED_PLATFORMS`: Longest-prefix-first platform matching order + +- **`routing/platform-transformers.js`**: Platform-specific path rewriting - `transformPath()`: Converts request paths to platform-specific URLs - - Special handling for crates.io (adds `/api/v1/crates` prefix), Jenkins (adds `/current/` prefix), and Homebrew + - Special handling for crates.io (adds `/api/v1/crates` prefix) and Jenkins + (adds `/current/` prefix) #### Protocol Handlers (`src/protocols/`) - **`git.js`**: Git protocol detection and header configuration - - Detects Git operations via User-Agent, endpoints (`/info/refs`, `/git-upload-pack`) + - Detects Git operations via User-Agent, endpoints (`/info/refs`, + `/git-upload-pack`) - Handles Git LFS via `Accept: application/vnd.git-lfs+json` - **`docker.js`**: Container registry protocol (OCI/Docker) @@ -117,17 +145,20 @@ npm run commitlint # Validate the latest commit message - Uses Cloudflare Cache API for GET requests (200 OK only) - Cache TTL controlled by `CACHE_DURATION` config - Skips cache for: Git operations, Docker operations, AI inference requests -- Range requests: First checks for range-specific cache, falls back to full content cache +- Range requests: First checks for range-specific cache, falls back to full + content cache ### Special Platform Handling #### npm -- Rewrites `https://registry.npmjs.org/` URLs in JSON responses to point to Xget instance +- Rewrites `https://registry.npmjs.org/` URLs in JSON responses to point to Xget + instance #### PyPI -- Rewrites `https://files.pythonhosted.org` URLs in HTML responses to point to Xget instance +- Rewrites `https://files.pythonhosted.org` URLs in HTML responses to point to + Xget instance - Uses separate `pypi-files` platform for file downloads #### crates.io @@ -153,17 +184,30 @@ npm run commitlint # Validate the latest commit message ``` src/ ├── index.js # Main Worker entry point +├── app/ +│ ├── handle-request.js # Shared request pipeline +│ └── request-context.js # Protocol-aware request classification ├── config/ -│ ├── index.js # Runtime configuration -│ └── platforms.js # Platform definitions +│ ├── index.js # Runtime configuration +│ ├── platform-catalog.js # Platform base URLs +│ └── platforms.js # Compatibility exports ├── protocols/ -│ ├── git.js # Git protocol handler -│ ├── docker.js # Docker/OCI handler -│ └── ai.js # AI inference handler +│ ├── git.js # Git protocol handler +│ ├── docker.js # Docker/OCI handler +│ └── ai.js # AI inference handler +├── response/ +│ └── finalize-response.js # Response shaping and cache writes +├── routing/ +│ ├── platform-index.js # Platform matching order +│ ├── platform-transformers.js +│ └── resolve-target.js # Upstream target resolution +├── upstream/ +│ ├── cache.js # Cache read helpers +│ └── fetch-upstream.js # Upstream transport and retries └── utils/ - ├── validation.js # Request validation - ├── security.js # Security utilities - └── performance.js # Performance monitoring + ├── validation.js # Request validation + ├── security.js # Security utilities + └── performance.js # Performance monitoring test/ ├── features/ # Feature tests @@ -185,8 +229,9 @@ test/ #### Adding a New Platform -1. Add platform entry to `PLATFORMS` object in `src/config/platforms.js` -2. If special path transformation needed, add case in `transformPath()` function +1. Add platform entry to `PLATFORM_CATALOG` in `src/config/platform-catalog.js` +2. If special path transformation needed, add a transformer in + `src/routing/platform-transformers.js` 3. Add platform tests in `test/platforms/` 4. Update README.md with platform documentation @@ -208,7 +253,8 @@ test/ ### Test Structure - **Unit tests** (`test/unit/`): Test individual functions in isolation -- **Feature tests** (`test/features/`): Test specific features (auth, caching, Git, performance) +- **Feature tests** (`test/features/`): Test specific features (auth, caching, + Git, performance) - **Platform tests** (`test/platforms/`): Test platform-specific transformations - **Integration tests** (`test/integration.test.js`): End-to-end request flows @@ -219,7 +265,7 @@ test/ npm run test:run test/unit/platforms.test.js # Run tests matching pattern -npm run test:run -- --grep "Docker" +npm run test:run -- --testNamePattern "Docker" # Run with coverage npm run test:coverage @@ -306,8 +352,9 @@ Configure in Cloudflare Workers dashboard or via `wrangler.toml`: ### Adding a New Platform -1. Add to `PLATFORMS` object in `src/config/platforms.js` -2. If special transformation needed, update `transformPath()` +1. Add to `PLATFORM_CATALOG` in `src/config/platform-catalog.js` +2. If special transformation needed, update + `src/routing/platform-transformers.js` 3. Add test in `test/platforms/` 4. Update README.md documentation 5. Test locally with `npm run dev` @@ -315,7 +362,8 @@ Configure in Cloudflare Workers dashboard or via `wrangler.toml`: ### Debugging Requests 1. Use `npm run dev` to start local server -2. Add `console.log()` statements in `src/index.js` +2. Add `console.log()` statements in `src/app/handle-request.js` or the relevant + extracted pipeline module 3. Check Wrangler dev server output 4. Inspect `X-Performance-Metrics` header in responses @@ -323,5 +371,6 @@ Configure in Cloudflare Workers dashboard or via `wrangler.toml`: 1. Run specific failing test: `npm run test:run test/path/to/test.js` 2. Check mock setup matches actual request pattern -3. Verify platform configuration in `src/config/platforms.js` +3. Verify platform configuration in `src/config/platform-catalog.js` and + `src/routing/platform-transformers.js` 4. Run all tests before committing: `npm run test:run` diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7c9b444..222da44 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -6,7 +6,8 @@ ### 报告问题 -- 使用 [Issue 模板](https://github.com/xixu-me/Xget/issues/new/choose)报告 bug 或提出功能请求 +- 使用 + [Issue 模板](https://github.com/xixu-me/Xget/issues/new/choose)报告 bug 或提出功能请求 - 搜索现有 issues 避免重复报告 - 提供详细的重现步骤和环境信息 @@ -15,7 +16,8 @@ - fork 存储库到您的 GitHub 账户 - 创建功能分支 (`git checkout -b feature/amazing-feature`) - 安装依赖以启用本地 Git hooks (`npm install`) -- 使用 Conventional Commits 提交更改 (`git commit -m 'feat(platforms): add amazing feature'`) +- 使用 Conventional + Commits 提交更改 (`git commit -m 'feat(platforms): add amazing feature'`) - 推送到分支 (`git push origin feature/amazing-feature`) - 创建 Pull Request @@ -50,8 +52,8 @@ npm install # 启动开发服务器 npm run dev -# 运行测试 -npm test +# 单次运行测试 +npm run test:run # 代码格式化 npm run format @@ -101,11 +103,14 @@ function exampleFunction(param1, param2) { ### 运行测试 ```bash -# 运行所有测试 -npm test +# 单次运行所有测试 +npm run test:run # 运行特定测试文件 -npm test -- --grep "platform" +npm run test:run test/platforms/jenkins.test.js + +# 按测试名称筛选 +npm run test:run -- --testNamePattern "platform" # 生成测试覆盖率报告 npm run test:coverage @@ -155,7 +160,8 @@ perf(proxy): optimize request handling performance ### 自动校验 - `npm install` 会自动安装 `commit-msg` hook，在本地阻止不符合规范的提交 -- GitHub Actions 会在 `push` 和 `pull_request` 中再次校验提交消息，防止绕过本地 hook +- GitHub Actions 会在 `push` 和 `pull_request` + 中再次校验提交消息，防止绕过本地 hook ## 🔍 Pull Request 流程 @@ -216,18 +222,19 @@ perf(proxy): optimize request handling performance ### 常见问题 -**Q: 如何添加新平台支持？** -A: 编辑 `src/config/platforms.js` 文件，添加平台配置，然后更新相关文档和测试。 +**Q: 如何添加新平台支持？** A: 在 `src/config/platform-catalog.js` +中添加平台地址；如果需要特殊路径转换，再更新 +`src/routing/platform-transformers.js`，然后补充相关文档和测试。 -**Q: 如何测试 Cloudflare Workers 功能？** -A: 使用 `npm run dev` 启动本地开发服务器，或部署到 Cloudflare Workers 测试环境。 +**Q: 如何测试 Cloudflare Workers 功能？** A: 使用 `npm run dev` +启动本地开发服务器，或部署到 Cloudflare Workers 测试环境。 -**Q: 如何处理跨域问题？** -A: 检查 CORS 配置，确保允许的源和方法设置正确。 +**Q: 如何处理跨域问题？** A: 检查 CORS 配置，确保允许的源和方法设置正确。 ## 📄 许可证 -通过贡献代码，您同意您的贡献将在与存储库相同的 [AGPL-3.0 许可证](LICENSE) 下发布。 +通过贡献代码，您同意您的贡献将在与存储库相同的 [AGPL-3.0 许可证](LICENSE) +下发布。 ## 🙏 致谢 diff --git a/README.md b/README.md index af826da..b8a58b0 100644 --- a/README.md +++ b/README.md @@ -125,9 +125,9 @@ standalone `/xget` directory in a skills installation - `Content-Security-Policy`: Strict content security policy - `Referrer-Policy`: Controls referrer information leakage - **Request Validation Mechanism**: - - HTTP method whitelist: Regular requests limited to GET/HEAD, while - Git/LFS, container registry, AI inference, and Hugging Face API traffic - allow `POST`, `PUT`, `PATCH`, and `DELETE` as needed + - HTTP method whitelist: Regular requests limited to GET/HEAD, while Git/LFS, + container registry, AI inference, and Hugging Face API traffic allow `POST`, + `PUT`, `PATCH`, and `DELETE` as needed - Path length limit: Prevents excessively long URL attacks (max 2048 characters) - Input sanitization: Prevents path traversal and injection attacks @@ -223,11 +223,17 @@ graph TD ```mermaid classDiagram class Worker { - +handleRequest(request) + +fetch(request) } - class Config { - +PLATFORMS + class AppHandler { + +handleRequest(request, env, ctx) + } + class PlatformCatalog { + +PLATFORM_CATALOG + } + class PlatformRouting { +transformPath() + +resolveTarget() } class Validation { +validateRequest() @@ -244,6 +250,13 @@ classDiagram class AIProtocol { +configureAIHeaders() } + class UpstreamPipeline { + +tryReadCachedResponse() + +fetchUpstreamResponse() + } + class ResponsePipeline { + +finalizeResponse() + } class Security { +addSecurityHeaders() } @@ -251,13 +264,18 @@ classDiagram +monitor() } - Worker --> Config - Worker --> Validation - Worker --> GitProtocol - Worker --> DockerProtocol - Worker --> AIProtocol - Worker --> Security - Worker --> Performance + Worker --> AppHandler + AppHandler --> PlatformCatalog + AppHandler --> PlatformRouting + AppHandler --> Validation + AppHandler --> GitProtocol + AppHandler --> DockerProtocol + AppHandler --> AIProtocol + AppHandler --> UpstreamPipeline + AppHandler --> ResponsePipeline + AppHandler --> Security + AppHandler --> Performance + PlatformRouting --> PlatformCatalog ``` ## 📖 URL Conversion Rules @@ -2879,17 +2897,19 @@ export const CONFIG = { ### Adding New Platforms -To add support for new platforms, edit `src/config/platforms.js`: +To add support for new platforms, update the platform catalog and, if needed, +the path transformers: ```javascript -export const PLATFORMS = { +// src/config/platform-catalog.js +export const PLATFORM_CATALOG = { // Existing platforms... + custom: 'https://example.com' +}; - // New platform example - custom: { - base: 'https://example.com', - transform: path => path.replace(/^\/custom\//, '/') - } +// src/routing/platform-transformers.js +const PLATFORM_PATH_TRANSFORMERS = { + custom: path => path.replace(/^\/custom\//, '/') }; ``` diff --git a/README.zh-Hans.md b/README.zh-Hans.md index 28f7e68..61a2d8a 100644 --- a/README.zh-Hans.md +++ b/README.zh-Hans.md @@ -102,7 +102,8 @@ Xget 已受邀入驻 - `Content-Security-Policy`：严格的内容安全策略 - `Referrer-Policy`：控制引用信息泄露 - **请求验证机制**： - - HTTP 方法白名单：常规请求限制为 GET/HEAD，而 Git/LFS、容器镜像仓库、AI 推理和 Hugging Face API 请求会按需允许 `POST`、`PUT`、`PATCH` 和 `DELETE` + - HTTP 方法白名单：常规请求限制为 GET/HEAD，而 Git/LFS、容器镜像仓库、AI 推理和 Hugging + Face API 请求会按需允许 `POST`、`PUT`、`PATCH` 和 `DELETE` - 路径长度限制：防止超长 URL 攻击（最大 2048 字符） - 输入清理：防止路径遍历和注入攻击 - **超时保护**：30 秒请求超时，防止资源耗尽和恶意请求 @@ -189,11 +190,17 @@ graph TD ```mermaid classDiagram class Worker { - +handleRequest(request) + +fetch(request) } - class Config { - +PLATFORMS + class AppHandler { + +handleRequest(request, env, ctx) + } + class PlatformCatalog { + +PLATFORM_CATALOG + } + class PlatformRouting { +transformPath() + +resolveTarget() } class Validation { +validateRequest() @@ -210,6 +217,13 @@ classDiagram class AIProtocol { +configureAIHeaders() } + class UpstreamPipeline { + +tryReadCachedResponse() + +fetchUpstreamResponse() + } + class ResponsePipeline { + +finalizeResponse() + } class Security { +addSecurityHeaders() } @@ -217,13 +231,18 @@ classDiagram +monitor() } - Worker --> Config - Worker --> Validation - Worker --> GitProtocol - Worker --> DockerProtocol - Worker --> AIProtocol - Worker --> Security - Worker --> Performance + Worker --> AppHandler + AppHandler --> PlatformCatalog + AppHandler --> PlatformRouting + AppHandler --> Validation + AppHandler --> GitProtocol + AppHandler --> DockerProtocol + AppHandler --> AIProtocol + AppHandler --> UpstreamPipeline + AppHandler --> ResponsePipeline + AppHandler --> Security + AppHandler --> Performance + PlatformRouting --> PlatformCatalog ``` ## 📖 URL 转换规则 @@ -2793,17 +2812,18 @@ export const CONFIG = { ### 添加新平台 -要添加对新平台的支持，编辑 `src/config/platforms.js`： +要添加对新平台的支持，请更新平台目录；如果需要特殊路径转换，再补充转换器： ```javascript -export const PLATFORMS = { +// src/config/platform-catalog.js +export const PLATFORM_CATALOG = { // 现有平台... + custom: 'https://example.com' +}; - // 新平台示例 - custom: { - base: 'https://example.com', - transform: path => path.replace(/^\/custom\//, '/') - } +// src/routing/platform-transformers.js +const PLATFORM_PATH_TRANSFORMERS = { + custom: path => path.replace(/^\/custom\//, '/') }; ``` diff --git a/README.zh-Hant.md b/README.zh-Hant.md index b39af77..3a6a9c7 100644 --- a/README.zh-Hant.md +++ b/README.zh-Hant.md @@ -102,7 +102,8 @@ Xget 已受邀入駐 - `Content-Security-Policy`：嚴格的內容安全策略 - `Referrer-Policy`：控制參照來源資訊洩露 - **請求驗證機制**： - - HTTP 方法白名單：常規請求限制為 GET/HEAD，而 Git/LFS、容器映像倉庫、AI 推理與 Hugging Face API 請求會按需允許 `POST`、`PUT`、`PATCH` 和 `DELETE` + - HTTP 方法白名單：常規請求限制為 GET/HEAD，而 Git/LFS、容器映像倉庫、AI 推理與 Hugging + Face API 請求會按需允許 `POST`、`PUT`、`PATCH` 和 `DELETE` - 路徑長度限制：防止超長 URL 攻擊（最大 2048 字元） - 輸入清理：防止路徑遍歷和注入攻擊 - **逾時保護**：30 秒請求逾時，防止資源耗盡和惡意請求 @@ -189,11 +190,17 @@ graph TD ```mermaid classDiagram class Worker { - +handleRequest(request) + +fetch(request) } - class Config { - +PLATFORMS + class AppHandler { + +handleRequest(request, env, ctx) + } + class PlatformCatalog { + +PLATFORM_CATALOG + } + class PlatformRouting { +transformPath() + +resolveTarget() } class Validation { +validateRequest() @@ -210,6 +217,13 @@ classDiagram class AIProtocol { +configureAIHeaders() } + class UpstreamPipeline { + +tryReadCachedResponse() + +fetchUpstreamResponse() + } + class ResponsePipeline { + +finalizeResponse() + } class Security { +addSecurityHeaders() } @@ -217,13 +231,18 @@ classDiagram +monitor() } - Worker --> Config - Worker --> Validation - Worker --> GitProtocol - Worker --> DockerProtocol - Worker --> AIProtocol - Worker --> Security - Worker --> Performance + Worker --> AppHandler + AppHandler --> PlatformCatalog + AppHandler --> PlatformRouting + AppHandler --> Validation + AppHandler --> GitProtocol + AppHandler --> DockerProtocol + AppHandler --> AIProtocol + AppHandler --> UpstreamPipeline + AppHandler --> ResponsePipeline + AppHandler --> Security + AppHandler --> Performance + PlatformRouting --> PlatformCatalog ``` ## 📖 URL 轉換規則 @@ -2792,17 +2811,18 @@ export const CONFIG = { ### 新增新平台 -要新增對新平台的支援，編輯 `src/config/platforms.js`： +要新增對新平台的支援，請更新平台目錄；如果需要特殊路徑轉換，再補上轉換器： ```javascript -export const PLATFORMS = { +// src/config/platform-catalog.js +export const PLATFORM_CATALOG = { // 現有平台... + custom: 'https://example.com' +}; - // 新平台範例 - custom: { - base: 'https://example.com', - transform: path => path.replace(/^\/custom\//, '/') - } +// src/routing/platform-transformers.js +const PLATFORM_PATH_TRANSFORMERS = { + custom: path => path.replace(/^\/custom\//, '/') }; ``` diff --git a/adapters/functions/api/index.js b/adapters/functions/api/index.js index 53f4b37..90df507 100644 --- a/adapters/functions/api/index.js +++ b/adapters/functions/api/index.js @@ -16,18 +16,33 @@ * along with this program. If not, see . */ +import { handleRequest } from '../../../src/app/handle-request.js'; +/** + * @typedef {{ + * ALLOWED_METHODS?: string, + * ALLOWED_ORIGINS?: string, + * CACHE_DURATION?: string, + * MAX_PATH_LENGTH?: string, + * MAX_RETRIES?: string, + * RETRY_DELAY_MS?: string, + * TIMEOUT_SECONDS?: string + * }} RuntimeEnv + */ -import { handleRequest } from '../src/index.js'; +/** + * @typedef {{ + * env?: RuntimeEnv, + * geo?: unknown, + * ip?: string, + * waitUntil?: (promise: Promise) => void + * }} FunctionAdapterContext + */ /** * Edge Function handler. * @param {Request} request - Standard Web API Request object - * @param {object} [context] - Platform-specific context (Netlify only) - * @param {object} [context.geo] - Geolocation data (Netlify) - * @param {string} [context.ip] - Client IP address (Netlify) - * @param {object} [context.env] - Environment variables (Netlify) - * @param {(promise: Promise) => void} [context.waitUntil] - Background task extension (Netlify) + * @param {FunctionAdapterContext} [context] - Platform-specific context (Netlify only) * @returns {Promise} Standard Web API Response * @example * // Netlify invokes with context @@ -37,14 +52,17 @@ import { handleRequest } from '../src/index.js'; * handler(request) */ export default async function handler(request, context) { + const runtimeContext = context || /** @type {FunctionAdapterContext} */ ({}); + // Detect runtime environment - const isNetlify = context && (context.geo !== undefined || context.ip !== undefined); + const isNetlify = runtimeContext.geo !== undefined || runtimeContext.ip !== undefined; // Normalize environment variables // Netlify provides context.env, Vercel Edge uses globalThis + /** @type {RuntimeEnv} */ let envSource; if (isNetlify) { - envSource = context.env || {}; + envSource = runtimeContext.env || {}; } else if (typeof process !== 'undefined' && process.env) { // Vercel or Node.js environment envSource = process.env; @@ -60,14 +78,23 @@ export default async function handler(request, context) { CACHE_DURATION: envSource.CACHE_DURATION, ALLOWED_METHODS: envSource.ALLOWED_METHODS, ALLOWED_ORIGINS: envSource.ALLOWED_ORIGINS, - MAX_PATH_LENGTH: envSource.MAX_PATH_LENGTH, + MAX_PATH_LENGTH: envSource.MAX_PATH_LENGTH }; // Create normalized execution context + const waitUntil = isNetlify && runtimeContext.waitUntil ? runtimeContext.waitUntil : null; const ctx = { - waitUntil: isNetlify && context.waitUntil - ? (promise) => context.waitUntil(promise) - : (_promise) => { + waitUntil: waitUntil + ? /** + * Forwards background work in runtimes that support waitUntil. + * @param {Promise} promise + */ + promise => waitUntil(promise) + : ( + /** @type {Promise} */ + _promise + ) => { + void _promise; // No-op on Vercel: background tasks not supported // Cache writes will run synchronously instead console.warn('waitUntil is not supported in Vercel Edge Runtime'); @@ -84,5 +111,5 @@ export default async function handler(request, context) { // Vercel Edge Runtime configuration (ignored by Netlify) export const config = { - runtime: 'edge', + runtime: 'edge' }; diff --git a/adapters/functions/deno.js b/adapters/functions/deno.js index 50b3747..0f7134c 100644 --- a/adapters/functions/deno.js +++ b/adapters/functions/deno.js @@ -16,9 +16,9 @@ * along with this program. If not, see . */ -/* eslint-disable no-undef, no-unused-vars */ +/* eslint-disable no-undef */ -import { handleRequest } from './src/index.js'; +import { handleRequest } from '../../src/app/handle-request.js'; /** * Deno Deploy handler. @@ -40,13 +40,17 @@ async function handler(request) { CACHE_DURATION: Deno.env.get('CACHE_DURATION'), ALLOWED_METHODS: Deno.env.get('ALLOWED_METHODS'), ALLOWED_ORIGINS: Deno.env.get('ALLOWED_ORIGINS'), - MAX_PATH_LENGTH: Deno.env.get('MAX_PATH_LENGTH'), + MAX_PATH_LENGTH: Deno.env.get('MAX_PATH_LENGTH') }; // Create minimal ExecutionContext-like object // Deno Deploy doesn't support waitUntil, so cache writes are synchronous const ctx = { - waitUntil: (promise) => { + waitUntil: ( + /** @type {Promise} */ + promise + ) => { + void promise; // No-op on Deno: background tasks not supported console.warn('waitUntil is not supported in Deno Deploy'); }, @@ -59,5 +63,9 @@ async function handler(request) { return handleRequest(request, env, ctx); } -// Start the server -Deno.serve(handler); +// Start the server only when executing inside Deno. +if (typeof Deno !== 'undefined' && typeof Deno.serve === 'function') { + Deno.serve(handler); +} + +export { handler }; diff --git a/adapters/pages/functions/[[path]].js b/adapters/pages/functions/[[path]].js index 95fb953..fc8cbb9 100644 --- a/adapters/pages/functions/[[path]].js +++ b/adapters/pages/functions/[[path]].js @@ -16,9 +16,18 @@ * along with this program. If not, see . */ +import { handleRequest } from '../../../src/app/handle-request.js'; - -import { handleRequest } from '../src/index.js'; +/** + * @typedef {{ + * request: Request, + * env: Record, + * params: object, + * waitUntil: (promise: Promise) => void, + * next: () => Promise, + * data: object + * }} PagesFunctionContext + */ /** * Pages Function handler for all routes. @@ -31,13 +40,7 @@ import { handleRequest } from '../src/index.js'; * The [[path]] syntax in the filename creates a catch-all route that matches * any path, allowing this single function to handle all requests to the Pages * application. - * @param {object} context - Pages Function context - * @param {Request} context.request - The incoming HTTP request - * @param {object} context.env - Environment variables and bindings (KV, secrets, etc.) - * @param {object} context.params - Route parameters (path segments from [[path]]) - * @param {(promise: Promise) => void} context.waitUntil - Extend function execution for background tasks - * @param {() => Promise} context.next - Call next middleware in chain (not used here) - * @param {object} context.data - Shared data between functions + * @param {PagesFunctionContext} context - Pages Function context * @returns {Promise} The HTTP response to return to the client * @example * // This is called automatically by Pages diff --git a/skills/xget/references/REFERENCE.md b/skills/xget/references/REFERENCE.md index cd1c427..6fee1a9 100644 --- a/skills/xget/references/REFERENCE.md +++ b/skills/xget/references/REFERENCE.md @@ -63,7 +63,7 @@ profile before retrying commands. The authoritative platform list for this skill comes from: -`https://raw.gitcode.com/xixu-me/xget/raw/main/src/config/platforms.js` +`https://raw.gitcode.com/xixu-me/xget/raw/main/src/config/platform-catalog.js` Fetch it from the repository root with: @@ -116,9 +116,9 @@ examples back: - `.env`, SDK initialization code, shell profile files Treat phrasing like "configure this", "change it", "wire it in", "switch to -Xget", "run this", "fix it", or "deploy it" as a cue to execute. Only fall -back to example commands when the user explicitly asks for examples or a -missing fact prevents safe execution. +Xget", "run this", "fix it", or "deploy it" as a cue to execute. Only fall back +to example commands when the user explicitly asks for examples or a missing fact +prevents safe execution. ## Deployment diff --git a/skills/xget/scripts/xget.mjs b/skills/xget/scripts/xget.mjs index 58db1c0..a27e148 100644 --- a/skills/xget/scripts/xget.mjs +++ b/skills/xget/scripts/xget.mjs @@ -4,9 +4,9 @@ import { get } from 'node:https'; import { relative } from 'node:path'; import process from 'node:process'; import { pathToFileURL } from 'node:url'; -import vm from 'node:vm'; -const DEFAULT_SOURCE_URL = 'https://raw.gitcode.com/xixu-me/xget/raw/main/src/config/platforms.js'; +const DEFAULT_SOURCE_URL = + 'https://raw.gitcode.com/xixu-me/xget/raw/main/src/config/platform-catalog.js'; const DEFAULT_README_URL = 'https://raw.githubusercontent.com/xixu-me/xget/main/README.md'; const DEFAULT_BASE_PLACEHOLDER = 'https://xget.example.com'; @@ -90,7 +90,7 @@ Commands: help Show this message. Global options: - --source-url URL Override the remote platforms.js URL. + --source-url URL Override the remote platform source URL. --format FORMAT json (default), text, or table when supported. --help Show command help. @@ -178,6 +178,39 @@ function fail(message, code = 1) { process.exit(code); } +/** + * Parses a platform map object literal from repository source. + * Supports the simple `key: 'value'` form used by the Xget platform catalog. + * @param {string} objectSource + * @returns {Record} + */ +function parsePlatformMapObject(objectSource) { + /** @type {Record} */ + const platforms = {}; + + for (const rawLine of objectSource.split(/\r?\n/)) { + const line = rawLine.trim(); + + if (!line || line === '{' || line === '}' || line.startsWith('//')) { + continue; + } + + const match = line.match( + /^(?:'([^']+)'|"([^"]+)"|([A-Za-z0-9_-]+))\s*:\s*(?:'([^']*)'|"([^"]*)")\s*,?$/ + ); + + if (!match) { + throw new Error(`unsupported platform entry: ${line}`); + } + + const key = match[1] || match[2] || match[3]; + const value = match[4] || match[5] || ''; + platforms[key] = value; + } + + return platforms; +} + /** * @param {string} url * @returns {Promise} @@ -216,17 +249,31 @@ function httpGet(url) { * @returns {Record} */ export function extractPlatformsModule(jsSource) { - const match = jsSource.match(/export const PLATFORMS = (\{[\s\S]*?\n\});/); + const platformExportPatterns = [ + { + name: 'PLATFORM_CATALOG', + pattern: /export const PLATFORM_CATALOG = (\{[\s\S]*?\n\});/ + }, + { + name: 'PLATFORMS', + pattern: /export const PLATFORMS = (\{[\s\S]*?\n\});/ + } + ]; - if (!match) { - fail('Could not find `export const PLATFORMS = {...}` in the remote source.'); + for (const { name, pattern } of platformExportPatterns) { + const match = jsSource.match(pattern); + if (!match) { + continue; + } + + try { + return parsePlatformMapObject(match[1]); + } catch (error) { + fail(`Could not parse remote ${name} object: ${getErrorMessage(error)}`); + } } - try { - return vm.runInNewContext(`(${match[1]})`); - } catch (error) { - fail(`Could not parse remote PLATFORMS object: ${getErrorMessage(error)}`); - } + fail('Could not find `export const PLATFORM_CATALOG = {...}` or `PLATFORMS = {...}`.'); } /** @@ -507,6 +554,7 @@ function isCodeFenceDelimiter(line) { * @returns {MarkdownHeading[]} */ function collectMarkdownHeadings(lines) { + /** @type {Array} */ const stack = []; let inCodeFence = false; @@ -528,7 +576,7 @@ function collectMarkdownHeadings(lines) { let parent = null; for (let level = heading.level - 1; level >= 1; level -= 1) { if (stack[level]) { - parent = stack[level]; + parent = stack[level] ?? null; break; } } diff --git a/src/app/handle-request.js b/src/app/handle-request.js new file mode 100644 index 0000000..6c20365 --- /dev/null +++ b/src/app/handle-request.js @@ -0,0 +1,186 @@ +/** + * Xget - High-performance acceleration engine for developer resources + * Copyright (C) 2025 Xi Xu + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + */ + +import { createRequestContext } from './request-context.js'; +import { + createHomepageRedirect, + normalizeEffectivePath, + resolveTarget +} from '../routing/resolve-target.js'; +import { finalizeResponse } from '../response/finalize-response.js'; +import { handleDockerAuth } from '../protocols/docker.js'; +import { getDefaultCache, tryReadCachedResponse } from '../upstream/cache.js'; +import { fetchUpstreamResponse } from '../upstream/fetch-upstream.js'; +import { PerformanceMonitor, addPerformanceHeaders } from '../utils/performance.js'; +import { addCorsHeaders, addSecurityHeaders, createErrorResponse } from '../utils/security.js'; +import { getAllowedMethods, isProtocolRequest, validateRequest } from '../utils/validation.js'; + +/** + * Main request handler with comprehensive caching, retry logic, and security measures. + * @param {Request} request - The incoming HTTP request + * @param {Record} env - Cloudflare Workers environment variables for runtime config overrides + * @param {ExecutionContext} ctx - Cloudflare Workers execution context for background tasks + * @returns {Promise} The HTTP response with appropriate headers and body + */ +export async function handleRequest(request, env, ctx) { + let response; + const monitor = new PerformanceMonitor(); + const requestContext = createRequestContext(request, env); + const { config, isCorsPreflight, isDocker, url } = requestContext; + + try { + if (isCorsPreflight) { + const requestedMethod = request.headers.get('Access-Control-Request-Method') || ''; + const allowedMethods = getAllowedMethods( + new Request(request.url, { method: requestedMethod || 'GET' }), + url, + config + ); + + if (!allowedMethods.includes(requestedMethod)) { + response = createErrorResponse('Method not allowed', 405); + } else { + const headers = addCorsHeaders(new Headers(), request, config); + if (!headers.has('Access-Control-Allow-Origin')) { + response = createErrorResponse('Origin not allowed', 403); + } else { + headers.set('Access-Control-Allow-Methods', allowedMethods.join(', ')); + headers.set('Access-Control-Max-Age', '86400'); + addSecurityHeaders(headers); + response = new Response(null, { status: 204, headers }); + } + } + } + + // Handle Docker API version check + else if (isDocker && (url.pathname === '/v2/' || url.pathname === '/v2')) { + const headers = new Headers({ + 'Docker-Distribution-Api-Version': 'registry/2.0', + 'Content-Type': 'application/json' + }); + addSecurityHeaders(headers); + response = new Response('{}', { status: 200, headers }); + } + // Redirect root path or invalid platforms to GitHub repository + else if (url.pathname === '/' || url.pathname === '') { + response = createHomepageRedirect(); + } else { + const validation = validateRequest(request, url, config, requestContext); + if (!validation.valid) { + response = createErrorResponse( + validation.error || 'Validation failed', + validation.status || 400 + ); + } else { + const normalizedPath = normalizeEffectivePath(url, isDocker); + let effectivePath = url.pathname; + + if ('response' in normalizedPath) { + const { response: normalizedResponse } = normalizedPath; + response = normalizedResponse; + } else { + const { effectivePath: normalizedEffectivePath } = normalizedPath; + effectivePath = normalizedEffectivePath; + } + + if (!response) { + // Handle Docker authentication explicitly + if ( + isDocker && + (url.pathname === '/v2/auth' || /^\/cr\/[^/]+\/v2\/auth\/?$/.test(url.pathname)) + ) { + response = await handleDockerAuth(request, url, config); + } else { + const resolvedTarget = resolveTarget(url, effectivePath, config.PLATFORMS); + + if ('response' in resolvedTarget) { + const { response: targetResponse } = resolvedTarget; + response = targetResponse; + } else { + const { cacheTargetUrl, platform, targetUrl } = resolvedTarget; + const authorization = request.headers.get('Authorization'); + const hasSensitiveHeaders = Boolean( + authorization || + request.headers.get('Cookie') || + request.headers.get('Proxy-Authorization') + ); + const canUseCache = request.method === 'GET' || request.method === 'HEAD'; + const shouldPassthroughRequest = isProtocolRequest(requestContext) || !canUseCache; + const cache = getDefaultCache(); + + response = await tryReadCachedResponse({ + cache, + cacheTargetUrl, + canUseCache, + hasSensitiveHeaders, + monitor, + request, + requestContext + }); + + if (!response) { + const { + response: upstreamResponse, + responseGeneratedLocally: upstreamResponseGeneratedLocally + } = await fetchUpstreamResponse({ + authorization, + canUseCache, + config, + effectivePath, + monitor, + platform, + request, + requestContext, + shouldPassthroughRequest, + targetUrl + }); + response = await finalizeResponse({ + cache, + cacheTargetUrl, + canUseCache, + config, + ctx, + effectivePath, + hasSensitiveHeaders, + monitor, + platform, + request, + requestContext, + response: upstreamResponse, + responseGeneratedLocally: upstreamResponseGeneratedLocally, + url + }); + } + } + } + } + } + } + } catch (error) { + console.error('Error handling request:', error); + response = createErrorResponse('Internal Server Error', 500); + } + + // Ensure performance headers are added to the final response + monitor.mark('complete'); + + const responseWithCors = (() => { + const headers = addCorsHeaders(new Headers(response.headers), request, config); + return new Response(response.body, { + status: response.status, + statusText: response.statusText, + headers + }); + })(); + + return isProtocolRequest(requestContext) + ? responseWithCors + : addPerformanceHeaders(responseWithCors, monitor); +} diff --git a/src/app/request-context.js b/src/app/request-context.js new file mode 100644 index 0000000..ae7d323 --- /dev/null +++ b/src/app/request-context.js @@ -0,0 +1,38 @@ +import { CONFIG, createConfig } from '../config/index.js'; +import { getRequestTraits } from '../utils/validation.js'; + +/** + * Builds the shared request context used by all runtime adapters. + * @param {Request} request + * @param {Record} env + * @returns {{ + * config: import('../config/index.js').ApplicationConfig, + * env: Record, + * isAI: boolean, + * isCorsPreflight: boolean, + * isDocker: boolean, + * isGit: boolean, + * isGitLFS: boolean, + * isHF: boolean, + * request: Request, + * url: URL + * }} Request context with parsed config, URL, and protocol traits. + */ +export function createRequestContext(request, env) { + const runtimeEnv = env && typeof env === 'object' ? env : {}; + const config = env === undefined ? CONFIG : createConfig(runtimeEnv); + const url = new URL(request.url); + const traits = getRequestTraits(request, url); + + return { + ...traits, + config, + env: runtimeEnv, + isCorsPreflight: + request.method === 'OPTIONS' && + request.headers.has('Origin') && + request.headers.has('Access-Control-Request-Method'), + request, + url + }; +} diff --git a/src/config/index.js b/src/config/index.js index 999bf79..ae61b76 100644 --- a/src/config/index.js +++ b/src/config/index.js @@ -16,7 +16,7 @@ * along with this program. If not, see . */ -import { PLATFORMS } from './platforms.js'; +import { PLATFORMS } from './platform-catalog.js'; /** * Security-related configuration options for request validation and CORS. diff --git a/src/config/platform-catalog.js b/src/config/platform-catalog.js new file mode 100644 index 0000000..a5a88ec --- /dev/null +++ b/src/config/platform-catalog.js @@ -0,0 +1,121 @@ +/** + * Xget - High-performance acceleration engine for developer resources + * Copyright (C) 2025 Xi Xu + * + * This program is free software: you can redistribute it and/or modify + * it under the terms of the GNU Affero General Public License as published by + * the Free Software Foundation, either version 3 of the License, or + * (at your option) any later version. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU Affero General Public License for more details. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ + +/** + * Platform base URLs used by request routing. + * @type {{ [key: string]: string }} + */ +export const PLATFORM_CATALOG = { + // Code Repositories & Version Control + gh: 'https://github.com', + gist: 'https://gist.github.com', + gl: 'https://gitlab.com', + gitea: 'https://gitea.com', + codeberg: 'https://codeberg.org', + sf: 'https://sourceforge.net', + aosp: 'https://android.googlesource.com', + hf: 'https://huggingface.co', + civitai: 'https://civitai.com', + + // Package Managers + npm: 'https://registry.npmjs.org', + pypi: 'https://pypi.org', + 'pypi-files': 'https://files.pythonhosted.org', + conda: 'https://repo.anaconda.com', + 'conda-community': 'https://conda.anaconda.org', + maven: 'https://repo1.maven.org', + apache: 'https://downloads.apache.org', + gradle: 'https://plugins.gradle.org', + homebrew: 'https://github.com/Homebrew', + 'homebrew-api': 'https://formulae.brew.sh/api', + 'homebrew-bottles': 'https://ghcr.io', + rubygems: 'https://rubygems.org', + cran: 'https://cran.r-project.org', + cpan: 'https://www.cpan.org', + ctan: 'https://tug.ctan.org', + golang: 'https://proxy.golang.org', + nuget: 'https://api.nuget.org', + crates: 'https://crates.io', + packagist: 'https://repo.packagist.org', + flathub: 'https://dl.flathub.org', + + // Linux Distributions + debian: 'https://deb.debian.org', + ubuntu: 'https://archive.ubuntu.com', + fedora: 'https://dl.fedoraproject.org', + rocky: 'https://download.rockylinux.org', + opensuse: 'https://download.opensuse.org', + arch: 'https://geo.mirror.pkgbuild.com', + + // Other Resources + arxiv: 'https://arxiv.org', + fdroid: 'https://f-droid.org', + jenkins: 'https://updates.jenkins.io', + + // AI Inference Providers + 'ip-openai': 'https://api.openai.com', + 'ip-anthropic': 'https://api.anthropic.com', + 'ip-gemini': 'https://generativelanguage.googleapis.com', + 'ip-vertexai': 'https://aiplatform.googleapis.com', + 'ip-cohere': 'https://api.cohere.ai', + 'ip-mistralai': 'https://api.mistral.ai', + 'ip-xai': 'https://api.x.ai', + 'ip-githubmodels': 'https://models.github.ai', + 'ip-nvidiaapi': 'https://integrate.api.nvidia.com', + 'ip-perplexity': 'https://api.perplexity.ai', + 'ip-braintrust': 'https://api.braintrust.dev', + 'ip-groq': 'https://api.groq.com', + 'ip-cerebras': 'https://api.cerebras.ai', + 'ip-sambanova': 'https://api.sambanova.ai', + 'ip-siray': 'https://api.siray.ai', + 'ip-huggingface': 'https://router.huggingface.co', + 'ip-together': 'https://api.together.xyz', + 'ip-replicate': 'https://api.replicate.com', + 'ip-fireworks': 'https://api.fireworks.ai', + 'ip-nebius': 'https://api.studio.nebius.ai', + 'ip-jina': 'https://api.jina.ai', + 'ip-voyageai': 'https://api.voyageai.com', + 'ip-falai': 'https://fal.run', + 'ip-novita': 'https://api.novita.ai', + 'ip-burncloud': 'https://ai.burncloud.com', + 'ip-openrouter': 'https://openrouter.ai', + 'ip-poe': 'https://api.poe.com', + 'ip-featherlessai': 'https://api.featherless.ai', + 'ip-hyperbolic': 'https://api.hyperbolic.xyz', + + // Container Registries + 'cr-docker': 'https://registry-1.docker.io', + 'cr-quay': 'https://quay.io', + 'cr-gcr': 'https://gcr.io', + 'cr-mcr': 'https://mcr.microsoft.com', + 'cr-ecr': 'https://public.ecr.aws', + 'cr-ghcr': 'https://ghcr.io', + 'cr-gitlab': 'https://registry.gitlab.com', + 'cr-redhat': 'https://registry.redhat.io', + 'cr-oracle': 'https://container-registry.oracle.com', + 'cr-cloudsmith': 'https://docker.cloudsmith.io', + 'cr-digitalocean': 'https://registry.digitalocean.com', + 'cr-vmware': 'https://projects.registry.vmware.com', + 'cr-k8s': 'https://registry.k8s.io', + 'cr-heroku': 'https://registry.heroku.com', + 'cr-suse': 'https://registry.suse.com', + 'cr-opensuse': 'https://registry.opensuse.org', + 'cr-gitpod': 'https://registry.gitpod.io' +}; + +export const PLATFORMS = PLATFORM_CATALOG; diff --git a/src/config/platforms.js b/src/config/platforms.js index 2c77928..746e46a 100644 --- a/src/config/platforms.js +++ b/src/config/platforms.js @@ -1,395 +1,11 @@ /** - * Xget - High-performance acceleration engine for developer resources - * Copyright (C) 2025 Xi Xu + * Compatibility exports for platform configuration and routing helpers. * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the GNU Affero General Public License as published by - * the Free Software Foundation, either version 3 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU Affero General Public License for more details. - * - * You should have received a copy of the GNU Affero General Public License - * along with this program. If not, see . + * New code should prefer: + * - `src/config/platform-catalog.js` for base URL data + * - `src/routing/platform-index.js` for matching order + * - `src/routing/platform-transformers.js` for path normalization */ - -/** - * Configuration object mapping platform prefixes to their base URLs. - * - * Supports 40+ platforms across multiple categories: - * - * **Code Repositories & Version Control:** - * - `gh` - GitHub (github.com) - * - `gist` - GitHub Gist (gist.github.com) - * - `gl` - GitLab (gitlab.com) - * - `gitea` - Gitea (gitea.com) - * - `codeberg` - Codeberg (codeberg.org) - * - `sf` - SourceForge (sourceforge.net) - * - `aosp` - Android Open Source Project (android.googlesource.com) - * - `hf` - Hugging Face (huggingface.co) - * - `civitai` - Civitai (civitai.com) - * - * **Package Managers:** - * - `npm` - Node Package Manager (registry.npmjs.org) - * - `pypi` - Python Package Index (pypi.org) - * - `pypi-files` - PyPI file hosting (files.pythonhosted.org) - * - `conda` - Anaconda packages (repo.anaconda.com) - * - `conda-community` - Community Conda (conda.anaconda.org) - * - `maven` - Maven Central (repo1.maven.org) - * - `apache` - Apache downloads (downloads.apache.org) - * - `gradle` - Gradle plugins (plugins.gradle.org) - * - `homebrew` - Homebrew repositories (github.com/Homebrew) - * - `homebrew-api` - Homebrew API (formulae.brew.sh/api) - * - `homebrew-bottles` - Homebrew bottles (ghcr.io) - * - `rubygems` - RubyGems (rubygems.org) - * - `cran` - R CRAN (cran.r-project.org) - * - `cpan` - Perl CPAN (cpan.org) - * - `ctan` - TeX CTAN (tug.ctan.org) - * - `golang` - Go proxy (proxy.golang.org) - * - `nuget` - NuGet (api.nuget.org) - * - `crates` - Rust crates.io (crates.io) - * - `packagist` - PHP Packagist (repo.packagist.org) - * - `flathub` - Flathub Flatpak repository (dl.flathub.org) - * - * **Linux Distributions:** - * - `debian` - Debian packages (deb.debian.org) - * - `ubuntu` - Ubuntu archives (archive.ubuntu.com) - * - `fedora` - Fedora downloads (dl.fedoraproject.org) - * - `rocky` - Rocky Linux (download.rockylinux.org) - * - `opensuse` - openSUSE downloads (download.opensuse.org) - * - `arch` - Arch Linux mirrors (geo.mirror.pkgbuild.com) - * - * **Other Resources:** - * - `arxiv` - arXiv papers (arxiv.org) - * - `fdroid` - F-Droid Android apps (f-droid.org) - * - `jenkins` - Jenkins plugins (updates.jenkins.io) - * - * **AI Inference Providers (prefix: ip-):** - * - `ip-openai` - OpenAI API - * - `ip-anthropic` - Claude API - * - `ip-gemini` - Google Gemini API - * - `ip-vertexai` - Google Vertex AI - * - `ip-cohere` - Cohere API - * - `ip-mistralai` - Mistral AI - * - `ip-xai` - X.AI - * - `ip-githubmodels` - GitHub Models - * - `ip-nvidiaapi` - NVIDIA API - * - `ip-perplexity` - Perplexity AI - * - `ip-braintrust` - Braintrust - * - `ip-groq` - Groq - * - `ip-cerebras` - Cerebras - * - `ip-sambanova` - SambaNova - * - `ip-siray` - Siray AI - * - `ip-huggingface` - Hugging Face Inference - * - `ip-together` - Together AI - * - `ip-replicate` - Replicate - * - `ip-fireworks` - Fireworks AI - * - `ip-nebius` - Nebius AI - * - `ip-jina` - Jina AI - * - `ip-voyageai` - Voyage AI - * - `ip-falai` - Fal AI - * - `ip-novita` - Novita AI - * - `ip-burncloud` - BurnCloud AI - * - `ip-openrouter` - OpenRouter - * - `ip-poe` - Poe - * - `ip-featherlessai` - Featherless AI - * - `ip-hyperbolic` - Hyperbolic - * - * **Container Registries (prefix: cr-):** - * - `cr-docker` - Docker Hub (registry-1.docker.io) - * - `cr-quay` - Quay.io - * - `cr-gcr` - Google Container Registry - * - `cr-mcr` - Microsoft Container Registry - * - `cr-ecr` - AWS Elastic Container Registry (public) - * - `cr-ghcr` - GitHub Container Registry - * - `cr-gitlab` - GitLab Container Registry - * - `cr-redhat` - Red Hat Registry - * - `cr-oracle` - Oracle Container Registry - * - `cr-cloudsmith` - Cloudsmith Docker Registry - * - `cr-digitalocean` - DigitalOcean Container Registry - * - `cr-vmware` - VMware Harbor - * - `cr-k8s` - Kubernetes Registry - * - `cr-heroku` - Heroku Container Registry - * - `cr-suse` - SUSE Registry - * - `cr-opensuse` - openSUSE Registry - * - `cr-gitpod` - Gitpod Registry - * @type {{ [key: string]: string }} - * @example - * // Access GitHub base URL - * const githubUrl = PLATFORMS.gh; // 'https://github.com' - * @example - * // Access OpenAI API base URL - * const openaiUrl = PLATFORMS['ip-openai']; // 'https://api.openai.com' - * @example - * // Check if platform exists - * if (PLATFORMS.npm) { - * console.log('npm registry available'); - * } - */ -export const PLATFORMS = { - // Code Repositories & Version Control - gh: 'https://github.com', - gist: 'https://gist.github.com', - gl: 'https://gitlab.com', - gitea: 'https://gitea.com', - codeberg: 'https://codeberg.org', - sf: 'https://sourceforge.net', - aosp: 'https://android.googlesource.com', - hf: 'https://huggingface.co', - civitai: 'https://civitai.com', - - // Package Managers - npm: 'https://registry.npmjs.org', - pypi: 'https://pypi.org', - 'pypi-files': 'https://files.pythonhosted.org', - conda: 'https://repo.anaconda.com', - 'conda-community': 'https://conda.anaconda.org', - maven: 'https://repo1.maven.org', - apache: 'https://downloads.apache.org', - gradle: 'https://plugins.gradle.org', - homebrew: 'https://github.com/Homebrew', - 'homebrew-api': 'https://formulae.brew.sh/api', - 'homebrew-bottles': 'https://ghcr.io', - rubygems: 'https://rubygems.org', - cran: 'https://cran.r-project.org', - cpan: 'https://www.cpan.org', - ctan: 'https://tug.ctan.org', - golang: 'https://proxy.golang.org', - nuget: 'https://api.nuget.org', - crates: 'https://crates.io', - packagist: 'https://repo.packagist.org', - flathub: 'https://dl.flathub.org', - - // Linux Distributions - debian: 'https://deb.debian.org', - ubuntu: 'https://archive.ubuntu.com', - fedora: 'https://dl.fedoraproject.org', - rocky: 'https://download.rockylinux.org', - opensuse: 'https://download.opensuse.org', - arch: 'https://geo.mirror.pkgbuild.com', - - // Other Resources - arxiv: 'https://arxiv.org', - fdroid: 'https://f-droid.org', - jenkins: 'https://updates.jenkins.io', - - // AI Inference Providers - 'ip-openai': 'https://api.openai.com', - 'ip-anthropic': 'https://api.anthropic.com', - 'ip-gemini': 'https://generativelanguage.googleapis.com', - 'ip-vertexai': 'https://aiplatform.googleapis.com', - 'ip-cohere': 'https://api.cohere.ai', - 'ip-mistralai': 'https://api.mistral.ai', - 'ip-xai': 'https://api.x.ai', - 'ip-githubmodels': 'https://models.github.ai', - 'ip-nvidiaapi': 'https://integrate.api.nvidia.com', - 'ip-perplexity': 'https://api.perplexity.ai', - 'ip-braintrust': 'https://api.braintrust.dev', - 'ip-groq': 'https://api.groq.com', - 'ip-cerebras': 'https://api.cerebras.ai', - 'ip-sambanova': 'https://api.sambanova.ai', - 'ip-siray': 'https://api.siray.ai', - 'ip-huggingface': 'https://router.huggingface.co', - 'ip-together': 'https://api.together.xyz', - 'ip-replicate': 'https://api.replicate.com', - 'ip-fireworks': 'https://api.fireworks.ai', - 'ip-nebius': 'https://api.studio.nebius.ai', - 'ip-jina': 'https://api.jina.ai', - 'ip-voyageai': 'https://api.voyageai.com', - 'ip-falai': 'https://fal.run', - 'ip-novita': 'https://api.novita.ai', - 'ip-burncloud': 'https://ai.burncloud.com', - 'ip-openrouter': 'https://openrouter.ai', - 'ip-poe': 'https://api.poe.com', - 'ip-featherlessai': 'https://api.featherless.ai', - 'ip-hyperbolic': 'https://api.hyperbolic.xyz', - - // Container Registries - 'cr-docker': 'https://registry-1.docker.io', - 'cr-quay': 'https://quay.io', - 'cr-gcr': 'https://gcr.io', - 'cr-mcr': 'https://mcr.microsoft.com', - 'cr-ecr': 'https://public.ecr.aws', - 'cr-ghcr': 'https://ghcr.io', - 'cr-gitlab': 'https://registry.gitlab.com', - 'cr-redhat': 'https://registry.redhat.io', - 'cr-oracle': 'https://container-registry.oracle.com', - 'cr-cloudsmith': 'https://docker.cloudsmith.io', - 'cr-digitalocean': 'https://registry.digitalocean.com', - 'cr-vmware': 'https://projects.registry.vmware.com', - 'cr-k8s': 'https://registry.k8s.io', - 'cr-heroku': 'https://registry.heroku.com', - 'cr-suse': 'https://registry.suse.com', - 'cr-opensuse': 'https://registry.opensuse.org', - 'cr-gitpod': 'https://registry.gitpod.io' -}; - -/** - * Pre-computed sorted platforms keys for efficient matching. - * Sorted by key length (descending) to prioritize more specific paths. - */ -export const SORTED_PLATFORMS = Object.keys(PLATFORMS).sort((a, b) => { - const pathA = `/${a.replace('-', '/')}/`; - const pathB = `/${b.replace('-', '/')}/`; - return pathB.length - pathA.length; -}); - -/** - * Unified path transformation function that converts request paths to platform-specific URLs. - * - * This function performs two primary operations: - * 1. Strips the platform prefix from the request path - * 2. Applies platform-specific transformations (crates.io, Homebrew, Jenkins) - * - * The function handles special cases for platforms that require API path prefixes or - * URL structure modifications to match their upstream API conventions. - * @param {string} path - The original request path including platform prefix (e.g., '/gh/user/repo') - * @param {string} platformKey - The platform key from PLATFORMS object (e.g., 'gh', 'crates', 'npm') - * @returns {string} The transformed path ready for upstream request - * @example - * // Basic transformation - strips platform prefix - * transformPath('/gh/torvalds/linux', 'gh') - * // Returns: '/torvalds/linux' - * @example - * // crates.io API transformation - adds API prefix - * transformPath('/crates/serde/1.0.0/download', 'crates') - * // Returns: '/api/v1/crates/serde/1.0.0/download' - * @example - * // crates.io search endpoint - * transformPath('/crates/?q=tokio', 'crates') - * // Returns: '/api/v1/crates?q=tokio' - * @example - * // Jenkins update center transformation - * transformPath('/jenkins/update-center.json', 'jenkins') - * // Returns: '/current/update-center.json' - * @example - * // Homebrew API paths (pass-through) - * transformPath('/homebrew/api/formula/git.json', 'homebrew-api') - * // Returns: '/formula/git.json' - * @example - * // Unknown platform (no transformation) - * transformPath('/unknown/path', 'nonexistent') - * // Returns: '/unknown/path' - * @example - * // Multi-part platform key (hyphens converted to slashes) - * transformPath('/ip/openai/v1/chat/completions', 'ip-openai') - * // Returns: '/v1/chat/completions' - */ -export function transformPath(path, platformKey) { - // Return original path if platform doesn't exist - if (!PLATFORMS[platformKey]) { - return path; - } - - // Convert platform key to path prefix (e.g., 'ip-openai' -> '/ip/openai/') - const prefix = `/${platformKey.replace(/-/g, '/')}/`; - let transformedPath = path.replace( - new RegExp(`^${prefix.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}`), - '/' - ); - - /** - * Special handling for crates.io API paths - * - * The Rust package registry requires an `/api/v1/crates` prefix for all API endpoints. - * This transformation adds the necessary prefix to match crates.io's API structure. - * - * Transformations: - * - `/serde/1.0.0/download` -> `/api/v1/crates/serde/1.0.0/download` - * - `/serde` -> `/api/v1/crates/serde` - * - `/?q=query` -> `/api/v1/crates?q=query` - */ - if (platformKey === 'crates') { - if (transformedPath.startsWith('/')) { - if (transformedPath === '/' || transformedPath.startsWith('/?')) { - // Search endpoint: /?q=query -> /api/v1/crates?q=query - transformedPath = transformedPath.replace('/', '/api/v1/crates'); - } else { - // Crate-specific endpoints: /serde -> /api/v1/crates/serde - transformedPath = `/api/v1/crates${transformedPath}`; - } - } - } - - /** - * Special handling for Homebrew API paths - * - * Homebrew API paths are already in the correct format (e.g., /formula/git.json), - * so we simply strip the prefix and return the path as-is. - * - * Supported endpoints: - * - `/formula/{name}.json` - Formula metadata - * - `/cask/{name}.json` - Cask metadata - */ - if (platformKey === 'homebrew-api') { - if (transformedPath.startsWith('/')) { - return transformedPath; - } - } - - /** - * Special handling for Homebrew bottles - * - * Homebrew bottles are served from GitHub Container Registry (ghcr.io). - * The paths follow OCI registry format (/v2/...) and are passed through as-is. - * - * Example: `/v2/homebrew/core/git/manifests/2.39.0` - */ - if (platformKey === 'homebrew-bottles') { - if (transformedPath.startsWith('/')) { - return transformedPath; - } - } - - /** - * Special handling for Jenkins plugins - * - * Jenkins update center requires paths to be prefixed with `/current/` for - * the default update center. Experimental and download paths are preserved. - * - * Transformations: - * - `/update-center.json` -> `/current/update-center.json` - * - `/update-center.actual.json` -> `/current/update-center.actual.json` - * - `/experimental/...` -> `/experimental/...` (preserved) - * - `/download/...` -> `/download/...` (preserved) - * - Other paths -> `/current/{path}` - */ - if (platformKey === 'jenkins') { - if (transformedPath.startsWith('/')) { - if (transformedPath === '/update-center.json') { - return '/current/update-center.json'; - } else if (transformedPath === '/update-center.actual.json') { - return '/current/update-center.actual.json'; - } else if ( - transformedPath.startsWith('/experimental/') || - transformedPath.startsWith('/download/') || - transformedPath.startsWith('/current/') - ) { - // Keep experimental, download, and current paths as-is - return transformedPath; - } else { - // For other paths, assume they are relative to current - return `/current${transformedPath}`; - } - } - } - - /** - * Special handling for Homebrew repositories - * - * Homebrew repositories are Git repos on GitHub (github.com/Homebrew). - * Paths are passed through as-is to access repos like homebrew-core, homebrew-cask. - * - * Example: `/brew`, `/homebrew-core`, `/homebrew-cask` - */ - if (platformKey === 'homebrew') { - if (transformedPath.startsWith('/')) { - return transformedPath; - } - } - - return transformedPath; -} +export { PLATFORM_CATALOG, PLATFORMS } from './platform-catalog.js'; +export { SORTED_PLATFORMS } from '../routing/platform-index.js'; +export { transformPath } from '../routing/platform-transformers.js'; diff --git a/src/index.js b/src/index.js index dd96f6f..1e0398d 100644 --- a/src/index.js +++ b/src/index.js @@ -8,713 +8,9 @@ * (at your option) any later version. */ -import { CONFIG, createConfig } from './config/index.js'; -import { SORTED_PLATFORMS, transformPath } from './config/platforms.js'; -import { configureAIHeaders, isAIInferenceRequest } from './protocols/ai.js'; -import { configureHuggingFaceHeaders, isHuggingFaceAPIRequest } from './protocols/huggingface.js'; -import { - fetchToken, - getScopeFromUrl, - handleDockerAuth, - normalizeRegistryApiPath, - parseAuthenticate, - readRegistryTokenResponse, - responseUnauthorized -} from './protocols/docker.js'; -import { configureGitHeaders, isGitLFSRequest, isGitRequest } from './protocols/git.js'; -import { PerformanceMonitor, addPerformanceHeaders } from './utils/performance.js'; -import { - isFlatpakReferenceFilePath, - rewriteTextResponse, - shouldRewriteTextResponse -} from './utils/rewrite.js'; -import { addCorsHeaders, addSecurityHeaders, createErrorResponse } from './utils/security.js'; -import { getAllowedMethods, isDockerRequest, validateRequest } from './utils/validation.js'; +import { handleRequest } from './app/handle-request.js'; -/** - * Main request handler with comprehensive caching, retry logic, and security measures. - * @param {Request} request - The incoming HTTP request - * @param {Record} env - Cloudflare Workers environment variables for runtime config overrides - * @param {ExecutionContext} ctx - Cloudflare Workers execution context for background tasks - * @returns {Promise} The HTTP response with appropriate headers and body - */ -async function handleRequest(request, env, ctx) { - let response; - let responseGeneratedLocally = false; - const monitor = new PerformanceMonitor(); - - try { - // Create config with environment variable overrides - const config = env ? createConfig(env) : CONFIG; - const url = new URL(request.url); - const isDocker = isDockerRequest(request, url); - const isCorsPreflight = - request.method === 'OPTIONS' && - request.headers.has('Origin') && - request.headers.has('Access-Control-Request-Method'); - - if (isCorsPreflight) { - const requestedMethod = request.headers.get('Access-Control-Request-Method') || ''; - const allowedMethods = getAllowedMethods( - new Request(request.url, { method: requestedMethod || 'GET' }), - url, - config - ); - - if (!allowedMethods.includes(requestedMethod)) { - response = createErrorResponse('Method not allowed', 405); - } else { - const headers = addCorsHeaders(new Headers(), request, config); - if (!headers.has('Access-Control-Allow-Origin')) { - response = createErrorResponse('Origin not allowed', 403); - } else { - headers.set('Access-Control-Allow-Methods', allowedMethods.join(', ')); - headers.set('Access-Control-Max-Age', '86400'); - addSecurityHeaders(headers); - response = new Response(null, { status: 204, headers }); - } - } - } - - // Handle Docker API version check - else if (isDocker && (url.pathname === '/v2/' || url.pathname === '/v2')) { - const headers = new Headers({ - 'Docker-Distribution-Api-Version': 'registry/2.0', - 'Content-Type': 'application/json' - }); - addSecurityHeaders(headers); - response = new Response('{}', { status: 200, headers }); - } - // Redirect root path or invalid platforms to GitHub repository - else if (url.pathname === '/' || url.pathname === '') { - const HOME_PAGE_URL = 'https://github.com/xixu-me/Xget'; - response = Response.redirect(HOME_PAGE_URL, 302); - } else { - const validation = validateRequest(request, url, config); - if (!validation.valid) { - response = createErrorResponse( - validation.error || 'Validation failed', - validation.status || 400 - ); - } else { - // Parse platform and path - let effectivePath = url.pathname; - - // Handle container registry paths specially - if (isDocker) { - // For Docker requests, check if they have /cr/ prefix - // but allow /v2/auth which handles authentication - if ( - !url.pathname.startsWith('/cr/') && - !url.pathname.startsWith('/v2/cr/') && - url.pathname !== '/v2/auth' - ) { - response = createErrorResponse('container registry requests must use /cr/ prefix', 400); - } else { - // Remove /v2 from the path for container registry API consistency if present - effectivePath = url.pathname.replace(/^\/v2/, ''); - - // Docker clients address Xget as a registry host, so image names like - // `xget.example/cr/ghcr/owner/image:tag` arrive as `/v2/cr/ghcr/...`. - // Normalize that shape into the canonical `/cr//v2/...` form - // used by our registry platform routing. - if (url.pathname.startsWith('/v2/cr/')) { - effectivePath = effectivePath.replace(/^\/cr\/([^/]+)\//, '/cr/$1/v2/'); - } - } - } - - if (!response) { - // Handle Docker authentication explicitly - if ( - isDocker && - (url.pathname === '/v2/auth' || /^\/cr\/[^/]+\/v2\/auth\/?$/.test(url.pathname)) - ) { - response = await handleDockerAuth(request, url, config); - } else { - // Platform detection using transform patterns - // Use pre-computed sorted platforms - const platform = - SORTED_PLATFORMS.find(key => { - const expectedPrefix = `/${key.replace('-', '/')}/`; - return effectivePath.startsWith(expectedPrefix); - }) || effectivePath.split('/')[1]; - - if (!platform || !config.PLATFORMS[platform]) { - const HOME_PAGE_URL = 'https://github.com/xixu-me/Xget'; - response = Response.redirect(HOME_PAGE_URL, 302); - } else { - // Check if the path only contains the platform prefix without any actual resource path - const platformPath = `/${platform.replace(/-/g, '/')}`; - if (effectivePath === platformPath || effectivePath === `${platformPath}/`) { - const HOME_PAGE_URL = 'https://github.com/xixu-me/Xget'; - response = Response.redirect(HOME_PAGE_URL, 302); - } else { - // Transform URL based on platform using unified logic - const targetPath = transformPath(effectivePath, platform); - - const finalTargetPath = platform.startsWith('cr-') - ? normalizeRegistryApiPath(platform, targetPath) - : targetPath; - - const targetUrl = `${config.PLATFORMS[platform]}${finalTargetPath}${url.search}`; - const authorization = request.headers.get('Authorization'); - const hasSensitiveHeaders = Boolean( - authorization || - request.headers.get('Cookie') || - request.headers.get('Proxy-Authorization') - ); - - // Check if this is a Git operation - const isGit = isGitRequest(request, url); - - // Check if this is a Git LFS operation - const isGitLFS = isGitLFSRequest(request, url); - - // Check if this is an AI inference request - const isAI = isAIInferenceRequest(request, url); - - // Check if this is a Hugging Face API request - const isHF = isHuggingFaceAPIRequest(request, url); - const shouldVaryCacheByOrigin = - platform === 'flathub' && isFlatpakReferenceFilePath(effectivePath); - const cacheTargetUrl = shouldVaryCacheByOrigin - ? `${targetUrl}${targetUrl.includes('?') ? '&' : '?'}__xget_origin=${encodeURIComponent(url.origin)}` - : targetUrl; - const canUseCache = request.method === 'GET' || request.method === 'HEAD'; - const shouldPassthroughRequest = - isGit || isGitLFS || isDocker || isAI || isHF || !canUseCache; - - // Check cache first (skip cache for Git, Git LFS, Docker, AI inference, and HF API operations) - /** @type {Cache | null} */ - // @ts-ignore - Cloudflare Workers cache API - const cache = - typeof caches !== 'undefined' && /** @type {any} */ (caches).default // eslint-disable-line jsdoc/reject-any-type - ? /** @type {any} */ (caches).default // eslint-disable-line jsdoc/reject-any-type - : null; - - if ( - cache && - canUseCache && - !isGit && - !isGitLFS && - !isDocker && - !isAI && - !isHF && - !hasSensitiveHeaders - ) { - try { - // For Range requests, try cache match first - const cacheKey = new Request(cacheTargetUrl, { - method: 'GET', - headers: request.headers - }); - const cachedResponse = await cache.match(cacheKey); - if (cachedResponse) { - monitor.mark('cache_hit'); - response = cachedResponse; - } else { - // If Range request missed cache, try with original request to see if we have full content cached - const rangeHeader = request.headers.get('Range'); - if (rangeHeader) { - const fullContentKey = new Request(cacheTargetUrl, { - method: 'GET', // Always use GET method for cache key consistency - headers: new Headers( - [...request.headers.entries()].filter( - ([k]) => k.toLowerCase() !== 'range' - ) - ) - }); - const fullCachedResponse = await cache.match(fullContentKey); - if (fullCachedResponse) { - monitor.mark('cache_hit_full_content'); - response = fullCachedResponse; - } - } - } - } catch (cacheError) { - console.warn('Cache API unavailable:', cacheError); - } - } - - if (!response) { - /** @type {RequestInit} */ - const fetchOptions = { - method: request.method, - headers: new Headers(), - redirect: 'follow' - }; - - if (request.body !== null && !canUseCache) { - fetchOptions.body = request.body; - } - - // Cast headers to Headers for proper typing - const requestHeaders = /** @type {Headers} */ (fetchOptions.headers); - - // Preserve caller-supplied headers for protocol requests and for - // explicitly enabled non-GET/HEAD methods on regular platforms. - if (shouldPassthroughRequest) { - for (const [key, value] of request.headers.entries()) { - // Skip headers that might cause issues with proxying - if ( - !['host', 'connection', 'upgrade', 'proxy-connection'].includes( - key.toLowerCase() - ) - ) { - requestHeaders.set(key, value); - } - } - - // Configure protocol-specific headers using modular helpers - if (isGit || isGitLFS) { - configureGitHeaders(requestHeaders, request, url, isGitLFS); - } - - if (isAI) { - configureAIHeaders(requestHeaders, request); - } - - if (isHF) { - configureHuggingFaceHeaders(requestHeaders, request); - } - } else { - // Regular GET/HEAD file download headers - Object.assign(fetchOptions, { - cf: { - http3: true, - cacheTtl: config.CACHE_DURATION, - cacheEverything: true, - preconnect: true - } - }); - - requestHeaders.set('Accept-Encoding', 'gzip, deflate, br'); - requestHeaders.set('Connection', 'keep-alive'); - requestHeaders.set('User-Agent', 'Wget/1.21.3'); - const origin = request.headers.get('Origin'); - if (origin) { - requestHeaders.set('Origin', origin); - } - - if (authorization) { - requestHeaders.set('Authorization', authorization); - } - - const rangeHeader = request.headers.get('Range'); - const isMediaFile = targetUrl.match( - /\.(mp4|avi|mkv|mov|wmv|flv|webm|mp3|wav|flac|aac|ogg|jpg|jpeg|png|gif|bmp|svg|pdf|zip|rar|7z|tar|gz|bz2|xz)$/i - ); - - if (isMediaFile || rangeHeader) { - requestHeaders.set('Accept-Encoding', 'identity'); - } - - if (rangeHeader) { - requestHeaders.set('Range', rangeHeader); - } - } - - // Implement retry mechanism - let attempts = 0; - while (attempts < config.MAX_RETRIES) { - /** @type {ReturnType | undefined} */ - let timeoutId; - try { - monitor.mark(`attempt_${attempts}`); - - const controller = new AbortController(); - timeoutId = setTimeout( - () => controller.abort(), - config.TIMEOUT_SECONDS * 1000 - ); - - const finalFetchOptions = { - ...fetchOptions, - signal: controller.signal - }; - - // Special handling for Docker redirects to avoid leaking Auth headers to S3 (blobs) - if (isDocker) { - finalFetchOptions.redirect = 'manual'; - } - - // Special handling for HEAD requests to ensure Content-Length header - if (request.method === 'HEAD') { - response = await fetch(targetUrl, finalFetchOptions); - - if (response.ok && !response.headers.get('Content-Length')) { - const rangeHeaders = new Headers(requestHeaders); - rangeHeaders.set('Range', 'bytes=0-0'); - - const rangeResponse = await fetch(targetUrl, { - ...finalFetchOptions, - method: 'GET', - headers: rangeHeaders - }); - - let contentLength = null; - - if (rangeResponse.status === 206) { - const contentRange = rangeResponse.headers.get('Content-Range'); - if (contentRange) { - const match = contentRange.match(/bytes\s+\d+-\d+\/(\d+)/); - if (match) { - [, contentLength] = match; - } - } - } else if (rangeResponse.ok) { - contentLength = rangeResponse.headers.get('Content-Length'); - } - - if (contentLength) { - const headHeaders = new Headers(response.headers); - headHeaders.set('Content-Length', contentLength); - response = new Response(null, { - status: response.status, - statusText: response.statusText, - headers: headHeaders - }); - } - } - } else { - response = await fetch(targetUrl, finalFetchOptions); - } - - // Handle manual redirect for Docker - if ( - isDocker && - (response.status === 301 || - response.status === 302 || - response.status === 303 || - response.status === 307 || - response.status === 308) - ) { - const location = response.headers.get('Location'); - if (location) { - // Fetch the new location without Authorization header - // Cloudflare Workers fetch should follow this automatically if we used 'follow', - // but we used 'manual' to strip headers. - const redirectHeaders = new Headers(finalFetchOptions.headers); - redirectHeaders.delete('Authorization'); - - response = await fetch(new URL(location, targetUrl), { - ...finalFetchOptions, - headers: redirectHeaders, - redirect: 'follow' // Follow subsequent redirects normally - }); - } - } - - if (response.ok || response.status === 206) { - monitor.mark('success'); - break; - } - - // For container registry, handle authentication challenges more intelligently - if (isDocker && response.status === 401) { - monitor.mark('docker_auth_challenge'); - - const authenticateStr = response.headers.get('WWW-Authenticate'); - - // Calculate scope for upstream token fetch - const scope = getScopeFromUrl(url, effectivePath, platform); - - if (authenticateStr) { - try { - const wwwAuthenticate = parseAuthenticate(authenticateStr); - - // Try to get a token for public access (without authorization) - const tokenResponse = await fetchToken( - wwwAuthenticate, - scope || '', - '' - ); - - if (tokenResponse.ok) { - const token = await readRegistryTokenResponse(tokenResponse); - if (token) { - const retryHeaders = new Headers(requestHeaders); - retryHeaders.set('Authorization', `Bearer ${token}`); - - const retryOptions = { - ...finalFetchOptions, - headers: retryHeaders - }; - - // Also use manual redirect for retry - if (isDocker) { - retryOptions.redirect = 'manual'; - } - - let retryResponse = await fetch(targetUrl, retryOptions); - - // Handle manual redirect for retry - if ( - isDocker && - (retryResponse.status === 301 || - retryResponse.status === 302 || - retryResponse.status === 303 || - retryResponse.status === 307 || - retryResponse.status === 308) - ) { - const location = retryResponse.headers.get('Location'); - if (location) { - const redirectHeaders = new Headers(retryOptions.headers); - redirectHeaders.delete('Authorization'); - - retryResponse = await fetch(new URL(location, targetUrl), { - ...retryOptions, - headers: redirectHeaders, - redirect: 'follow' - }); - } - } - - if (retryResponse.ok) { - response = retryResponse; - monitor.mark('success'); - break; - } - } - } - } catch (error) { - console.warn('Token fetch failed:', error); - } - } - - response = responseUnauthorized(url, platform); - break; - } - - if (response.status >= 400 && response.status < 500) { - monitor.mark('client_error'); - break; - } - - attempts++; - if (attempts < config.MAX_RETRIES) { - await new Promise(resolve => - setTimeout(resolve, config.RETRY_DELAY_MS * attempts) - ); - } - } catch (error) { - attempts++; - if (error instanceof Error && error.name === 'AbortError') { - response = createErrorResponse('Request timeout', 408); - responseGeneratedLocally = true; - break; - } - if (attempts >= config.MAX_RETRIES) { - response = createErrorResponse('Upstream request failed', 502); - responseGeneratedLocally = true; - break; - } - await new Promise(resolve => - setTimeout(resolve, config.RETRY_DELAY_MS * attempts) - ); - } finally { - if (timeoutId !== undefined) { - clearTimeout(timeoutId); - } - } - } - - if (!response) { - response = createErrorResponse( - 'No response received after all retry attempts', - 500 - ); - responseGeneratedLocally = true; - } else if (!responseGeneratedLocally && !response.ok && response.status !== 206) { - if (isDocker && response.status === 401) { - if (!response.headers.has('WWW-Authenticate')) { - // Handle Docker 401 responses that might not have been caught by the retry loop - const isCustomError = - response.headers.get('content-type') === 'application/json' && - (await response.clone().text()).includes('UNAUTHORIZED'); - - if (!isCustomError) { - const errorText = await response.text().catch(() => ''); - response = createErrorResponse( - `Authentication required for this container registry resource. This may be a private repository. Original error: ${errorText}`, - 401, - true - ); - } - } - } else { - const errorText = await response.text().catch(() => 'Unknown error'); - response = createErrorResponse( - `Upstream server error (${response.status}): ${errorText}`, - response.status, - true - ); - } - } else { - // Success case processing (rewriting URLs etc) - /** @type {string | ReadableStream | null} */ - let responseBody = response.body; - let rewrittenContentLength = null; - let hasOriginBoundRewrite = false; - - if ( - shouldRewriteTextResponse( - platform, - effectivePath, - response.headers.get('content-type') || '' - ) - ) { - const originalText = - platform === 'flathub' && isFlatpakReferenceFilePath(effectivePath) - ? new TextDecoder().decode(await response.arrayBuffer()) - : await response.text(); - const rewrittenText = rewriteTextResponse( - platform, - effectivePath, - originalText, - url.origin - ); - responseBody = rewrittenText; - rewrittenContentLength = new TextEncoder().encode(rewrittenText).byteLength; - hasOriginBoundRewrite = platform === 'pypi'; - } - - const headers = new Headers(response.headers); - - if (rewrittenContentLength !== null) { - headers.set('Content-Length', String(rewrittenContentLength)); - } - - if (!isGit && !isGitLFS && !isDocker && !isAI && !isHF) { - if (!canUseCache) { - headers.set('Cache-Control', 'no-store'); - } else if (hasOriginBoundRewrite) { - headers.set('Cache-Control', 'no-store'); - } else if (hasSensitiveHeaders) { - headers.set('Cache-Control', 'private, no-store'); - const existingVary = headers.get('Vary'); - headers.set( - 'Vary', - existingVary - ? `${existingVary}, Authorization, Cookie` - : 'Authorization, Cookie' - ); - } else { - headers.set('Cache-Control', `public, max-age=${config.CACHE_DURATION}`); - } - - headers.set('X-Content-Type-Options', 'nosniff'); - headers.set('Accept-Ranges', 'bytes'); - - if (!headers.has('Content-Length') && response.status === 200) { - try { - const contentLength = response.headers.get('Content-Length'); - if (contentLength) { - headers.set('Content-Length', contentLength); - } - } catch (error) { - console.warn('Could not set Content-Length header:', error); - } - } - - addSecurityHeaders(headers); - } - - response = new Response(responseBody, { - status: response.status, - headers - }); - - // Cache success logic - if ( - cache && - !isGit && - !isGitLFS && - !isDocker && - !isAI && - !isHF && - !hasOriginBoundRewrite && - !hasSensitiveHeaders && - request.method === 'GET' && - response.ok && - response.status === 200 - ) { - const rangeHeader = request.headers.get('Range'); - const cacheKey = rangeHeader - ? new Request(cacheTargetUrl, { - method: 'GET', - headers: new Headers( - [...request.headers.entries()].filter( - ([k]) => k.toLowerCase() !== 'range' - ) - ) - }) - : new Request(cacheTargetUrl, { method: 'GET' }); - - try { - if (ctx && typeof ctx.waitUntil === 'function') { - ctx.waitUntil(cache.put(cacheKey, response.clone())); - } else { - cache.put(cacheKey, response.clone()).catch(error => { - console.warn('Cache put failed:', error); - }); - } - - if (rangeHeader && response.status === 200) { - const rangedResponse = await cache.match( - new Request(cacheTargetUrl, { - method: 'GET', - headers: request.headers - }) - ); - if (rangedResponse) { - monitor.mark('range_cache_hit_after_full_cache'); - response = rangedResponse; - } - } - } catch (cacheError) { - console.warn('Cache put/match failed:', cacheError); - } - } - } - } - } - } - } - } - } - } - } catch (error) { - console.error('Error handling request:', error); - response = createErrorResponse('Internal Server Error', 500); - } - - // Ensure performance headers are added to the final response - monitor.mark('complete'); - const isGit = isGitRequest(request, new URL(request.url)); - const isDocker = isDockerRequest(request, new URL(request.url)); - const isAI = isAIInferenceRequest(request, new URL(request.url)); - const isGitLFS = isGitLFSRequest(request, new URL(request.url)); - const isHF = isHuggingFaceAPIRequest(request, new URL(request.url)); - - const responseWithCors = (() => { - const headers = addCorsHeaders( - new Headers(response.headers), - request, - env ? createConfig(env) : CONFIG - ); - return new Response(response.body, { - status: response.status, - statusText: response.statusText, - headers - }); - })(); - - return isGit || isGitLFS || isDocker || isAI || isHF - ? responseWithCors - : addPerformanceHeaders(responseWithCors, monitor); -} +export { handleRequest } from './app/handle-request.js'; export default { /** diff --git a/src/protocols/docker.js b/src/protocols/docker.js index 723d527..c058ab3 100644 --- a/src/protocols/docker.js +++ b/src/protocols/docker.js @@ -20,7 +20,7 @@ * Docker/OCI Registry protocol handler for Xget */ -import { SORTED_PLATFORMS } from '../config/platforms.js'; +import { SORTED_PLATFORMS } from '../routing/platform-index.js'; import { createErrorResponse } from '../utils/security.js'; /** diff --git a/src/response/finalize-response.js b/src/response/finalize-response.js new file mode 100644 index 0000000..c338424 --- /dev/null +++ b/src/response/finalize-response.js @@ -0,0 +1,284 @@ +import { + isFlatpakReferenceFilePath, + rewriteTextResponse, + shouldRewriteTextResponse +} from '../utils/rewrite.js'; +import { addSecurityHeaders, createErrorResponse } from '../utils/security.js'; + +/** + * Wraps an unsuccessful upstream response into the user-facing error contract. + * @param {{ + * effectivePath: string, + * platform: string, + * request: Request, + * requestContext: { + * isAI: boolean, + * isDocker: boolean, + * isGit: boolean, + * isGitLFS: boolean, + * isHF: boolean + * }, + * response: Response, + * responseGeneratedLocally: boolean, + * url: URL + * }} options + * @returns {Promise} Final error response. + */ +async function finalizeErrorResponse({ requestContext, response, responseGeneratedLocally }) { + if (responseGeneratedLocally || response.ok || response.status === 206) { + return response; + } + + if (requestContext.isDocker && response.status === 401) { + if (!response.headers.has('WWW-Authenticate')) { + const isCustomError = + response.headers.get('content-type') === 'application/json' && + (await response.clone().text()).includes('UNAUTHORIZED'); + + if (!isCustomError) { + const errorText = await response.text().catch(() => ''); + return createErrorResponse( + `Authentication required for this container registry resource. This may be a private repository. Original error: ${errorText}`, + 401, + true + ); + } + } + + return response; + } + + const errorText = await response.text().catch(() => 'Unknown error'); + return createErrorResponse( + `Upstream server error (${response.status}): ${errorText}`, + response.status, + true + ); +} + +/** + * Finalizes a successful upstream response, including rewriting, cache headers, and background cache writes. + * @param {{ + * cache: Cache | null, + * cacheTargetUrl: string, + * canUseCache: boolean, + * config: import('../config/index.js').ApplicationConfig, + * ctx: ExecutionContext, + * effectivePath: string, + * hasSensitiveHeaders: boolean, + * monitor: import('../utils/performance.js').PerformanceMonitor, + * platform: string, + * request: Request, + * requestContext: { + * isAI: boolean, + * isDocker: boolean, + * isGit: boolean, + * isGitLFS: boolean, + * isHF: boolean + * }, + * response: Response, + * url: URL + * }} options + * @returns {Promise} Final proxied response. + */ +async function finalizeSuccessfulResponse({ + cache, + cacheTargetUrl, + canUseCache, + config, + ctx, + effectivePath, + hasSensitiveHeaders, + monitor, + platform, + request, + requestContext, + response, + url +}) { + const { isAI, isDocker, isGit, isGitLFS, isHF } = requestContext; + + /** @type {string | ReadableStream | null} */ + let responseBody = response.body; + let rewrittenContentLength = null; + let hasOriginBoundRewrite = false; + + if ( + shouldRewriteTextResponse(platform, effectivePath, response.headers.get('content-type') || '') + ) { + const originalText = + platform === 'flathub' && isFlatpakReferenceFilePath(effectivePath) + ? new TextDecoder().decode(await response.arrayBuffer()) + : await response.text(); + const rewrittenText = rewriteTextResponse(platform, effectivePath, originalText, url.origin); + responseBody = rewrittenText; + rewrittenContentLength = new TextEncoder().encode(rewrittenText).byteLength; + hasOriginBoundRewrite = platform === 'pypi'; + } + + const headers = new Headers(response.headers); + + if (rewrittenContentLength !== null) { + headers.set('Content-Length', String(rewrittenContentLength)); + } + + if (!isGit && !isGitLFS && !isDocker && !isAI && !isHF) { + if (!canUseCache || hasOriginBoundRewrite) { + headers.set('Cache-Control', 'no-store'); + } else if (hasSensitiveHeaders) { + headers.set('Cache-Control', 'private, no-store'); + const existingVary = headers.get('Vary'); + headers.set( + 'Vary', + existingVary ? `${existingVary}, Authorization, Cookie` : 'Authorization, Cookie' + ); + } else { + headers.set('Cache-Control', `public, max-age=${config.CACHE_DURATION}`); + } + + headers.set('X-Content-Type-Options', 'nosniff'); + headers.set('Accept-Ranges', 'bytes'); + + if (!headers.has('Content-Length') && response.status === 200) { + try { + const contentLength = response.headers.get('Content-Length'); + if (contentLength) { + headers.set('Content-Length', contentLength); + } + } catch (error) { + console.warn('Could not set Content-Length header:', error); + } + } + + addSecurityHeaders(headers); + } + + let finalizedResponse = new Response(responseBody, { + status: response.status, + headers + }); + + if ( + cache && + !isGit && + !isGitLFS && + !isDocker && + !isAI && + !isHF && + !hasOriginBoundRewrite && + !hasSensitiveHeaders && + request.method === 'GET' && + finalizedResponse.ok && + finalizedResponse.status === 200 + ) { + const rangeHeader = request.headers.get('Range'); + const cacheKey = rangeHeader + ? new Request(cacheTargetUrl, { + method: 'GET', + headers: new Headers( + [...request.headers.entries()].filter(([key]) => key.toLowerCase() !== 'range') + ) + }) + : new Request(cacheTargetUrl, { method: 'GET' }); + + try { + if (ctx && typeof ctx.waitUntil === 'function') { + ctx.waitUntil(cache.put(cacheKey, finalizedResponse.clone())); + } else { + cache.put(cacheKey, finalizedResponse.clone()).catch(error => { + console.warn('Cache put failed:', error); + }); + } + + if (rangeHeader && finalizedResponse.status === 200) { + const rangedResponse = await cache.match( + new Request(cacheTargetUrl, { + method: 'GET', + headers: request.headers + }) + ); + if (rangedResponse) { + monitor.mark('range_cache_hit_after_full_cache'); + finalizedResponse = rangedResponse; + } + } + } catch (cacheError) { + console.warn('Cache put/match failed:', cacheError); + } + } + + return finalizedResponse; +} + +/** + * Finalizes the upstream response after cache lookup and fetch execution. + * @param {{ + * cache: Cache | null, + * cacheTargetUrl: string, + * canUseCache: boolean, + * config: import('../config/index.js').ApplicationConfig, + * ctx: ExecutionContext, + * effectivePath: string, + * hasSensitiveHeaders: boolean, + * monitor: import('../utils/performance.js').PerformanceMonitor, + * platform: string, + * request: Request, + * requestContext: { + * isAI: boolean, + * isDocker: boolean, + * isGit: boolean, + * isGitLFS: boolean, + * isHF: boolean + * }, + * response: Response, + * responseGeneratedLocally: boolean, + * url: URL + * }} options + * @returns {Promise} Final response returned to the client. + */ +export async function finalizeResponse({ + cache, + cacheTargetUrl, + canUseCache, + config, + ctx, + effectivePath, + hasSensitiveHeaders, + monitor, + platform, + request, + requestContext, + response, + responseGeneratedLocally, + url +}) { + const errorResponse = await finalizeErrorResponse({ + effectivePath, + platform, + request, + requestContext, + response, + responseGeneratedLocally, + url + }); + + if (errorResponse !== response || !errorResponse.ok) { + return errorResponse; + } + + return await finalizeSuccessfulResponse({ + cache, + cacheTargetUrl, + canUseCache, + config, + ctx, + effectivePath, + hasSensitiveHeaders, + monitor, + platform, + request, + requestContext, + response: errorResponse, + url + }); +} diff --git a/src/routing/platform-index.js b/src/routing/platform-index.js new file mode 100644 index 0000000..13eea24 --- /dev/null +++ b/src/routing/platform-index.js @@ -0,0 +1,17 @@ +import { PLATFORM_CATALOG } from '../config/platform-catalog.js'; + +/** + * Converts a platform key into its matching URL prefix. + * @param {string} platformKey + * @returns {string} Platform prefix, for example `/ip/openai/`. + */ +export function getPlatformPathPrefix(platformKey) { + return `/${platformKey.replace(/-/g, '/')}/`; +} + +/** + * Pre-computed sorted platform keys for efficient path matching. + */ +export const SORTED_PLATFORMS = Object.keys(PLATFORM_CATALOG).sort((a, b) => { + return getPlatformPathPrefix(b).length - getPlatformPathPrefix(a).length; +}); diff --git a/src/routing/platform-transformers.js b/src/routing/platform-transformers.js new file mode 100644 index 0000000..8641fd4 --- /dev/null +++ b/src/routing/platform-transformers.js @@ -0,0 +1,91 @@ +import { PLATFORM_CATALOG } from '../config/platform-catalog.js'; +import { getPlatformPathPrefix } from './platform-index.js'; + +/** + * Escapes a string for safe use inside a regular expression. + * @param {string} value + * @returns {string} Escaped string. + */ +function escapeRegex(value) { + return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +/** + * Removes the platform prefix from a request path. + * @param {string} path + * @param {string} platformKey + * @returns {string} Path without the leading platform segment. + */ +function stripPlatformPrefix(path, platformKey) { + const prefix = getPlatformPathPrefix(platformKey); + return path.replace(new RegExp(`^${escapeRegex(prefix)}`), '/'); +} + +/** + * Applies crates.io-specific API path normalization. + * @param {string} transformedPath + * @returns {string} Normalized crates.io API path. + */ +function transformCratesPath(transformedPath) { + if (!transformedPath.startsWith('/')) { + return transformedPath; + } + + if (transformedPath === '/' || transformedPath.startsWith('/?')) { + return transformedPath.replace('/', '/api/v1/crates'); + } + + return `/api/v1/crates${transformedPath}`; +} + +/** + * Applies Jenkins update-center path normalization. + * @param {string} transformedPath + * @returns {string} Normalized Jenkins path. + */ +function transformJenkinsPath(transformedPath) { + if (!transformedPath.startsWith('/')) { + return transformedPath; + } + + if (transformedPath === '/update-center.json') { + return '/current/update-center.json'; + } + + if (transformedPath === '/update-center.actual.json') { + return '/current/update-center.actual.json'; + } + + if ( + transformedPath.startsWith('/experimental/') || + transformedPath.startsWith('/download/') || + transformedPath.startsWith('/current/') + ) { + return transformedPath; + } + + return `/current${transformedPath}`; +} + +/** @type {{ [key: string]: (transformedPath: string) => string }} */ +const PLATFORM_PATH_TRANSFORMERS = { + crates: transformCratesPath, + jenkins: transformJenkinsPath +}; + +/** + * Converts a routed request path into the upstream path expected by the platform. + * @param {string} path + * @param {string} platformKey + * @returns {string} Upstream-ready request path. + */ +export function transformPath(path, platformKey) { + if (!PLATFORM_CATALOG[platformKey]) { + return path; + } + + const transformedPath = stripPlatformPrefix(path, platformKey); + const transformPlatformPath = PLATFORM_PATH_TRANSFORMERS[platformKey]; + + return transformPlatformPath ? transformPlatformPath(transformedPath) : transformedPath; +} diff --git a/src/routing/resolve-target.js b/src/routing/resolve-target.js new file mode 100644 index 0000000..a71c143 --- /dev/null +++ b/src/routing/resolve-target.js @@ -0,0 +1,96 @@ +import { SORTED_PLATFORMS } from './platform-index.js'; +import { transformPath } from './platform-transformers.js'; +import { normalizeRegistryApiPath } from '../protocols/docker.js'; +import { isFlatpakReferenceFilePath } from '../utils/rewrite.js'; +import { createErrorResponse } from '../utils/security.js'; + +export const HOME_PAGE_URL = 'https://github.com/xixu-me/Xget'; + +/** + * Creates the canonical homepage redirect response. + * @returns {Response} Redirect response to the Xget homepage. + */ +export function createHomepageRedirect() { + return Response.redirect(HOME_PAGE_URL, 302); +} + +/** + * Normalizes request paths before platform routing. + * @param {URL} url + * @param {boolean} isDocker + * @returns {{ effectivePath: string } | { response: Response }} Normalized path or an early error response. + */ +export function normalizeEffectivePath(url, isDocker) { + let effectivePath = url.pathname; + + if (!isDocker) { + return { effectivePath }; + } + + if ( + !url.pathname.startsWith('/cr/') && + !url.pathname.startsWith('/v2/cr/') && + url.pathname !== '/v2/auth' + ) { + return { + response: createErrorResponse('container registry requests must use /cr/ prefix', 400) + }; + } + + effectivePath = url.pathname.replace(/^\/v2/, ''); + + if (url.pathname.startsWith('/v2/cr/')) { + effectivePath = effectivePath.replace(/^\/cr\/([^/]+)\//, '/cr/$1/v2/'); + } + + return { effectivePath }; +} + +/** + * Resolves an effective request path to an upstream target URL. + * @param {URL} url + * @param {string} effectivePath + * @param {{ [key: string]: string }} platforms + * @returns {{ + * cacheTargetUrl: string, + * platform: string, + * shouldVaryCacheByOrigin: boolean, + * targetPath: string, + * targetUrl: string + * } | { response: Response }} Target metadata or an early redirect response. + */ +export function resolveTarget(url, effectivePath, platforms) { + const platform = + SORTED_PLATFORMS.find(key => { + const expectedPrefix = `/${key.replace('-', '/')}/`; + return effectivePath.startsWith(expectedPrefix); + }) || effectivePath.split('/')[1]; + + if (!platform || !platforms[platform]) { + return { response: createHomepageRedirect() }; + } + + const platformPath = `/${platform.replace(/-/g, '/')}`; + if (effectivePath === platformPath || effectivePath === `${platformPath}/`) { + return { response: createHomepageRedirect() }; + } + + const transformedPath = transformPath(effectivePath, platform); + const targetPath = platform.startsWith('cr-') + ? normalizeRegistryApiPath(platform, transformedPath) + : transformedPath; + const targetUrl = `${platforms[platform]}${targetPath}${url.search}`; + const shouldVaryCacheByOrigin = + platform === 'flathub' && isFlatpakReferenceFilePath(effectivePath); + const cacheTargetUrl = shouldVaryCacheByOrigin + ? `${targetUrl}${targetUrl.includes('?') ? '&' : '?'}__xget_origin=${encodeURIComponent(url.origin)}` + : targetUrl; + + return { + cacheTargetUrl, + platform, + shouldVaryCacheByOrigin, + targetPath, + targetUrl + }; +} diff --git a/src/types.d.ts b/src/types.d.ts index 4c5676f..2973039 100644 --- a/src/types.d.ts +++ b/src/types.d.ts @@ -18,3 +18,23 @@ interface ExecutionContext { */ passThroughOnException(): void; } + +interface DenoEnv { + /** + * Reads an environment variable from Deno Deploy. + * @param name - Environment variable name + */ + get(name: string): string | undefined; +} + +interface DenoGlobal { + env: DenoEnv; + + /** + * Starts the Deno Deploy HTTP server. + * @param handler - Request handler callback + */ + serve(handler: (request: Request) => Promise | Response): void; +} + +declare const Deno: DenoGlobal; diff --git a/src/upstream/cache.js b/src/upstream/cache.js new file mode 100644 index 0000000..cce5a2e --- /dev/null +++ b/src/upstream/cache.js @@ -0,0 +1,92 @@ +/** + * Cache helpers for upstream request handling. + */ + +/** + * Reads the default Cloudflare cache when available. + * @returns {Cache | null} Default runtime cache, or null when unavailable. + */ +export function getDefaultCache() { + // @ts-ignore - Cloudflare Workers cache API + return typeof caches !== 'undefined' && /** @type {any} */ (caches).default // eslint-disable-line jsdoc/reject-any-type + ? // @ts-ignore - Cloudflare Workers cache API + /** @type {any} */ (caches).default // eslint-disable-line jsdoc/reject-any-type + : null; +} + +/** + * Attempts to satisfy a request from cache before reaching the upstream. + * @param {{ + * cache: Cache | null, + * cacheTargetUrl: string, + * canUseCache: boolean, + * hasSensitiveHeaders: boolean, + * monitor: import('../utils/performance.js').PerformanceMonitor, + * request: Request, + * requestContext: { + * isAI: boolean, + * isDocker: boolean, + * isGit: boolean, + * isGitLFS: boolean, + * isHF: boolean + * } + * }} options + * @returns {Promise} Cached response when one can be reused, otherwise null. + */ +export async function tryReadCachedResponse({ + cache, + cacheTargetUrl, + canUseCache, + hasSensitiveHeaders, + monitor, + request, + requestContext +}) { + const { isAI, isDocker, isGit, isGitLFS, isHF } = requestContext; + + if ( + !cache || + !canUseCache || + isGit || + isGitLFS || + isDocker || + isAI || + isHF || + hasSensitiveHeaders + ) { + return null; + } + + try { + const cacheKey = new Request(cacheTargetUrl, { + method: 'GET', + headers: request.headers + }); + const cachedResponse = await cache.match(cacheKey); + if (cachedResponse) { + monitor.mark('cache_hit'); + return cachedResponse; + } + + const rangeHeader = request.headers.get('Range'); + if (!rangeHeader) { + return null; + } + + const fullContentKey = new Request(cacheTargetUrl, { + method: 'GET', + headers: new Headers( + [...request.headers.entries()].filter(([key]) => key.toLowerCase() !== 'range') + ) + }); + const fullCachedResponse = await cache.match(fullContentKey); + if (fullCachedResponse) { + monitor.mark('cache_hit_full_content'); + return fullCachedResponse; + } + } catch (cacheError) { + console.warn('Cache API unavailable:', cacheError); + } + + return null; +} diff --git a/src/upstream/fetch-upstream.js b/src/upstream/fetch-upstream.js new file mode 100644 index 0000000..157cd6f --- /dev/null +++ b/src/upstream/fetch-upstream.js @@ -0,0 +1,415 @@ +import { configureAIHeaders } from '../protocols/ai.js'; +import { + fetchToken, + getScopeFromUrl, + parseAuthenticate, + readRegistryTokenResponse, + responseUnauthorized +} from '../protocols/docker.js'; +import { configureGitHeaders } from '../protocols/git.js'; +import { configureHuggingFaceHeaders } from '../protocols/huggingface.js'; +import { createErrorResponse } from '../utils/security.js'; + +const MEDIA_FILE_PATTERN = + /\.(mp4|avi|mkv|mov|wmv|flv|webm|mp3|wav|flac|aac|ogg|jpg|jpeg|png|gif|bmp|svg|pdf|zip|rar|7z|tar|gz|bz2|xz)$/i; + +/** + * Creates upstream fetch options for the current request. + * @param {{ + * authorization: string | null, + * canUseCache: boolean, + * config: import('../config/index.js').ApplicationConfig, + * request: Request, + * requestContext: { + * isAI: boolean, + * isDocker: boolean, + * isGit: boolean, + * isGitLFS: boolean, + * isHF: boolean, + * url: URL + * }, + * shouldPassthroughRequest: boolean, + * targetUrl: string + * }} options + * @returns {{ fetchOptions: RequestInit, requestHeaders: Headers }} Fetch options and mutable headers. + */ +function createFetchOptions({ + authorization, + canUseCache, + config, + request, + requestContext, + shouldPassthroughRequest, + targetUrl +}) { + const { isAI, isGit, isGitLFS, isHF, url } = requestContext; + + /** @type {RequestInit} */ + const fetchOptions = { + method: request.method, + headers: new Headers(), + redirect: 'follow' + }; + + if (request.body !== null && !canUseCache) { + fetchOptions.body = request.body; + } + + const requestHeaders = /** @type {Headers} */ (fetchOptions.headers); + + if (shouldPassthroughRequest) { + for (const [key, value] of request.headers.entries()) { + if (!['host', 'connection', 'upgrade', 'proxy-connection'].includes(key.toLowerCase())) { + requestHeaders.set(key, value); + } + } + + if (isGit || isGitLFS) { + configureGitHeaders(requestHeaders, request, url, isGitLFS); + } + + if (isAI) { + configureAIHeaders(requestHeaders, request); + } + + if (isHF) { + configureHuggingFaceHeaders(requestHeaders, request); + } + + return { fetchOptions, requestHeaders }; + } + + Object.assign(fetchOptions, { + cf: { + http3: true, + cacheTtl: config.CACHE_DURATION, + cacheEverything: true, + preconnect: true + } + }); + + requestHeaders.set('Accept-Encoding', 'gzip, deflate, br'); + requestHeaders.set('Connection', 'keep-alive'); + requestHeaders.set('User-Agent', 'Wget/1.21.3'); + + const origin = request.headers.get('Origin'); + if (origin) { + requestHeaders.set('Origin', origin); + } + + if (authorization) { + requestHeaders.set('Authorization', authorization); + } + + const rangeHeader = request.headers.get('Range'); + if (MEDIA_FILE_PATTERN.test(targetUrl) || rangeHeader) { + requestHeaders.set('Accept-Encoding', 'identity'); + } + + if (rangeHeader) { + requestHeaders.set('Range', rangeHeader); + } + + return { fetchOptions, requestHeaders }; +} + +/** + * Follows a Docker redirect without forwarding credentials to the redirected host. + * @param {Response} response + * @param {string} targetUrl + * @param {RequestInit} finalFetchOptions + * @returns {Promise} Redirect-followed response, or the original response when no redirect is needed. + */ +async function followDockerRedirectIfNeeded(response, targetUrl, finalFetchOptions) { + if ( + response.status !== 301 && + response.status !== 302 && + response.status !== 303 && + response.status !== 307 && + response.status !== 308 + ) { + return response; + } + + const location = response.headers.get('Location'); + if (!location) { + return response; + } + + const redirectHeaders = new Headers(finalFetchOptions.headers); + redirectHeaders.delete('Authorization'); + + const redirectOptions = /** @type {RequestInit} */ ({ + ...finalFetchOptions, + headers: redirectHeaders, + redirect: 'follow' + }); + + return await fetch(new URL(location, targetUrl), redirectOptions); +} + +/** + * Executes the upstream fetch, including HEAD fallback probing and Docker redirect handling. + * @param {{ + * fetchOptions: RequestInit, + * request: Request, + * requestContext: { + * isDocker: boolean + * }, + * requestHeaders: Headers, + * targetUrl: string + * }} options + * @returns {Promise} Upstream response. + */ +async function executeFetch({ fetchOptions, request, requestContext, requestHeaders, targetUrl }) { + const finalFetchOptions = /** @type {RequestInit} */ ({ + ...fetchOptions, + signal: /** @type {AbortSignal} */ (fetchOptions.signal) + }); + + if (requestContext.isDocker) { + finalFetchOptions.redirect = 'manual'; + } + + let response; + if (request.method === 'HEAD') { + response = await fetch(targetUrl, finalFetchOptions); + + if (response.ok && !response.headers.get('Content-Length')) { + const rangeHeaders = new Headers(requestHeaders); + rangeHeaders.set('Range', 'bytes=0-0'); + + const rangeResponse = await fetch(targetUrl, { + ...finalFetchOptions, + method: 'GET', + headers: rangeHeaders + }); + + let contentLength = null; + + if (rangeResponse.status === 206) { + const contentRange = rangeResponse.headers.get('Content-Range'); + if (contentRange) { + const match = contentRange.match(/bytes\s+\d+-\d+\/(\d+)/); + if (match) { + [, contentLength] = match; + } + } + } else if (rangeResponse.ok) { + contentLength = rangeResponse.headers.get('Content-Length'); + } + + if (contentLength) { + const headHeaders = new Headers(response.headers); + headHeaders.set('Content-Length', contentLength); + response = new Response(null, { + status: response.status, + statusText: response.statusText, + headers: headHeaders + }); + } + } + } else { + response = await fetch(targetUrl, finalFetchOptions); + } + + if (requestContext.isDocker) { + response = await followDockerRedirectIfNeeded(response, targetUrl, finalFetchOptions); + } + + return response; +} + +/** + * Retries a Docker request with an anonymous bearer token when the registry challenges first. + * @param {{ + * effectivePath: string, + * platform: string, + * requestHeaders: Headers, + * requestContext: { + * isDocker: boolean, + * url: URL + * }, + * response: Response, + * targetUrl: string, + * finalFetchOptions: RequestInit + * }} options + * @returns {Promise} Successful retried response, or a synthesized auth challenge response. + */ +async function retryDockerWithAnonymousToken({ + effectivePath, + finalFetchOptions, + platform, + requestContext, + requestHeaders, + response, + targetUrl +}) { + const authenticateStr = response.headers.get('WWW-Authenticate'); + const scope = getScopeFromUrl(requestContext.url, effectivePath, platform); + + if (authenticateStr) { + try { + const wwwAuthenticate = parseAuthenticate(authenticateStr); + const tokenResponse = await fetchToken(wwwAuthenticate, scope || '', ''); + + if (tokenResponse.ok) { + const token = await readRegistryTokenResponse(tokenResponse); + if (token) { + const retryHeaders = new Headers(requestHeaders); + retryHeaders.set('Authorization', `Bearer ${token}`); + + const retryOptions = /** @type {RequestInit} */ ({ + ...finalFetchOptions, + headers: retryHeaders, + redirect: 'manual' + }); + + let retryResponse = await fetch(targetUrl, retryOptions); + retryResponse = await followDockerRedirectIfNeeded( + retryResponse, + targetUrl, + retryOptions + ); + + if (retryResponse.ok) { + return retryResponse; + } + } + } + } catch (error) { + console.warn('Token fetch failed:', error); + } + } + + return responseUnauthorized(requestContext.url, platform); +} + +/** + * Fetches an upstream resource with retries and protocol-specific handling. + * @param {{ + * authorization: string | null, + * canUseCache: boolean, + * config: import('../config/index.js').ApplicationConfig, + * effectivePath: string, + * monitor: import('../utils/performance.js').PerformanceMonitor, + * platform: string, + * request: Request, + * requestContext: { + * isAI: boolean, + * isDocker: boolean, + * isGit: boolean, + * isGitLFS: boolean, + * isHF: boolean, + * url: URL + * }, + * shouldPassthroughRequest: boolean, + * targetUrl: string + * }} options + * @returns {Promise<{ response: Response, responseGeneratedLocally: boolean }>} Upstream or synthesized response. + */ +export async function fetchUpstreamResponse({ + authorization, + canUseCache, + config, + effectivePath, + monitor, + platform, + request, + requestContext, + shouldPassthroughRequest, + targetUrl +}) { + let response; + let responseGeneratedLocally = false; + const { fetchOptions, requestHeaders } = createFetchOptions({ + authorization, + canUseCache, + config, + request, + requestContext, + shouldPassthroughRequest, + targetUrl + }); + + let attempts = 0; + while (attempts < config.MAX_RETRIES) { + /** @type {ReturnType | undefined} */ + let timeoutId; + + try { + monitor.mark(`attempt_${attempts}`); + + const controller = new AbortController(); + timeoutId = setTimeout(() => controller.abort(), config.TIMEOUT_SECONDS * 1000); + + fetchOptions.signal = controller.signal; + response = await executeFetch({ + fetchOptions, + request, + requestContext, + requestHeaders, + targetUrl + }); + + if (response.ok || response.status === 206) { + monitor.mark('success'); + break; + } + + if (requestContext.isDocker && response.status === 401) { + monitor.mark('docker_auth_challenge'); + response = await retryDockerWithAnonymousToken({ + effectivePath, + finalFetchOptions: fetchOptions, + platform, + requestContext, + requestHeaders, + response, + targetUrl + }); + + if (response.ok) { + monitor.mark('success'); + } + break; + } + + if (response.status >= 400 && response.status < 500) { + monitor.mark('client_error'); + break; + } + + attempts++; + if (attempts < config.MAX_RETRIES) { + await new Promise(resolve => setTimeout(resolve, config.RETRY_DELAY_MS * attempts)); + } + } catch (error) { + attempts++; + if (error instanceof Error && error.name === 'AbortError') { + response = createErrorResponse('Request timeout', 408); + responseGeneratedLocally = true; + break; + } + + if (attempts >= config.MAX_RETRIES) { + response = createErrorResponse('Upstream request failed', 502); + responseGeneratedLocally = true; + break; + } + + await new Promise(resolve => setTimeout(resolve, config.RETRY_DELAY_MS * attempts)); + } finally { + if (timeoutId !== undefined) { + clearTimeout(timeoutId); + } + } + } + + if (!response) { + response = createErrorResponse('No response received after all retry attempts', 500); + responseGeneratedLocally = true; + } + + return { response, responseGeneratedLocally }; +} diff --git a/src/utils/validation.js b/src/utils/validation.js index edf5c75..fa04c1a 100644 --- a/src/utils/validation.js +++ b/src/utils/validation.js @@ -27,6 +27,43 @@ import { isAIInferenceRequest } from '../protocols/ai.js'; import { isGitLFSRequest, isGitRequest } from '../protocols/git.js'; import { isHuggingFaceAPIRequest } from '../protocols/huggingface.js'; +/** + * Computes protocol and request traits used across validation, routing, and response handling. + * @param {Request} request + * @param {URL} url + * @returns {{ + * isAI: boolean, + * isDocker: boolean, + * isGit: boolean, + * isGitLFS: boolean, + * isHF: boolean + * }} Request traits for the current request. + */ +export function getRequestTraits(request, url) { + return { + isAI: isAIInferenceRequest(request, url), + isDocker: isDockerRequest(request, url), + isGit: isGitRequest(request, url), + isGitLFS: isGitLFSRequest(request, url), + isHF: isHuggingFaceAPIRequest(request, url) + }; +} + +/** + * Checks whether a request should use protocol passthrough behavior. + * @param {{ + * isAI: boolean, + * isDocker: boolean, + * isGit: boolean, + * isGitLFS: boolean, + * isHF: boolean + * }} traits + * @returns {boolean} True when request handling should follow protocol passthrough rules. + */ +export function isProtocolRequest(traits) { + return traits.isGit || traits.isGitLFS || traits.isDocker || traits.isAI || traits.isHF; +} + /** * Best-effort decode for security validation. * @@ -141,13 +178,9 @@ export { isAIInferenceRequest, isGitLFSRequest, isGitRequest, isHuggingFaceAPIRe * @returns {string[]} Allowed HTTP methods for this request shape. */ export function getAllowedMethods(request, url, config = CONFIG) { - const isGit = isGitRequest(request, url); - const isGitLFS = isGitLFSRequest(request, url); - const isDocker = isDockerRequest(request, url); - const isAI = isAIInferenceRequest(request, url); - const isHF = isHuggingFaceAPIRequest(request, url); + const traits = getRequestTraits(request, url); - return isGit || isGitLFS || isDocker || isAI || isHF + return isProtocolRequest(traits) ? ['GET', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE'] : config.SECURITY.ALLOWED_METHODS; } @@ -165,10 +198,24 @@ export function getAllowedMethods(request, url, config = CONFIG) { * @param {Request} request - The incoming request object * @param {URL} url - Parsed URL object * @param {import('../config/index.js').ApplicationConfig} config - Configuration object + * @param {{ + * isAI: boolean, + * isDocker: boolean, + * isGit: boolean, + * isGitLFS: boolean, + * isHF: boolean + * }} traits - Pre-computed request traits to avoid repeated protocol detection. * @returns {{valid: boolean, error?: string, status?: number}} Validation result object */ -export function validateRequest(request, url, config = CONFIG) { - const allowedMethods = getAllowedMethods(request, url, config); +export function validateRequest( + request, + url, + config = CONFIG, + traits = getRequestTraits(request, url) +) { + const allowedMethods = isProtocolRequest(traits) + ? ['GET', 'HEAD', 'POST', 'PUT', 'PATCH', 'DELETE'] + : config.SECURITY.ALLOWED_METHODS; if (!allowedMethods.includes(request.method)) { return { valid: false, error: 'Method not allowed', status: 405 }; diff --git a/test/platforms/cran.test.js b/test/platforms/cran.test.js index 10587c7..edb0a46 100644 --- a/test/platforms/cran.test.js +++ b/test/platforms/cran.test.js @@ -1,5 +1,6 @@ import { describe, expect, it } from 'vitest'; -import { PLATFORMS, transformPath } from '../../src/config/platforms.js'; +import { PLATFORM_CATALOG as PLATFORMS } from '../../src/config/platform-catalog.js'; +import { transformPath } from '../../src/routing/platform-transformers.js'; describe('CRAN Platform Configuration', () => { it('should have CRAN platform configured', () => { diff --git a/test/platforms/crates.test.js b/test/platforms/crates.test.js index f438060..07ded27 100644 --- a/test/platforms/crates.test.js +++ b/test/platforms/crates.test.js @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import { transformPath } from '../../src/config/platforms.js'; +import { transformPath } from '../../src/routing/platform-transformers.js'; describe('crates.io path transformation', () => { it('should transform crate download URLs correctly', () => { diff --git a/test/platforms/flathub.test.js b/test/platforms/flathub.test.js index 92aa383..dbb6cd3 100644 --- a/test/platforms/flathub.test.js +++ b/test/platforms/flathub.test.js @@ -1,5 +1,6 @@ import { describe, expect, it } from 'vitest'; -import { PLATFORMS, transformPath } from '../../src/config/platforms.js'; +import { PLATFORM_CATALOG as PLATFORMS } from '../../src/config/platform-catalog.js'; +import { transformPath } from '../../src/routing/platform-transformers.js'; describe('Flathub Platform Configuration', () => { it('should have Flathub platform configured', () => { diff --git a/test/platforms/homebrew.test.js b/test/platforms/homebrew.test.js index b309eee..ad66d40 100644 --- a/test/platforms/homebrew.test.js +++ b/test/platforms/homebrew.test.js @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import { transformPath } from '../../src/config/platforms.js'; +import { transformPath } from '../../src/routing/platform-transformers.js'; describe('Homebrew path transformation', () => { describe('homebrew-api platform', () => { diff --git a/test/platforms/jenkins.test.js b/test/platforms/jenkins.test.js index c93e45f..d50ca6e 100644 --- a/test/platforms/jenkins.test.js +++ b/test/platforms/jenkins.test.js @@ -1,5 +1,5 @@ import { describe, expect, it } from 'vitest'; -import { transformPath } from '../../src/config/platforms.js'; +import { transformPath } from '../../src/routing/platform-transformers.js'; describe('Jenkins Plugin Support', () => { describe('Update Center Transformations', () => { diff --git a/test/platforms/opensuse.test.js b/test/platforms/opensuse.test.js index e7e0fe2..a3928ae 100644 --- a/test/platforms/opensuse.test.js +++ b/test/platforms/opensuse.test.js @@ -1,5 +1,6 @@ import { describe, expect, it } from 'vitest'; -import { PLATFORMS, transformPath } from '../../src/config/platforms.js'; +import { PLATFORM_CATALOG as PLATFORMS } from '../../src/config/platform-catalog.js'; +import { transformPath } from '../../src/routing/platform-transformers.js'; describe('openSUSE Platform Configuration', () => { it('should have openSUSE platform configured', () => { diff --git a/test/unit/app-structure.test.js b/test/unit/app-structure.test.js new file mode 100644 index 0000000..4a152fb --- /dev/null +++ b/test/unit/app-structure.test.js @@ -0,0 +1,52 @@ +import { describe, expect, it } from 'vitest'; + +import apiHandler, { config as vercelConfig } from '../../adapters/functions/api/index.js'; +import { handler as denoHandler } from '../../adapters/functions/deno.js'; +import { onRequest } from '../../adapters/pages/functions/[[path]].js'; +import { createRequestContext } from '../../src/app/request-context.js'; +import { PLATFORM_CATALOG } from '../../src/config/platform-catalog.js'; +import { normalizeEffectivePath, resolveTarget } from '../../src/routing/resolve-target.js'; + +describe('Application structure', () => { + it('builds a shared request context for protocol-aware routing', () => { + const request = new Request('https://example.com/ip/openai/v1/chat/completions', { + method: 'OPTIONS', + headers: { + Origin: 'https://app.example.com', + 'Access-Control-Request-Method': 'POST' + } + }); + + const context = createRequestContext(request, { + ALLOWED_METHODS: 'GET,HEAD,POST' + }); + + expect(context.isAI).toBe(true); + expect(context.isCorsPreflight).toBe(true); + expect(context.config.SECURITY.ALLOWED_METHODS).toContain('POST'); + }); + + it('normalizes Docker host-style paths before resolving upstream targets', () => { + const url = new URL('https://example.com/v2/cr/ghcr/xixu-me/xget/manifests/latest'); + const normalized = normalizeEffectivePath(url, true); + + expect('effectivePath' in normalized).toBe(true); + if ('effectivePath' in normalized) { + expect(normalized.effectivePath).toBe('/cr/ghcr/v2/xixu-me/xget/manifests/latest'); + + const target = resolveTarget(url, normalized.effectivePath, PLATFORM_CATALOG); + expect('response' in target).toBe(false); + if (!('response' in target)) { + expect(target.platform).toBe('cr-ghcr'); + expect(target.targetUrl).toBe('https://ghcr.io/v2/xixu-me/xget/manifests/latest'); + } + } + }); + + it('exposes thin runtime adapter entrypoints', () => { + expect(typeof apiHandler).toBe('function'); + expect(typeof denoHandler).toBe('function'); + expect(typeof onRequest).toBe('function'); + expect(vercelConfig).toEqual({ runtime: 'edge' }); + }); +}); diff --git a/test/unit/pipeline-modules.test.js b/test/unit/pipeline-modules.test.js new file mode 100644 index 0000000..c44022a --- /dev/null +++ b/test/unit/pipeline-modules.test.js @@ -0,0 +1,116 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { createRequestContext } from '../../src/app/request-context.js'; +import { CONFIG } from '../../src/config/index.js'; +import { finalizeResponse } from '../../src/response/finalize-response.js'; +import { tryReadCachedResponse } from '../../src/upstream/cache.js'; +import { fetchUpstreamResponse } from '../../src/upstream/fetch-upstream.js'; +import { PerformanceMonitor } from '../../src/utils/performance.js'; + +afterEach(() => { + vi.restoreAllMocks(); +}); + +describe('Pipeline modules', () => { + it('reuses cached full content for range requests through the cache helper', async () => { + const cache = { + match: vi + .fn() + .mockResolvedValueOnce(null) + .mockResolvedValueOnce( + new Response('full-body', { + status: 200, + headers: { 'Content-Type': 'text/plain' } + }) + ) + }; + const monitor = new PerformanceMonitor(); + const markSpy = vi.spyOn(monitor, 'mark'); + const request = new Request('https://example.com/gh/user/repo/file.txt', { + headers: { Range: 'bytes=0-3' } + }); + + const response = await tryReadCachedResponse({ + cache: /** @type {Cache} */ (/** @type {unknown} */ (cache)), + cacheTargetUrl: 'https://github.com/user/repo/file.txt', + canUseCache: true, + hasSensitiveHeaders: false, + monitor, + request, + requestContext: createRequestContext(request, {}) + }); + + expect(await response?.text()).toBe('full-body'); + expect(markSpy).toHaveBeenCalledWith('cache_hit_full_content'); + }); + + it('retries upstream fetches through the transport helper before succeeding', async () => { + const request = new Request('https://example.com/gh/user/repo/file.txt'); + const requestContext = createRequestContext(request, {}); + const fetchSpy = vi + .spyOn(globalThis, 'fetch') + .mockRejectedValueOnce(new Error('temporary-network-error')) + .mockResolvedValueOnce( + new Response('ok', { + status: 200, + headers: { 'Content-Type': 'text/plain' } + }) + ); + + const result = await fetchUpstreamResponse({ + authorization: null, + canUseCache: true, + config: { ...CONFIG, MAX_RETRIES: 2, RETRY_DELAY_MS: 0 }, + effectivePath: '/gh/user/repo/file.txt', + monitor: new PerformanceMonitor(), + platform: 'gh', + request, + requestContext, + shouldPassthroughRequest: false, + targetUrl: 'https://github.com/user/repo/file.txt' + }); + + expect(result.responseGeneratedLocally).toBe(false); + expect(result.response.status).toBe(200); + expect(fetchSpy).toHaveBeenCalledTimes(2); + }); + + it('rewrites npm metadata and refreshes content length during response finalization', async () => { + const request = new Request('https://example.com/npm/pkg'); + const requestContext = createRequestContext(request, {}); + const upstreamBody = JSON.stringify({ + dist: { + tarball: 'https://registry.npmjs.org/pkg/-/pkg-1.0.0.tgz' + } + }); + + const response = await finalizeResponse({ + cache: null, + cacheTargetUrl: 'https://registry.npmjs.org/pkg', + canUseCache: true, + config: CONFIG, + ctx: /** @type {ExecutionContext} */ ({ waitUntil() {}, passThroughOnException() {} }), + effectivePath: '/npm/pkg', + hasSensitiveHeaders: false, + monitor: new PerformanceMonitor(), + platform: 'npm', + request, + requestContext, + response: new Response(upstreamBody, { + status: 200, + headers: { + 'Content-Type': 'application/json', + 'Content-Length': String(upstreamBody.length) + } + }), + responseGeneratedLocally: false, + url: new URL(request.url) + }); + const body = await response.text(); + + expect(body).toContain('https://example.com/npm/pkg/-/pkg-1.0.0.tgz'); + expect(response.headers.get('Content-Length')).toBe( + String(new TextEncoder().encode(body).byteLength) + ); + }); +}); diff --git a/test/unit/platform-boundaries.test.js b/test/unit/platform-boundaries.test.js new file mode 100644 index 0000000..394b686 --- /dev/null +++ b/test/unit/platform-boundaries.test.js @@ -0,0 +1,31 @@ +import { describe, expect, it } from 'vitest'; + +import { PLATFORM_CATALOG } from '../../src/config/platform-catalog.js'; +import { PLATFORMS, SORTED_PLATFORMS, transformPath } from '../../src/config/platforms.js'; +import { getPlatformPathPrefix } from '../../src/routing/platform-index.js'; +import { transformPath as transformPlatformPath } from '../../src/routing/platform-transformers.js'; + +describe('Platform module boundaries', () => { + it('keeps the compatibility export wired to the platform catalog', () => { + expect(PLATFORMS).toBe(PLATFORM_CATALOG); + }); + + it('sorts platform keys by the longest routable prefix first', () => { + const prefixLengths = SORTED_PLATFORMS.map( + platformKey => getPlatformPathPrefix(platformKey).length + ); + + prefixLengths.forEach((length, index) => { + if (index < prefixLengths.length - 1) { + expect(length).toBeGreaterThanOrEqual(prefixLengths[index + 1]); + } + }); + }); + + it('routes legacy transform imports through the dedicated transformer module', () => { + expect(transformPath('/crates/?q=tokio', 'crates')).toBe( + transformPlatformPath('/crates/?q=tokio', 'crates') + ); + expect(transformPath('/jenkins/test-path', 'jenkins')).toBe('/current/test-path'); + }); +}); diff --git a/test/unit/platforms.test.js b/test/unit/platforms.test.js index 4e58bd5..703ff77 100644 --- a/test/unit/platforms.test.js +++ b/test/unit/platforms.test.js @@ -1,5 +1,6 @@ import { describe, expect, it } from 'vitest'; -import { PLATFORMS, transformPath } from '../../src/config/platforms.js'; +import { PLATFORM_CATALOG as PLATFORMS } from '../../src/config/platform-catalog.js'; +import { transformPath } from '../../src/routing/platform-transformers.js'; describe('Platform Configuration', () => { describe('Platform Definitions', () => { diff --git a/test/unit/xget-skill-script.test.js b/test/unit/xget-skill-script.test.js new file mode 100644 index 0000000..7c5ba9c --- /dev/null +++ b/test/unit/xget-skill-script.test.js @@ -0,0 +1,52 @@ +import { describe, expect, it } from 'vitest'; + +import { + createPlatformEntries, + extractPlatformsModule, + loadPlatformsFromSource +} from '../../skills/xget/scripts/xget.mjs'; + +describe('xget skill script', () => { + it('extracts platform data from the new platform catalog source', () => { + const source = `export const PLATFORM_CATALOG = { + gh: 'https://github.com', + 'cr-ghcr': 'https://ghcr.io' +}; + +export const PLATFORMS = PLATFORM_CATALOG; +`; + + expect(extractPlatformsModule(source)).toEqual({ + gh: 'https://github.com', + 'cr-ghcr': 'https://ghcr.io' + }); + }); + + it('still accepts the legacy PLATFORMS object source', () => { + const source = `export const PLATFORMS = { + npm: 'https://registry.npmjs.org' +}; +`; + + expect(extractPlatformsModule(source)).toEqual({ + npm: 'https://registry.npmjs.org' + }); + }); + + it('loads categorized platform entries from the extracted source', () => { + const entries = loadPlatformsFromSource(`export const PLATFORM_CATALOG = { + gh: 'https://github.com', + 'ip-openai': 'https://api.openai.com', + 'cr-ghcr': 'https://ghcr.io' +}; +`); + + expect(entries).toEqual( + createPlatformEntries({ + gh: 'https://github.com', + 'ip-openai': 'https://api.openai.com', + 'cr-ghcr': 'https://ghcr.io' + }) + ); + }); +});