diff --git a/skills/xget/SKILL.md b/skills/xget/SKILL.md index 9d23f11..86a6a62 100644 --- a/skills/xget/SKILL.md +++ b/skills/xget/SKILL.md @@ -29,8 +29,8 @@ self-hosted option exists. - `XGET_BASE_URL` from the environment - if neither exists and the task is not just writing docs, ask the user for their self-hosted Xget domain and configure `XGET_BASE_URL` - - `https://xget.example.com` only for docs or templates when a real domain - is not available yet + - `https://xget.example.com` only for docs or templates when a real domain is + not available yet - `https://xget.xi-xu.me` only as an explicitly labeled fallback after the user declines or cannot provide a self-hosted domain 2. Keep platform data fresh. Do not hardcode the full prefix list from memory. @@ -51,7 +51,8 @@ node scripts/xget.mjs convert --base-url https://xget.example.com --url https:// 1. Identify the user's goal: - convert one or more upstream URLs - - generate config snippets for npm, pip, Go, NuGet, Cargo, Docker, or AI SDKs + - generate config snippets for npm, pip, Go, NuGet, Docker, or AI SDKs + - explain the current Cargo limitation for registry source replacement - explain which Xget prefix to use - propose or document a self-hosted deployment 2. Refresh the live platform map with `scripts/xget.mjs` if the answer depends @@ -62,6 +63,7 @@ node scripts/xget.mjs convert --base-url https://xget.example.com --url https:// 5. Before finishing, sanity-check that every example uses the right Xget path shape: - repo/content: `/{prefix}/...` + - crates.io HTTP URLs: `/crates/...` rather than `/crates/api/v1/crates/...` - inference APIs: `/ip/{provider}/...` - OCI registries: `/cr/{registry}/...` @@ -94,7 +96,10 @@ node scripts/xget.mjs platforms --format table [references/REFERENCE.md](references/REFERENCE.md). - If an upstream URL does not match any known platform, do not invent a prefix. Report that no current Xget mapping was found. -- When writing pip config for HTTPS domains, keep `trusted-host` aligned with - the actual host only if the user really needs it. +- The default pip snippet should omit `trusted-host`; add it only when the + deployment really needs it and keep it aligned with the actual host. +- The `cargo` preset is informational for now. Xget can rewrite direct + `crates.io` HTTP URLs under `/crates/...`, but this skill should not emit + Cargo source replacement config until Xget exposes a registry index endpoint. - When generating docs or templates without a real domain, prefer `https://xget.example.com` over the public demo. diff --git a/skills/xget/references/REFERENCE.md b/skills/xget/references/REFERENCE.md index ae5a0a0..fbfb48b 100644 --- a/skills/xget/references/REFERENCE.md +++ b/skills/xget/references/REFERENCE.md @@ -9,7 +9,9 @@ Use these defaults in order: 3. `https://xget.example.com` only for templates and docs 4. `https://xget.xi-xu.me` only as a clearly labeled public-demo fallback -The Xget README explicitly labels `xget.xi-xu.me` as a pre-deployed instance with no reliability guarantee, while the self-hosting docs and DigitalOcean guide show recommended self-hosted domains such as `xget.example.com`. +The Xget README explicitly labels `xget.xi-xu.me` as a pre-deployed instance +with no reliability guarantee, while the self-hosting docs and DigitalOcean +guide show recommended self-hosted domains such as `xget.example.com`. ## Live platform source @@ -44,7 +46,11 @@ The script derives these path shapes from platform keys: - pip simple index: `https://{base}/pypi/simple/` - Go proxy: `https://{base}/golang` - NuGet v3 index: `https://{base}/nuget/v3/index.json` -- Cargo registry: `https://{base}/crates/` +- Direct crates.io HTTP URLs: `https://{base}/crates/...` + +Cargo registry source replacement is not currently emitted by this skill. Xget's +`/crates/...` route rewrites direct crates.io HTTP API and download URLs, but it +does not expose a Cargo registry index endpoint. ### Container registries @@ -76,7 +82,7 @@ Representative presets: - `pip` - `go` - `nuget` -- `cargo` +- `cargo` (returns the current limitation instead of source replacement config) - `docker-ghcr` - `openai` - `anthropic` @@ -98,7 +104,7 @@ services: image: ghcr.io/xixu-me/xget:latest container_name: xget ports: - - "127.0.0.1:8080:8080" + - '127.0.0.1:8080:8080' restart: unless-stopped ``` @@ -109,8 +115,14 @@ Representative reverse-proxy outcome: ## Troubleshooting heuristics -- `404` on converted URLs often means the wrong prefix or an unmatched upstream platform. -- pip issues often come from mixing the right `index-url` with the wrong host in `trusted-host`. +- `404` on converted URLs often means the wrong prefix or an unmatched upstream + platform. +- crates.io conversions should strip the upstream `/api/v1/crates` prefix before + adding `/crates/...`. +- pip issues often come from adding `trusted-host` unnecessarily or pointing it + at the wrong host. - Docker examples must use `/cr/{registry}` prefixes, not plain `/{prefix}`. -- AI SDK examples usually need the Xget base URL changed but keep the original API key behavior. -- If the user asks for the “latest” supported platform, refresh the live platform map before answering. +- AI SDK examples usually need the Xget base URL changed but keep the original + API key behavior. +- If the user asks for the “latest” supported platform, refresh the live + platform map before answering. diff --git a/skills/xget/scripts/xget.mjs b/skills/xget/scripts/xget.mjs index 08abe69..483dac5 100644 --- a/skills/xget/scripts/xget.mjs +++ b/skills/xget/scripts/xget.mjs @@ -2,12 +2,51 @@ import { get } from 'node:https'; import process from 'node:process'; +import { pathToFileURL } from 'node:url'; import vm from 'node:vm'; const DEFAULT_SOURCE_URL = 'https://raw.gitcode.com/xixu-me/xget/raw/main/src/config/platforms.js'; const DEFAULT_BASE_PLACEHOLDER = 'https://xget.example.com'; +const CRATES_API_PREFIX = '/api/v1/crates'; + +/** + * @typedef {'resource' | 'registry' | 'inference'} PlatformCategory + */ + +/** + * @typedef {{ key: string, upstream: string, pathPrefix: string, category: PlatformCategory }} PlatformEntry + */ + +/** + * @typedef {{ + * help?: boolean, + * format?: string, + * url?: string, + * preset?: string, + * 'source-url'?: string, + * 'base-url'?: string, + * [key: string]: string | boolean | undefined + * }} CliOptions + */ + +/** + * @typedef {{ command: string, options: CliOptions }} ParsedArgs + */ + +/** + * @typedef {{ + * preset: string, + * summary: string, + * commands?: string[], + * env?: Record, + * files?: Record, + * notes?: string[], + * supported?: boolean + * }} Snippet + */ + function printHelp() { console.log(`Usage: node scripts/xget.mjs [options] @@ -33,7 +72,9 @@ convert options: snippet options: --base-url URL Xget base URL. Defaults to XGET_BASE_URL. --preset NAME One of: npm, pip, go, nuget, cargo, docker-ghcr, - openai, anthropic, gemini. + openai, anthropic, gemini. The cargo preset + explains the current limitation instead of + emitting source replacement config. --format json|text Examples: @@ -43,12 +84,17 @@ Examples: `); } +/** + * @param {string[]} argv + * @returns {ParsedArgs} + */ function parseArgs(argv) { const [command = 'help', ...rest] = argv; if (command === '--help') { return { command: 'help', options: { help: true } }; } + /** @type {CliOptions} */ const options = {}; for (let index = 0; index < rest.length; index += 1) { @@ -75,11 +121,28 @@ function parseArgs(argv) { return { command, options }; } +/** + * @param {unknown} error + * @returns {string} + */ +function getErrorMessage(error) { + return error instanceof Error ? error.message : String(error); +} + +/** + * @param {string} message + * @param {number} [code] + * @returns {never} + */ function fail(message, code = 1) { console.error(`Error: ${message}`); process.exit(code); } +/** + * @param {string} url + * @returns {Promise} + */ function httpGet(url) { return new Promise((resolve, reject) => { get(url, response => { @@ -109,7 +172,11 @@ function httpGet(url) { }); } -function extractPlatformsModule(jsSource) { +/** + * @param {string} jsSource + * @returns {Record} + */ +export function extractPlatformsModule(jsSource) { const match = jsSource.match(/export const PLATFORMS = (\{[\s\S]*?\n\});/); if (!match) { @@ -119,13 +186,15 @@ function extractPlatformsModule(jsSource) { try { return vm.runInNewContext(`(${match[1]})`); } catch (error) { - fail(`Could not parse remote PLATFORMS object: ${error.message}`); + fail(`Could not parse remote PLATFORMS object: ${getErrorMessage(error)}`); } } -async function loadPlatforms(sourceUrl) { - const jsSource = await httpGet(sourceUrl); - const platforms = extractPlatformsModule(jsSource); +/** + * @param {Record} platforms + * @returns {PlatformEntry[]} + */ +export function createPlatformEntries(platforms) { return Object.entries(platforms) .sort(([left], [right]) => left.localeCompare(right)) .map(([key, upstream]) => ({ @@ -140,8 +209,30 @@ async function loadPlatforms(sourceUrl) { })); } +/** + * @param {string} jsSource + * @returns {PlatformEntry[]} + */ +export function loadPlatformsFromSource(jsSource) { + const platforms = extractPlatformsModule(jsSource); + return createPlatformEntries(platforms); +} + +/** + * @param {string} sourceUrl + * @returns {Promise} + */ +async function loadPlatforms(sourceUrl) { + const jsSource = await httpGet(sourceUrl); + return loadPlatformsFromSource(jsSource); +} + +/** + * @param {string | undefined} value + * @returns {string | null} + */ function normalizeBaseUrl(value) { - if (!value) { + if (typeof value !== 'string' || !value) { return null; } @@ -156,6 +247,11 @@ function normalizeBaseUrl(value) { } } +/** + * @param {string} value + * @param {string} flagName + * @returns {URL} + */ function normalizeAbsoluteUrl(value, flagName) { try { return new URL(value); @@ -164,19 +260,168 @@ function normalizeAbsoluteUrl(value, flagName) { } } -function findPlatformForUrl(platforms, originUrl) { - const origin = originUrl.origin; - return platforms.find(({ upstream }) => upstream === origin) ?? null; +/** + * @param {string} pathname + * @returns {string} + */ +function normalizePathname(pathname) { + if (!pathname || pathname === '/') { + return ''; + } + + return pathname.replace(/\/+$/, ''); } -function buildConvertedUrl(baseUrl, platform, originUrl) { - const suffix = originUrl.pathname + originUrl.search + originUrl.hash; +/** + * @param {string} pathname + * @param {string} prefix + * @param {boolean} [caseInsensitive] + * @returns {boolean} + */ +function matchesPathPrefix(pathname, prefix, caseInsensitive = false) { + const normalizedPath = normalizePathname(pathname); + const normalizedPrefix = normalizePathname(prefix); + + if (!normalizedPrefix) { + return true; + } + + if (!normalizedPath) { + return false; + } + + if (caseInsensitive) { + const lowerPath = normalizedPath.toLowerCase(); + const lowerPrefix = normalizedPrefix.toLowerCase(); + return lowerPath === lowerPrefix || lowerPath.startsWith(`${lowerPrefix}/`); + } + + return normalizedPath === normalizedPrefix || normalizedPath.startsWith(`${normalizedPrefix}/`); +} + +/** + * @param {string} pathname + * @param {string} prefix + * @param {boolean} [caseInsensitive] + * @returns {string} + */ +function stripPathPrefix(pathname, prefix, caseInsensitive = false) { + const normalizedPrefix = normalizePathname(prefix); + if (!normalizedPrefix) { + return pathname; + } + + const flags = caseInsensitive ? 'i' : ''; + const escapedPrefix = normalizedPrefix.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + return pathname.replace(new RegExp(`^${escapedPrefix}(?=/|$)`, flags), ''); +} + +/** + * @param {PlatformEntry[]} platforms + * @param {string} key + * @returns {PlatformEntry | null} + */ +function findPlatformByKey(platforms, key) { + return platforms.find(platform => platform.key === key) ?? null; +} + +/** + * @param {PlatformEntry[]} platforms + * @param {URL} originUrl + * @returns {PlatformEntry | null} + */ +function findSpecialPlatformForUrl(platforms, originUrl) { + if (originUrl.hostname === 'ghcr.io') { + if (originUrl.pathname.startsWith('/v2/homebrew/')) { + return findPlatformByKey(platforms, 'homebrew-bottles'); + } + + return findPlatformByKey(platforms, 'cr-ghcr'); + } + + return null; +} + +/** + * @param {PlatformEntry[]} platforms + * @param {URL} originUrl + * @returns {PlatformEntry | null} + */ +export function findPlatformForUrl(platforms, originUrl) { + const specialPlatform = findSpecialPlatformForUrl(platforms, originUrl); + if (specialPlatform) { + return specialPlatform; + } + + const matchingPlatforms = platforms + .filter(platform => { + const upstreamUrl = new URL(platform.upstream); + if (upstreamUrl.origin !== originUrl.origin) { + return false; + } + + const caseInsensitive = platform.key === 'homebrew' || platform.key === 'homebrew-api'; + return matchesPathPrefix(originUrl.pathname, upstreamUrl.pathname, caseInsensitive); + }) + .sort((left, right) => { + const leftPathLength = normalizePathname(new URL(left.upstream).pathname).length; + const rightPathLength = normalizePathname(new URL(right.upstream).pathname).length; + return rightPathLength - leftPathLength; + }); + + return matchingPlatforms[0] ?? null; +} + +/** + * @param {PlatformEntry} platform + * @param {URL} originUrl + * @returns {string} + */ +export function getConvertedSuffix(platform, originUrl) { + let pathname = originUrl.pathname; + + if (platform.key === 'homebrew') { + pathname = stripPathPrefix(pathname, '/Homebrew', true); + } else if (platform.key === 'homebrew-api') { + pathname = stripPathPrefix(pathname, '/api', true); + } else if (platform.key === 'crates') { + pathname = stripPathPrefix(pathname, CRATES_API_PREFIX, true); + } else { + const upstreamPath = new URL(platform.upstream).pathname; + pathname = stripPathPrefix(pathname, upstreamPath); + } + + if (!pathname) { + pathname = '/'; + } + + if (!pathname.startsWith('/')) { + pathname = `/${pathname}`; + } + + return `${pathname}${originUrl.search}${originUrl.hash}`; +} + +/** + * @param {string} baseUrl + * @param {PlatformEntry} platform + * @param {URL} originUrl + * @returns {string} + */ +export function buildConvertedUrl(baseUrl, platform, originUrl) { + const suffix = getConvertedSuffix(platform, originUrl); return `${baseUrl}${platform.pathPrefix}${suffix.replace(/^\/+/, '')}`; } -function createSnippet(baseUrl, preset) { +/** + * @param {string} baseUrl + * @param {string} preset + * @returns {Snippet} + */ +export function createSnippet(baseUrl, preset) { const host = new URL(baseUrl).host; + /** @type {Record} */ const snippets = { npm: { preset, @@ -186,10 +431,9 @@ function createSnippet(baseUrl, preset) { pip: { preset, summary: 'Configure pip to use the Xget PyPI simple index.', - commands: [ - `pip config set global.index-url ${baseUrl}/pypi/simple/`, - `pip config set global.trusted-host ${host}`, - 'pip config list' + commands: [`pip config set global.index-url ${baseUrl}/pypi/simple/`, 'pip config list'], + notes: [ + `Only add "pip config set global.trusted-host ${host}" when the deployment really needs it.` ] }, go: { @@ -207,16 +451,12 @@ function createSnippet(baseUrl, preset) { }, cargo: { preset, - summary: 'Route crates.io traffic through Xget.', - files: { - '~/.cargo/config.toml': [ - '[source.crates-io]', - 'replace-with = "xget"', - '', - '[source.xget]', - `registry = "${baseUrl}/crates/"` - ].join('\n') - } + supported: false, + summary: 'Cargo registry source replacement is not currently supported by Xget.', + notes: [ + 'Xget can convert direct crates.io HTTP URLs under /crates/..., but it does not expose a Cargo registry index.', + 'Do not generate ~/.cargo/config.toml source replacement entries until Xget provides a registry index endpoint.' + ] }, 'docker-ghcr': { preset, @@ -257,16 +497,29 @@ function createSnippet(baseUrl, preset) { return result; } +/** + * @param {unknown} value + * @returns {void} + */ function renderJson(value) { console.log(JSON.stringify(value, null, 2)); } +/** + * @param {PlatformEntry[]} rows + * @returns {void} + */ function renderTable(rows) { + /** @type {Array} */ const headers = ['key', 'category', 'pathPrefix', 'upstream']; const widths = headers.map(header => Math.max(header.length, ...rows.map(row => String(row[header]).length)) ); + /** + * @param {Record} row + * @returns {string} + */ const formatRow = row => headers.map((header, index) => String(row[header]).padEnd(widths[index])).join(' '); @@ -275,6 +528,10 @@ function renderTable(rows) { rows.forEach(row => console.log(formatRow(row))); } +/** + * @param {Snippet} snippet + * @returns {void} + */ function renderTextSnippet(snippet) { console.log(snippet.summary); @@ -295,6 +552,21 @@ function renderTextSnippet(snippet) { console.log(content); }); } + + if (snippet.notes) { + console.log('\nNotes:'); + snippet.notes.forEach(note => console.log(note)); + } +} + +/** + * @param {CliOptions} options + * @param {string} key + * @returns {string | undefined} + */ +function getStringOption(options, key) { + const value = options[key]; + return typeof value === 'string' ? value : undefined; } async function main() { @@ -305,8 +577,8 @@ async function main() { return; } - const sourceUrl = options['source-url'] ?? DEFAULT_SOURCE_URL; - const format = options.format ?? 'json'; + const sourceUrl = getStringOption(options, 'source-url') ?? DEFAULT_SOURCE_URL; + const format = getStringOption(options, 'format') ?? 'json'; if (command === 'platforms') { const platforms = await loadPlatforms(sourceUrl); @@ -329,10 +601,10 @@ async function main() { if (command === 'convert') { const baseUrl = - normalizeBaseUrl(options['base-url'] ?? process.env.XGET_BASE_URL) ?? + normalizeBaseUrl(getStringOption(options, 'base-url') ?? process.env.XGET_BASE_URL) ?? fail(`Missing --base-url and XGET_BASE_URL. For docs, use ${DEFAULT_BASE_PLACEHOLDER}.`, 2); - const rawUrl = options.url; + const rawUrl = getStringOption(options, 'url'); if (!rawUrl) { fail('Missing --url for convert.', 2); } @@ -345,12 +617,13 @@ async function main() { fail(`No current Xget platform matched upstream origin ${originUrl.origin}.`, 3); } + const convertedUrl = buildConvertedUrl(baseUrl, platform, originUrl); const payload = { sourceUrl, baseUrl, upstreamUrl: originUrl.toString(), matchedPlatform: platform, - convertedUrl: buildConvertedUrl(baseUrl, platform, originUrl) + convertedUrl }; if (format === 'json') { @@ -368,10 +641,10 @@ async function main() { if (command === 'snippet') { const baseUrl = - normalizeBaseUrl(options['base-url'] ?? process.env.XGET_BASE_URL) ?? + normalizeBaseUrl(getStringOption(options, 'base-url') ?? process.env.XGET_BASE_URL) ?? fail(`Missing --base-url and XGET_BASE_URL. For docs, use ${DEFAULT_BASE_PLACEHOLDER}.`, 2); - const preset = options.preset; + const preset = getStringOption(options, 'preset'); if (!preset) { fail('Missing --preset for snippet.', 2); } @@ -393,4 +666,8 @@ async function main() { fail(`Unknown command "${command}". Use --help for supported commands.`, 2); } -main().catch(error => fail(error.message)); +const entryHref = process.argv[1] ? pathToFileURL(process.argv[1]).href : null; + +if (entryHref === import.meta.url) { + main().catch(error => fail(getErrorMessage(error))); +} diff --git a/test/unit/xget-skill.test.js b/test/unit/xget-skill.test.js new file mode 100644 index 0000000..24ebe65 --- /dev/null +++ b/test/unit/xget-skill.test.js @@ -0,0 +1,87 @@ +import { describe, expect, it } from 'vitest'; + +import { PLATFORMS } from '../../src/config/platforms.js'; +import { + buildConvertedUrl, + createPlatformEntries, + createSnippet, + findPlatformForUrl +} from '../../skills/xget/scripts/xget.mjs'; + +const BASE_URL = 'https://xget.example.com'; +const platforms = createPlatformEntries(PLATFORMS); + +/** + * Convert an upstream URL with the xget skill helpers. + * @param {string} url + * @returns {{ platform: { key: string } | null, convertedUrl: string | null }} Converted result. + */ +function convert(url) { + const upstreamUrl = new URL(url); + const platform = findPlatformForUrl(platforms, upstreamUrl); + + return { + platform, + convertedUrl: platform ? buildConvertedUrl(BASE_URL, platform, upstreamUrl) : null + }; +} + +describe('xget skill helpers', () => { + describe('URL conversion', () => { + it('converts Homebrew repository URLs to the homebrew prefix', () => { + const result = convert('https://github.com/Homebrew/homebrew-core/raw/HEAD/Formula/g/git.rb'); + + expect(result.platform?.key).toBe('homebrew'); + expect(result.convertedUrl).toBe( + 'https://xget.example.com/homebrew/homebrew-core/raw/HEAD/Formula/g/git.rb' + ); + }); + + it('converts Homebrew API URLs to the homebrew/api prefix', () => { + const result = convert('https://formulae.brew.sh/api/formula/git.json'); + + expect(result.platform?.key).toBe('homebrew-api'); + expect(result.convertedUrl).toBe('https://xget.example.com/homebrew/api/formula/git.json'); + }); + + it('routes Homebrew bottle URLs to homebrew/bottles instead of cr/ghcr', () => { + const result = convert('https://ghcr.io/v2/homebrew/core/git/manifests/2.39.0'); + + expect(result.platform?.key).toBe('homebrew-bottles'); + expect(result.convertedUrl).toBe( + 'https://xget.example.com/homebrew/bottles/v2/homebrew/core/git/manifests/2.39.0' + ); + }); + + it('normalizes crates.io API URLs to the documented /crates path shape', () => { + const result = convert('https://crates.io/api/v1/crates/serde/1.0.0/download'); + + expect(result.platform?.key).toBe('crates'); + expect(result.convertedUrl).toBe('https://xget.example.com/crates/serde/1.0.0/download'); + }); + }); + + describe('snippets', () => { + it('omits trusted-host from the default pip preset', () => { + const snippet = createSnippet(BASE_URL, 'pip'); + + expect(snippet.commands).toEqual([ + 'pip config set global.index-url https://xget.example.com/pypi/simple/', + 'pip config list' + ]); + expect(snippet.notes).toContain( + 'Only add "pip config set global.trusted-host xget.example.com" when the deployment really needs it.' + ); + }); + + it('returns an explicit unsupported notice for cargo instead of a broken config file', () => { + const snippet = createSnippet(BASE_URL, 'cargo'); + + expect(snippet.supported).toBe(false); + expect(snippet.files).toBeUndefined(); + expect(snippet.notes).toContain( + 'Do not generate ~/.cargo/config.toml source replacement entries until Xget provides a registry index endpoint.' + ); + }); + }); +});