Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: dynamic subdomain gateway detection #53

Merged
merged 5 commits into from
Feb 26, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions public/_redirects
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💭 I was not able to find where ?helia-sw is handled – assumed it is not wired up yet, but I've updated _redirects so it will work on web (subdomain or dnslink) gateway once we add it.

@SgtPooki what was the plan around ?helia-sw-subdomain and ?helia-sw? I want to add origin isolation enforcement in follow-up PR and most likely will have to touch this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?helia-sw & ?helia-sw-subdomain are no longer required and should be able to be removed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they were used to help with registering the service worker originally, but now, if the service worker is hit, we never render the jsx, and if the service worker isn't hit, we determine (in app.tsx) if we need to load the or not

Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
/ipns/* /?helia-sw=/ipns/:splat 302
/ipfs/* /?helia-sw=/ipfs/:splat 302
/* /?helia-sw=/:splat 302
4 changes: 2 additions & 2 deletions src/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ import './app.css'
import App from './app.tsx'
import { loadConfigFromLocalStorage } from './lib/config-db.ts'
import { isPathOrSubdomainRequest } from './lib/path-or-subdomain.ts'
import { BASE_URL } from './lib/webpack-constants.ts'
import RedirectPage from './redirectPage.tsx'

await loadConfigFromLocalStorage()
Expand All @@ -15,13 +14,14 @@ const sw = await navigator.serviceWorker.register(new URL('sw.ts', import.meta.u
const root = ReactDOMClient.createRoot(container)

// SW did not trigger for this request
if (isPathOrSubdomainRequest(BASE_URL, window.location)) {
if (isPathOrSubdomainRequest(window.location)) {
// but the requested path is something it should, so show redirect and redirect to the same URL
root.render(
<RedirectPage />
)
window.location.replace(window.location.href)
} else {
// TODO: add detection of DNSLink gateways (alowing use with Host: en.wikipedia-on-ipfs.org)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if en.wikipedia-on-ipfs.org.ipns.sw-host.tld is requested, this will never be hit.

// the requested path is not recognized as a path or subdomain request, so render the app UI
if (window.location.pathname !== '/') {
// pathname is not blank, but is invalid. redirect to the root
Expand Down
28 changes: 25 additions & 3 deletions src/lib/dns-link-labels.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,35 @@
import { CID } from 'multiformats/cid'

/**
* For dnslinks see https://specs.ipfs.tech/http-gateways/subdomain-gateway/#host-request-header
* DNSLink names include . which means they must be inlined into a single DNS label to provide unique origin and work with wildcard TLS certificates.
*/

// DNS label can have up to 63 characters, consisting of alphanumeric
// characters or hyphens -, but it must not start or end with a hyphen.
const dnsLabelRegex = /^[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?$/

/**
* We can receive either IPNS Name string or DNSLink label string here.
* IPNS Names do not have dots or dashes.
*/
export function isValidDnsLabel (label: string): boolean {
// If string is not a valid IPNS Name (CID)
// then we assume it may be a valid DNSLabel.
try {
CID.parse(label)
return false
} catch {
return dnsLabelRegex.test(label)
}
}

/**
* We can receive either a peerId string or dnsLink label string here. PeerId strings do not have dots or dashes.
* Checks if label looks like inlined DNSLink.
* (https://specs.ipfs.tech/http-gateways/subdomain-gateway/#host-request-header)
*/
export function isDnsLabel (label: string): boolean {
return ['-', '.'].some((char) => label.includes(char))
export function isInlinedDnsLink (label: string): boolean {
return isValidDnsLabel(label) && label.includes('-') && !label.includes('.')
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ to avoid false-positives added code to ensure an inlined DNSLink

  • does not parse as a valid CID
  • does not include .
  • includes at least one -

}

/**
Expand Down
22 changes: 9 additions & 13 deletions src/lib/heliaFetch.ts
Original file line number Diff line number Diff line change
@@ -1,14 +1,13 @@
import { createVerifiedFetch, type ContentTypeParser } from '@helia/verified-fetch'
import { fileTypeFromBuffer } from '@sgtpooki/file-type'
import { dnsLinkLabelDecoder, isDnsLabel } from './dns-link-labels.ts'
import type { Helia } from '@helia/interface'

export interface HeliaFetchOptions {
path: string
helia: Helia
signal?: AbortSignal
headers?: Headers
origin?: string | null
id?: string | null
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ had to rename, would be confusing for web devs ("origin" in web security context, including window.location, means at minimum both protocol and the authority identifier).

protocol?: string | null
}

Expand Down Expand Up @@ -116,29 +115,26 @@ function changeCssFontPath (path: string): string {
* heliaFetch should have zero awareness of whether it's being used inside a service worker or not.
*
* The `path` supplied should be either:
* * /ipfs/CID
* * /ipns/DNSLink
* * /ipns/key
* * /ipfs/CID (https://docs.ipfs.tech/concepts/content-addressing/)
* * /ipns/DNSLink (https://dnslink.dev/)
* * /ipns/IPNSName (https://specs.ipfs.tech/ipns/ipns-record/#ipns-name)
Comment on lines +118 to +120
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

*
* Things to do:
* * TODO: implement as much of the gateway spec as possible.
* * TODO: why we would be better than ipfs.io/other-gateway
* * TODO: have error handling that renders 404/500/other if the request is bad.
*
*/
export async function heliaFetch ({ path, helia, signal, headers, origin, protocol }: HeliaFetchOptions): Promise<Response> {
export async function heliaFetch ({ path, helia, signal, headers, id, protocol }: HeliaFetchOptions): Promise<Response> {
const verifiedFetch = await createVerifiedFetch(helia, {
contentTypeParser
})

let verifiedFetchUrl: string
if (origin != null && protocol != null) {
if (protocol === 'ipns' && isDnsLabel(origin)) {
verifiedFetchUrl = `${protocol}://${dnsLinkLabelDecoder(origin)}/${path}`
} else {
// likely a peerId instead of a dnsLink label
verifiedFetchUrl = `${protocol}://${origin}${path}`
}

if (id != null && protocol != null) {
verifiedFetchUrl = `${protocol}://${id}${path}`

// eslint-disable-next-line no-console
console.log('subdomain fetch for ', verifiedFetchUrl)
} else {
Expand Down
7 changes: 4 additions & 3 deletions src/lib/path-or-subdomain.ts
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💭 I did not clean this up to make the PR smaller and easier to review.

My plan is to move majority of subdomain handling to @helia/verified-fetch in future PRs and then get back here and decide what minimal handling makes sense to remain in this project.

Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
const subdomainRegex = /^(?<origin>[^/]+)\.(?<protocol>ip[fn]s)?$/
// TODO: dry, this is same regex code as in getSubdomainParts
const subdomainRegex = /^(?<id>[^/]+)\.(?<protocol>ip[fn]s)\.[^/]+$/
const pathRegex = /^\/(?<protocol>ip[fn]s)\/(?<path>.*)$/

export const isPathOrSubdomainRequest = (baseUrl: string, location: Pick<Location, 'hostname' | 'pathname'>): boolean => {
const subdomain = location.hostname.replace(`.${baseUrl}`, '')
export const isPathOrSubdomainRequest = (location: Pick<Location, 'hostname' | 'pathname'>): boolean => {
const subdomain = location.hostname
const subdomainMatch = subdomain.match(subdomainRegex)

const pathMatch = location.pathname.match(pathRegex)
Expand Down
10 changes: 0 additions & 10 deletions src/lib/webpack-constants.ts

This file was deleted.

39 changes: 24 additions & 15 deletions src/sw.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@
// import { clientsClaim } from 'workbox-core'
import mime from 'mime-types'
import { getHelia } from './get-helia.ts'
import { dnsLinkLabelDecoder, isInlinedDnsLink } from './lib/dns-link-labels.ts'
import { heliaFetch } from './lib/heliaFetch.ts'
import { BASE_URL } from './lib/webpack-constants.ts'
import type { Helia } from '@helia/interface'

declare let self: ServiceWorkerGlobalScope
Expand Down Expand Up @@ -51,8 +51,8 @@ const fetchHandler = async ({ path, request }: FetchHandlerArg): Promise<Respons
// 5 minute timeout
const abortController = AbortSignal.timeout(5 * 60 * 1000)
try {
const { origin, protocol } = getSubdomainParts(request)
return await heliaFetch({ path, helia, signal: abortController, headers: request.headers, origin, protocol })
const { id, protocol } = getSubdomainParts(request)
return await heliaFetch({ path, helia, signal: abortController, headers: request.headers, id, protocol })
} catch (err: unknown) {
const errorMessages: string[] = []
if (isAggregateError(err)) {
Expand Down Expand Up @@ -91,23 +91,34 @@ const isRootRequestForContent = (event: FetchEvent): boolean => {
return isRootRequest // && getCidFromUrl(event.request.url) != null
}

function getSubdomainParts (request: Request): { origin: string | null, protocol: string | null } {
function getSubdomainParts (request: Request): { id: string | null, protocol: string | null } {
const urlString = request.url
const url = new URL(urlString)
const subdomain = url.hostname.replace(`.${BASE_URL}`, '')
const subdomainRegex = /^(?<origin>[^/]+)\.(?<protocol>ip[fn]s)?$/
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ this produced invalid results for docs.ipfs.tech.ipns.example.com, the new code walks labels from right to left, which is better order for avoiding false-positive matches.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we redirect from subdomains like "docs.ipfs.tech" to "docs-ipfs-tech" ?

const subdomainMatch = subdomain.match(subdomainRegex)
const { origin, protocol } = subdomainMatch?.groups ?? { origin: null, protocol: null }
const labels = new URL(urlString).hostname.split('.')
let id: string | null = null; let protocol: string | null = null

// DNS label inspection happens from from right to left
// to work fine with edge cases like docs.ipfs.tech.ipns.foo.localhost
for (let i = labels.length - 1; i >= 0; i--) {
if (labels[i].startsWith('ipfs') || labels[i].startsWith('ipns')) {
protocol = labels[i]
id = labels.slice(0, i).join('.')
if (protocol === 'ipns' && isInlinedDnsLink(id)) {
// un-inline DNSLink names according to https://specs.ipfs.tech/http-gateways/subdomain-gateway/#host-request-header
id = dnsLinkLabelDecoder(id)
}
break
}
}

return { origin, protocol }
return { id, protocol }
}

function isSubdomainRequest (event: FetchEvent): boolean {
const { origin, protocol } = getSubdomainParts(event.request)
console.log('isSubdomainRequest.origin: ', origin)
const { id, protocol } = getSubdomainParts(event.request)
console.log('isSubdomainRequest.id: ', id)
console.log('isSubdomainRequest.protocol: ', protocol)

return origin != null && protocol != null
return id != null && protocol != null
}

const isValidRequestForSW = (event: FetchEvent): boolean =>
Expand Down Expand Up @@ -151,8 +162,6 @@ self.addEventListener('fetch', event => {
const newUrlString = newParts.join('/') + '/' + destinationParts.slice(index).join('/')
const newUrl = new URL(newUrlString)

// const { origin, protocol } = getSubdomainParts(event)

/**
* respond with redirect to newUrl
*/
Expand Down
23 changes: 17 additions & 6 deletions tests/path-or-subdomain.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,33 +4,44 @@ import { isPathOrSubdomainRequest } from '../src/lib/path-or-subdomain.ts'

describe('isPathOrSubdomainRequest', () => {
it('returns true for path-based request', () => {
expect(isPathOrSubdomainRequest('example.com', {
expect(isPathOrSubdomainRequest({
hostname: 'example.com',
pathname: '/ipfs/bafyFoo'
})).to.equal(true)
expect(isPathOrSubdomainRequest('example.com', {
expect(isPathOrSubdomainRequest({
hostname: 'example.com',
pathname: '/ipns/specs.ipfs.tech'
})).to.equal(true)
})

it('returns true for subdomain request', () => {
expect(isPathOrSubdomainRequest('example.com', {
expect(isPathOrSubdomainRequest({
hostname: 'bafyFoo.ipfs.example.com',
pathname: '/'
})).to.equal(true)
expect(isPathOrSubdomainRequest('example.com', {
expect(isPathOrSubdomainRequest({
hostname: 'docs.ipfs.tech.ipns.example.com',
pathname: '/'
})).to.equal(true)
})

it('returns true for inlined dnslink subdomain request', () => {
expect(isPathOrSubdomainRequest({
hostname: 'bafyFoo.ipfs.example.com',
pathname: '/'
})).to.equal(true)
expect(isPathOrSubdomainRequest({
hostname: 'specs-ipfs-tech.ipns.example.com',
pathname: '/'
})).to.equal(true)
})

it('returns false for non-path and non-subdomain request', () => {
expect(isPathOrSubdomainRequest('example.com', {
expect(isPathOrSubdomainRequest({
hostname: 'example.com',
pathname: '/foo/bar'
})).to.equal(false)
expect(isPathOrSubdomainRequest('example.com', {
expect(isPathOrSubdomainRequest({
hostname: 'foo.bar.example.com',
pathname: '/'
})).to.equal(false)
Expand Down
2 changes: 1 addition & 1 deletion webpack.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ const dev = {
// Only update what has changed on hot reload
hot: true,
port: 3000,
allowedHosts: [process.env.BASE_URL ?? 'helia-sw-gateway.localhost', 'localhost']
allowedHosts: ['helia-sw-gateway.localhost', 'localhost']
},

plugins: [
Expand Down
Loading