165 Commits
Author SHA1 Message Date
Julien HerrandClaude Opus 4.7 97ce9a62b4 feat: reader-rendering correctness + privacy hardening (P1·S batch)
Close the five open P1·S items from TODO.md:
- X-Robots-Tag: noindex on rss/atom/entries/files + a /robots.txt
- absolutize relative content URLs against the sender's site
- promote lazy-loaded images (data-src → src, strip loading="lazy")
- strip XML-illegal control chars from generated feeds (keep emoji)
- plain-text feed <title> (strip HTML, decode entities)

Sender-base derivation lives on the EmailAddress value object
(siteBaseUrl) instead of a misplaced favicon helper. Bump to 0.2.1
and document the changes in README + CLAUDE.md.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 17:47:46 +02:00
Julien HerrandClaude Opus 4.7 81e46c9026 feat(stats): count emails forwarded to the catch-all fallback
Adds an emails_forwarded counter (a subset of emails_rejected) bumped on a
successful FALLBACK_FORWARD_ADDRESS forward. Dropped = rejected − forwarded.
Surfaced in the /api/v1/stats response and the public status page.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 17:19:12 +02:00
Julien HerrandClaude Opus 4.7 1583e95875 docs(landing): add Catch-All Fallback feature card
Surfaces FALLBACK_FORWARD_ADDRESS on the landing page feature grid.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 17:15:42 +02:00
Julien HerrandClaude Opus 4.7 2c450817df feat(email): forward non-feed mail to FALLBACK_FORWARD_ADDRESS
Lets you point a domain's catch-all at the worker without losing personal
mail: inbound mail that isn't a feed (invalid_address / feed_not_found) is
forwarded to an optional verified destination instead of being dropped.
Expired feeds and blocked senders are still dropped so newsletters never
leak to the fallback inbox. Unset env keeps the original drop-and-log path.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 17:14:04 +02:00
Julien HerrandClaude Opus 4.7 6cb036fe2c docs(todo): catalog feature gaps with origin refs and priority/size tags
Survey of kill-the-newsletter issues/PRs, competitors, RSS readers, and a
code audit. Each idea carries an origin reference (so we can notify the
requester on ship) and a Pn·Size badge (user value × implementation effort).
Adds the FALLBACK_FORWARD_ADDRESS catch-all fallback-forwarding idea.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 17:11:11 +02:00
Julien HerrandClaude Opus 4.7 b002f8ad43 refactor: extract inline view CSS into dedicated stylesheets
Move the hardcoded <style> blocks from the single-email view and the admin
email preview iframe into src/styles/*.css so they benefit from Prettier,
linting, and syntax highlighting like the rest of the design system.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 14:54:33 +02:00
Julien HerrandClaude Opus 4.7 5137637181 feat(attachments): render inline cid images in place, not as attachments
Inline images (referenced by src="cid:…") are now classified at ingest and
kept out of the downloadable attachment lists, RSS/Atom enclosures, and the
API — while still stored in R2 and cleaned up with the email. Fixes the admin
email preview, which injected raw HTML into the data: iframe so cid refs never
resolved; it now rewrites them to absolute /files URLs.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 14:39:59 +02:00
Julien HerrandClaude Opus 4.7 be45e70571 docs(landing): add auto-expiring feeds feature card
Surface the feed TTL / auto-deletion capability on the landing page,
framed as a disposable temporary-mailbox use case.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 14:13:45 +02:00
Julien HerrandClaude Opus 4.7 06c436c36a refactor: separate Feed domain state from persistence DTO
Move four DDD tensions on the Feed aggregate to ground:

- #1 The aggregate now holds a domain FeedState (camelCase) instead of the
  snake_case FeedConfig DTO; infrastructure/feed-mapper.ts owns the
  FeedState<->FeedConfig/FeedListItem translation as the sole snake_case site
  outside the HTTP edge.
- #3 Replace the edit() recomputeExpiry control flag with a Lifetime VO:
  passing a lifetime recomputes expiry, omitting it preserves the current one
  (the dashboard quick-edit path).
- #4 Domain events carry their own feedId; dispatchFeedEvents centralizes the
  drain+dispatch in the application layer (no more manual pullEvents at call
  sites), keeping infra->application dependency direction intact.
- #6 Rename FeedId.fromTrusted to FeedId.unchecked to make the absence of
  revalidation explicit.

Adds Lifetime + feed-mapper round-trip tests. 353 tests green, tsc clean,
wrangler dry-run OK. Docs (CLAUDE.md) synced.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 14:10:04 +02:00
Julien HerrandClaude Opus 4.7 d68a24867d docs(claude): sync layering rules with the aggregate refactor
The aggregate no longer exposes raw config/metadata (intention-revealing
accessors + snapshots), feeds:list is maintained by the repository from
feed.summary(), edit() is the single config mutation, and FeedId now flows
through the application and infrastructure layers.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 13:45:30 +02:00
Julien HerrandClaude Opus 4.7 ad196f1761 refactor: tighten DDD boundaries on the Feed aggregate
Address five modeling tensions in one pass:

- Encapsulation: the Feed aggregate no longer exposes raw config/metadata
  (a shallow Readonly still leaked mutable arrays). It now offers
  intention-revealing accessors that return copies, plus
  toConfigSnapshot/toMetadataSnapshot for the repository and summary() for
  the global registry.
- feeds:list consistency: FeedRepository.save/saveConfig upsert the registry
  entry from feed.summary(), so services no longer mirror title/description/
  expiry by hand (the old add/updateInList footgun is gone).
- domain/feed.ts: drop the dead applySenderPolicy, internalise resolveExpiresAt
  and trimToByteBudget into the aggregate; feed.ts keeps only the shared
  isExpired predicate used by the read-model routes.
- Single edit path: remove editDetails; edit(patch, deps) is the sole config
  mutation, with a systematic expired guard. Renaming an expired feed now 403s.
- FeedId flows through the application and infrastructure signatures;
  fromTrusted/parse happen once at the edge, .value only at the serialisation
  boundaries (urls, feed-generator, feed-keys, logs, JSON).

347 tests green, tsc clean, Worker bundle builds.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 13:45:13 +02:00
Julien HerrandClaude Opus 4.7 b3d42f6c50 refactor: introduce domain events for feed side effects (Track E — point 5)
Light "collect + dispatch" variant: the Feed aggregate records FeedEvents
(FeedCreated, EmailIngested) on the mutations that have consequences, exposed via
pullEvents(). A new application dispatcher (feed-events.applyFeedEvents) maps
those events to their side effects — counters (awaited) plus WebSub pings and
favicon fetches handed to a BackgroundScheduler. This removes the inline,
scattered side effects from the ingest hot path (email-processor) and from
createFeedRecord; the aggregate is now the source of truth for "what happened".

Side effects with no aggregate mutation (rejected email, feed deletion bypassing
the aggregate, bulk admin ops, the cron, unsubscribes-sent) stay imperative by
design — there is no aggregate event for them to ride on.

BackgroundScheduler type moved to infrastructure/worker.ts (shared). CLAUDE.md
updated. 355 tests pass (+4 event tests); tsc --noEmit clean.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 13:12:42 +02:00
Julien HerrandClaude Opus 4.7 46af982c40 refactor: invert application↔routes boundary (Track B — points 3, 6a)
- Point 3: move the feed/email storage-cleanup helpers (purgeFeedKeysStep,
  collectUnsubscribeUrls, purgeExpiredFeeds, deleteKeysWithConcurrency,
  deleteAttachmentsForEmails) out of routes/admin/helpers.ts into
  src/application/feed-cleanup.ts, so the application layer no longer imports
  from routes/. deleteFeedRecord no longer takes a Hono Context: it accepts a
  BackgroundScheduler ((task) => void) and the HTTP edge passes
  (p) => waitUntilSafe(c, p). Application/domain are now Hono-Context-free.
- Point 6a: rename the misleadingly-named Feed.rename → Feed.editDetails (it
  edits title + description), and feed-service.renameFeed → editFeedDetails.

CLAUDE.md source layout updated. 351 tests pass; tsc --noEmit clean.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 10:05:21 +02:00
Julien HerrandClaude Opus 4.7 f823a5f222 refactor: move KV repositories to infrastructure (Track P — points 2, 6c)
Make the domain stop depending on infrastructure ("imports point inward").

- Point 2: relocate the four KV adapters (FeedRepository, IconRepository,
  WebSubSubscriptionRepository, CountersRepository) from domain/ to
  infrastructure/, where the logger import is legitimate. The domain now keeps
  only the pure key schema (feed-keys.ts), the Feed aggregate and value objects;
  it imports nothing outward. Deliberately no hand-rolled 24-method port
  interface (YAGNI without DI) — relocation alone fixes the direction.
- Point 6c: EmailParser.extractFeedId now returns a validated FeedId value
  object instead of a raw string, so the most untrusted input (an inbound
  recipient address) is guarded at the parse boundary and no longer round-trips
  through FeedId.fromTrusted in the ingest path.

All import paths updated; CLAUDE.md source layout/KV-schema notes updated.
351 tests pass; tsc --noEmit clean.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 10:02:23 +02:00
Julien HerrandClaude Opus 4.7 23dd0a0c96 refactor(domain): purify the Feed aggregate (Track D — points 1, 4, 6b)
Remove the infrastructure Env leak and ambient time from the domain core, and
model the sender policy as a value object.

- Point 1: Feed.create/edit no longer receive Env. The application layer resolves
  the effective lifetime (parsing FEED_TTL_HOURS and applying the server override)
  via feed-service.resolveTtlHours and hands the domain a plain ttlHours.
  resolveExpiresAt(ttlHours, now) is now pure.
- Point 4: introduce a Clock port (systemClock default), injected at
  create/reconstitute. The aggregate uses clock.now() instead of Date.now().
  The isExpired edge helper keeps its Date.now() default for routes.
- Point 6b: extract SenderPolicy value object built once from the lists
  (decide(senders)) instead of re-parsing per sender; applySenderPolicy is now a
  thin wrapper over it.

Coverage moved with the logic: the FEED_TTL_HOURS override is now pinned by
feed-service.test.ts; aggregate tests use an injected fixed clock.

351 tests pass; tsc --noEmit clean.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 09:55:55 +02:00
Julien HerrandClaude Opus 4.7 e324571122 docs(claude): document the DDD layering, Feed aggregate and repo split
Refresh the source layout for domain/application/infrastructure, replace
the single-repository rule with the four-repository split + feed-keys,
and add domain rules: the Feed aggregate is the only writer of config +
the email index, and FeedId circulates through domain and repository.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 00:48:13 +02:00
Julien HerrandClaude Opus 4.7 7bf0f71f86 refactor: split src into domain / application / infrastructure layers
Replace the history-driven lib/ + utils/ split with DDD layers:
- domain/: aggregate, repositories, value objects, pure parsers/format
- application/: feed-service, email-processor, feed-fetcher, stats
- infrastructure/: logging, auth, KV/R2 adapters, HTTP, framework glue

Pure file relocation; imports updated mechanically. Behaviour unchanged.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 00:46:56 +02:00
Julien HerrandClaude Opus 4.7 ab1c15e69a refactor(domain): make FeedId circulate through the domain and repository
FeedId is now the type of Feed.id and of every single-feed method on
FeedRepository; callers wrap raw strings via FeedId.fromTrusted at the
repository boundary. String-medium operations (URLs, logs, JSON,
list registry, email keys) stay string. Drop the redundant
generateFeedId wrapper in favour of FeedId.generate.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 00:44:24 +02:00
Julien HerrandClaude Opus 4.7 05388b45c8 refactor(domain): split updateFeedRecord into renameFeed and editFeed
The inPlace boolean hid two distinct intentions. Replace it with two
intention-revealing operations backed by Feed.rename (presentational,
never touches expiry) and Feed.edit (full edit, recomputes expiry,
rejects expired). Add FeedRepository.saveConfig so these config-only
edits don't re-write (and risk clobbering) the email index.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 00:35:07 +02:00
Julien HerrandClaude Opus 4.7 c45f6677fe refactor(domain): introduce the Feed aggregate as the write-path API
Add a Feed aggregate class owning config + the email index, with create,
ingest, removeEmails, isExpired and accepts delegating to the existing
pure invariant functions. FeedRepository gains load/save/saveMetadata
that reconstitute and persist the aggregate.

All write paths now go through it: createFeedRecord (Feed.create),
email ingestion (feed.ingest), and every email deletion in the admin UI
and REST API (feed.removeEmails) — no route mutates metadata.emails
directly anymore. KV key strings unchanged.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 00:33:14 +02:00
Julien HerrandClaude Opus 4.7 a31ff42f59 refactor(domain): split Icon/WebSub/Counters out of FeedRepository
FeedRepository no longer owns favicons, WebSub subscriber lists or the
monitoring counters singleton. Each concern gets its own repository
(IconRepository, WebSubSubscriptionRepository, CountersRepository),
sharing the key schema via feed-keys. KV key strings are unchanged;
counters increment policy stays in utils/stats.ts.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 00:27:33 +02:00
Julien HerrandClaude Opus 4.7 b347f2f625 refactor(domain): extract the KV key schema into feed-keys.ts
Move the feed:/icon:/websub: key builders out of FeedRepository into a
pure feed-keys module so the wire format lives in one place, shared by
the repositories to come. Strings are byte-identical; behaviour unchanged.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 00:24:43 +02:00
Julien HerrandClaude Opus 4.7 49f69ff19e docs(claude): document the domain layer and FeedRepository KV access rule
Reflect the refactor: add the src/domain/ tree (feed-repository, feed, value
objects), drop the deleted storage.ts, and update the KV-schema note to point at
FeedRepository as the single key-access layer. Correct the websub key shape and
add the icon: key.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 00:06:48 +02:00
Julien HerrandClaude Opus 4.7 c65aabe7f4 refactor(domain): add FeedId, EmailAddress and Domain value objects
Encapsulate the email/domain/feed-id parsing that was scattered as ad-hoc
regexes and split("@") calls into three small immutable value objects under
src/domain/value-objects/. EmailParser.extractFeedId and generateFeedId now
delegate to FeedId; the sender policy, favicon domain extraction and the admin
SenderField parse through EmailAddress/Domain.

Left as-is on purpose: forwardemail's multi-address free-text extraction and the
admin allow/block list normaliser, which operate on mixed email-or-domain input
that the single-address value objects would reject.

Behaviour-preserving; adds unit tests for each value object.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 00:05:46 +02:00
Julien HerrandClaude Opus 4.7 8f036cf223 refactor(ingest): return a domain result from processEmail, map HTTP at the edge
processEmail/validateEmail now return an IngestResult discriminated union
({ ok } | { ok: false; reason }) instead of an HTTP Response. The status mapping
moves to the edge (ingestResultToResponse in forwardemail.ts), and the Cloudflare
email handler now logs the rejection reason instead of silently discarding it.

The ingestion core is transport-agnostic. End-to-end status mapping stays covered
by inbound.test.ts (now incl. 410 expired); email-processor.test asserts on the
domain result directly.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-24 00:02:05 +02:00
Julien HerrandClaude Opus 4.7 6b51173722 refactor(domain): consolidate Feed aggregate invariants in domain/feed.ts
Gather the feed's scattered business rules — expiry, sender allow/block policy,
and the email byte-size budget — into one framework-agnostic module. Expiry was
duplicated across feed-service, email-processor and the rss/atom/entries routes;
the sender policy and trim loop lived inline in email-processor. Each now calls
a single function (isExpired, applySenderPolicy, trimToByteBudget,
resolveExpiresAt). Drops the now-unused MAX_METADATA_EMAILS constant.

Behaviour-preserving; adds feed.test.ts covering every invariant.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 23:59:15 +02:00
Julien HerrandClaude Opus 4.7 2b3f00f7e3 refactor(domain): introduce FeedRepository as the single KV access layer
Centralise the KV key schema and all get/put access behind a FeedRepository
class under src/domain/. Every feed/email/list/icon/websub/counter key was
previously inlined across ~12 modules with two divergent storeEmail and
addFeedToList implementations; the dead src/utils/storage.ts write path is
removed and the email key convention unified on feed:<id>:<ts>.

Behaviour-preserving: existing tests pass unchanged in logic, plus a new
feed-repository.test.ts covering CRUD, key builders, list ops and counters.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 23:56:44 +02:00
Julien HerrandClaude Opus 4.7 a0eaebe749 docs(landing): add a REST API & OpenAPI feature card
Surface the new versioned REST API on the marketing landing, linking to the
live Scalar reference on the demo.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 23:16:54 +02:00
Julien HerrandClaude Opus 4.7 c2a0a68058 refactor(api): remove the deprecated /api/stats endpoint
The only consumer (the marketing landing) now uses /api/v1/stats, so drop
the legacy /api/stats route and its handler. Delete src/routes/stats.ts and
its test; repoint the index CORS test at /api/v1/stats.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 23:15:08 +02:00
Julien HerrandClaude Opus 4.7 daa93d8093 feat(api): make /api/v1/stats public and point the landing at it
Unify the monitoring stats on the versioned API: /api/v1/stats is now public
(no auth) and CORS-enabled, mirroring the legacy /api/stats. The marketing
landing (docs/index.html) now fetches /api/v1/stats; /api/stats is kept as a
deprecated alias for existing monitors. Feed/email routes remain token-gated.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 23:12:43 +02:00
Julien HerrandClaude Opus 4.7 45d2a14a12 feat(api): add versioned REST API with OpenAPI 3.1 spec
Expose /api/v1/* for feed and email management (feeds CRUD, email
list/get/delete, stats) so the service can be automated without scraping
the admin UI. Built on @hono/zod-openapi; the OpenAPI 3.1 spec is served at
/api/openapi.json with a Scalar reference at /api/docs.

Auth is token-based (Authorization: Bearer <ADMIN_PASSWORD>) plus the
existing reverse-proxy headers — no cookie, no CSRF. Extracted the auth
primitives into src/lib/auth.ts and the feed create/update/delete
orchestration into src/lib/feed-service.ts so the admin UI and the REST API
share a single source of truth.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 23:01:15 +02:00
Julien HerrandClaude Opus 4.7 7f5b913576 docs: add SECURITY.md and CONTRIBUTING.md
Add a security policy with private reporting channels and project-specific
scope, plus a contributor guide covering dev setup, testing, and commit
conventions. Drop the stale AGENTS.md reference from CLAUDE.md.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 22:18:57 +02:00
Julien HerrandClaude Opus 4.7 db31e33a8b docs: extract install/deploy/config guide into INSTALL.md
Slim README down to project overview (why, features, architecture,
security) with a short Installation quick-start that links to the new
INSTALL.md. Repoint setup.sh references and CLAUDE.md maintenance list.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 21:49:19 +02:00
Julien HerrandClaude Opus 4.7 8ffa4ad5e7 docs(license): add Julien Herr copyright alongside original author
Cumulative MIT copyright reflecting the substantial rewrite while
preserving the original author's notice as required by the license.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 21:49:13 +02:00
Julien HerrandClaude Opus 4.7 7f4afa3ec8 feat(admin): add status page link to dashboard header
Add a "Status" link in the admin header pointing to the public status page
(/), mirroring the existing "Go to admin" link on that page. Add a gap to
.header-actions so the new link and Logout button are spaced apart.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 21:43:24 +02:00
Julien HerrandClaude Opus 4.7 3368b0d1d2 feat(admin): collapse create-feed form into accordion
Wrap the "Create New Feed" form in a native <details> accordion, collapsed
by default and auto-opened when no feeds exist. After creating a feed,
redirect to the "Your Feeds" anchor so the new feed is immediately visible.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 21:41:25 +02:00
Julien HerrandClaude Opus 4.7 5fc91a0be4 refactor(html-processor): isolate cid rewrite from sanitization
Keep sanitizeElement single-purpose and run the cid: rewrite as a
separate guarded pass over [src] elements. Use a type-only import for
AttachmentData.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 18:47:20 +02:00
Julien HerrandClaude Opus 4.7 debbfc623e fix(attachments): render inline cid: images in emails and feeds
Capture each attachment's Content-ID at ingestion (postal-mime and
mailparser paths) and rewrite cid: image refs to the stored /files URL
in processEmailContent, shared by the entry view and RSS/Atom feeds.
Bodyless HTML fragments are now serialized so sanitization and the cid
rewrite apply to them too.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 18:42:04 +02:00
Julien HerrandClaude Opus 4.7 6cd2d425a2 feat(attachments): list downloadable attachments on admin email detail page
The admin email detail view loaded the full email but never rendered its
attachments, so there was no way to download them from the admin UI (only
the public entry view and the feed enclosure exposed them).

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 18:11:29 +02:00
Julien HerrandClaude Opus 4.7 9141cf89bd fix(attachments): purge R2 attachments on no-JS bulk email delete
The form-based bulk-delete fallback removed KV entries but left R2
attachments orphaned. Extract a shared deleteAttachmentsForEmails helper
and use it across single, JSON bulk, and form bulk delete paths.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 17:57:31 +02:00
Julien HerrandClaude Opus 4.7 ddde0e26a2 docs(readme): add Continuous deployment section with CI secrets and token permissions
Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 17:51:50 +02:00
Julien HerrandClaude Opus 4.7 20c9bca34a docs(readme): note R2 permission needed for scoped deploy tokens
Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 17:50:04 +02:00
Julien HerrandClaude Opus 4.7 2de09b2a5d refactor(home): dedupe byte formatting in storage cards
Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 17:39:55 +02:00
Julien HerrandClaude Opus 4.7 f150d40c45 feat(attachments): R2 toggle, storage metrics, and demo R2 config
Add an ATTACHMENTS_ENABLED switch (default on when R2 is bound) via a
central getAttachmentBucket helper, surface R2 + estimated KV usage
against the free tier on the status page and /api/stats (refreshed by the
hourly cron), let setup.sh create and wire the R2 bucket, and bind the
demo bucket so the deployed demo has attachments.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 17:33:50 +02:00
Julien HerrandClaude Opus 4.7 7226e718f7 feat(admin): paperclip indicator for emails with attachments
Show an inline paperclip icon before the subject in the admin email
list when an email has attachments, with the count in a tooltip. Uses
the attachmentIds already stored in metadata, so no extra fetch.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 15:10:24 +02:00
Julien HerrandClaude Opus 4.7 f4e751e40b fix(favicon): add landing-page favicon to fix /favicon.ico 404
Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 14:49:57 +02:00
Julien HerrandClaude Opus 4.7 766f2717a7 feat(entries): list email attachments with download links
The email detail page loaded the full EmailData (including attachments)
but never rendered them, so attachments were invisible. Add a conditional
"Attachments" section linking each file to /files/:id/:filename with name
and human-readable size.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 14:46:25 +02:00
Julien HerrandClaude Opus 4.7 d322bc1e92 docs(todo): add REST API with OpenAPI description item
Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 14:36:16 +02:00
Julien HerrandClaude Opus 4.7 3ad0188bc0 feat(unsubscribe): RFC 8058 one-click unsubscribe on feed deletion
Capture each sender's List-Unsubscribe one-click URL during ingestion
(stored per sender in feed metadata, mirroring the iconDomain pattern) and
fire one-click POSTs via ctx.waitUntil when a feed is deleted, so newsletters
stop mailing the now-dead address. Tracked with a new unsubscribes_sent
counter surfaced on the status page and /api/stats.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 14:35:05 +02:00
Julien HerrandClaude Opus 4.7 eb12f21894 feat(favicon): per-feed icon from the last sender's domain
Resolve each feed's most recent sender domain and serve its favicon at
GET /favicon/:feedId, falling back to the project icon. Icons are fetched
in the background on ingestion (direct /favicon.ico then a DuckDuckGo
fallback), cached base64 in KV keyed by domain with a 1-week TTL so the
fetch only fires when absent. Exposed via RSS <image> / Atom <icon>/<logo>
and rendered in the admin feed list, plus a landing-page feature card.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 14:05:14 +02:00
Julien HerrandClaude Opus 4.7 d299c8891d feat(favicon): serve project favicon reusing the header envelope logo
Serve an inline SVG icon at /favicon.svg and /favicon.ico and link it
from the shared Layout and the standalone entry view, so the admin UI,
status page, and entry pages stop emitting /favicon.ico 404s. Doubles
as the fallback for the upcoming per-feed favicon feature.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 13:13:44 +02:00
Julien HerrandClaude Opus 4.7 a67baa71f4 feat(landing): add FAQ section and trim demo URL
Add a native details/summary accordion FAQ inspired by kill-the-newsletter,
rewritten for self-hosted differentiators; drop /admin from the demo URL.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 12:59:27 +02:00
Julien HerrandClaude Opus 4.7 b985e2c643 feat(status): redesign status page with hero, themed sections, responsive grid
Rework the public / status page from a flat uniform grid into a hero
featured metric plus four themed sections (Feeds, Emails, Distribution,
Instance). Add semantic colors (green success, red rejects/deletes),
relative timestamps with UTC tooltips, and derived metrics (net feeds,
acceptance rate, avg emails/feed, humanized uptime). Grid is fluid above
640px (auto-fit) and locks to two columns on mobile.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 12:39:57 +02:00
Julien HerrandClaude Opus 4.7 81d05e5774 feat(landing): move live counter above demo banner, point CTAs to demo root
Surface the live stats counter directly under the hero, ahead of the
"Try it live" banner. Demo CTAs (hero + banner) now open the demo root
instead of /admin so visitors land on the public status page first.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 10:43:30 +02:00
Julien HerrandClaude Opus 4.7 4db9fc1b8a fix(lint): close type-check gaps in client scripts and tooling
Remove unused import flagged by CI lint, then harden the toolchain so
such issues are caught before push:

- lint-staged now also matches .tsx/.jsx (previously .tsx files skipped
  the pre-commit eslint pass, which is how the error reached CI)
- eslint ignores generated client bundles (gitignored, not worth linting)
- typecheck now also runs the client tsconfig; the hand-written browser
  source was excluded from the root config and never type-checked
- consolidate the window global augmentations (showToast,
  parseJsonResponseOrThrow) into a single client globals.d.ts; the inline
  declare-global blocks failed (non-module files) and masked real errors

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 10:38:01 +02:00
Julien HerrandClaude Opus 4.7 6bfaa4dec7 fix(landing): eliminate mobile horizontal scroll
Fix the install step grid track (48px minmax(0,1fr)) so wide code blocks
and the WAF table no longer blow out the page width on mobile. Transpose
the WAF rate-limit table to a vertical layout (endpoints as columns,
settings as rows) and reclaim horizontal space with tighter mobile
padding.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 10:23:11 +02:00
Julien HerrandClaude Opus 4.7 fd6a1a945f feat(landing): show animated live demo stats counters
Add a "Live from the demo instance" section to the landing page that
fetches feeds_created and emails_received from the demo /api/stats and
counts them up on scroll into view. Make /api/stats publicly readable
(CORS *) and refresh the stale allowlist origins to kill-the.news.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 10:09:13 +02:00
Julien HerrandClaude Opus 4.7 d1959acad1 docs(todo): add favicon and RFC 8058 unsubscribe ideas
Add backlog items for a project favicon (also used as the per-feed
fallback), per-feed favicons resolved from the last sender's domain with
aggressive caching, and RFC 8058 one-click unsubscribe on feed deletion.
Include a detailed design breakdown for the per-feed favicon feature.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 09:57:59 +02:00
Julien HerrandClaude Opus 4.7 b534ce5bf8 feat(monitoring): add stats counters API and public status page
Add GET /api/stats exposing cumulative counters (feeds created/deleted,
emails received/rejected, recent date-times) plus live values (active
feeds, active WebSub subscriptions). Counters persist in a stats:counters
KV singleton and are incremented at the email-processing chokepoint and
feed create/delete paths. Replace the / → /admin redirect with a public
status page rendering these figures with a link to the admin.

Co-Authored-By: Claude Opus 4.7 <[email protected]>
2026-05-23 09:50:51 +02:00
Julien HerrandClaude Sonnet 4.6 f4d5edda0e feat(feeds): add configurable per-feed lifetime (TTL)
Replace the demo nightly KV wipe with a per-feed expiry. Feeds can be
given a lifetime at creation (and edited later); FEED_TTL_HOURS locks the
value server-side and greys out the UI field. Expired feeds stay visible
in admin (greyed, actions disabled), return 410 on rss/atom/entries, and
reject inbound emails. The scheduled handler now purges only expired
feeds (KV + R2 attachments) on an hourly global cron.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-23 09:05:48 +02:00
Julien HerrandClaude Sonnet 4.6 24c7d2a53e fix(release): create release before uploading bundle
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 23:50:39 +02:00
Julien HerrandClaude Sonnet 4.6 7e72022ffc fix(release): remove redundant cp and use env var for bundle path
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 23:49:25 +02:00
Julien HerrandClaude Sonnet 4.6 78117421fd fix(build): add wrangler.build.toml for CI and fix autoconfig
Wrangler 4.94+ introduced --experimental-autoconfig (default: true) which
fails in non-interactive CI environments. Without a committed wrangler.toml,
the release action build was broken.

- Add wrangler.build.toml with minimal config (placeholder KV ID, no secrets)
- Update build script to use wrangler.build.toml + --no-experimental-autoconfig

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 23:47:48 +02:00
Julien HerrandClaude Sonnet 4.6 1b31665b4f docs: update WAF rate limiting instructions for free tier accuracy
- Fix menu path: Security → Security rules (not Security → WAF)
- Add free tier limitations note: 1 rule max, 10s period/block cap
- Show recommended vs free tier limits side by side in table
- Remove HTTP method filter from conditions (not available in rate limiting rules)
- Note Terraform supports method filtering and longer periods (paid plan)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 23:38:41 +02:00
Julien HerrandClaude Sonnet 4.6 75a557d542 feat(admin): add site header and footer to all admin pages
Adds a minimal header with a branded link to kill-the.news and an
"admin" badge, plus a discreet footer with site link and GitHub
Sponsors link.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 23:23:48 +02:00
Julien HerrandClaude Sonnet 4.5 4a4c276859 feat: add sender blocklist with priority matching and quick-add dropdown
- Add `blocked_senders` field to FeedConfig (alongside existing `allowed_senders`)
- Refactor sender matching to priority-based logic: exact block > exact allow > domain block > domain allow, enabling exceptions (e.g. allow [email protected] despite blocking gmail.com)
- Add `POST /admin/feeds/:feedId/sender-filter` endpoint for quick allow/block from email detail view; returns 409 on conflict with opposite list
- Add ⋮ dropdown on From field in email detail with 4 options (allow/block sender/domain), inline success/error feedback
- Add blocked_senders textarea to create/edit feed forms
- 209 tests passing

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
2026-05-22 23:09:53 +02:00
Julien HerrandClaude Sonnet 4.6 7b2b98d693 refactor: extract url helpers, add EMAIL_DOMAIN support
- Add src/utils/urls.ts with baseUrl, feedRssUrl, feedAtomUrl, feedUrl,
  feedEmailAddress, feedTopicPattern
- Add optional EMAIL_DOMAIN env var so web domain and email domain can
  differ (e.g. demo.kill-the.news serves feeds, @kill-the.news receives mail)
- Replace all inline domain template literals with the new helpers
- Remove unused site_url/feed_url fields from FeedConfig
- Remove unused feedPath param from fetchFeedData
- Extract verifyCallback() to deduplicate verifyAndStoreSubscription /
  verifyAndDeleteSubscription

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 22:38:29 +02:00
Julien HerrandClaude Sonnet 4.6 c64ceb4ee7 feat(site): add GitHub Sponsors links (nav, sponsor section, footer)
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 21:56:56 +02:00
Julien HerrandClaude Sonnet 4.6 6bf5ae0356 feat: landing page install guide, demo banner, WAF docs, nightly demo reset
- docs/index.html: nav links (Features/How it works/Install), hero CTAs
  (Try demo primary, Self-host, GitHub), demo banner with credentials,
  full 7-step installation section with WAF rate limiting guide (dashboard
  + Terraform) integrated as step 7
- wrangler-example.toml: cron trigger on demo env for nightly KV reset at 03:00 UTC
- src/index.ts: scheduled handler that wipes all EMAIL_STORAGE KV keys
- TODO.md: mark WAF rate limiting as done

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 21:50:42 +02:00
Julien Herr af721c081a chore: mark WebSub as done in TODO.md 2026-05-22 21:14:00 +02:00
Julien HerrandClaude Sonnet 4.5 a29e9ab372 feat: WebSub Atom support, HTML processing via linkedom, W3C badges
WebSub / PubSubHubbub:
- Hub now accepts both /rss/:id and /atom/:id topic URLs
- WebSubSubscription stores format ("rss" | "atom")
- notifySubscribers sends RSS or Atom XML with correct Content-Type
- verifyAndStoreSubscription sends correct topic URL per format
- CI paths-ignore docs/** to skip deploy on docs-only changes

HTML processing (linkedom + escape-html):
- New html-processor.ts: body extraction, script/iframe/object removal,
  event handler + javascript: URL stripping, mso-* style cleanup,
  plain text → <pre> with HTML escaping via escape-html
- feed-generator.ts and entries.ts use processEmailContent

Admin UI:
- W3C validation badges (Atom + RSS) on feed detail page

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
2026-05-22 21:12:22 +02:00
Julien HerrandClaude Sonnet 4.6 1789870f27 fix(feed): use permalink URL as Atom entry id, strip mso-* inline styles
- Entry <id> was a non-URL string (timestamp + base64 snippet), which
  is invalid per the Atom spec; now uses the entry permalink URL which
  is both valid and stable across feed regeneration
- Strip mso-* properties from inline style attributes in extracted body
  content to eliminate the feed validator DangerousStyleAttr warning
  caused by Microsoft Office HTML in newsletter emails

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 21:12:22 +02:00
Julien HerrandClaude Sonnet 4.6 afed4464cf fix(feed): use permalink URL as Atom entry id, strip mso-* inline styles
- Entry <id> was a non-URL string (timestamp + base64 snippet), which
  is invalid per the Atom spec; now uses the entry permalink URL which
  is both valid and stable across feed regeneration
- Strip mso-* properties from inline style attributes in extracted body
  content to eliminate the feed validator DangerousStyleAttr warning
  caused by Microsoft Office HTML in newsletter emails

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 18:43:06 +02:00
Julien HerrandClaude Sonnet 4.6 4428f35dd4 fix(feed): add Atom link in emails page, fix HTML stripping, use request URL for self-link
- Add Atom Feed URL to the Feed Details card in the emails page
- Fix extractBodyContent to handle emails without a closing </body> tag
  (regex now falls back to capturing everything after the opening <body>)
- Use the actual request URL origin for atom:link rel="self" in RSS/Atom
  feeds, guaranteeing it always matches the document location regardless
  of how DOMAIN is configured

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 18:41:21 +02:00
Julien HerrandClaude Sonnet 4.6 bcc9640591 fix(feed): correct feed link, canonical id, and strip html wrapper from content
- link: computed as /admin/feeds/:id/emails instead of stale site_url from KV
- id: computed dynamically from baseUrl instead of stale feed_url from KV
- item description/content: strip <html><head><body> wrapper via extractBodyContent()
  so feed readers receive a body fragment, not a full HTML document

Fixes RSS validator warnings: SelfDoesntMatchLocation (stale KV domain) and
InvalidHTML (full HTML document inside <description>/<content:encoded>).
Adds 8 tests covering extractBodyContent and the new feed/atom link assertions.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 18:28:13 +02:00
Julien HerrandClaude Sonnet 4.6 780bf6c190 fix(admin): add Atom feed URL, unify container width, update branding
- Atom feed URL shown in both list and table views (new Atom column)
- Remove container-wide toggle — both views now use max-width 1200px
- Update dashboard title and login title to kill-the-news

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 18:21:49 +02:00
Julien HerrandClaude Sonnet 4.6 41f70143d1 design(admin): add Inter font, update page title to kill-the-news
Imports Inter 400/500/600/700 from Google Fonts to match the landing
page typography. Updates browser tab title format.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 18:21:42 +02:00
Julien HerrandClaude Sonnet 4.6 b95fbb468f design(admin): align with kill-the-news landing page visual identity
- variables.css: orange primary (#f6821f), dark bg (#0a0a0a), Inter font
- layout.css: orange radial glow, unified container 1200px (no width jump)
- components.css: orange buttons, remove backdrop-filter on inputs/cards

Fixes blurred form fields (double backdrop-filter), jarring width shift
between list/table views, and mismatched blue iOS aesthetic vs orange
Cloudflare identity of the site.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 18:21:34 +02:00
Julien HerrandClaude Sonnet 4.6 ba7add5f53 fix(demo): correct DOMAIN env var to demo.kill-the.news
The [env.demo] section pointed DOMAIN to kill-the.news while the
custom_domain route was demo.kill-the.news, causing feed/email URLs
to show the wrong domain in the admin UI.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 18:21:26 +02:00
Julien Herr 3ccbd876b6 build(deps-dev): bump dev-dependencies group with 8 updates
Includes TypeScript 5→6 migration: moduleResolution node→bundler.
2026-05-22 17:35:01 +02:00
Julien Herr 03d069bfa3 build(deps): bump hono from 4.11.7 to 4.12.22
Fix TypeScript errors from stricter c.req.param() types (string | undefined).
2026-05-22 17:33:47 +02:00
Julien Herr ea400883dd build(deps): bump zod from 4.3.6 to 4.4.3 2026-05-22 17:32:50 +02:00
Julien Herr 7dd1225cde build(deps): bump feed from 5.2.0 to 5.2.1 2026-05-22 17:32:37 +02:00
Julien Herr 3b4a1ce98c build(deps): bump actions/setup-node from v4 to v6.4.0 2026-05-22 17:32:19 +02:00
Julien Herr 3fa17228a3 build(deps): bump actions/checkout from v4 to v6.0.2 2026-05-22 17:32:04 +02:00
Julien Herr ebfb8d4a60 fix(config): silence CSS module rule fallthrough warning 2026-05-22 17:28:02 +02:00
Julien Herr d657875322 feat(ci): auto-deploy demo instance on main push 2026-05-22 17:20:46 +02:00
Julien Herr 4d9418057e feat(deploy): add demo environment to wrangler config 2026-05-22 17:20:08 +02:00
Julien HerrandClaude Sonnet 4.6 edc1183e61 docs: rewrite CLAUDE.md to match actual codebase, remove .cursor
CLAUDE.md now reflects the real route set (atom, entries, files, hub,
email handler), src/lib/ layout, admin sub-modules, client script
pipeline, full Env bindings, and WebSub KV schema.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 15:53:22 +02:00
Julien HerrandClaude Sonnet 4.6 4316354ee5 refactor(styles): migrate TS template literal styles to real CSS files
Extract CSS from TypeScript template literals into standalone .css files
(variables.css, layout.css, components.css, utilities.css) and update
src/routes/admin/ui.tsx to import them directly via Wrangler text imports,
concatenating the strings at runtime for the inline <style> tag.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 15:44:08 +02:00
Julien HerrandClaude Sonnet 4.6 5d75682702 build: add Wrangler text rule for CSS imports
Add [[rules]] type = "Text" globs = ["**/*.css"] to wrangler-example.toml
so Wrangler bundles .css files as raw text strings importable in TypeScript.
Add src/types/css.d.ts to provide the module declaration for `import css from "*.css"`.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 15:43:49 +02:00
Julien HerrandClaude Sonnet 4.5 d64f703820 fix(ui): replace children cast to any with hono/jsx Child type
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
2026-05-22 15:35:10 +02:00
Julien HerrandClaude Sonnet 4.5 0e1592f8c6 build: gitignore generated client scripts, rebuild on npm install
Move src/scripts/generated/ to .gitignore — files are deterministic
build artefacts and don't belong in version control. Wire build:client
into prepare so they're regenerated automatically after npm install.

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
2026-05-22 13:53:33 +02:00
Julien HerrandClaude Sonnet 4.6 0f6670d0e9 refactor(admin): extract emailsScript to src/scripts/client/emails-page.ts
Moves the inline JS template literal from emails.tsx into a typed
TypeScript source file. The dynamic feedId value (previously interpolated
directly) is now passed via a window.__APP_CONFIG__ bootstrap script
injected immediately before the compiled static script in the HTML.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 13:50:54 +02:00
Julien HerrandClaude Sonnet 4.6 40f4b42cd5 refactor(admin): extract dashboardScript to src/scripts/client/dashboard.ts
Moves the 650-line inline JS template literal from admin.tsx into a
proper TypeScript source file with full type annotations. esbuild
compiles it to a minified IIFE committed in src/scripts/generated/,
which is imported and inlined into the HTML response as before.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 13:50:47 +02:00
Julien HerrandClaude Sonnet 4.6 0663861471 build: add client-side TypeScript compilation pipeline
Adds scripts/build-client.mjs which uses esbuild to compile TypeScript
files in src/scripts/client/ into minified IIFE bundles, then writes
them as TypeScript string-constant modules in src/scripts/generated/.

- Adds build:client npm script; wires it as prebuild and predev hooks
- Adds src/scripts/client/tsconfig.json with DOM lib for IDE support
- Excludes src/scripts/client/ from the root Worker tsconfig to avoid
  DOM type conflicts with the Cloudflare Workers runtime types

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 13:50:41 +02:00
Julien HerrandClaude Sonnet 4.6 671179acc7 build: add esbuild dev dependency
Installs esbuild to support compiling client-side TypeScript into
minified IIFE bundles that can be inlined into Worker HTML responses.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 13:50:31 +02:00
Julien HerrandClaude Sonnet 4.6 9bfebf4aa4 refactor(admin): migrate admin.ts to admin.tsx with JSX login and dashboard
Convert login page and dashboard GET routes from hono/html tagged template
literals to typed JSX using the <Layout> component. Extracts reusable
CopyIcon, CheckIcon, and CopyFieldInline components. Dashboard inline
script (~650 lines) preserved exactly via dangerouslySetInnerHTML constant.
All auth logic, CSRF middleware, and API routes are unchanged.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 13:17:46 +02:00
Julien HerrandClaude Sonnet 4.6 c9ab3839e4 refactor(admin): migrate emails.ts to emails.tsx with JSX rendering
Convert feed emails list and single email view GET routes from hono/html
tagged template literals to typed JSX. Extracts reusable CopyField and
SVG icon components. Inline page scripts are preserved verbatim via
dangerouslySetInnerHTML. Raw HTML display in single email view uses
dangerouslySetInnerHTML to avoid double-escaping pre-escaped content.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 13:17:38 +02:00
Julien HerrandClaude Sonnet 4.6 ecb85730e0 refactor(admin): migrate feeds.ts to feeds.tsx with JSX rendering
Convert the edit feed GET route from hono/html tagged template literals
to typed JSX using the <Layout> component. All CRUD routes and business
logic are preserved unchanged. textarea placeholder special characters
are now handled via JSX attribute escaping rather than &#10; entities.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 13:17:31 +02:00
Julien HerrandClaude Sonnet 4.6 996aa5e211 refactor(admin): migrate ui.ts to ui.tsx with JSX Layout component
Replace hono/html tagged template layout() function with a typed JSX
<Layout> component. CSS and interactive scripts are injected via
dangerouslySetInnerHTML to preserve exact output. clampText() is
preserved and re-exported for consumers.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 13:17:24 +02:00
Julien HerrandClaude Sonnet 4.6 f5bac1bd7d build: add hono/jsx support to tsconfig
Add jsx and jsxImportSource compiler options to enable hono/jsx server-side
rendering in .tsx files without per-file JSX pragmas.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 13:17:16 +02:00
Julien HerrandClaude Sonnet 4.6 d82344ac67 chore: remove TECH_DEBT.md — all tasks completed
All three phases fully remediated. P1-5 (WAF rate limiting) is
infrastructure-only and tracked in TODO.md.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 12:01:42 +02:00
Julien HerrandClaude Sonnet 4.6 427eac973c docs: mark Phase 3 tasks done; document WAF rate limiting in TODO
P1-4, P2-10, P2-11 marked DONE in TECH_DEBT.md.
P1-5 (rate limiting) is infrastructure-only — documented in TODO.md with
the recommended Cloudflare WAF rule targets and thresholds.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 10:56:55 +02:00
Julien HerrandClaude Sonnet 4.6 205d4ef5bb refactor: split admin.ts into sub-modules (P2-11)
Extracts 3833-line admin.ts into focused modules:
- src/routes/admin/ui.ts       — layout() and clampText() helpers
- src/routes/admin/helpers.ts  — KV list helpers (listAllFeeds, addFeedToList, etc.)
- src/routes/admin/feeds.ts    — feed CRUD routes (feedsRouter)
- src/routes/admin/emails.ts   — email view/delete routes (emailsRouter)

admin.ts now mounts the sub-routers and retains only auth middleware,
dashboard, login/logout, and the in-place feed API update route.
All 163 tests continue to pass.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 10:56:20 +02:00
Julien HerrandClaude Sonnet 4.6 a9501d6e44 feat: add structured JSON logger and worker utility (P1-4)
Introduces src/lib/logger.ts emitting JSON lines (level, message, data)
compatible with Cloudflare Logpush. Replaces all console.log/warn/error
calls in email-processor.ts, index.ts, and hub.ts with structured logger
calls. Extracts waitUntilSafe into src/utils/worker.ts to avoid duplicating
the executionCtx guard across routes.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 10:55:15 +02:00
Julien HerrandClaude Sonnet 4.6 0c0669c473 feat: extract constants module (P2-10)
Centralises magic numbers and string literals into src/config/constants.ts
(FEED_MAX_BYTES, MAX_FEED_ITEMS, MAX_METADATA_EMAILS, ADMIN_COOKIE_MAX_AGE,
FORWARD_EMAIL_IPS_CACHE_TTL_MS, DEFAULT_LEASE_SECONDS, MAX_LEASE_SECONDS,
FEEDS_LIST_KEY). Consumers updated in storage.ts and feed-fetcher.ts.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 10:55:01 +02:00
Julien HerrandClaude Sonnet 4.6 582108a8e3 docs: move Durable Objects migration from TECH_DEBT to TODO
The race condition is an accepted limitation of KV; the DO migration
is a future feature rather than active tech debt to remediate.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 10:31:41 +02:00
Julien HerrandClaude Sonnet 4.6 a0415cdc41 refactor: replace custom HMAC CSRF with hono/csrf middleware
Removes 38-line hand-rolled HMAC-SHA256 implementation in favour of
the built-in hono/csrf, which validates the Origin header natively.

- Delete src/utils/csrf.ts
- Replace custom CSRF middleware with hono/csrf (Origin-header check)
- Remove csrfToken from ContextVariableMap, layout(), forms, and JS fetch() calls
- Update admin tests: swap X-CSRF-Token for Origin header

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 10:28:26 +02:00
Julien HerrandClaude Sonnet 4.6 7d375693b9 feat: complete Phase 2 tech debt remediation
- Extract shared RSS/Atom fetch logic into feed-fetcher utility (P1-3)
- Split email-processor into validateEmail/storeEmail functions (P1-6)
- Add stateless HMAC-SHA256 CSRF protection to admin forms (P2-8)
- Fix Hono<{ Bindings: Env }> type safety across all routes (P3-13)
- Add entries.test.ts and files.test.ts with full coverage (P1-7)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-22 09:46:55 +02:00
Julien Herr f2981eec31 test: add inbound route tests covering IP auth, validation, and R2 upload 2026-05-22 07:39:48 +02:00
Julien Herr e874906291 docs: inline Durable Objects migration TODO in email-processor 2026-05-22 07:39:43 +02:00
Julien Herr 10404efffa fix: delete orphaned KV entries when trimming feed metadata 2026-05-22 07:39:37 +02:00
Julien Herr dae5db2524 feat: add GET /health endpoint 2026-05-22 07:39:24 +02:00
Julien HerrandClaude Sonnet 4.6 bde06dd3e4 fix(websub): add missing WebSub Link header to Atom feed
The RSS feed already advertised hub and self via the Link response
header, but the Atom feed was missing it entirely. Subscribers using
Atom had no way to discover the hub for real-time push notifications.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 23:46:51 +02:00
Julien HerrandClaude Sonnet 4.6 5723fd36f9 refactor(admin): validate JSON feed update via @hono/zod-validator
Moves validation of POST /api/feeds/:feedId/update from inline
schema.parse() to zValidator middleware. The route now receives
typed validated data via c.req.valid("json"), and returns a
structured {success: false, error: ZodIssue[]} on invalid input.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 23:46:51 +02:00
Julien HerrandClaude Sonnet 4.6 57e0cc5413 refactor(cors): replace manual CORS middleware with hono/cors
Fixes a bug where routes returning raw `new Response()` (RSS, Atom,
entries) were not receiving CORS headers — hono/cors applies headers
after next(), covering all response paths.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 23:46:51 +02:00
Julien Herr ece237c0af chore: ignore coverage/ directory 2026-05-21 23:46:51 +02:00
Julien HerrandClaude Sonnet 4.6 68151cbb5f fix(websub): require feed existence for subscriptions, remove atom hub header, simplify router mounting
- Add KV feed existence check in hub.ts to prevent SSRF via non-existent feeds (returns 404)
- Treat empty string hub.secret as absent (|| instead of ??)
- Remove misleading hub Link header from atom.ts (hub only supports RSS topics)
- Simplify double-layered hub router in index.ts (direct app.route instead of nested Hono)
- Update hub.test.ts to seed KV with feed config for tests requiring valid subscribe/unsubscribe

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 23:46:50 +02:00
Julien HerrandClaude Sonnet 4.6 0d00e003d4 test(websub): add ctx.waitUntil coverage for processEmail notification wiring
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 23:46:50 +02:00
Julien HerrandClaude Sonnet 4.6 d0764ddd8e feat(websub): wire real-time push notifications on email ingest
Pass ExecutionContext through the email processing chain so notifySubscribers
is called via ctx.waitUntil after a new email is stored.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 23:46:50 +02:00
Julien Herr 6d221a07dd feat(websub): add hub discovery Link headers to RSS and Atom feeds 2026-05-21 23:46:50 +02:00
Julien Herr aa41337c6b feat(websub): mount /hub route 2026-05-21 23:46:50 +02:00
Julien HerrandClaude Sonnet 4.6 09db52bb4d test(websub): add hub route tests
Add comprehensive tests for POST /hub validation (missing fields, unknown mode, non-HTTPS callback, invalid URL, wrong domain, secret > 200 bytes) and happy-path subscribe/unsubscribe (202). Also fix hub.ts to use a waitUntilSafe wrapper so executionCtx.waitUntil doesn't throw in Node test environments.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 23:46:50 +02:00
Julien Herr 4165774667 fix(websub): validate callback URL (HTTPS), fix domain regex, enforce secret length 2026-05-21 23:46:50 +02:00
Julien Herr ee4b9d8fdc feat(websub): add hub route (subscribe/unsubscribe) 2026-05-21 23:46:49 +02:00
Julien Herr e8f5af8b87 test(websub): add missing error-path tests for verify functions 2026-05-21 23:46:49 +02:00
Julien Herr c6785554d4 test(websub): add unit tests for WebSub utilities 2026-05-21 23:46:49 +02:00
Julien Herr 0b4ed0fbec fix(websub): fix signature header, add delivery logging, parallelize KV reads 2026-05-21 23:46:49 +02:00
Julien Herr 8810109e6f feat(websub): add WebSub utilities (HMAC, KV helpers, notify, verify) 2026-05-21 23:46:49 +02:00
Julien Herr ed6d2b4a0c feat(websub): add WebSubSubscription type 2026-05-21 23:46:49 +02:00
Julien Herr 20dc20ed1d Update FUNDING.yml to include GitHub sponsor 2026-05-21 21:44:13 +02:00
Julien HerrandClaude Sonnet 4.6 6fddbd40f2 feat: rename project to kill-the-news, add GitHub Pages landing page
- Rename all references from Email-to-RSS/email-to-rss to kill-the-news
  across README.md, AGENTS.md, package.json, wrangler-example.toml, setup.sh
- Add docs/index.html: dark-themed landing page for GitHub Pages
  covering features, how it works, quick start, and tech stack

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 20:45:25 +02:00
Julien Herr aede0c9d16 Merge pull request #1 from juherr/feat/add-eslint-lint-staged
feat: add ESLint, lint-staged, and update pre-commit hook + CI
2026-05-21 20:41:15 +02:00
Julien Herr b0c31ac8ef Create CNAME 2026-05-21 20:32:22 +02:00
Julien HerrandClaude Sonnet 4.6 b26990a875 fix: address PR review comments
- Fix KV json overload to return Promise<unknown | null> (null on missing keys)
- Add shebang to Husky pre-commit hook
- Explicitly add eslint ^10.0.0 to devDependencies

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 12:09:26 +02:00
Julien HerrandClaude Sonnet 4.6 b24ee969d1 style: fix Prettier formatting on 11 files
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 11:35:37 +02:00
Julien HerrandClaude Sonnet 4.6 3aea41f862 feat: add ESLint, lint-staged, and update pre-commit hook + CI
- Add ESLint 9 flat config (eslint.config.mjs) with typescript-eslint
  recommended rules and eslint-config-prettier
- Add lint-staged to run eslint+prettier only on staged files
- Update pre-commit hook to use lint-staged instead of full prettier check
- Add `lint` and `format:check` scripts to package.json
- Add Lint step to CI workflow
- Fix resulting lint errors: unused vars (_ctx, _options, catch binding),
  any→unknown in type declarations, stale eslint-disable comments

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 09:49:20 +02:00
Julien HerrandClaude Sonnet 4.6 e93bbb8d3e feat: store email attachments in R2 and expose as RSS enclosures
Attachments from incoming emails are uploaded to an optional Cloudflare R2
bucket and exposed as <enclosure> elements in RSS and <link rel="enclosure">
in Atom feeds, served at /files/{id}/{filename} with immutable caching.

R2 is opt-in: if ATTACHMENT_BUCKET is not bound the feature is a no-op.
Attachments are cleaned up from R2 on email/feed deletion and during
size-based feed trimming. Adds MockR2 to the test setup.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 09:09:37 +02:00
Julien Herr 3e28246c61 docs: document FEED_MAX_SIZE_BYTES and proxy auth setup in README 2026-05-21 08:42:48 +02:00
Julien HerrandClaude Sonnet 4.6 caaa6a7ba6 feat: add external proxy auth support (Authelia/Authentik)
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 08:39:10 +02:00
Julien HerrandClaude Sonnet 4.6 9eba4c34c6 feat: replace fixed 50-entry cap with size-based feed trimming
Emails are now trimmed from the oldest end when total serialised size
exceeds FEED_MAX_SIZE_BYTES (default 512 KB). Each EmailMetadata entry
stores its size so future trims are computed without re-reading KV.
Adds FEED_MAX_SIZE_BYTES, PROXY_TRUSTED_IPS and PROXY_AUTH_SECRET to Env.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 08:28:07 +02:00
Julien HerrandClaude Sonnet 4.6 984362f637 docs: add CLAUDE.md with project guidance for Claude Code
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 07:40:00 +02:00
Julien Herr 19b590a568 docs: mark Atom feed format as done in TODO 2026-05-21 07:36:14 +02:00
Julien Herr 1acc2f952c feat: mount /atom route in main app 2026-05-21 07:36:05 +02:00
Julien Herr b107803177 feat: add Atom 1.0 feed route at /atom/:feedId 2026-05-21 07:35:32 +02:00
Julien Herr 4d8d1fdc82 refactor: extract buildFeed helper, add generateAtomFeed 2026-05-21 07:34:29 +02:00
Julien HerrandClaude Sonnet 4.6 ad5bd9a79a docs: document Cloudflare Email Workers as primary ingestion method
Add setup instructions for Cloudflare Email Routing (Option A, recommended)
and reposition ForwardEmail as an optional alternative (Option B). Clarify
Cloudflare free plan includes Workers, KV, and Email Routing. Fix security
note scope and remove redundant/misleading content.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 00:36:44 +02:00
Julien HerrandClaude Sonnet 4.6 1c8b0ca194 ci: add CI workflow for format, typecheck, and test
Runs on push/PR to main. Also adds skipLibCheck to tsconfig to silence
pre-existing errors in vite/vitest declaration files that don't affect
the Worker build.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 00:07:25 +02:00
Julien HerrandClaude Sonnet 4.6 2e9cdd4364 test: add unit tests for EmailParser
Covers extractFeedId, decodeEncodedWords, and parseForwardEmailPayload
including edge cases: missing fields, structured vs text from, headerLines
vs raw headers string, RFC 2047 subject decoding.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 00:06:13 +02:00
Julien HerrandClaude Sonnet 4.6 41efee44ca refactor: replace custom escapeHtml with Hono's html template
Hono's `html` tagged template auto-escapes all interpolated values;
`raw()` is used for the email body which must render as HTML.
This removes the ad-hoc utility and aligns entries.ts with the
same pattern already used in admin.ts.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-21 00:05:35 +02:00
Julien HerrandClaude Sonnet 4.6 5308544672 refactor: simplify quick-win code after review
- Make feedId required in generateRssFeed (removes dead /emails/ fallback)
- Hoist loop-invariant conditional and remove intermediate variable
- Extract normalizeAllowedSenders() so JSON and form paths share same logic
- Move escapeHtml to src/utils/html.ts for reuse by admin.ts
- Parallelize the two independent KV puts in feed creation

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:55:17 +02:00
Julien HerrandClaude Sonnet 4.6 e140b433d8 docs: mark quick wins as done in TODO.md
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:52:26 +02:00
Julien HerrandClaude Sonnet 4.6 fdedbe13c4 feat: accept JSON on POST /admin/feeds/create and return {feedId, email, feedUrl}
When Content-Type is application/json, parse the request body as JSON and
return a JSON response instead of redirecting. Useful for automation tools
(e.g. Terraform/OpenTofu provisioning).

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:52:14 +02:00
Julien HerrandClaude Sonnet 4.6 298765527c feat: add HTML view for individual email entries at /entries/:feedId/:receivedAt
Serves each email as a standalone HTML page with a Content-Security-Policy
header, useful for reading emails outside a feed reader and for debugging.
Also updates RSS item links to point to this route.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:51:28 +02:00
Julien HerrandClaude Sonnet 4.6 54e7a1bfa0 feat: parse <author> from From header in RSS items
Parse the From header into name + email parts so the feed library
renders proper RFC 2822 format (email (Name)) in <author> elements.
Also passes feedId to the generator so item links can point to the
upcoming /entries/:feedId/:receivedAt route.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:50:54 +02:00
Julien HerrandClaude Sonnet 4.6 6a6614fb26 docs: add TODO.md with feature gaps vs kill-the-newsletter
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:44:44 +02:00
Julien HerrandClaude Sonnet 4.6 da6494cbd8 ci: pin action versions to SHA + add github-actions to Dependabot
Dependabot will now open PRs when new versions of actions/checkout and
actions/setup-node are released, keeping the pinned SHAs up to date.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:34:59 +02:00
Julien HerrandClaude Sonnet 4.6 5694498d89 ci: publish dist/index.js to GitHub Release on version tag
Triggers on v* tags. Builds with wrangler --dry-run, falls back to
dist/worker.js and dist/_worker.js if index.js is not produced.
Uploads as index.js so homelab Tofu can fetch a stable URL.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:28:47 +02:00
Julien HerrandClaude Sonnet 4.6 50b1911854 chore: add Dependabot config for weekly npm updates
Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:15:18 +02:00
Julien HerrandClaude Sonnet 4.6 243eec0cce refactor: extract ForwardEmail adapter to src/lib/forwardemail.ts
Mirrors the same pattern as cloudflare-email.ts — each provider has its
own adapter that translates provider-specific input into ProcessEmailInput
and delegates to processEmail(). inbound.ts is now a thin HTTP handler.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:13:43 +02:00
Julien HerrandClaude Sonnet 4.6 71a5c20e62 refactor: parallelize KV writes, cap metadata, remove redundant normalization
- email-processor: run email put + metadata get in parallel (saves one KV round-trip per email)
- email-processor: add missing 50-email metadata cap (was unbounded unlike storage.ts)
- email-processor: remove redundant normalizeEmail call on already-normalized allowedSender
- email-parser: strip WHAT-comments and boilerplate JSDoc throughout

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
2026-05-20 23:05:26 +02:00
Julien HerrandClaude Sonnet 4.5 093efe7fc9 feat: add Cloudflare Email Workers support alongside ForwardEmail
Both email providers now work in parallel on the same Worker:
- ForwardEmail: existing POST /api/inbound webhook (unchanged)
- Cloudflare Email Routing: native `email` handler using postal-mime

New files:
- src/lib/email-processor.ts  shared business logic (feed lookup,
  sender allowlist, KV storage) extracted from inbound.ts
- src/lib/cloudflare-email.ts  Cloudflare `email` handler; parses
  raw RFC 2822 email with postal-mime, delegates to processEmail()
- src/lib/email-processor.test.ts  9 unit tests
- src/lib/cloudflare-email.test.ts  5 integration tests

Also fixes pre-existing CORS 204 response: c.text("", 204) →
c.body(null, 204) to match Hono's EmptyStatusCode constraint.

To enable: configure Cloudflare Email Routing with a catch-all rule
`*@domain.com` pointing to this Worker.

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
2026-05-20 22:54:46 +02:00
Julien HerrandClaude Sonnet 4.5 29446a2aac chore: add typecheck script and fix pre-existing TypeScript errors
- Add `typecheck` script (`tsc --noEmit`) to package.json
- Remove conflicting `declare global` from test/setup.ts (superseded
  by @cloudflare/workers-types); use `globalThis as any` for test globals
  and declare minimal `require` locally to avoid pulling in @types/node
- Cast `createMockEnv()` and `deleteRes.json()` results in admin.test.ts
  to silence strict `unknown` / MockKV-vs-KVNamespace errors

Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
2026-05-20 22:54:32 +02:00
Julien HerrandClaude Sonnet 4.5 3ed9d2ee22 chore: apply Prettier formatting to entire codebase
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
2026-05-20 22:01:53 +02:00
140 changed files with 23072 additions and 6824 deletions

No files matched your search

+30
View File
@@ -0,0 +1,30 @@
{
"version": "0.0.1",
"configurations": [
{
"name": "docs",
"runtimeExecutable": "npx",
"runtimeArgs": ["serve", "docs", "-p", "4321", "--no-clipboard"],
"port": 4321
},
{
"name": "dev",
"runtimeExecutable": "npm",
"runtimeArgs": ["run", "dev"],
"port": 8787
},
{
"name": "dev-build",
"runtimeExecutable": "npx",
"runtimeArgs": [
"wrangler",
"dev",
"--config",
"wrangler.build.toml",
"--port",
"8788"
],
"port": 8788
}
]
}
@@ -1,57 +0,0 @@
---
description: Project Implementation Plan
globs: *
alwaysApply: false
---
# Detailed Project Implementation Plan
This section is a top-level "blueprint" describing the solution architecture and how each component fits together.
## Overview
**Goal:** Build a service that turns email newsletters into RSS feeds, so you can subscribe in an RSS reader like Reeder. The service should provide unique email addresses per feed, a front-end admin panel, indefinite (or long-term) storage of newsletters, and minimal cost—preferably using Cloudflare services plus ForwardEmail.net.
## Key Components
1. **ForwardEmail.net**
- Accept incoming newsletters on your custom domain’s email addresses.
- Forward them (via webhook) to your API endpoint for processing.
- Free inbound plan includes JSON + raw MIME data.
2. **Cloudflare Workers**
- **Inbound Worker:** Receives the webhook from ForwardEmail.net, parses/stores newsletter data in KV (or R2).
- **RSS Worker:** Serves RSS feeds by reading from KV and outputting XML.
- **Admin Worker (potential):** Could serve a small UI or JSON API for feed management.
3. **Cloudflare KV**
- Key-value store for storing newsletter items (subject, date, HTML, etc.).
- Minimal cost for text data.
- Indefinite retention if you keep usage under limits.
4. **Cloudflare Pages (Optional)**
- Could host a separate front-end for admin tasks.
- Alternatively, build a simple admin UI directly within the Worker.
5. **Admin Dashboard**
- Basic login and feed creation (generate random email addresses).
- List newsletters and optionally delete them or rename feed titles.
- For a simple approach, implement a minimal password-protected area or JSON endpoints.
6. **Domain / DNS Setup**
- Use your custom domain (e.g. `mynewsletters.dev`).
- Add DNS records so ForwardEmail.net is the MX handler.
- Configure Cloudflare for general DNS (with “Orange Cloud” or not, depending on your proxying preferences).
- Verify your domain following ForwardEmail.net’s instructions.
## Data Flow
1. A newsletter arrives at `[email protected]`.
2. ForwardEmail.net triggers a webhook to `https://your-worker.example.com/api/inbound?feed=XYZ` with JSON + raw MIME.
3. The Worker parses the email, extracts relevant information (date, subject, HTML body), and stores it in KV under a key like `feed:XYZ:timestamp`.
4. When your RSS reader (e.g. Reeder) requests `GET https://your-worker.example.com/rss/XYZ`, the Worker fetches all items from KV for that feed, builds an RSS XML response, and returns it.
5. *(Optional)* The Admin Dashboard (via a password-protected route or a separate Cloudflare Pages front-end) can create new feed IDs, display items, etc.
## Summary of Implementation Steps
1. Set up the Domain and ForwardEmail.net for inbound mail.
2. Create a Cloudflare Worker to handle the inbound webhook.
3. Parse the email (using ForwardEmail.net’s parsed data or parsing the raw MIME if necessary).
4. Store the data in KV.
5. Create an RSS Worker endpoint to retrieve the data and output XML.
6. *(Optional)* Develop an Admin UI to create new feeds, list items, and manage them.
7. Deploy and test the solution. Subscribe to the feed with Reeder.
+1
View File
@@ -0,0 +1 @@
github: [juherr]
+14
View File
@@ -0,0 +1,14 @@
version: 2
updates:
- package-ecosystem: "npm"
directory: "/"
schedule:
interval: "weekly"
groups:
dev-dependencies:
dependency-type: "development"
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
+37
View File
@@ -0,0 +1,37 @@
name: CI
on:
push:
branches: ["main"]
paths-ignore:
- "docs/**"
pull_request:
branches: ["main"]
paths-ignore:
- "docs/**"
jobs:
ci:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 22
cache: npm
- run: npm ci
- name: Format check
run: npm run format:check
- name: Lint
run: npm run lint
- name: Type check
run: npm run typecheck
- name: Test
run: npm test
+48
View File
@@ -0,0 +1,48 @@
name: Deploy Demo
on:
workflow_run:
workflows: ["CI"]
types: [completed]
branches: [main]
jobs:
deploy-demo:
runs-on: ubuntu-latest
if: ${{ github.event.workflow_run.conclusion == 'success' }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 22
cache: npm
- run: npm ci
- name: Generate wrangler.toml
env:
DEMO_KV_NAMESPACE_ID: ${{ secrets.DEMO_KV_NAMESPACE_ID }}
run: |
COMPAT_DATE=$(date +%Y-%m-%d)
sed \
-e "s/REPLACE_WITH_COMPATIBILITY_DATE/${COMPAT_DATE}/g" \
-e "s/REPLACE_WITH_YOUR_KV_NAMESPACE_ID/${DEMO_KV_NAMESPACE_ID}/g" \
-e "s/REPLACE_WITH_YOUR_PREVIEW_KV_NAMESPACE_ID/${DEMO_KV_NAMESPACE_ID}/g" \
-e "s/REPLACE_WITH_DEMO_KV_NAMESPACE_ID/${DEMO_KV_NAMESPACE_ID}/g" \
-e "s/REPLACE_WITH_YOUR_DOMAIN/kill-the.news/g" \
wrangler-example.toml > wrangler.toml
- name: Deploy to demo
run: npx wrangler deploy --env demo
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
- name: Set demo admin password
run: echo "$DEMO_ADMIN_PASSWORD" | npx wrangler secret put ADMIN_PASSWORD --env demo
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
DEMO_ADMIN_PASSWORD: ${{ secrets.DEMO_ADMIN_PASSWORD }}
+45
View File
@@ -0,0 +1,45 @@
name: Release
on:
push:
tags:
- "v*"
jobs:
release:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: 22
cache: npm
- run: npm ci
- run: npm run build
- name: Locate bundled output
id: bundle
run: |
for f in dist/index.js dist/worker.js dist/_worker.js; do
if [ -f "$f" ]; then
echo "path=$f" >> "$GITHUB_OUTPUT"
exit 0
fi
done
echo "No bundled output found in dist/" >&2
exit 1
- name: Create GitHub Release and upload bundle
env:
GH_TOKEN: ${{ github.token }}
TAG_NAME: ${{ github.ref_name }}
BUNDLE_PATH: ${{ steps.bundle.outputs.path }}
run: |
gh release create "$TAG_NAME" --generate-notes --verify-tag || true
gh release upload "$TAG_NAME" "$BUNDLE_PATH" --clobber
+2
View File
@@ -4,8 +4,10 @@ yarn.lock
# Build
dist/
coverage/
.wrangler/
.dev.vars
src/scripts/generated/
# Environment
.env
+2
View File
@@ -0,0 +1,2 @@
#!/bin/sh
npx lint-staged && npm run typecheck && npm test
-99
View File
@@ -1,99 +0,0 @@
# AGENTS.md
This file gives coding agents fast context for working in this repository.
## Project summary
Email-to-RSS is a Cloudflare Worker that ingests newsletters from ForwardEmail and exposes them as RSS feeds.
Core goals:
- Self-hosted and private
- Free-tier-friendly (Cloudflare + ForwardEmail)
- Minimal operational overhead
## Runtime and stack
- Runtime: Cloudflare Workers
- Framework: Hono (`src/index.ts` + `src/routes/*`)
- Storage: Cloudflare KV (`EMAIL_STORAGE` binding)
- Typescript + Vitest for development/testing
## Important files
- `setup.sh`: bootstraps local setup, KV namespaces, secrets, and local Wrangler config
- `wrangler-example.toml`: template used by setup
- `src/index.ts`: app boot + CORS + inbound IP allowlist middleware
- `src/routes/inbound.ts`: email ingestion endpoint
- `src/routes/rss.ts`: RSS rendering endpoint
- `src/routes/admin.ts`: admin UI and feed/email management
- `src/test/setup.ts`: test runtime mocks (KV + Cache)
## KV data model
Current keys used by routes:
- `feeds:list` -> `{ feeds: Array<{ id, title }> }`
- `feeds:list.feeds[].description` -> optional description (used to keep the dashboard fast; older data may omit it)
- `feed:<feedId>:config` -> feed config object
- `feed:<feedId>:config.allowed_senders` -> optional sender allowlist (email or domain)
- `feed:<feedId>:metadata` -> `{ emails: Array<{ key, subject, receivedAt }> }`
- `feed:<feedId>:<timestamp>` -> stored email body/metadata
Notes:
- Some utility files contain alternate key helpers not used by routes (`src/utils/storage.ts`).
- Keep route behavior and key schema consistent when refactoring.
## Setup/deploy workflow
1. `npx wrangler login`
2. `bash setup.sh`
3. Configure ForwardEmail DNS records in Cloudflare
4. `npm run deploy`
`setup.sh` assumes Wrangler v4 command syntax (`wrangler kv namespace ...`).
## Development workflow
- Install: `npm install`
- Test: `npm test`
- Build (dry-run deploy bundle): `npm run build`
- Dev server: `npm run dev`
## Testing notes
- Tests run in Node environment (`vitest.config.ts`), not DOM.
- Hono v4 test requests pass env as the 3rd arg: `app.request(path, init, env)`.
- Some tests intentionally hit validation errors; stderr logs are expected.
## Security assumptions
- Inbound endpoint only accepts requests from ForwardEmail source IPs.
- Admin access uses a signed cookie gate and password stored in Worker secret (`ADMIN_PASSWORD`).
- Admin pages set `Cache-Control: no-store`.
- Prefer setting `allowed_senders` on legitimate feeds to reduce inbound spam.
- Do not hardcode credentials or domain-specific secrets into tracked files.
## Spam cleanup workflow
- First choice: use dashboard bulk actions (`/admin`) with search + checkbox selection.
- Use **Table** view for bulk delete.
- Table columns are resizable and sortable; widths persist per-browser via localStorage.
- **Select Results** selects all rows currently shown by the search filter; **Clear Selection** unselects everything.
- Bulk deletes are performed asynchronously (batched requests) so the UI stays responsive.
- Avoid wildcard deletion; prefer search + small batches to reduce risk of deleting legitimate feeds.
## Cloudflare/Wrangler conventions
- `wrangler.toml` is generated locally from `wrangler-example.toml`.
- Keep `compatibility_date` current on meaningful runtime upgrades.
- Prefer explicit `--env production` for deploy/secret commands.
## If you change behavior
Update all of the following together:
- `README.md`
- `setup.sh` (if setup/deploy assumptions changed)
- tests under `src/routes/*.test.ts` and `src/test/setup.ts`
+205
View File
@@ -0,0 +1,205 @@
# CLAUDE.md
This file provides guidance to Claude Code when working in this repository.
## Commands
```bash
npm install # Install dependencies (also builds client scripts via prepare)
npm run dev # Start local dev server (wrangler dev)
npm test # Run all tests once
npm run test:watch # Run tests in watch mode
npm run test:coverage # Run tests with coverage report
npm run build # Dry-run deploy bundle (wrangler deploy --dry-run)
npm run build:client # Compile client scripts only (src/scripts/client → src/scripts/generated)
npm run deploy # Deploy to Cloudflare production
npm run format # Format with Prettier
```
Run a single test file:
```bash
npx vitest run src/routes/admin.test.ts
```
## Project summary
kill-the-news is a Cloudflare Worker that ingests email newsletters and exposes them as private RSS/Atom feeds. Self-hosted, free-tier-friendly (Cloudflare + ForwardEmail).
## Development approach
Work **test-first (TDD)** and **domain-driven (DDD)** in this repo — both are first-class, not optional.
**TDD.** Write or extend a test before/with the change, then make it pass. Mirror the existing test layout (`*.test.ts` next to the source, `createMockEnv()` from `src/test/setup.ts`, MSW for outbound HTTP). End every change green: `npx tsc --noEmit`, `npm test`, and `npm run build` (dry-run deploy) must all pass before declaring done.
**DDD.** Before adding logic, check whether the domain already models the concept — reach for the value objects in `src/domain/value-objects/` (`EmailAddress`, `Domain`, `FeedId`, `Lifetime`, `SenderPolicy`) and the `Feed` aggregate rather than re-deriving things ad hoc. New behavior belongs on the type that owns the data (e.g. "sender site URL" lives on `EmailAddress`, not in a helper). Respect the layering and aggregate rules below — imports point inward (routes → application → domain; infrastructure implements ports), and never reach across a layer for convenience (e.g. importing a favicon/infra helper just to parse a domain). When the same derivation appears twice, that's the signal to push it onto a domain type.
## Architecture
Single Cloudflare Worker built with Hono. Routes:
| Method | Path | Purpose |
| ------------------------------------ | ---------------------------------------------------------------------- | ------- |
| `GET /` | Public status page (monitoring counters + link to admin) |
| `POST /api/inbound` | Webhook from ForwardEmail; IP-allowlisted to their MX sources |
| `/api/v1/feeds*` | Versioned REST API (Bearer/proxy auth) — feeds + emails CRUD |
| `GET /api/v1/stats` | Public monitoring counters (JSON, CORS); canonical stats endpoint |
| `GET /api/openapi.json` | OpenAPI 3.1 spec (public) |
| `GET /api/docs` | Rendered API reference (Scalar, public) |
| `GET /rss/:feedId` | Public RSS 2.0 feed |
| `GET /atom/:feedId` | Public Atom feed (with WebSub hub header) |
| `GET /entries/:feedId/:entryId` | Individual email HTML view |
| `GET /files/:attachmentId/:filename` | R2 attachment serving |
| `GET /admin` | Password-protected admin UI |
| `/hub` | WebSub hub (subscribe/publish) |
| `GET /favicon.svg`, `/favicon.ico` | Project favicon (envelope logo); fallback for per-feed favicons |
| `GET /favicon/:feedId` | Per-feed favicon from the last sender's domain (falls back to project) |
| `GET /health` | Health check |
| `email` | Cloudflare Email routing handler (alternative to ForwardEmail webhook) |
### Source layout
```
src/
index.ts # App entrypoint: CORS, IP middleware, route mounting, email handler export
config/constants.ts # Shared constants (TTLs, limits)
types/index.ts # Env, FeedConfig, EmailData, WebSubSubscription, etc.
domain/ # Framework-agnostic core (no Hono/infra imports leak out)
feed.aggregate.ts # Feed aggregate: consistency boundary; holds domain FeedState (camelCase), exposes intention-revealing reads, never raw state/metadata
feed-state.ts # FeedState: the aggregate's config in domain (camelCase) vocabulary — NOT the snake_case persistence DTO
feed.ts # The expiry predicate (`isExpired`) — the one invariant shared with the read-model routes
feed-keys.ts # The KV key schema (pure string builders), shared by every repository
clock.ts # Clock port (systemClock) — injected into the aggregate; no ambient Date.now()
events.ts # FeedEvent union (FeedCreated, EmailIngested) — each carries its feedId
email-parser.ts # Email parsing (addresses, headers, encoded words)
format.ts # Pure formatting helpers (formatBytes)
value-objects/ # FeedId, EmailAddress, Domain, SenderPolicy, Lifetime (immutable, self-validating)
application/ # Use-cases / orchestration (wires domain + infrastructure)
feed-service.ts # createFeedRecord / editFeedDetails / editFeed / deleteFeedRecord (admin UI + REST API)
feed-cleanup.ts # Feed/email storage cleanup: purgeFeedKeysStep, collectUnsubscribeUrls, attachment+key deletion
feed-events.ts # Dispatcher: maps aggregate FeedEvents to side effects (counters, WebSub, favicon)
email-processor.ts # Core ingestion: load aggregate → accepts? → feed.ingest → persist
feed-fetcher.ts # Read model for RSS/Atom rendering (config + email bodies; bypasses the aggregate)
stats.ts # Monitoring counters increment policy + storage scans
infrastructure/ # Adapters: KV/R2, outbound HTTP, logging, framework glue
logger.ts # JSON structured logger
feed-repository.ts # KV adapter for the Feed aggregate + global feed list + email bodies (load/save)
feed-mapper.ts # Translation seam: domain FeedState ↔ persistence DTOs (FeedConfig/FeedListItem); sole owner of snake_case outside the edge
icon-repository.ts # KV adapter for cached favicons (icon:*)
websub-subscription-repository.ts # KV adapter for WebSub subscriber lists (websub:subs:*)
counters-repository.ts # KV adapter for the monitoring counters singleton (stats:counters)
auth.ts # timingSafeEqual, proxy-auth check, API bearer middleware
cloudflare-email.ts # Cloudflare Email routing handler
forwardemail.ts # ForwardEmail webhook types/parsing
worker.ts # Typed worker / waitUntil helper
attachments.ts # R2 bucket accessor
favicon-fetcher.ts # Outbound favicon fetch + cache (uses IconRepository)
feed-generator.ts # RSS/Atom XML generation
html-processor.ts # Email HTML sanitization / inline cid: rewriting
websub.ts # WebSub subscription management + delivery
unsubscribe.ts # RFC 8058 one-click unsubscribe dispatch
urls.ts # URL builders
routes/
inbound.ts # ForwardEmail webhook handler
rss.ts # RSS feed renderer
atom.ts # Atom feed renderer
entries.ts # Single email HTML view
files.ts # R2 attachment serving
hub.ts # WebSub hub
home.tsx # Public status page (GET /)
admin.tsx # Admin UI entrypoint (hono/jsx)
admin/ # Admin sub-modules
feeds.tsx # Feeds CRUD UI
emails.tsx # Emails list/delete UI
ui.tsx # Shared UI components
helpers.ts # Shared admin helpers
api/ # Versioned REST API (@hono/zod-openapi)
index.ts # OpenAPIHono app: /v1 routes + /openapi.json + /docs
schemas.ts # Zod schemas (validation + OpenAPI source of truth)
scripts/
client/ # TypeScript client scripts (compiled by esbuild)
dashboard.ts # Admin dashboard interactions
emails-page.ts # Emails page interactions
generated/ # Compiled output (gitignored, rebuilt on npm install)
styles/ # CSS files bundled into the Worker
variables.css
layout.css
components.css
utilities.css
data/nouns.ts # Word list for ID generation
test/setup.ts # Test mocks: MockKV, createMockEnv()
```
### KV schema
All data lives in the `EMAIL_STORAGE` KV namespace:
| Key | Value |
| --------------------------- | ---------------------------------------------------------------------------------------------- |
| `feeds:list` | `{ feeds: Array<{ id, title, description?, expires_at? }> }` |
| `feed:<feedId>:config` | `FeedConfig` |
| `feed:<feedId>:metadata` | `{ emails: Array<{ key, subject, receivedAt, size?, attachmentIds?, inlineAttachmentIds? }> }` |
| `feed:<feedId>:<timestamp>` | Full `EmailData` |
| `websub:subs:<feedId>` | `WebSubSubscription[]` (per-feed subscriber list) |
| `icon:<domain>` | Cached favicon record (base64 + content type; negative entries allowed) |
| `stats:counters` | `Counters` (cumulative monitoring counters singleton) |
The KV key schema lives in `src/domain/feed-keys.ts` (pure, framework-agnostic) — never inline a `feed:`/`feeds:list`/`websub:`/`icon:`/`stats:counters` key string anywhere else. KV access is owned by four repository **adapters** in `src/infrastructure/`, each for one concern: `FeedRepository` (the Feed aggregate + global list + email bodies), `IconRepository` (`icon:*`), `WebSubSubscriptionRepository` (`websub:subs:*`), and `CountersRepository` (`stats:counters`). Go through a repository, never `env.EMAIL_STORAGE.get/put` directly. The domain depends only on the key schema, not on these adapters.
### Domain & layering rules
- **Layers**: `domain/` is framework-agnostic (no Hono). `application/` orchestrates use-cases. `infrastructure/` holds adapters (KV/R2, HTTP, logging). `routes/` is the HTTP edge. Imports point inward: routes → application → domain; infrastructure implements ports the inner layers call.
- **The `Feed` aggregate is the only writer of feed config + the email index.** Load it with `FeedRepository.load(feedId)`, mutate via its methods (`ingest`, `removeEmails`, `edit`), then persist with `save`/`saveMetadata`/`saveConfig`. No route or service mutates `metadata.emails` directly. Email **bodies** are large blobs outside the aggregate — flush them (`putEmail`/`deleteEmail`) alongside the metadata save.
- **The domain never speaks the storage dialect.** The aggregate holds its config as domain `FeedState` (camelCase), never the snake_case `FeedConfig` DTO. The translation `FeedState ↔ FeedConfig/FeedListItem` lives in `infrastructure/feed-mapper.ts` — the only place outside the HTTP edge that knows the persisted field names. `FeedRepository.load` maps DTO→state on the way in; `save`/`saveConfig` map state→DTO on the way out.
- **The aggregate never exposes its raw state.** It has no `state`/`metadata` getters (a shallow `Readonly<…>` would still leak mutable arrays). Read named accessors (`title`, `expiresAt`, `emails`, `allowedSenders()`, …) which return copies; the repository reads `state()`/`toMetadataSnapshot()` (copies) and runs them through the mapper.
- **One edit path.** `edit(patch, { lifetime? })` is the single mutation for config. A `Lifetime` VO is resolved by the application (env `FEED_TTL_HOURS` override + client request); its **presence recomputes expiry, its absence preserves it** — which is exactly the dashboard's title/description quick-edit (no lifetime passed). It rejects an already-expired feed, so a quick-edit can no more touch an expired feed than a full edit can.
- **`feeds:list` stays in sync automatically.** `FeedRepository.save`/`saveConfig` upsert the registry entry via `toListItemDTO(feed.id, feed.state())` — services never mirror title/description/expiry into the list by hand.
- Read-only RSS/Atom rendering uses the `feed-fetcher` read model, not the aggregate (no invariant to enforce on the hot path).
- KV has no multi-key transaction; the aggregate is the seam a future Durable Object would wrap to serialise concurrent ingests (see `email-processor.ts`).
- **Side effects via domain events.** Mutations with consequences record a `FeedEvent` (`FeedCreated`, `EmailIngested`), each carrying its own `feedId`. After persisting, the caller hands the aggregate to `application/feed-events.dispatchFeedEvents(feed, env, schedule)` — the single dispatch entry point that drains `pullEvents()` and runs the counters/WebSub/favicon. Don't pull events or thread the feed id by hand at call sites. Side effects with no aggregate mutation (a rejected email, feed deletion that bypasses the aggregate, bulk admin ops, the cron) stay imperative — they have no event to ride on.
- **`FeedId` flows through the layers.** It is the identity type taken by the domain (`Feed.id`), the application use-cases (`editFeed`, `editFeedDetails`, `deleteFeedRecord`, `fetchFeedData`, the cleanup steps) and the infrastructure repositories/services (`FeedRepository`, `WebSubSubscriptionRepository`, `notifySubscribers`, …). Mint it **once** at the edge — `FeedId.parse(address)` for inbound email (validates), `FeedId.unchecked(param)` at the HTTP edge (no revalidation: a bad id just misses in KV and 404s), `FeedId.generate()` for a new feed — then pass the VO inward. Unwrap to `.value` (string) only at the true serialisation edges: URL builders (`urls.ts`), XML generation (`feed-generator.ts`), the KV key schema (`feed-keys.ts`), logs and JSON responses.
### Worker bindings (`Env`)
```ts
EMAIL_STORAGE: KVNamespace; // All feed/email data
ADMIN_PASSWORD: string; // Worker secret — never in config files
DOMAIN: string; // e.g. "getmynews.app"
ATTACHMENT_BUCKET?: R2Bucket; // R2 for email attachments
FEED_MAX_SIZE_BYTES?: string; // Optional email size cap
PROXY_TRUSTED_IPS?: string; // Trusted reverse-proxy IPs
PROXY_AUTH_SECRET?: string; // Shared secret for proxy auth
```
### Client scripts
`src/scripts/client/` contains TypeScript that runs in the browser. It is compiled by esbuild into `src/scripts/generated/` (gitignored) and bundled into the Worker as inline `<script>` tags. The `prepare` npm hook rebuilds them on `npm install`. Run `npm run build:client` to rebuild manually.
### Testing
Tests run in Node (not a Worker runtime). Hono test requests pass the mock env as the 3rd argument:
```ts
const res = await app.request("/path", init, createMockEnv());
```
MSW (`msw/node`) handles external HTTP mocks. Tests that hit validation paths intentionally produce stderr output — expected.
## Configuration
- `wrangler.toml` is generated locally from `wrangler-example.toml` by `setup.sh` — do not commit it
- `ADMIN_PASSWORD` is set via `wrangler secret put` — never in config files
- Keep `compatibility_date` current on runtime upgrades
## When changing behavior
**Always document evolutions** — treat docs as part of the change, not a follow-up. When you add or change a feature, update the relevant docs in the same change:
- `README.md`
- `INSTALL.md` (setup, deployment, and configuration guide)
- `setup.sh` (if setup/deploy assumptions changed)
- Tests under `src/routes/*.test.ts` and `src/test/setup.ts`
Keep it proportionate: user-facing or config changes warrant doc updates; purely internal refactors usually don't.
**Marketing landing page (`docs/index.html`).** This is the public GH Pages site (served at the `CNAME` domain), not the in-app status page (`src/routes/home.tsx`). When a feature is also a selling point — something a prospective self-hoster would care about (privacy guarantees, full-body capture, burnable aliases, reader compatibility, automation/API, AI features…) — surface it there too (hero copy or a feature card), matching the existing section/card style. Internal correctness fixes don't belong on the landing page; differentiators do.
+86
View File
@@ -0,0 +1,86 @@
# Contributing to kill-the-news
Thanks for your interest in contributing! This is a small, self-hosted
Cloudflare Worker project. Issues, bug fixes, and well-scoped features are all
welcome.
## Getting started
Requirements: Node.js (LTS) and npm. A Cloudflare account is only needed to
deploy, not to run tests locally.
```bash
git clone https://github.com/juherr/kill-the-news.git
cd kill-the-news
npm install # installs deps and builds client scripts (prepare hook)
npm run dev # start the local dev server (wrangler dev)
```
For full setup, deployment, and configuration details, see
[INSTALL.md](INSTALL.md).
## Development workflow
```bash
npm test # run all tests once
npm run test:watch # run tests in watch mode
npm run build # dry-run deploy bundle (wrangler deploy --dry-run)
npm run format # format with Prettier
```
Run a single test file:
```bash
npx vitest run src/routes/admin.test.ts
```
Client-side scripts live in `src/scripts/client/` and are compiled by esbuild
into `src/scripts/generated/` (gitignored). They rebuild on `npm install`; to
rebuild manually:
```bash
npm run build:client
```
The architecture, source layout, and KV schema are documented in
[CLAUDE.md](CLAUDE.md) — a good orientation before making changes.
## Before opening a pull request
- **Add or update tests** for any behavior change. Tests live in
`src/routes/*.test.ts`, with shared mocks in `src/test/setup.ts`.
- **Run the checks**: `npm test`, `npm run build`, and `npm run format`.
Pre-commit hooks (husky + lint-staged) run lint/format on staged files.
- **Keep docs in sync.** When you change behavior, update the relevant files
together:
- `README.md`
- `INSTALL.md` (setup, deployment, configuration)
- `setup.sh` (if setup/deploy assumptions changed)
- **Keep PRs focused.** One logical change per PR is easier to review.
## Commit messages
This project follows [Conventional Commits](https://www.conventionalcommits.org/)
with a scope, matching the existing history. Examples:
```
feat(admin): collapse create-feed form into accordion
fix(attachments): render inline cid: images in emails and feeds
refactor(home): dedupe byte formatting in storage cards
docs(readme): add Continuous deployment section
```
Common types: `feat`, `fix`, `refactor`, `docs`, `test`, `chore`.
## Reporting bugs and requesting features
Open an issue at
[github.com/juherr/kill-the-news/issues](https://github.com/juherr/kill-the-news/issues).
For bugs, include reproduction steps, expected vs. actual behavior, and your
environment (ingestion method, relevant config). For security issues, follow
[SECURITY.md](SECURITY.md) instead of opening a public issue.
## License
By contributing, you agree that your contributions will be licensed under the
[MIT License](LICENSE).
+236
View File
@@ -0,0 +1,236 @@
# Installation & deployment
How to set up, run, deploy, and configure kill-the-news. For an overview of what the project does, see [README.md](README.md).
## Requirements
- Node.js 20+
- A Cloudflare account (free plan works — Workers, KV, and Email Routing are all included)
- A domain added to Cloudflare as a zone (DNS managed by Cloudflare)
- A ForwardEmail account _(Option B only)_
## Cloudflare setup
If your domain is not yet on Cloudflare: in the [Cloudflare dashboard](https://dash.cloudflare.com/), go to _Add a site_, enter your domain, choose the Free plan, and follow the instructions to update your nameservers at your registrar. Wait for the zone to become active (usually a few minutes).
## Setup
1. Clone this repository.
2. Authenticate Wrangler:
```bash
npx wrangler login
```
3. Run setup:
```bash
bash setup.sh
```
The script will prompt for an admin password and your domain, then:
- install npm dependencies
- verify Cloudflare auth (`wrangler whoami`)
- create KV namespaces (`EMAIL_STORAGE` + preview) in your account
- set the `ADMIN_PASSWORD` secret in the `production` environment
- generate `wrangler.toml` from `wrangler-example.toml` with your KV IDs, domain, and today's compatibility date
4. Configure email ingestion — choose **one** of the two options below.
### Option A — Cloudflare Email Workers (recommended)
No third-party service required. Cloudflare receives the email and hands it directly to the Worker.
1. In the Cloudflare dashboard, go to _Email → Email Routing_ for your zone and click **Enable Email Routing**. Cloudflare will prompt you to add MX and SPF records — accept and it adds them automatically.
2. Under _Email Routing → Routing Rules_, add a **Catch-all** rule:
- Action: **Send to Worker**
- Worker: `kill-the-news` (the name from `wrangler.toml`)
That's it. No webhook configuration is needed.
### Option B — ForwardEmail (alternative)
Use this if you prefer ForwardEmail's additional features (sender filtering, open-tracking, etc.).
Add these DNS records in Cloudflare (_DNS → Records_):
| Type | Name | Content | Notes |
| ---- | ---- | ---------------------------------------------------- | ----------------------- |
| MX | @ | `mx1.forwardemail.net` | Priority `10`, DNS only |
| MX | @ | `mx2.forwardemail.net` | Priority `10`, DNS only |
| TXT | @ | `"forward-email=https://yourdomain.com/api/inbound"` | webhook target |
| TXT | @ | `"v=spf1 include:spf.forwardemail.net -all"` | SPF |
Replace `yourdomain.com` with your actual domain.
The Worker verifies each webhook request against ForwardEmail's published MX IP list before processing it.
5. Deploy:
```bash
npm run deploy
```
Wrangler will create the Worker and register `yourdomain.com` (and `www.yourdomain.com`) as custom domains pointing to it. Cloudflare handles TLS automatically.
6. Open `https://yourdomain.com/admin` and sign in.
> **Tip:** To verify the Worker is running, check _Workers & Pages → kill-the-news_ in the Cloudflare dashboard. The _Custom Domains_ tab should list your domain once the deploy succeeds.
## Development
```bash
npm install
npm run dev
npm test
npm run build
```
## Continuous deployment (GitHub Actions)
The repo ships a [`Deploy Demo`](.github/workflows/demo.yml) workflow that generates `wrangler.toml` from `wrangler-example.toml` and runs `wrangler deploy --env demo` after CI passes on `main`. To wire up your own automated deploys, set these repository secrets (_Settings → Secrets and variables → Actions_):
| Secret | Purpose |
| ----------------------- | ------------------------------------------------------------------- |
| `CLOUDFLARE_API_TOKEN` | Scoped API token used by Wrangler to deploy (see permissions below) |
| `CLOUDFLARE_ACCOUNT_ID` | Target Cloudflare account ID |
| `DEMO_KV_NAMESPACE_ID` | KV namespace ID substituted into the generated `wrangler.toml` |
| `DEMO_ADMIN_PASSWORD` | Admin password set via `wrangler secret put` |
### Deploy token permissions
Local `npx wrangler login` uses OAuth and already has every permission, so the gaps below only bite **scoped API tokens** (i.e. CI). Create the token at <https://dash.cloudflare.com/profile/api-tokens> — the **"Edit Cloudflare Workers"** template is the easiest base — and make sure it carries the permissions matching the bindings you actually deploy:
| Permission | Needed for |
| ------------------------------------------------- | -------------------------------------------------------------------------- |
| Account · **Workers Scripts** · Edit | Deploying the Worker and running `wrangler secret put` |
| Account · **Workers KV Storage** · Edit | The `EMAIL_STORAGE` KV binding |
| Account · **Workers R2 Storage** · Edit | The `ATTACHMENT_BUCKET` R2 binding (only when attachments are enabled) |
| Zone · **Workers Routes** · Edit + **DNS** · Edit | The `custom_domain` routes (e.g. `demo.kill-the.news`), scoped to its zone |
Scope the token to the relevant **account** and, for custom domains, the relevant **zone**. A missing R2 permission fails with `Authentication error [code: 10000]` on `/r2/buckets/...`; a missing routes/DNS permission fails while provisioning the custom domain. The `User Details`/`Memberships` warnings Wrangler prints are only for `whoami` display and are not fatal.
## Configuration notes
- `wrangler-example.toml` is the template; `wrangler.toml` is generated locally.
- Keep `compatibility_date` fresh when doing runtime upgrades.
- `ADMIN_PASSWORD` is a Cloudflare Worker secret, not a plain env var in config.
### Catch-all fallback forwarding
By default, inbound mail that doesn't match a feed is dropped (logged, then discarded). If you want to point a domain's **catch-all** at this worker without losing your personal mail, set an optional fallback address — non-feed mail is forwarded there instead of dropped:
```toml
[vars]
FALLBACK_FORWARD_ADDRESS = "[email protected]"
```
**Prerequisite:** the address must be a **verified destination** in _Email → Email Routing → Destination addresses_ (Cloudflare won't forward to an unverified address — `message.forward()` fails, and the worker just logs a warning). This only applies to the Cloudflare Email Workers path (Option A).
What gets forwarded vs dropped:
| Situation | Action |
| -------------------------------------------------- | --------------------- |
| Address isn't a feed (e.g. `you@`, typo) | forward |
| Well-formed feed address but no such feed | forward |
| Feed exists but is **expired** | drop |
| Feed exists but the sender is **blocked/filtered** | drop |
| Delivered to a live feed | ingested (no forward) |
Expired feeds and blocked senders are dropped on purpose, so a real newsletter never leaks into your fallback inbox. Leave the variable unset to keep the original drop-and-log behavior.
### Feed size limit
By default the worker keeps emails until the feed's stored data exceeds **512 KB**, then drops the oldest entries (and their KV records) to stay under the limit. This is more robust than a fixed entry count for HTML-heavy newsletters.
To override the threshold, add to `wrangler.toml` under `[vars]`:
```toml
FEED_MAX_SIZE_BYTES = "524288" # 512 KB — adjust as needed
```
### Email attachments (R2)
When an incoming email contains attachments, the Worker can store them in a Cloudflare R2 bucket and expose them as `<enclosure>` elements in the RSS feed (and `<link rel="enclosure">` in Atom). Each attachment is served at `/files/{id}/{filename}` with an immutable cache header. Attachments are also listed with download links on the admin email detail page and the public entry view.
Inline images (the ones an email references with `src="cid:…"`) are handled separately: they are still stored in R2 (and deleted with the email), but instead of appearing in the attachment list they render in place — the `cid:` reference is rewritten to the stored `/files/{id}/{filename}` URL in the feed, the admin preview, and the public entry view.
This feature is **optional**. If no R2 bucket is bound, attachments are silently ignored and nothing else changes.
**Setup (automated):** `setup.sh` now asks _"Enable email attachments stored in R2?"_. Answer yes and it creates the buckets (`<worker>-attachments` and `<worker>-attachments-preview`) and wires the binding into the generated `wrangler.toml` for you.
**Setup (manual):**
1. Create an R2 bucket in the Cloudflare dashboard (_R2 Object Storage → Create bucket_), or with Wrangler:
```bash
npx wrangler r2 bucket create your-bucket-name
```
2. In `wrangler.toml`, uncomment and fill in the R2 binding (the commented block from `wrangler-example.toml`):
```toml
r2_buckets = [
{ binding = "ATTACHMENT_BUCKET", bucket_name = "your-bucket-name", preview_bucket_name = "your-bucket-name-preview" }
]
```
The binding is **per environment**: add it under every env you deploy (`[env.production]`, `[env.demo]`, …), each pointing at its own bucket.
3. Redeploy:
```bash
npm run deploy
```
> **Deploy token permission:** with an R2 binding, `wrangler deploy` verifies the bucket exists, so a scoped CI token also needs **Account → Workers R2 Storage** — see [Continuous deployment](#continuous-deployment-github-actions). Local `npx wrangler login` already has it.
**Turning it off:** set `ATTACHMENTS_ENABLED = "false"` in `[vars]` to disable attachments even while the R2 bucket stays bound (useful to cap usage on a demo). Any other value (or leaving it unset) keeps the feature on whenever R2 is configured.
Attachments are deleted from R2 automatically when the corresponding email is deleted from the admin UI, or when an email is dropped during feed size trimming.
**Monitoring storage / free tier:** the status page (`/`) and `/api/v1/stats` report R2 space used (against the **10 GB** R2 free tier) and an estimate of KV space used (against the **1 GB** KV free tier). The figures are refreshed hourly by the cron trigger. KV usage is an estimate based on stored email sizes, so treat it as a lower bound.
### External auth provider (Authelia / Authentik / reverse proxy)
Instead of the built-in password login you can delegate admin authentication to a reverse proxy that sets a trusted user header (`Remote-User` or `X-Forwarded-User`).
**Required Worker secrets** (set with `wrangler secret put`, never in `[vars]`):
| Secret | Description |
| ------------------- | ---------------------------------------------- |
| `PROXY_AUTH_SECRET` | Shared secret between the proxy and the Worker |
**Required `[vars]`** in `wrangler.toml`:
```toml
PROXY_TRUSTED_IPS = "10.0.0.1" # comma-separated IPs of your reverse proxy
```
When both are configured, the Worker authenticates a request if:
1. `CF-Connecting-IP` is in `PROXY_TRUSTED_IPS`
2. The `X-Auth-Proxy-Secret` header matches `PROXY_AUTH_SECRET`
3. `Remote-User` or `X-Forwarded-User` is non-empty
Password login remains available as a fallback when the proxy check fails.
> **Security note:** `CF-Connecting-IP` can be spoofed on direct `workers.dev` requests. Disable the `workers.dev` subdomain in production (`workers_dev = false` in `[env.production]`).
### REST API authentication
The versioned REST API (`/api/v1/*`) is authenticated independently of the cookie-based
admin UI — there is no CSRF check, so it is suited to server-to-server automation. A
request is authorized when **either**:
- it carries `Authorization: Bearer <ADMIN_PASSWORD>` (the same admin password secret), **or**
- it passes the reverse-proxy check above (`PROXY_TRUSTED_IPS` + `X-Auth-Proxy-Secret` + `Remote-User`).
The OpenAPI 3.1 spec (`/api/openapi.json`) and the Scalar reference (`/api/docs`) are
public. In the Scalar UI, click **Authorize** and paste the admin password as the bearer
token to try requests. See the route table in [README.md](README.md#rest-api).
## Upgrading dependencies
To refresh dependencies to latest:
```bash
npm outdated
npm install
npm test
npm run build
```
Then update `compatibility_date` and redeploy.
+1
View File
@@ -1,6 +1,7 @@
MIT License
Copyright (c) 2025 Young Lee
Copyright (c) 2025-2026 Julien Herr
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
+100 -77
View File
@@ -1,14 +1,14 @@
# Email-to-RSS
# kill-the-news
Convert email newsletters into a private RSS feed using Cloudflare Workers + ForwardEmail.
Convert email newsletters into private RSS feeds using Cloudflare Workers.
This project is self-hosted, uses your own domain, and keeps your data in your own Cloudflare account.
Self-hosted, uses your own domain, and keeps your data in your own Cloudflare account. Live at [kill-the.news](https://kill-the.news).
## Why this exists
Many newsletters only support email delivery. RSS readers offer a better reading experience, but getting email-only newsletters into RSS usually means relying on shared third-party infrastructure.
Email-to-RSS keeps the same workflow while avoiding shared domains and shared data stores.
kill-the-news keeps the same workflow while avoiding shared domains and shared data stores.
## Features
@@ -17,107 +17,130 @@ Email-to-RSS keeps the same workflow while avoiding shared domains and shared da
- Inline double-confirm delete interactions with toast feedback in the admin dashboard
- Resizable + sortable table columns in the admin dashboard (Table view)
- Unique newsletter addresses per feed (for example `[email protected]`)
- ForwardEmail webhook ingestion with source-IP verification
- Cloudflare Email Workers ingestion (no third-party service)
- ForwardEmail webhook ingestion with source-IP verification (optional alternative)
- Optional per-feed sender allowlist (`[email protected]` or `domain.com`)
- RSS generation on demand (`/rss/:feedId`)
- Atom feed at `/atom/:feedId`
- Reader-friendly output: relative links/images absolutized to the sender's site, lazy-loaded images promoted (`data-src` → `src`), plain-text feed titles, and XML-illegal control characters stripped so feeds parse in strict readers
- Per-feed favicon derived from the last sender's domain (`/favicon/:feedId`), cached and shown in feeds + admin
- Automatic RFC 8058 one-click unsubscribe when a feed is deleted — stops newsletters from mailing the now-dead address
- Email attachments stored in Cloudflare R2 and exposed as RSS enclosures (optional)
- Cloudflare KV storage for feed config + email metadata/content
- Password-protected admin UI
- Fully self-hosted on your Cloudflare account
- Versioned REST API (`/api/v1/*`) with an OpenAPI 3.1 spec and Scalar docs for automation
## Architecture
1. ForwardEmail forwards inbound messages to `https://yourdomain.com/api/inbound`.
2. The Worker validates the request source against ForwardEmail MX IP ranges.
3. The Worker parses and stores incoming content in KV.
4. `https://yourdomain.com/rss/:feedId` renders RSS from stored items.
5. `/admin` provides feed management and email deletion.
Two ingestion methods are supported — pick one or use both:
| Method | How it works |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| **Cloudflare Email Workers** | Cloudflare Email Routing delivers the raw message directly to the Worker via the `email()` handler — no outbound webhook needed |
| **ForwardEmail webhook** | ForwardEmail parses the message and POSTs a JSON payload to `POST /api/inbound`; the Worker verifies the source IP before processing |
Common path:
1. Incoming email arrives at `[email protected]`.
2. The Worker resolves the feed from the recipient address and stores the email in KV.
3. `https://yourdomain.com/rss/:feedId` renders RSS from stored items.
4. `/admin` provides feed management and email deletion.
5. `https://yourdomain.com/` shows a public status page with monitoring counters and a link to the admin.
Main routes:
- `src/routes/inbound.ts`: webhook ingestion
- `src/lib/cloudflare-email.ts`: Cloudflare Email Workers ingestion
- `src/routes/inbound.ts`: ForwardEmail webhook ingestion
- `src/routes/rss.ts`: RSS rendering
- `src/routes/admin.ts`: admin UI + feed CRUD
- `src/routes/atom.ts`: Atom feed rendering
- `src/routes/files.ts`: attachment file serving from R2
- `src/routes/admin.tsx`: admin UI + feed CRUD
- `src/routes/api/`: versioned REST API + OpenAPI spec/docs (`/api/v1/*`, `/api/openapi.json`, `/api/docs`)
- `src/lib/feed-service.ts`: shared feed create/update/delete (used by the admin UI and the REST API)
- `src/routes/home.tsx`: public status page (`GET /`)
## Requirements
### Monitoring
- Node.js 20+
- A Cloudflare account
- A domain managed in Cloudflare DNS
- A ForwardEmail account
`GET /api/v1/stats` returns JSON counters (public, no auth, CORS-enabled) for
uptime/monitoring tools and the landing page:
## Setup
| Field | Meaning |
| ----------------------------- | -------------------------------------------------------- |
| `active_feeds` | Feeds currently configured (live) |
| `feeds_created` | Total feeds ever created (cumulative) |
| `feeds_deleted` | Total feeds ever deleted (cumulative) |
| `emails_received` | Total emails ingested successfully (cumulative) |
| `emails_rejected` | Total emails rejected during validation (cumulative) |
| `websub_subscriptions_active` | Active WebSub subscriptions (live) |
| `last_email_at` | ISO 8601 date-time of the last ingested email |
| `last_feed_created_at` | ISO 8601 date-time of the last feed creation |
| `first_seen` | ISO 8601 date-time the instance first recorded a counter |
The same figures are rendered on the public status page at `GET /`. Cumulative counters
are persisted in the `EMAIL_STORAGE` KV under the `stats:counters` key.
### REST API
A versioned REST API lets you automate feed and email management without scraping the
admin UI. The OpenAPI 3.1 spec is served at `GET /api/openapi.json` and a rendered
reference (Scalar) at `GET /api/docs` — both public.
The feed and email endpoints require authentication, using either:
- **Bearer token**: `Authorization: Bearer <ADMIN_PASSWORD>`, or
- **Reverse-proxy auth**: the same trusted-IP + `X-Auth-Proxy-Secret` + `Remote-User`
headers as the admin UI (see [INSTALL.md](INSTALL.md)).
`GET /api/v1/stats`, the OpenAPI spec, and the docs page are public.
| Method | Path | Auth | Purpose |
| -------- | ------------------------------------ | ------ | ------------------------ |
| `GET` | `/api/v1/feeds` | yes | List feeds |
| `POST` | `/api/v1/feeds` | yes | Create a feed |
| `GET` | `/api/v1/feeds/{feedId}` | yes | Get a feed |
| `PATCH` | `/api/v1/feeds/{feedId}` | yes | Update a feed |
| `DELETE` | `/api/v1/feeds/{feedId}` | yes | Delete a feed |
| `GET` | `/api/v1/feeds/{feedId}/emails` | yes | List a feed's emails |
| `GET` | `/api/v1/feeds/{feedId}/emails/{id}` | yes | Get a single email |
| `DELETE` | `/api/v1/feeds/{feedId}/emails/{id}` | yes | Delete a single email |
| `GET` | `/api/v1/stats` | public | Read monitoring counters |
The email `{id}` is the email's `receivedAt` timestamp (as returned by the list endpoint).
```bash
# Create a feed
curl -X POST https://yourdomain.com/api/v1/feeds \
-H "Authorization: Bearer $ADMIN_PASSWORD" \
-H 'Content-Type: application/json' \
-d '{"title":"Daily Digest","allowedSenders":["[email protected]"]}'
```
## Installation
See **[INSTALL.md](INSTALL.md)** for the full setup, deployment, and configuration guide. Quick start:
1. Clone this repository.
2. Authenticate Wrangler:
```bash
npx wrangler login
```
3. Run setup:
```bash
bash setup.sh
bash setup.sh # prompts for admin password + domain, provisions KV, generates wrangler.toml
npm run deploy # deploys the Worker and registers your custom domain
```
`setup.sh` will:
- install npm dependencies
- verify Cloudflare auth (`wrangler whoami`)
- create KV namespaces (`EMAIL_STORAGE` + preview)
- set the `ADMIN_PASSWORD` secret in `production`
- generate `wrangler.toml` from `wrangler-example.toml`
- stamp `compatibility_date` to the current date
4. Configure ForwardEmail DNS records in Cloudflare:
| Type | Name | Content | Notes |
| ---- | ---- | ---------------------------------------------------- | ----------------------- |
| MX | @ | `mx1.forwardemail.net` | Priority `10`, DNS only |
| MX | @ | `mx2.forwardemail.net` | Priority `10`, DNS only |
| TXT | @ | `"forward-email=https://yourdomain.com/api/inbound"` | webhook target |
| TXT | @ | `"v=spf1 include:spf.forwardemail.net -all"` | SPF |
5. Deploy:
```bash
npm run deploy
```
6. Open `https://yourdomain.com/admin` and sign in.
## Development
```bash
npm install
npm run dev
npm test
npm run build
```
## Configuration notes
- `wrangler-example.toml` is the template; `wrangler.toml` is generated locally.
- Keep `compatibility_date` fresh when doing runtime upgrades.
- `ADMIN_PASSWORD` is a Cloudflare Worker secret, not a plain env var in config.
Then enable email ingestion (Cloudflare Email Workers or ForwardEmail) and open `https://yourdomain.com/admin`. Details, options, and configuration knobs (feed size limit, R2 attachments, reverse-proxy auth, CI deploys) are all in [INSTALL.md](INSTALL.md).
## Security notes
- Inbound webhook access is IP-restricted to ForwardEmail MX sources.
- When using Option B (ForwardEmail), inbound webhook access is IP-restricted to ForwardEmail MX sources.
- Admin auth uses a signed, `HttpOnly`, `Secure`, `SameSite=Strict` cookie.
- Admin responses are `no-store` to avoid cache leakage.
- Feed, entry, and attachment responses send `X-Robots-Tag: noindex`, and `/robots.txt` disallows `/rss`, `/atom`, `/entries`, `/files`, and `/admin`, so private feeds and emails are kept out of search engines.
- For high-value feeds, set `Allowed senders` so only known sender addresses/domains are accepted.
- You should use a strong admin password and rotate periodically.
- All secret comparisons (admin password, proxy secret) use constant-time comparison to prevent timing attacks.
## Upgrading dependencies
## Acknowledgements
To refresh dependencies to latest:
```bash
npm outdated
npm install
npm test
npm run build
```
Then update `compatibility_date` and redeploy.
- [kill-the-newsletter](https://github.com/leafac/kill-the-newsletter) by Leandro Facchinetti — the inspiration for this project and the reference implementation for feature ideas (Atom feeds, attachment enclosures, entry HTML views, and more).
- [Email-to-RSS](https://github.com/yl8976/Email-to-RSS) by yl8976 — the initial codebase this project is based on.
## License
+61
View File
@@ -0,0 +1,61 @@
# Security Policy
## Supported versions
kill-the-news is a self-hosted, single-Worker application. Only the latest
release on the `main` branch receives security fixes. If you run a fork or an
older deployment, update to the latest `main` before reporting an issue.
## Reporting a vulnerability
**Please do not open a public GitHub issue for security problems.**
Report privately through one of:
- [GitHub Security Advisories](https://github.com/juherr/kill-the-news/security/advisories/new) (preferred)
- Email: me@juherr.dev
Please include:
- A description of the issue and its impact
- Steps to reproduce (proof-of-concept if possible)
- Affected route, file, or configuration
- Any suggested remediation
You can expect an acknowledgement within a few days. Since this is a
volunteer-maintained project, fix timelines depend on severity and
availability, but credible reports are taken seriously. Coordinated disclosure
is appreciated — please give a reasonable window for a fix before going public.
## Scope
Because this Worker ingests email and exposes feeds, the security-sensitive
surface includes:
- **Admin authentication** — password handling, the signed session cookie, and
constant-time secret comparison (`ADMIN_PASSWORD`, `PROXY_AUTH_SECRET`).
- **Inbound ingestion** — the ForwardEmail webhook (`POST /api/inbound`),
IP allowlisting, and the Cloudflare Email Workers handler.
- **Email rendering** — HTML sanitization for stored emails and feed output
(XSS via `entries/`, `rss/`, `atom/`, inline `cid:` images).
- **Public endpoints** — `GET /`, `GET /api/stats`, `GET /rss/:feedId`,
`GET /atom/:feedId`, `GET /files/:attachmentId/:filename`,
`GET /favicon/:feedId`.
### Out of scope
- Vulnerabilities in Cloudflare, ForwardEmail, or other third-party
infrastructure (report those to the respective vendor).
- Misconfiguration of a self-hosted deployment (e.g. a weak `ADMIN_PASSWORD`,
an exposed `workers.dev` subdomain, or committing secrets). See the security
notes in [README.md](README.md) and [INSTALL.md](INSTALL.md).
- Denial of service from sending large volumes of email.
## Hardening reminders for operators
- Use a strong, unique `ADMIN_PASSWORD` and rotate it periodically.
- Set `ADMIN_PASSWORD` via `wrangler secret put` — never in config files.
- Disable the `workers.dev` subdomain in production (`workers_dev = false`),
since `CF-Connecting-IP` can be spoofed on direct `workers.dev` requests.
- Set per-feed `Allowed senders` for high-value feeds.
- Never commit `wrangler.toml` or `.dev.vars` (both are gitignored).
+179
View File
@@ -0,0 +1,179 @@
# TODO
Feature gaps identified by comparing with [kill-the-newsletter](https://github.com/leafac/kill-the-newsletter).
> **Origin tags.** Every idea carries an `_origin:_` reference so we can notify the source when it ships.
>
> - `ktn#N` → a [kill-the-newsletter issue/PR](https://github.com/leafac/kill-the-newsletter/issues) — **comment there when implemented** to close the loop with the requester.
> - A tool/spec URL → external inspiration (a competitor or standard); no individual to notify, but the rationale is traceable.
> - `internal` → our own design/code audit; no external requester.
> **Priority × size.** Each idea is tagged `Pn·Size`. Priority by user value: **P1** (high) / **P2** (medium) / **P3** (nice-to-have). Effort by implementation size: **S** (hours) / **M** (~1–2 days) / **L** (several days) / **XL** (week+). Done items keep the tag as a retrospective estimate.
## Quick wins
- [x] `P1·S` **Author field in RSS entries** — expose the `from` address as `<author>` in each RSS `<item>`. The value is already stored in KV, just not rendered in the feed XML. — _origin: [ktn#102](https://github.com/leafac/kill-the-newsletter/issues/102) (ktn CHANGELOG 2.0.6 "author to entry")_
- [x] `P1·M` **HTML view for individual entries** — serve each email as an HTML page at e.g. `/entries/:feedId/:timestamp`. Useful for reading emails outside a feed reader and for debugging. kill-the-newsletter serves these at `/feeds/{feedId}/entries/{entryId}.html` with a Content-Security-Policy header. — _origin: upstream alternate-HTML view; gives each item a valid URL ([ktn#17](https://github.com/leafac/kill-the-newsletter/issues/17), [ktn#40](https://github.com/leafac/kill-the-newsletter/issues/40))_
- [x] `P2·S` **JSON API for feed creation** — accept `Content-Type: application/json` on `POST /admin/feeds` and return `{ feedId, email, feedUrl }`. Useful for automation (e.g. Tofu/OpenTofu provisioning). — _origin: [ktn#43](https://github.com/leafac/kill-the-newsletter/issues/43) (ktn CHANGELOG 2.0.5)_
- [x] `P2·S` **Project favicon** — serve a single bundled icon at `/favicon.ico` and add a `<link rel="icon">` in the shared `Layout` so the admin UI, status page, and entry views stop 404-ing. Doubles as the default/fallback icon for the per-feed favicon feature below. — _origin: internal (404 fix); related [ktn#131](https://github.com/leafac/kill-the-newsletter/issues/131)_
## Medium effort
- [x] `P2·M` **Size-based feed trimming** — instead of a fixed 50-entry cap, drop the oldest entries when the feed exceeds a size threshold (kill-the-newsletter uses ~512 KB). More robust for HTML-heavy newsletters where one entry can dominate. — _origin: upstream size limit (ktn CHANGELOG 2.0.8); related [ktn#59](https://github.com/leafac/kill-the-newsletter/issues/59), [ktn#115](https://github.com/leafac/kill-the-newsletter/issues/115)_
- [x] `P1·M` **Atom feed format** — expose feeds as Atom (`application/atom+xml`) in addition to or instead of RSS 2.0. Atom has better native support for HTML content and author metadata. — _origin: upstream (Atom-native product) / internal parity_
- [x] `P3·M` **Authelia / external auth provider support** — allow delegating admin authentication to an external identity provider (e.g. Authelia, Authentik) via a trusted header (`Remote-User`, `X-Forwarded-User`) set by a reverse proxy. The Worker would accept the header as proof of authentication instead of checking the cookie, with a configurable secret or IP allowlist to trust only the proxy. — _origin: internal_
- [x] `P2·M` **Per-feed favicon from the last sender's domain** — give each feed an icon by fetching the favicon of the last sender's domain, so feeds are visually distinguishable in readers and the admin UI. Resolve the domain from the most recent email's `from`, fetch its favicon (e.g. `https://<domain>/favicon.ico` or a parsed `<link rel="icon">`, with a fallback service), and cache the result aggressively (KV/R2 + Cache API with a long TTL) so it isn't re-fetched on every request. Expose it via the RSS `<image>` / Atom `<icon>` and the admin feed list. — _origin: [ktn#92](https://github.com/leafac/kill-the-newsletter/issues/92) (ktn CHANGELOG 2.0.6/2.0.7)_
- [x] `P2·M` **RFC 8058 one-click unsubscribe on feed deletion** — when a feed is deleted, automatically unsubscribe from the newsletters that fed it so messages stop arriving at the now-dead address. Parse and store the `List-Unsubscribe` / `List-Unsubscribe-Post` headers ([RFC 8058](https://www.rfc-editor.org/rfc/rfc8058.txt)) from incoming emails, then on deletion POST `List-Unsubscribe=One-Click` to each stored unsubscribe URL. Requires capturing the headers during ingestion (`src/lib/email-processor.ts`) and firing the outbound requests from the feed-delete paths (`src/routes/admin/feeds.tsx`), ideally via `ctx.waitUntil`. — _origin: internal ([RFC 8058](https://www.rfc-editor.org/rfc/rfc8058.txt))_
## Heavy
- [x] `P1·L` **Email attachments as RSS enclosures** — store attachments in Cloudflare R2 and expose them as `<enclosure>` elements in the feed. kill-the-newsletter serves them at `/files/{enclosureId}/{filename}`. — _origin: [ktn#66](https://github.com/leafac/kill-the-newsletter/issues/66), [ktn#86](https://github.com/leafac/kill-the-newsletter/issues/86) (ktn CHANGELOG 2.0.5)_
- [x] `P2·L` **WebSub (PubSubHubbub) push notifications** — notify subscribers in real time when a new email arrives, instead of requiring them to poll the feed. Requires either integrating a public WebSub hub or implementing the hub protocol directly. — _origin: [ktn#68](https://github.com/leafac/kill-the-newsletter/issues/68) (ktn CHANGELOG 2.0.4)_
- [x] `P2·S` **Rate limiting via Cloudflare WAF rules** — protect `/api/inbound` and `/admin` against abuse. Configure WAF custom rules in the Cloudflare dashboard (or via Terraform): rate-limit `/api/inbound` to ~60 req/min per IP, and `/admin` to ~20 req/min per IP. No code changes required; this is pure infrastructure configuration. — _origin: upstream parity (ktn CHANGELOG 2.0.3) / internal_
- [x] `P2·L` **REST API with OpenAPI description** — expose a documented, machine-consumable REST API for feed/email management (create/list/update/delete feeds, list/read/delete emails, read stats) so the service can be automated without scraping the admin UI. Implemented as a versioned `/api/v1/*` surface (Bearer-token auth with the admin password, plus the existing proxy-auth) built on `@hono/zod-openapi`; the OpenAPI 3.1 spec is served at `/api/openapi.json` with a Scalar docs page at `/api/docs`. Feed create/update/delete logic was extracted into `src/lib/feed-service.ts` so the admin UI and the REST API share a single source of truth. — _origin: [ktn#43](https://github.com/leafac/kill-the-newsletter/issues/43)_
- [ ] `P3·XL` **Migrate feed metadata to Durable Objects for atomic writes** — the current KV-based metadata store has a read-modify-write race condition: two concurrent emails to the same feed can silently overwrite each other's changes. Cloudflare Durable Objects serialise access per feed and eliminate the race entirely. Requires replacing `feed:<feedId>:metadata` KV writes in `src/lib/email-processor.ts` with a Durable Object that exposes an `appendEmail()` RPC, updating `wrangler.toml` with a DO binding, and migrating existing metadata at deploy time. — _origin: internal; same race behind [ktn#6](https://github.com/leafac/kill-the-newsletter/issues/6), [ktn#31](https://github.com/leafac/kill-the-newsletter/issues/31)_
## From upstream issues/PRs (2026-05-24 review)
Gaps found by reading every open/closed issue + PR on [kill-the-newsletter](https://github.com/leafac/kill-the-newsletter/issues). These are requests we do **not** yet satisfy (many other recurring requests — dark mode, copy buttons, favicon, expiration, attachments, API, WebSub, sender-in-author — we already cover).
- [ ] `P1·M` **Subscription confirmation handling** — _the single most recurring upstream request_ ([#5](https://github.com/leafac/kill-the-newsletter/issues/5), [#23](https://github.com/leafac/kill-the-newsletter/issues/23), [#57](https://github.com/leafac/kill-the-newsletter/issues/57), [#73](https://github.com/leafac/kill-the-newsletter/issues/73), [#89](https://github.com/leafac/kill-the-newsletter/issues/89), [#95](https://github.com/leafac/kill-the-newsletter/issues/95), [#97](https://github.com/leafac/kill-the-newsletter/issues/97)). Newsletters require a "click to confirm your email" step; users can't easily find/click the link buried in a feed reader. Our admin already lists emails, but nothing **surfaces** the confirmation link or shows the first email inline right after feed creation. Low effort, high payoff (admin UX in `src/routes/admin/feeds.tsx` + maybe extract candidate confirm links during ingestion in `src/application/email-processor.ts`).
- [ ] `P1·M` **Separate write (email) / read (feed) IDs** — _most-requested privacy gap, still open upstream_ ([#114](https://github.com/leafac/kill-the-newsletter/issues/114), [#93](https://github.com/leafac/kill-the-newsletter/issues/93), [#75](https://github.com/leafac/kill-the-newsletter/issues/75)). Today `feedEmailAddress = <feedId>@domain` and `/rss/<feedId>` reuse the **same** id (`src/infrastructure/urls.ts`), so anyone with the inbound address can read the feed (and vice-versa) — you can't share a feed without leaking its subscribe address. Add a distinct read id alongside the write id: touch `FeedState`, id generation (`FeedId`), `urls.ts`, the `inbound` parse, and the feed-list/registry. Medium effort.
- [ ] `P2·M` **Proxy/prefetch remote images** ([#69](https://github.com/leafac/kill-the-newsletter/issues/69)). We already proxy inline `cid:` images via R2, but remote `<img src="https://…">` stay remote → tracking pixels fire on read. Extend `src/infrastructure/html-processor.ts` to rewrite remote image src through a worker proxy/cache endpoint (reuse the R2 + Cache API pattern from favicons).
- [ ] `P3·M` **Tracking-link redirect resolver** ([#36](https://github.com/leafac/kill-the-newsletter/issues/36)). Unwrap marketing/tracking URLs (e.g. `click.convertkit-mail…`) to their final destination so the redirect/tracking happens server-side (or is stripped) instead of from the reader. Lives in `src/infrastructure/html-processor.ts`. Mind SSRF/abuse surface when following redirects.
- [ ] `P2·S` **Strip-styles / plaintext rendering option** ([#74](https://github.com/leafac/kill-the-newsletter/issues/74), [#119](https://github.com/leafac/kill-the-newsletter/issues/119)). Some readers render newsletter HTML/CSS poorly. Offer an opt-in to strip `<style>` + inline styles (keeping links), or to prefer the `text/plain` part. Per-feed setting + `src/infrastructure/html-processor.ts`.
- [ ] `P2·S` **Optional sender in entry title** ([#123 — open PR upstream](https://github.com/leafac/kill-the-newsletter/pull/123), [#124](https://github.com/leafac/kill-the-newsletter/issues/124)). We already emit `<author>`, but some users want `[Sender] Subject` as the entry title for at-a-glance scanning in the reader. Per-feed toggle + `src/infrastructure/feed-generator.ts`.
- [ ] `P2·S` **Detect a newsletter's native Atom/RSS feed** — _top item on upstream's own [TODO](https://github.com/leafac/kill-the-newsletter/blob/main/TODO.md), not yet built there_. When an incoming email's HTML contains `<link rel="alternate" type="application/atom+xml">` (or `application/rss+xml`), surface it: "this newsletter already publishes a feed — subscribe to it directly instead." We already parse HTML with linkedom in `src/infrastructure/html-processor.ts`, so detection is cheap; store the discovered URL on the feed and show it in the admin UI / a feed entry. A genuine differentiator — we'd ship it before upstream.
- [x] `P1·S` **`X-Robots-Tag: none` on feed + entry routes** ([#33](https://github.com/leafac/kill-the-newsletter/issues/33)). Private feeds/emails should never be search-indexed. Upstream sets `X-Robots-Tag: none` on its responses; we set a CSP on `/entries` but **no** robots header anywhere. Add `X-Robots-Tag: noindex` to `rss.ts`, `atom.ts`, `entries.ts`, `files.ts` (and optionally a `/robots.txt`). Low effort, real privacy gap.
## From similar projects & RSS readers (2026-05-24 review)
Ideas from competitors (Feedbin, Readwise Reader, Inoreader, Omnivore, LetterFeed, Mailbrew, mail2rss) and from what leading readers (NetNewsWire, Reeder, Feedly, Inoreader, NewsBlur, Miniflux, FreshRSS) can consume. Deduplicated against the upstream-issues section above. Tagged **[table-stakes]** vs **[differentiating]**.
### Feed-output enrichments (small XML wins — we use the `feed` lib, which already emits `content:encoded`, `atom:link rel="self"`, stable `<guid>`)
- [ ] `P2·S` **JSON Feed 1.1 endpoint** `GET /json/:feedId` **[differentiating, cheap]** — the `feed` lib already supports `.json1()`; we only expose `.rss2()`/`.atom1()` (`src/infrastructure/feed-generator.ts`). Natively consumed by NetNewsWire, Reeder, NewsBlur, Feedly. ~1 route + 1 generator fn. — _origin: [JSON Feed 1.1 spec](https://www.jsonfeed.org/version/1.1/) (reader ecosystem)_
- [ ] `P2·M` **Per-item `<category>` + per-feed tags/categories** **[differentiating]** — we set no categories today. Tag entries by sender (or a user-set feed category) so readers (Inoreader, Feedly, NewsBlur) can filter/mute subsets. Pairs with the filtering item below; touches `FeedState`, `feed-generator.ts`. — _origin: [RSS best practices (kevincox)](https://kevincox.ca/2022/05/06/rss-feed-best-practices/); Inoreader/Feedly filtering_
- [ ] `P3·S` **Reader cadence hints: `<ttl>` + `sy:updatePeriod`/`sy:updateFrequency`** **[table-stakes, niche]** — advertise the feed's real update rhythm so pollers (FreshRSS, Miniflux, Inoreader) back off; complements our WebSub push. Support is uneven, so keep it as a hint alongside WebSub. Also advertise the WebSub hub link _inside_ the XML (`<atom:link rel="hub">`), not only the HTTP `Link` header. — _origin: [FreshRSS TTL #6721](https://github.com/FreshRSS/FreshRSS/issues/6721)_
- [ ] `P2·M` **Media RSS lead image (`<media:content>`/`<media:thumbnail>`)** **[differentiating]** — extract the first image of each email as a thumbnail so card/story layouts (Feedly, Inoreader, NewsBlur) show a preview. The `feed` lib doesn't emit Media RSS, so this needs post-processing or a custom serializer. — _origin: [Media RSS spec](https://www.rssboard.org/media-rss); Feedly/Inoreader consume it_
### Ingestion & processing
- [ ] `P2·M` **Keyword/subject filtering rules (keep/drop)** **[differentiating]** — we already have _sender_ allow/block (`SenderPolicy`), but no content rules. Add per-feed keep/drop rules by subject or body keyword (Inoreader/Omnivore-style), applied in `src/application/email-processor.ts` at the same gate as the sender policy. — _origin: [Inoreader rules](https://www.inoreader.com/blog/2020/02/declutter-your-inbox-subscribe-to-email-newsletters-straight-into-inoreader.html); Omnivore filters_
- [ ] `P2·M` **Confirmation-code relay** **[differentiating]** — _extends the "Subscription confirmation handling" item above_. Readwise Reader auto-detects "reply with code X" / "click to confirm" emails and surfaces (or relays) the code. Beyond just showing the link: detect the confirm pattern and present a one-tap action in admin. — _origin: [Readwise Reader docs](https://docs.readwise.io/reader/docs/faqs/email-newsletters); also [ktn#89](https://github.com/leafac/kill-the-newsletter/issues/89) (reply-to-confirm)_
- [ ] `P3·XL` **IMAP-pull ingestion option** **[differentiating for self-hosters]** — alternative to the ForwardEmail/Cloudflare-Email webhook: poll an existing IMAP mailbox and route allow-listed senders to feeds (LetterFeed model). Big lift on a Worker (needs a scheduled fetch + IMAP over a TCP socket / external relay); evaluate feasibility before committing. — _origin: [LetterFeed](https://github.com/LeonMusCoden/LetterFeed); also [ktn#26](https://github.com/leafac/kill-the-newsletter/issues/26) (use IMAP instead of hosting a mail server)_
### Reading experience
- [ ] `P2·S` **OPML export** `GET /opml` **[table-stakes, easy]** — export all feeds as an OPML outline so users can bulk-import every feed into their reader in one shot. Every reader imports OPML; strong onboarding/migration win. Pure read over the feed registry. — _origin: reader ecosystem ([NetNewsWire](https://github.com/Ranchero-Software/NetNewsWire/)); Feedbin OPML export_
- [ ] `P2·L` **Full-text search across received emails** **[differentiating]** — admin-side search over subjects + bodies (Omnivore/Feedbin have this). On KV this means an index or scan; consider scope (subject-only first) before building. — _origin: [Omnivore](https://www.timeatlas.com/omnivore-newsletters/); Feedbin search_
- [ ] `P3·L` **Readability / clean-text view toggle** **[differentiating]** — _related to "strip-styles" above but distinct_: run a readability extraction (article body only) as an opt-in per feed, remembered per sender (Readwise pattern), rather than just stripping CSS. — _origin: [Readwise Reader feed docs](https://docs.readwise.io/reader/docs/faqs/feed)_
### Greenfield differentiators
- [ ] `P2·L` **AI per-newsletter summarization** **[differentiating]** — generate a short TL;DR per email (or a daily digest summary) using Cloudflare Workers AI (no new vendor, no key to manage). Almost no competitor ships this well. Add an `AI` binding + an opt-in per-feed flag; render the summary atop the entry content. — _origin: [Precis](https://github.com/leozqin/precis), [babarot AI reader](https://dev.to/babarot/i-built-a-self-hosted-rss-reader-with-ai-summarization-translation-and-an-mcp-server-316c)_
- [ ] `P3·L` **Digest / bundling mode** **[differentiating]** — for low-volume feeds, batch N emails into a single periodic digest entry (Mailbrew model) so readers aren't flooded. Per-feed cadence setting; runs on the existing cron. — _origin: [Mailbrew](https://www.readless.app/blog/mailbrew-pricing-2026)_
## Robustness, delivery, auth & integrations (2026-05-24 deep dig)
Verified-missing in our code, deduplicated against the sections above. From a code audit + a sweep of niche/recent tools (Precis, changedetection.io+Apprise, MailCast email-to-podcast, FreshRSS/Miniflux token auth, RFC 5005, postly dedup).
### Delivery / bandwidth
- [ ] `P2·S` **Conditional GET on feeds (ETag + Last-Modified + 304)** **[table-stakes, easy]** — `rss.ts`/`atom.ts` only send `Cache-Control: max-age=1800`; no validators. Emit a strong `ETag` (hash of the latest entry id + count) and `Last-Modified` (newest `receivedAt`), and return `304 Not Modified` on `If-None-Match`/`If-Modified-Since`. Cuts bandwidth for every polling reader. Generate the ETag _before_ compression. — _origin: internal code audit ([RFC 9110 conditional requests](https://www.rfc-editor.org/rfc/rfc9110#name-conditional-requests))_
- [ ] `P3·L` **RFC 5005 paged / archived feeds** **[differentiating, niche]** — readers only ever see the capped current window; older entries vanish. Mark the subscription document `fh:complete` and expose `prev-archive` pages so readers can backfill history. Pairs naturally with our expiring-feed model (an expired feed = a sealed archive). ([RFC 5005](https://www.rfc-editor.org/rfc/rfc5005.html))
### Ingestion robustness
- [ ] `P1·M` **Duplicate-send dedup** **[differentiating]** — the same newsletter resent (or delivered twice) creates two entries today (key = `receivedAt`). Dedup by `Message-ID` first, then a SHA-256 of normalized subject+body within a short window, in `src/application/email-processor.ts`. Fixes the upstream "duplicate posts" complaint ([#31](https://github.com/leafac/kill-the-newsletter/issues/31), [#6](https://github.com/leafac/kill-the-newsletter/issues/6)).
- [ ] `P3·M` **Calendar (.ics) invite extraction** **[differentiating, novel]** — no email→feed tool does this. Detect `text/calendar` parts, parse the event, and surface it in the entry (summary + an `.ics` enclosure / add-to-calendar link). Useful for event/booking newsletters. — _origin: internal (novel; no external requester)_
- [x] `P2·S` **`FALLBACK_FORWARD_ADDRESS` — catch-all fallback forwarding** **[differentiating for self-hosters]** — today `handleCloudflareEmail` silently drops (just `logger.warn`) any address that isn't a feed, so you can't point a domain's _catch-all_ at KTN without swallowing your personal mail. Add an optional `FALLBACK_FORWARD_ADDRESS` env var: after `processEmail`, forward non-feed mail to it based on `result.reason` — **forward** on `invalid_address` (not a `noun.noun.NN` address) and `feed_not_found` (well-formed but no such feed); **drop** on `feed_expired` and `sender_blocked` (don't leak a newsletter to the fallback box); nothing on `ok`. Unset env → current drop+log behavior unchanged. The destination must be a _verified_ Cloudflare Email Routing address or `message.forward()` fails; `await` it in a `try/catch` (`logger.warn` on failure), forward at most once. Touch: `Env` (`src/types/index.ts`), `src/infrastructure/cloudflare-email.ts` (`result.reason` already available), `cloudflare-email.test.ts` (forwarded for `feed_not_found`/`invalid_address` when set; not for `feed_expired`/`sender_blocked`; not when unset), `wrangler-example.toml` (commented `# FALLBACK_FORWARD_ADDRESS` under `[vars]`), `INSTALL.md` ("Catch-all fallback forwarding" section: verified-destination prerequisite + use case). — _origin: internal (juherr — self-host on juherr.dev catch-all); generic "use KTN as my domain's catch-all"_
### Auth & privacy
- [ ] `P2·M` **Scoped / multiple API tokens** **[security]** — the REST API currently accepts the single `ADMIN_PASSWORD` as the bearer (`src/infrastructure/auth.ts`). Add named, independently-revocable tokens (optionally read-only or feed-scoped) so automation doesn't hold the master password. — _origin: internal security audit_
- [ ] `P2·M` **Token-protected private feeds** **[security, differentiating]** — `/rss` and `/atom` are public-by-obscurity (anyone with the URL reads it). Offer an opt-in `?token=…` (FreshRSS-style) or HMAC-signed, optionally expiring URL (fits our expiring-feed model) so a feed can be truly private and shareable without leaking the inbound address. Complements the _separate write/read IDs_ item above. ([FreshRSS](https://freshrss.github.io/FreshRSS/en/admins/09_AccessControl.html))
### Push & integrations
- [ ] `P2·L` **Push new items to chat (per-feed)** **[differentiating]** — for users who don't run a reader, push each new email to Telegram / Discord / ntfy / a generic webhook, routed per feed, instant-vs-digest toggle (Precis / changedetection.io+Apprise pattern). Fires from the existing event dispatcher (`src/application/feed-events.ts`) via `ctx.waitUntil`. ([Precis](https://github.com/leozqin/precis))
### Novel / stretch (Cloudflare-native)
- [ ] `P3·M` **MCP server over your feeds** **[differentiating, novel]** — expose feeds/emails to AI agents via a Model Context Protocol endpoint on the Worker, so an assistant can read/search a user's newsletters. Cheap to add on a Worker, genuinely new in this space. — _origin: [babarot AI reader + MCP](https://dev.to/babarot/i-built-a-self-hosted-rss-reader-with-ai-summarization-translation-and-an-mcp-server-316c)_
- [ ] `P3·L` **Email-to-podcast (TTS audio enclosure)** **[differentiating, novel]** — opt-in: synthesize each newsletter to audio (Cloudflare Workers AI TTS), store in R2, attach as an `<enclosure>` so the feed doubles as a private podcast. Reframes feed item = audio. ([prior art](https://github.com/tcanfarotta/email-to-podcast-rss))
> Framing notes (no code, worth surfacing in docs/landing): we already deliver several things competitors charge for — **full-body capture bypasses Substack/"read more" truncation** (we ingest the email, not the scraped page), and each feed's inbound address is effectively a **burnable alias** (delete the feed + RFC 8058 one-click unsubscribe already kills the sender). Market these explicitly.
## Feed namespaces & reader-rendering correctness (2026-05-24 deep dig)
Two final angles: (1) less-common RSS/Atom namespaces that visibly improve feeds in real readers, and (2) generator-side correctness fixes that stop feeds breaking in self-hosted readers. The `feed` lib emits `content:encoded`/`atom:link rel=self`/stable `<guid>` but does **not** handle the items below — they need its custom-namespace/extension hooks or a post-process pass.
### Namespaces worth emitting
- [ ] `P2·S` **WebFeeds branding (`webfeeds:accentColor`, `webfeeds:icon`, `webfeeds:logo`, `webfeeds:cover`)** **[differentiating, high visible payoff]** — Feedly puts your SVG logo on every story and recolors links to your accent color. We already derive a per-feed favicon; add an accent + logo for branded-looking feeds. — _origin: [Working With Web Feeds (CSS-Tricks)](https://css-tricks.com/working-with-web-feeds-its-more-than-rss/)_
- [ ] `P2·M` **Media RSS thumbnail/credit (`media:thumbnail`, `media:description`, `media:credit`)** **[differentiating]** — richer than the lead-image item above: gives readers a card image, alt text, and attribution. — _origin: [Media RSS spec](https://www.rssboard.org/media-rss)_
- [ ] `P3·S` **Dublin Core `dc:creator`** **[niche, cheap]** — credits the newsletter sender **without** an email address (RSS `<author>` requires one); safer than a synthetic `noreply@`. — _origin: [RSS Best Practices Profile](https://www.rssboard.org/rss-profile), [mod_dublincore](https://www.oreilly.com/library/view/developing-feeds-with/0596008813/re08.html)_
- [ ] `P3·M` **Podcast namespace (`itunes:*` + `podcast:transcript`/`chapters`)** **[stretch]** — only if the email-to-podcast item ships; turns the audio feed into a real Podcasting 2.0 feed. — _origin: [Podcast Namespace](https://podcasting2.org/docs/podcast-namespace)_
### Reader-rendering correctness (turn these into hardening tasks)
- [x] `P1·S` **Rewrite relative URLs in content to absolute** **[correctness]** — most readers ignore `xml:base`; relative `src`/`href` in `content:encoded` break in Miniflux/NetNewsWire. Absolutize every link/image before emitting (`src/infrastructure/html-processor.ts`). — _origin: [W3C ContainsRelRef](https://validator.w3.org/feed/docs/warning/ContainsRelRef.html)_
- [x] `P1·S` **Promote lazy-loaded images (`data-src` → `src`, strip `loading="lazy"`)** **[correctness]** — newsletters with lazy images render blank in readers. — _origin: [Hugo RSS & lazy images](https://brainbaking.com/post/2021/01/hugo-rss-feeds-and-lazy-image-loading/)_
- [x] `P1·S` **Strip XML-illegal control chars + guarantee valid UTF-8** **[correctness]** — a single bad codepoint fails the _whole_ feed parse in strict readers (newsboat). Sanitize before serialization. — _origin: [newsboat #2328](https://github.com/newsboat/newsboat/issues/2328), [W3C SAXError](https://validator.w3.org/feed/docs/error/SAXError.html); upstream hit this too ([ktn#1](https://github.com/leafac/kill-the-newsletter/issues/1) cyrillic, [ktn#9](https://github.com/leafac/kill-the-newsletter/issues/9) invalid XML char)_
- [ ] `P2·S` **Real `enclosure` byte length + correct type (never `length="0"`)** **[correctness]** — zero/missing length makes podcast clients reject the enclosure; use the actual R2 object size. — _origin: [AzuraCast #7809](https://github.com/AzuraCast/AzuraCast/issues/7809)_
- [x] `P1·S` **Plain-text `<title>` (strip HTML, decode entities)** **[correctness]** — raw tags in titles show literally in readers; keep markup only in `content`. — _origin: [RSS.app feed output guide](https://help.rss.app/en/articles/10769849-guide-to-feed-output); upstream [ktn#11](https://github.com/leafac/kill-the-newsletter/issues/11) (subject placed as link)_
## Per-feed favicon — design notes
Breakdown of the _"Per-feed favicon from the last sender's domain"_ item above (the parent is `P2·M`; these sub-tasks are each ~`S`). Goal: each feed shows an icon derived from its newsletter source, fetched once and cached so it never re-fetches on a normal request.
- [x] `P2·S` **Resolve the sender domain** — on ingestion, extract the domain from the latest email's `from` address (`extractEmailDomain` in `src/utils/favicon-fetcher.ts`) and persist it as `iconDomain` on the feed metadata so the icon tracks the most recent sender.
- [x] `P2·S` **Fetch the favicon** — resolve an icon URL for the domain: try `https://<domain>/favicon.ico`, then fall back to `https://icons.duckduckgo.com/ip3/<domain>.ico`. Runs async via `ctx.waitUntil` so it never blocks email processing.
- [x] `P2·S` **Cache aggressively** — store the fetched bytes (base64) keyed by domain in KV with a 1-week TTL (`ICON_TTL_SECONDS`). The domain is the cache key so feeds from the same sender share one fetch; the fetch only fires when the cache entry is absent/expired.
- [x] `P2·S` **Serve endpoint** — `GET /favicon/:feedId` returns the cached bytes with the correct `Content-Type` and a long `Cache-Control`, falling back to the project favicon when no domain icon is found.
- [x] `P2·S` **Expose in outputs** — the icon is referenced from the RSS `<image>` and Atom `<icon>`/`<logo>` in `src/utils/feed-generator.ts`, and rendered next to each feed in the admin list/table (`src/routes/admin.tsx`).
- [x] `P2·S` **Failure handling** — missing/blocked favicons degrade gracefully to the project favicon fallback (negative cache entry); icon fetch errors never surface to ingestion or feed rendering.
+1
View File
@@ -0,0 +1 @@
kill-the.news
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 4.2 KiB

+7
View File
@@ -0,0 +1,7 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32" width="32" height="32">
<rect width="32" height="32" rx="7" fill="#f6821f"/>
<g fill="none" stroke="#ffffff" stroke-width="2.2" stroke-linecap="round" stroke-linejoin="round">
<path d="M7 9h18c1.1 0 2 .9 2 2v10c0 1.1-.9 2-2 2H7c-1.1 0-2-.9-2-2V11c0-1.1.9-2 2-2z"/>
<polyline points="27,11 16,18.5 5,11"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 391 B

+1308
View File
File diff suppressed because it is too large. Load diff
+22
View File
@@ -0,0 +1,22 @@
import prettier from "eslint-config-prettier";
import tseslint from "typescript-eslint";
export default tseslint.config(
{ ignores: ["dist/", "coverage/", "src/scripts/generated/"] },
...tseslint.configs.recommended,
prettier,
{
rules: {
"@typescript-eslint/no-unused-vars": [
"error",
{ argsIgnorePattern: "^_", varsIgnorePattern: "^_" },
],
},
},
{
files: ["src/**/*.test.ts", "src/test/**/*.ts"],
rules: {
"@typescript-eslint/no-explicit-any": "off",
},
},
);
+3168 -826
View File
File diff suppressed because it is too large. Load diff
+46 -16
View File
@@ -1,34 +1,64 @@
{
"name": "email-to-rss",
"version": "0.1.0",
"description": "A service that converts email newsletters to RSS feeds using Cloudflare Workers",
"name": "kill-the-news",
"version": "0.2.1",
"description": "Convert email newsletters into private RSS feeds using Cloudflare Workers",
"main": "dist/worker.js",
"scripts": {
"build": "wrangler deploy --dry-run --outdir=dist",
"build:client": "node scripts/build-client.mjs",
"build": "wrangler deploy --config wrangler.build.toml --dry-run --outdir=dist --no-experimental-autoconfig",
"prebuild": "npm run build:client",
"format": "prettier --write '**/*.{js,ts,css,json,md}'",
"format:check": "prettier --check '**/*.{js,ts,css,json,md}'",
"lint": "eslint src",
"dev": "wrangler dev",
"predev": "npm run build:client",
"deploy": "wrangler deploy --env production",
"test": "vitest run",
"test:watch": "vitest",
"test:coverage": "vitest run --coverage"
"test:coverage": "vitest run --coverage",
"typecheck": "tsc --noEmit && npm run typecheck:client",
"typecheck:client": "tsc -p src/scripts/client/tsconfig.json --noEmit",
"prepare": "husky && npm run build:client"
},
"lint-staged": {
"*.{ts,tsx,js,jsx}": [
"eslint --fix",
"prettier --write"
],
"*.{json,md,css}": [
"prettier --write"
]
},
"author": "",
"license": "MIT",
"devDependencies": {
"@cloudflare/workers-types": "^4.20260206.0",
"@cloudflare/workers-types": "4.20260522.1",
"@types/escape-html": "^1.0.4",
"@types/mailparser": "^3.4.6",
"@types/rss": "^0.0.32",
"@vitest/coverage-v8": "^4.0.18",
"happy-dom": "^20.5.0",
"msw": "^2.12.8",
"prettier": "^3.8.1",
"typescript": "^5.9.3",
"vitest": "^4.0.18",
"wrangler": "^4.63.0"
"@vitest/coverage-v8": "4.1.7",
"esbuild": "^0.28.0",
"eslint": "^10.0.0",
"eslint-config-prettier": "^10.1.8",
"happy-dom": "20.9.0",
"husky": "^9.1.7",
"lint-staged": "^17.0.5",
"msw": "2.14.6",
"prettier": "3.8.3",
"typescript": "6.0.3",
"typescript-eslint": "^8.59.4",
"vitest": "4.1.7",
"wrangler": "4.94.0"
},
"dependencies": {
"feed": "^5.2.0",
"hono": "^4.11.7",
"zod": "^4.3.6"
"@hono/zod-openapi": "^1.4.0",
"@hono/zod-validator": "^0.8.0",
"@scalar/hono-api-reference": "^0.10.19",
"escape-html": "^1.0.3",
"feed": "5.2.1",
"hono": "4.12.22",
"linkedom": "^0.18.12",
"postal-mime": "^2.7.4",
"zod": "4.4.3"
}
}
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env node
// Compiles client-side TypeScript files with esbuild and writes the result
// as TypeScript string-constant modules importable by the Cloudflare Worker.
import { build } from 'esbuild';
import { writeFileSync, mkdirSync } from 'fs';
import { fileURLToPath } from 'url';
import { dirname, join } from 'path';
const __dirname = dirname(fileURLToPath(import.meta.url));
const root = join(__dirname, '..');
const entries = [
{
input: 'src/scripts/client/dashboard.ts',
exportName: 'dashboardScript',
output: 'src/scripts/generated/dashboard.ts',
},
{
input: 'src/scripts/client/emails-page.ts',
exportName: 'emailsPageScript',
output: 'src/scripts/generated/emails-page.ts',
},
];
mkdirSync(join(root, 'src/scripts/generated'), { recursive: true });
for (const { input, exportName, output } of entries) {
const result = await build({
entryPoints: [join(root, input)],
bundle: true,
minify: true,
write: false,
target: 'es2020',
platform: 'browser',
format: 'iife',
});
const code = new TextDecoder().decode(result.outputFiles[0].contents);
const content = [
'// AUTO-GENERATED by scripts/build-client.mjs — do not edit directly.',
'// Source: ' + input,
'// Run `npm run build:client` to regenerate.',
'/* eslint-disable */',
'export const ' + exportName + ' = ' + JSON.stringify(code) + ';',
'',
].join('\n');
writeFileSync(join(root, output), content);
console.log('Built:', output);
}
+49 -3
View File
@@ -2,7 +2,7 @@
set -euo pipefail
echo "🚀 Setting up Email to RSS service..."
echo "🚀 Setting up kill-the-news..."
if ! command -v npm >/dev/null 2>&1 || ! command -v npx >/dev/null 2>&1 || ! command -v node >/dev/null 2>&1; then
echo "❌ Error: Node.js (with npm and npx) is required but not found."
@@ -17,7 +17,7 @@ fi
WORKER_NAME="$(grep -E '^name = "' wrangler-example.toml | head -1 | cut -d'"' -f2)"
if [ -z "$WORKER_NAME" ]; then
WORKER_NAME="email-to-rss"
WORKER_NAME="kill-the-news"
fi
echo "📦 Installing dependencies..."
@@ -134,6 +134,34 @@ if [ -z "$domain" ]; then
fi
echo "✅ Domain: $domain"
ENABLE_R2=false
R2_BUCKET=""
R2_PREVIEW_BUCKET=""
read -r -p "Enable email attachments stored in R2? [y/N]: " enable_r2
if [[ "$enable_r2" =~ ^[Yy]$ ]]; then
R2_BUCKET="${WORKER_NAME}-attachments"
R2_PREVIEW_BUCKET="${R2_BUCKET}-preview"
echo "🪣 Creating R2 buckets..."
set +e
R2_OUT="$(npx wrangler r2 bucket create "$R2_BUCKET" 2>&1)"
R2_STATUS=$?
R2_PREVIEW_OUT="$(npx wrangler r2 bucket create "$R2_PREVIEW_BUCKET" 2>&1)"
R2_PREVIEW_STATUS=$?
set -e
# An existing bucket is fine; only treat real failures as blocking.
echo "$R2_OUT" | grep -qi "already exists" && R2_STATUS=0
echo "$R2_PREVIEW_OUT" | grep -qi "already exists" && R2_PREVIEW_STATUS=0
if [ "$R2_STATUS" -eq 0 ] && [ "$R2_PREVIEW_STATUS" -eq 0 ]; then
ENABLE_R2=true
echo " ✅ R2 bucket: $R2_BUCKET"
echo " ✅ R2 preview bucket: $R2_PREVIEW_BUCKET"
else
echo " ⚠️ Could not create R2 buckets (is R2 enabled on your account?)."
echo " Attachments will stay disabled — see INSTALL.md → 'Email attachments (R2)'."
echo "$R2_OUT"
fi
fi
escape_sed_replacement() {
printf '%s' "$1" | sed -e 's/[\/&]/\\&/g'
}
@@ -158,8 +186,26 @@ else
sed -i "s/REPLACE_WITH_COMPATIBILITY_DATE/$COMPATIBILITY_DATE_ESCAPED/g" wrangler.toml
fi
if [ "$ENABLE_R2" = true ]; then
echo "🔗 Enabling R2 attachment binding in wrangler.toml..."
node - "wrangler.toml" "$R2_BUCKET" "$R2_PREVIEW_BUCKET" <<'NODE'
const fs = require("node:fs");
const [file, bucket, previewBucket] = process.argv.slice(2);
let txt = fs.readFileSync(file, "utf8");
txt = txt.split("REPLACE_WITH_YOUR_PREVIEW_BUCKET_NAME").join(previewBucket);
txt = txt.split("REPLACE_WITH_YOUR_BUCKET_NAME").join(bucket);
// Uncomment the commented r2_buckets blocks (global + [env.production]).
txt = txt.replace(
/# r2_buckets = \[\n#(\s+\{ binding = "ATTACHMENT_BUCKET".*\})\n# \]/g,
'r2_buckets = [\n$1\n]',
);
fs.writeFileSync(file, txt);
NODE
echo " ✅ ATTACHMENT_BUCKET bound to $R2_BUCKET"
fi
echo "✅ wrangler.toml has been created and configured successfully!"
echo ""
echo "✅ Setup complete! Next steps:"
echo "1. Set up MX records for your domain with ForwardEmail.net (see README for details)"
echo "1. Configure email ingestion — Cloudflare Email Workers or ForwardEmail (see INSTALL.md for details)"
echo "2. Deploy with 'npm run deploy'"
+765
View File
@@ -0,0 +1,765 @@
import { describe, it, expect, beforeEach } from "vitest";
import { http, HttpResponse } from "msw";
import { createMockEnv, MockR2, server } from "../test/setup";
import {
processEmail,
ProcessEmailInput,
RawAttachment,
} from "./email-processor";
import { getCounters } from "../application/stats";
const iconKey = (domain: string) => `icon:${domain}`;
const VALID_FEED_ID = "apple.mountain.42";
const VALID_TO = `${VALID_FEED_ID}@test.getmynews.app`;
function makeInput(
overrides: Partial<ProcessEmailInput> = {},
): ProcessEmailInput {
return {
toAddress: VALID_TO,
from: "Sender <[email protected]>",
senders: ["[email protected]"],
subject: "Test Subject",
content: "<p>Hello</p>",
receivedAt: 1700000000000,
...overrides,
};
}
describe("processEmail", () => {
let env: ReturnType<typeof createMockEnv>;
beforeEach(() => {
env = createMockEnv();
});
it("returns 400 when toAddress has no valid feedId", async () => {
const res = await processEmail(
makeInput({ toAddress: "[email protected]" }),
env as any,
);
expect(res).toMatchObject({ ok: false, reason: "invalid_address" });
});
it("returns 404 when feed does not exist", async () => {
const res = await processEmail(makeInput(), env as any);
expect(res).toMatchObject({ ok: false, reason: "feed_not_found" });
});
it("returns 403 when sender is not in allowlist", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ allowed_senders: ["[email protected]"] }),
);
const res = await processEmail(
makeInput({ senders: ["[email protected]"] }),
env as any,
);
expect(res).toMatchObject({ ok: false, reason: "sender_blocked" });
});
it("returns 200 and stores email when sender is allowed by exact match", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ allowed_senders: ["[email protected]"] }),
);
const res = await processEmail(makeInput(), env as any);
expect(res.ok).toBe(true);
});
it("returns 200 and stores email when sender matches by domain", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ allowed_senders: ["example.com"] }),
);
const res = await processEmail(
makeInput({ senders: ["[email protected]"] }),
env as any,
);
expect(res.ok).toBe(true);
});
it("returns 200 when no allowlist is set", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ allowed_senders: [] }),
);
const res = await processEmail(makeInput(), env as any);
expect(res.ok).toBe(true);
});
it("returns 403 when sender is in blocklist by exact address", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ blocked_senders: ["[email protected]"] }),
);
const res = await processEmail(makeInput(), env as any);
expect(res).toMatchObject({ ok: false, reason: "sender_blocked" });
});
it("returns 403 when sender is in blocklist by domain", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ blocked_senders: ["example.com"] }),
);
const res = await processEmail(makeInput(), env as any);
expect(res).toMatchObject({ ok: false, reason: "sender_blocked" });
});
it("returns 200 when sender is not in blocklist", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ blocked_senders: ["[email protected]"] }),
);
const res = await processEmail(makeInput(), env as any);
expect(res.ok).toBe(true);
});
it("exact block takes precedence over domain allow", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({
allowed_senders: ["example.com"],
blocked_senders: ["[email protected]"],
}),
);
const res = await processEmail(makeInput(), env as any);
expect(res).toMatchObject({ ok: false, reason: "sender_blocked" });
});
it("exact allow overrides domain block (exception use case)", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({
allowed_senders: ["[email protected]"],
blocked_senders: ["example.com"],
}),
);
const res = await processEmail(makeInput(), env as any);
expect(res.ok).toBe(true);
});
it("exact block takes precedence over exact allow", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({
allowed_senders: ["[email protected]"],
blocked_senders: ["[email protected]"],
}),
);
const res = await processEmail(makeInput(), env as any);
expect(res).toMatchObject({ ok: false, reason: "sender_blocked" });
});
it("stores email data and updates metadata in KV", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
const input = makeInput({ subject: "My Subject", content: "<b>body</b>" });
await processEmail(input, env as any);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
expect(metadata.emails).toHaveLength(1);
expect(metadata.emails[0].subject).toBe("My Subject");
const emailData = await env.EMAIL_STORAGE.get(
metadata.emails[0].key,
"json",
);
expect(emailData.subject).toBe("My Subject");
expect(emailData.content).toBe("<b>body</b>");
expect(emailData.from).toBe("Sender <[email protected]>");
});
it("prepends to existing metadata", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:metadata`,
JSON.stringify({
emails: [{ key: "old-key", subject: "Old", receivedAt: 1, size: 100 }],
}),
);
await processEmail(makeInput({ subject: "New" }), env as any);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
expect(metadata.emails).toHaveLength(2);
expect(metadata.emails[0].subject).toBe("New");
expect(metadata.emails[1].subject).toBe("Old");
});
it("trims oldest emails when total size exceeds FEED_MAX_SIZE_BYTES", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
const oldKey1 = `feed:${VALID_FEED_ID}:111`;
const oldKey2 = `feed:${VALID_FEED_ID}:222`;
const bigContent = "x".repeat(200);
const email1 = JSON.stringify({
subject: "Old1",
from: "[email protected]",
content: bigContent,
receivedAt: 111,
headers: {},
});
const email2 = JSON.stringify({
subject: "Old2",
from: "[email protected]",
content: bigContent,
receivedAt: 222,
headers: {},
});
await env.EMAIL_STORAGE.put(oldKey1, email1);
await env.EMAIL_STORAGE.put(oldKey2, email2);
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:metadata`,
JSON.stringify({
emails: [
{
key: oldKey2,
subject: "Old2",
receivedAt: 222,
size: email2.length,
},
{
key: oldKey1,
subject: "Old1",
receivedAt: 111,
size: email1.length,
},
],
}),
);
const tinyEnv = { ...env, FEED_MAX_SIZE_BYTES: "50" };
const res = await processEmail(
makeInput({ subject: "New" }),
tinyEnv as any,
);
expect(res.ok).toBe(true);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
expect(metadata.emails).toHaveLength(1);
expect(metadata.emails[0].subject).toBe("New");
const deleted1 = await env.EMAIL_STORAGE.get(oldKey1, "json");
const deleted2 = await env.EMAIL_STORAGE.get(oldKey2, "json");
expect(deleted1).toBeNull();
expect(deleted2).toBeNull();
});
it("keeps entries within size budget untouched", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
const bigEnv = { ...env, FEED_MAX_SIZE_BYTES: String(10 * 1024 * 1024) };
await processEmail(makeInput({ subject: "First" }), bigEnv as any);
await processEmail(makeInput({ subject: "Second" }), bigEnv as any);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
expect(metadata.emails).toHaveLength(2);
});
it("calls ctx.waitUntil with notifySubscribers when ctx is provided", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({
title: "Test",
language: "en",
created_at: Date.now(),
}),
);
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:metadata`,
JSON.stringify({ emails: [] }),
);
let waitUntilCalled = false;
const ctx = {
waitUntil: (p: Promise<unknown>) => {
waitUntilCalled = true;
void p; // don't actually await it
},
passThroughOnException: () => {},
} as unknown as ExecutionContext;
const res = await processEmail(makeInput(), env as any, ctx);
expect(res.ok).toBe(true);
expect(waitUntilCalled).toBe(true);
});
it("does not call ctx.waitUntil on error paths (feed not found)", async () => {
let waitUntilCalled = false;
const ctx = {
waitUntil: (p: Promise<unknown>) => {
waitUntilCalled = true;
void p;
},
passThroughOnException: () => {},
} as unknown as ExecutionContext;
// Feed ID is valid format but config doesn't exist → 404
const res = await processEmail(
makeInput({ toAddress: `[email protected]` }),
env as any,
ctx,
);
expect(res).toMatchObject({ ok: false, reason: "feed_not_found" });
expect(waitUntilCalled).toBe(false);
});
});
describe("processEmail — attachments", () => {
const pdfContent = new TextEncoder().encode("PDF bytes")
.buffer as ArrayBuffer;
const pdfAttachment: RawAttachment = {
filename: "report.pdf",
contentType: "application/pdf",
content: pdfContent,
};
it("skips R2 upload when ATTACHMENT_BUCKET is not configured", async () => {
const env = createMockEnv();
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
const res = await processEmail(
makeInput({ attachments: [pdfAttachment] }),
env as any,
);
expect(res.ok).toBe(true);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
const emailData = await env.EMAIL_STORAGE.get(
metadata.emails[0].key,
"json",
);
expect(emailData.attachments).toBeUndefined();
});
it("skips R2 upload when ATTACHMENTS_ENABLED is 'false' even with R2 bound", async () => {
const env = createMockEnv({ withR2: true });
(env as any).ATTACHMENTS_ENABLED = "false";
const mockR2 = (env as any).ATTACHMENT_BUCKET as unknown as MockR2;
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
const res = await processEmail(
makeInput({ attachments: [pdfAttachment] }),
env as any,
);
expect(res.ok).toBe(true);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
const emailData = await env.EMAIL_STORAGE.get(
metadata.emails[0].key,
"json",
);
expect(emailData.attachments).toBeUndefined();
expect((await mockR2.list()).objects).toHaveLength(0);
});
it("uploads attachments to R2 and stores AttachmentData in emailData", async () => {
const env = createMockEnv({ withR2: true });
const mockR2 = (env as any).ATTACHMENT_BUCKET as unknown as MockR2;
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
const res = await processEmail(
makeInput({ attachments: [pdfAttachment] }),
env as any,
);
expect(res.ok).toBe(true);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
const emailData = await env.EMAIL_STORAGE.get(
metadata.emails[0].key,
"json",
);
expect(emailData.attachments).toHaveLength(1);
expect(emailData.attachments[0].filename).toBe("report.pdf");
expect(emailData.attachments[0].contentType).toBe("application/pdf");
expect(emailData.attachments[0].size).toBe(pdfContent.byteLength);
const id = emailData.attachments[0].id;
expect(mockR2._has(id)).toBe(true);
});
it("stores attachmentIds in EmailMetadata for trim-time cleanup", async () => {
const env = createMockEnv({ withR2: true });
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
await processEmail(makeInput({ attachments: [pdfAttachment] }), env as any);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
expect(metadata.emails[0].attachmentIds).toHaveLength(1);
expect(typeof metadata.emails[0].attachmentIds[0]).toBe("string");
});
it("classifies a cid-referenced image as inline, not a downloadable attachment", async () => {
const env = createMockEnv({ withR2: true });
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
const inlineImage: RawAttachment = {
filename: "logo.png",
contentType: "image/png",
content: new TextEncoder().encode("PNG").buffer as ArrayBuffer,
contentId: "logo123",
};
await processEmail(
makeInput({
content: '<p>Hi</p><img src="cid:logo123"/>',
attachments: [inlineImage, pdfAttachment],
}),
env as any,
);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
const emailData = await env.EMAIL_STORAGE.get(
metadata.emails[0].key,
"json",
);
const inline = emailData.attachments.find(
(a: any) => a.filename === "logo.png",
);
const pdf = emailData.attachments.find(
(a: any) => a.filename === "report.pdf",
);
expect(inline.inline).toBe(true);
expect(pdf.inline).toBeUndefined();
// Metadata splits ids: the pdf is downloadable, the logo is inline-only.
expect(metadata.emails[0].attachmentIds).toEqual([pdf.id]);
expect(metadata.emails[0].inlineAttachmentIds).toEqual([inline.id]);
});
it("deletes inline image R2 objects when a trimmed email had them", async () => {
const env = createMockEnv({ withR2: true });
const mockR2 = (env as any).ATTACHMENT_BUCKET as unknown as MockR2;
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
const oldKey = `feed:${VALID_FEED_ID}:111`;
const inlineId = "old-inline-uuid";
const oldEmail = JSON.stringify({
subject: "Old",
from: "[email protected]",
content: "x".repeat(200) + '<img src="cid:c"/>',
receivedAt: 111,
headers: {},
attachments: [
{
id: inlineId,
filename: "logo.png",
contentType: "image/png",
size: 100,
contentId: "c",
inline: true,
},
],
});
await env.EMAIL_STORAGE.put(oldKey, oldEmail);
await mockR2.put(inlineId, new ArrayBuffer(100));
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:metadata`,
JSON.stringify({
emails: [
{
key: oldKey,
subject: "Old",
receivedAt: 111,
size: oldEmail.length,
inlineAttachmentIds: [inlineId],
},
],
}),
);
const tinyEnv = { ...env, FEED_MAX_SIZE_BYTES: "50" };
const res = await processEmail(
makeInput({ subject: "New" }),
tinyEnv as any,
);
expect(res.ok).toBe(true);
expect(mockR2._has(inlineId)).toBe(false);
});
it("deletes R2 objects when a trimmed email had attachments", async () => {
const env = createMockEnv({ withR2: true });
const mockR2 = (env as any).ATTACHMENT_BUCKET as unknown as MockR2;
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
// Store an old email with attachment in KV and metadata
const oldKey = `feed:${VALID_FEED_ID}:111`;
const oldAttachmentId = "old-attachment-uuid";
const bigContent = "x".repeat(200);
const oldEmail = JSON.stringify({
subject: "Old",
from: "[email protected]",
content: bigContent,
receivedAt: 111,
headers: {},
attachments: [
{
id: oldAttachmentId,
filename: "old.pdf",
contentType: "application/pdf",
size: 100,
},
],
});
await env.EMAIL_STORAGE.put(oldKey, oldEmail);
// Also put the attachment in mock R2
await mockR2.put(oldAttachmentId, new ArrayBuffer(100));
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:metadata`,
JSON.stringify({
emails: [
{
key: oldKey,
subject: "Old",
receivedAt: 111,
size: oldEmail.length,
attachmentIds: [oldAttachmentId],
},
],
}),
);
// Process with tight size budget to force trimming
const tinyEnv = { ...env, FEED_MAX_SIZE_BYTES: "50" };
const res = await processEmail(
makeInput({ subject: "New" }),
tinyEnv as any,
);
expect(res.ok).toBe(true);
// Old attachment should be deleted from R2
expect(mockR2._has(oldAttachmentId)).toBe(false);
});
});
describe("processEmail — monitoring counters", () => {
it("increments emails_received and sets last_email_at on success", async () => {
const env = createMockEnv();
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
await processEmail(makeInput(), env as any);
const counters = await getCounters(env.EMAIL_STORAGE as any);
expect(counters.emails_received).toBe(1);
expect(counters.emails_rejected).toBe(0);
expect(counters.last_email_at).toBeDefined();
});
it("increments emails_rejected when validation fails", async () => {
const env = createMockEnv();
// No feed config → 404 rejection
await processEmail(makeInput(), env as any);
const counters = await getCounters(env.EMAIL_STORAGE as any);
expect(counters.emails_rejected).toBe(1);
expect(counters.emails_received).toBe(0);
});
});
describe("processEmail — feed icon", () => {
let env: ReturnType<typeof createMockEnv>;
beforeEach(async () => {
env = createMockEnv();
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
});
it("persists the latest sender domain on the feed metadata", async () => {
await processEmail(
makeInput({ from: "News <[email protected]>" }),
env as any,
);
const metadata = (await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
)) as { iconDomain?: string };
expect(metadata.iconDomain).toBe("github.com");
});
it("triggers a background favicon fetch via ctx.waitUntil", async () => {
let fetched = false;
server.use(
http.get("https://github.com/favicon.ico", () => {
fetched = true;
return new HttpResponse(new Uint8Array([1, 2, 3]), {
headers: { "Content-Type": "image/png" },
});
}),
);
const pending: Promise<unknown>[] = [];
const ctx = {
waitUntil: (p: Promise<unknown>) => pending.push(p),
passThroughOnException: () => {},
} as unknown as ExecutionContext;
await processEmail(makeInput({ from: "[email protected]" }), env as any, ctx);
await Promise.all(pending);
expect(fetched).toBe(true);
expect(
await env.EMAIL_STORAGE.get(iconKey("github.com"), "json"),
).toMatchObject({ contentType: "image/png" });
});
});
describe("processEmail — unsubscribe capture", () => {
let env: ReturnType<typeof createMockEnv>;
beforeEach(async () => {
env = createMockEnv();
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
});
it("stores the one-click unsubscribe URL on the feed metadata, keyed by sender", async () => {
await processEmail(
makeInput({
senders: ["[email protected]"],
headers: {
"list-unsubscribe": "<https://example.com/u?t=abc>",
"list-unsubscribe-post": "List-Unsubscribe=One-Click",
},
}),
env as any,
);
const metadata = (await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
)) as { unsubscribe?: Record<string, string> };
expect(metadata.unsubscribe).toEqual({
"[email protected]": "https://example.com/u?t=abc",
});
});
it("keeps one entry per sender and overwrites with the latest URL", async () => {
await processEmail(
makeInput({
senders: ["[email protected]"],
headers: {
"list-unsubscribe": "<https://one.com/u/1>",
"list-unsubscribe-post": "List-Unsubscribe=One-Click",
},
}),
env as any,
);
await processEmail(
makeInput({
senders: ["[email protected]"],
headers: {
"list-unsubscribe": "<https://two.com/u/1>",
"list-unsubscribe-post": "List-Unsubscribe=One-Click",
},
}),
env as any,
);
await processEmail(
makeInput({
senders: ["[email protected]"],
headers: {
"list-unsubscribe": "<https://one.com/u/2>",
"list-unsubscribe-post": "List-Unsubscribe=One-Click",
},
}),
env as any,
);
const metadata = (await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
)) as { unsubscribe?: Record<string, string> };
expect(metadata.unsubscribe).toEqual({
"[email protected]": "https://one.com/u/2",
"[email protected]": "https://two.com/u/1",
});
});
it("does not store anything without the one-click Post header", async () => {
await processEmail(
makeInput({
headers: { "list-unsubscribe": "<https://example.com/u/1>" },
}),
env as any,
);
const metadata = (await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
)) as { unsubscribe?: Record<string, string> };
expect(metadata.unsubscribe).toBeUndefined();
});
});
+216
View File
@@ -0,0 +1,216 @@
import { EmailParser } from "../domain/email-parser";
import { AttachmentData, EmailMetadata, Env } from "../types";
import { bumpCounters } from "../application/stats";
import { dispatchFeedEvents } from "../application/feed-events";
import { extractEmailDomain } from "../infrastructure/favicon-fetcher";
import { parseOneClickUnsubscribe } from "../infrastructure/unsubscribe";
import { getAttachmentBucket } from "../infrastructure/attachments";
import { extractInlineCids } from "../infrastructure/html-processor";
import { attachmentIdsForCleanup } from "./feed-cleanup";
import { FeedRepository } from "../infrastructure/feed-repository";
import { BackgroundScheduler } from "../infrastructure/worker";
import { Feed } from "../domain/feed.aggregate";
import { logger } from "../infrastructure/logger";
import { FEED_MAX_BYTES } from "../config/constants";
export interface RawAttachment {
filename: string;
contentType: string;
content: ArrayBuffer;
contentId?: string;
}
export interface ProcessEmailInput {
toAddress: string;
from: string;
senders: string[];
subject: string;
content: string;
receivedAt: number;
headers?: Record<string, string>;
attachments?: RawAttachment[];
}
export type IngestRejectionReason =
| "invalid_address"
| "feed_not_found"
| "feed_expired"
| "sender_blocked";
/**
* Outcome of ingesting an email — a domain result, not an HTTP concern. The edge
* (forwardemail.ts) maps this to a status code; the Cloudflare email handler
* logs the reason. Keeping HTTP out of the core keeps ingestion transport-agnostic.
*/
export type IngestResult =
| { ok: true; feedId: string }
| { ok: false; reason: IngestRejectionReason };
async function uploadAttachments(
attachments: RawAttachment[],
bucket: R2Bucket,
inlineCids: Set<string>,
): Promise<AttachmentData[]> {
return Promise.all(
attachments.map(async (att) => {
const id = crypto.randomUUID();
const inline = att.contentId ? inlineCids.has(att.contentId) : false;
await bucket.put(id, att.content, {
httpMetadata: {
contentType: att.contentType,
contentDisposition: `${inline ? "inline" : "attachment"}; filename="${att.filename}"`,
},
});
return {
id,
filename: att.filename,
contentType: att.contentType,
size: att.content.byteLength,
...(att.contentId ? { contentId: att.contentId } : {}),
...(inline ? { inline: true } : {}),
};
}),
);
}
async function loadAcceptingFeed(
input: ProcessEmailInput,
env: Env,
): Promise<
{ ok: true; feed: Feed } | { ok: false; reason: IngestRejectionReason }
> {
const feedId = EmailParser.extractFeedId(input.toAddress);
if (!feedId) {
logger.error("Invalid email address format", {
toAddress: input.toAddress,
});
return { ok: false, reason: "invalid_address" };
}
const feed = await FeedRepository.from(env).load(feedId);
if (!feed) {
logger.error("Feed not found", { feedId: feedId.value });
return { ok: false, reason: "feed_not_found" };
}
if (feed.isExpired()) {
logger.warn("Rejected email: feed expired", { feedId: feedId.value });
return { ok: false, reason: "feed_expired" };
}
if (feed.accepts(input.senders) === "blocked") {
logger.warn("Rejected email: sender filter", {
feedId: feedId.value,
senders: input.senders,
allowedSenders: feed.allowedSenders(),
blockedSenders: feed.blockedSenders(),
});
return { ok: false, reason: "sender_blocked" };
}
return { ok: true, feed };
}
async function storeEmail(
feed: Feed,
input: ProcessEmailInput,
env: Env,
ctx?: ExecutionContext,
): Promise<void> {
const attachmentBucket = getAttachmentBucket(env);
const inlineCids = extractInlineCids(input.content);
const storedAttachments: AttachmentData[] =
attachmentBucket && input.attachments?.length
? await uploadAttachments(input.attachments, attachmentBucket, inlineCids)
: [];
const emailData = {
subject: input.subject,
from: input.from,
content: input.content,
receivedAt: input.receivedAt,
headers: input.headers ?? {},
...(storedAttachments.length > 0 ? { attachments: storedAttachments } : {}),
};
const repo = FeedRepository.from(env);
const emailKey = repo.newEmailKey(feed.id);
await repo.putEmail(emailKey, emailData);
const serialisedSize = new TextEncoder().encode(
JSON.stringify(emailData),
).byteLength;
const downloadableIds = storedAttachments
.filter((a) => !a.inline)
.map((a) => a.id);
const inlineIds = storedAttachments.filter((a) => a.inline).map((a) => a.id);
const newEntry: EmailMetadata = {
key: emailKey,
subject: emailData.subject,
receivedAt: emailData.receivedAt,
size: serialisedSize,
...(downloadableIds.length > 0 ? { attachmentIds: downloadableIds } : {}),
...(inlineIds.length > 0 ? { inlineAttachmentIds: inlineIds } : {}),
};
// Track the latest sender's domain (feed icon) and capture the RFC 8058
// one-click unsubscribe link, keyed by sender so each newsletter keeps its
// own latest URL (fired when the feed is deleted).
const iconDomain = extractEmailDomain(input.from);
const unsubUrl = parseOneClickUnsubscribe(input.headers ?? {});
const unsub = unsubUrl
? {
senderKey: input.senders[0] || iconDomain || input.from,
url: unsubUrl,
}
: undefined;
const maxBytes =
parseInt(env.FEED_MAX_SIZE_BYTES ?? "", 10) || FEED_MAX_BYTES;
const { dropped } = feed.ingest(newEntry, {
maxBytes,
iconDomain: iconDomain ?? undefined,
unsub,
});
const r2Deletions =
attachmentBucket && dropped.length > 0
? dropped
.flatMap((e) => attachmentIdsForCleanup(e))
.map((id) => attachmentBucket.delete(id))
: [];
// KV has no compare-and-swap: the load (in loadAcceptingFeed) and this write
// are not serialised, so concurrent ingests for one feed can lose updates.
// Accepted under KV's eventual-consistency model; the Feed aggregate is the
// seam a Durable Object would later wrap to serialise these writers.
await Promise.all([
repo.saveMetadata(feed),
...dropped.map((e) => repo.deleteEmail(e.key)),
...r2Deletions,
]);
logger.info("Email processed", { feedId: feed.id.value });
// The aggregate recorded an EmailIngested event; the dispatcher applies its
// side effects (received counter, WebSub ping, favicon fetch). Background work
// rides on ctx.waitUntil when present, and is skipped in its absence (tests).
const schedule: BackgroundScheduler = ctx
? (p) => ctx.waitUntil(p)
: () => {};
await dispatchFeedEvents(feed, env, schedule);
}
export async function processEmail(
input: ProcessEmailInput,
env: Env,
ctx?: ExecutionContext,
): Promise<IngestResult> {
const validation = await loadAcceptingFeed(input, env);
if (!validation.ok) {
await bumpCounters(env.EMAIL_STORAGE, { emails_rejected: 1 });
return validation;
}
await storeEmail(validation.feed, input, env, ctx);
return { ok: true, feedId: validation.feed.id.value };
}
+147
View File
@@ -0,0 +1,147 @@
import { EmailData, EmailMetadata, Env } from "../types";
import { logger } from "../infrastructure/logger";
import { getAttachmentBucket } from "../infrastructure/attachments";
import { FeedRepository } from "../infrastructure/feed-repository";
import { FeedId } from "../domain/value-objects/feed-id";
// All R2 object ids an email owns — both downloadable attachments and inline
// images. Inline images are hidden from the user-facing lists but must still be
// purged from the bucket when the email is deleted.
export function attachmentIdsForCleanup(e: EmailMetadata): string[] {
return [...(e.attachmentIds ?? []), ...(e.inlineAttachmentIds ?? [])];
}
// Delete the R2 attachments belonging to the given email keys. Call before the
// emails are removed from feed metadata, while `emails` still carries their
// attachment ids.
export async function deleteAttachmentsForEmails(
env: Env,
emails: readonly EmailMetadata[],
keys: Iterable<string>,
): Promise<void> {
const keySet = new Set(keys);
const attachmentIds = emails
.filter((e) => keySet.has(e.key))
.flatMap((e) => attachmentIdsForCleanup(e));
if (attachmentIds.length === 0) return;
const bucket = getAttachmentBucket(env);
if (!bucket) return;
await Promise.allSettled(attachmentIds.map((id) => bucket.delete(id)));
}
export async function deleteKeysWithConcurrency(
emailStorage: KVNamespace,
keys: string[],
concurrency: number,
): Promise<{ ok: string[]; failed: string[] }> {
const uniqueKeys = Array.from(new Set(keys.filter(Boolean)));
const ok: string[] = [];
const failed: string[] = [];
const limit = Math.max(1, Math.floor(concurrency) || 1);
for (let i = 0; i < uniqueKeys.length; i += limit) {
const batch = uniqueKeys.slice(i, i + limit);
const results = await Promise.allSettled(
batch.map((key) => emailStorage.delete(key)),
);
results.forEach((result, idx) => {
const key = batch[idx];
if (result.status === "fulfilled") {
ok.push(key);
} else {
failed.push(key);
}
});
}
return { ok, failed };
}
/**
* Read a feed's stored RFC 8058 one-click unsubscribe URLs (one per sender).
* Must be called before the feed metadata is deleted. Never throws.
*/
export async function collectUnsubscribeUrls(
emailStorage: KVNamespace,
feedId: FeedId,
): Promise<string[]> {
try {
const metadata = await new FeedRepository(emailStorage).getMetadata(feedId);
return Object.values(metadata?.unsubscribe ?? {});
} catch (error) {
logger.error("Error reading unsubscribe URLs", {
feedId: feedId.value,
error: String(error),
});
return [];
}
}
export async function purgeFeedKeysStep(
emailStorage: KVNamespace,
feedId: FeedId,
options: { cursor?: string; limit?: number; bucket?: R2Bucket } = {},
): Promise<{
deletedKeys: string[];
failedKeys: string[];
cursor: string;
listComplete: boolean;
}> {
const repo = new FeedRepository(emailStorage);
const listed = await repo.listFeedKeys(feedId, {
cursor: options.cursor,
limit: options.limit,
});
const keys = listed.names;
if (options.bucket && keys.length > 0) {
const emailKeys = keys.filter((k) => repo.isEmailKey(feedId, k));
if (emailKeys.length > 0) {
const emailDataResults = await Promise.allSettled(
emailKeys.map((k) => repo.getEmail(k)),
);
const attachmentIds = emailDataResults
.filter(
(r): r is PromiseFulfilledResult<EmailData | null> =>
r.status === "fulfilled",
)
.flatMap((r) => r.value?.attachments?.map((a) => a.id) ?? []);
if (attachmentIds.length > 0) {
await Promise.allSettled(
attachmentIds.map((id) => options.bucket!.delete(id)),
);
}
}
}
const { ok, failed } = await deleteKeysWithConcurrency(
emailStorage,
keys,
35,
);
return {
deletedKeys: ok,
failedKeys: failed,
cursor: listed.cursor,
listComplete: listed.listComplete,
};
}
export async function purgeExpiredFeeds(
emailStorage: KVNamespace,
feedId: FeedId,
bucket?: R2Bucket,
): Promise<void> {
let cursor: string | undefined;
do {
const step = await purgeFeedKeysStep(emailStorage, feedId, {
bucket,
limit: 100,
cursor,
});
cursor = step.listComplete ? undefined : step.cursor;
} while (cursor);
}
+55
View File
@@ -0,0 +1,55 @@
import { Env } from "../types";
import { FeedEvent } from "../domain/events";
import { Feed } from "../domain/feed.aggregate";
import { BackgroundScheduler } from "../infrastructure/worker";
import { bumpCounters } from "./stats";
import { notifySubscribers } from "../infrastructure/websub";
import { cacheFaviconForDomain } from "../infrastructure/favicon-fetcher";
/**
* Apply the side effects of a feed's domain events — the single place that maps
* "what happened" (FeedCreated, EmailIngested) to its consequences. Each event
* carries its own `feedId`, so nothing has to be threaded in. Counter writes are
* awaited (they must land); WebSub pings and favicon fetches are handed to the
* caller's background scheduler (`ctx.waitUntil` at the edge, a no-op when none
* is available).
*/
export async function applyFeedEvents(
events: FeedEvent[],
env: Env,
schedule: BackgroundScheduler,
): Promise<void> {
for (const event of events) {
switch (event.type) {
case "FeedCreated":
await bumpCounters(env.EMAIL_STORAGE, {
feeds_created: 1,
last_feed_created_at: new Date().toISOString(),
});
break;
case "EmailIngested":
await bumpCounters(env.EMAIL_STORAGE, {
emails_received: 1,
last_email_at: new Date().toISOString(),
});
schedule(notifySubscribers(event.feedId, env));
if (event.iconDomain) {
schedule(cacheFaviconForDomain(event.iconDomain, env));
}
break;
}
}
}
/**
* Drain a freshly-persisted aggregate's events and apply their side effects. The
* single dispatch entry point: callers persist the `Feed`, then call this — no
* caller pulls events or passes the feed id by hand.
*/
export async function dispatchFeedEvents(
feed: Feed,
env: Env,
schedule: BackgroundScheduler,
): Promise<void> {
await applyFeedEvents(feed.pullEvents(), env, schedule);
}
+35
View File
@@ -0,0 +1,35 @@
import { Env, FeedConfig, EmailData } from "../types";
import { MAX_FEED_ITEMS } from "../config/constants";
import { FeedRepository } from "../infrastructure/feed-repository";
import { FeedId } from "../domain/value-objects/feed-id";
export interface FeedData {
feedConfig: FeedConfig;
emails: EmailData[];
}
export async function fetchFeedData(
feedId: FeedId,
env: Env,
): Promise<FeedData | null> {
const repo = FeedRepository.from(env);
const feedMetadata = await repo.getMetadata(feedId);
if (!feedMetadata) return null;
const feedConfig = (await repo.getConfig(feedId)) ?? {
title: `Newsletter Feed ${feedId.value}`,
description: "Converted email newsletter",
language: "en",
created_at: Date.now(),
};
const emailRefs = feedMetadata.emails.slice(0, MAX_FEED_ITEMS);
const emails: EmailData[] = [];
for (const ref of emailRefs) {
const data = await repo.getEmail(ref.key);
if (data) emails.push(data);
}
return { feedConfig, emails };
}
+90
View File
@@ -0,0 +1,90 @@
import { describe, it, expect } from "vitest";
import { createMockEnv } from "../test/setup";
import { createFeedRecord, editFeed } from "./feed-service";
import { getCounters } from "./stats";
import { FeedId } from "../domain/value-objects/feed-id";
import type { Env } from "../types";
const mkEnv = (overrides: Partial<Env> = {}) =>
({ ...createMockEnv(), ...overrides }) as unknown as Env;
const baseInput = {
title: "N",
language: "en",
allowedSenders: [],
blockedSenders: [],
};
const TWO_HOURS = 2 * 3_600_000;
// The lifetime policy (parse env, apply the server-side FEED_TTL_HOURS override)
// lives here in the application layer; the domain only receives a resolved
// ttlHours. These tests pin that policy at the public service boundary.
describe("createFeedRecord — TTL policy", () => {
it("never expires when neither server nor client lifetime is set", async () => {
const { config } = await createFeedRecord(mkEnv(), { ...baseInput });
expect(config.expires_at).toBeUndefined();
});
it("uses the client lifetimeHours when there is no server override", async () => {
const before = Date.now();
const { config } = await createFeedRecord(mkEnv(), {
...baseInput,
lifetimeHours: 2,
});
expect(config.expires_at!).toBeGreaterThanOrEqual(before + TWO_HOURS);
});
it("lets a server FEED_TTL_HOURS override a larger client lifetime", async () => {
const before = Date.now();
const { config } = await createFeedRecord(mkEnv({ FEED_TTL_HOURS: "1" }), {
...baseInput,
lifetimeHours: 9999,
});
// 1h (server) wins over 9999h (client).
expect(config.expires_at!).toBeLessThan(before + TWO_HOURS);
});
it("bumps the feeds_created counter via the FeedCreated domain event", async () => {
const env = mkEnv();
await createFeedRecord(env, { ...baseInput });
const counters = await getCounters(env.EMAIL_STORAGE);
expect(counters.feeds_created).toBe(1);
expect(counters.last_feed_created_at).toBeDefined();
});
});
describe("editFeed — TTL policy", () => {
it("recomputes expiry from the server override on edit", async () => {
const env = mkEnv({ FEED_TTL_HOURS: "1" });
const { feedId } = await createFeedRecord(env, { ...baseInput });
const before = Date.now();
const result = await editFeed(env, FeedId.unchecked(feedId), {
title: "renamed",
});
expect(result.status).toBe("ok");
if (result.status === "ok") {
expect(result.config.title).toBe("renamed");
expect(result.config.expires_at!).toBeLessThan(before + TWO_HOURS);
}
});
it("preserves expiry when neither server TTL nor client lifetime is given", async () => {
const env = mkEnv();
const { feedId, config } = await createFeedRecord(env, {
...baseInput,
lifetimeHours: 5,
});
const result = await editFeed(env, FeedId.unchecked(feedId), {
title: "x",
});
expect(result.status).toBe("ok");
if (result.status === "ok") {
expect(result.config.expires_at).toBe(config.expires_at);
}
});
});
+185
View File
@@ -0,0 +1,185 @@
import { Env, FeedConfig } from "../types";
import { bumpCounters } from "../application/stats";
import { dispatchFeedEvents } from "./feed-events";
import { sendUnsubscribes } from "../infrastructure/unsubscribe";
import { getAttachmentBucket } from "../infrastructure/attachments";
import { FeedRepository } from "../infrastructure/feed-repository";
import { toConfigDTO } from "../infrastructure/feed-mapper";
import { BackgroundScheduler } from "../infrastructure/worker";
import { FeedId } from "../domain/value-objects/feed-id";
import { Lifetime } from "../domain/value-objects/lifetime";
import {
Feed,
CreateFeedInput,
UpdateFeedInput,
} from "../domain/feed.aggregate";
import { purgeFeedKeysStep, collectUnsubscribeUrls } from "./feed-cleanup";
export type { CreateFeedInput, UpdateFeedInput };
/**
* Resolve the effective feed `Lifetime` from a client request and the
* server-side `FEED_TTL_HOURS` override. Parsing the env string and applying the
* override is application/config policy — the domain only receives the resolved
* VO. Returns `Lifetime.never` when the feed should never expire.
*/
function resolveLifetime(env: Env, requestedHours?: number): Lifetime {
const hours = env.FEED_TTL_HOURS
? parseInt(env.FEED_TTL_HOURS, 10)
: (requestedHours ?? NaN);
return Number.isFinite(hours) && hours > 0
? Lifetime.ofHours(hours)
: Lifetime.never;
}
/**
* Create a feed: write its config + empty metadata, register it in the global
* list, and bump the `feeds_created` counter. Returns the new feed id + config.
*/
export async function createFeedRecord(
env: Env,
input: CreateFeedInput,
): Promise<{ feedId: string; config: FeedConfig }> {
const repo = FeedRepository.from(env);
const feed = Feed.create(FeedId.generate(), input, {
lifetime: resolveLifetime(env, input.lifetimeHours),
});
await repo.save(feed);
// FeedCreated → bumps the feeds_created counter (no background work to schedule).
await dispatchFeedEvents(feed, env, () => {});
return { feedId: feed.id.value, config: toConfigDTO(feed.state()) };
}
export type UpdateFeedResult =
| { status: "ok"; config: FeedConfig }
| { status: "not_found" }
| { status: "expired" };
/**
* Quick-edit of title/description only — never recomputes expiry. Used by the
* dashboard's minimal edit. Delegates to the aggregate's single `edit` path, so
* an expired feed is rejected here too. The list projection is kept in sync by
* the repository on `saveConfig`.
*/
export async function editFeedDetails(
env: Env,
feedId: FeedId,
patch: { title?: string; description?: string },
): Promise<UpdateFeedResult> {
const repo = FeedRepository.from(env);
const feed = await repo.load(feedId);
if (!feed) return { status: "not_found" };
// No lifetime passed ⇒ expiry preserved (quick-edit never recomputes it).
if (feed.edit(patch).status === "expired") {
return { status: "expired" };
}
await repo.saveConfig(feed);
return { status: "ok", config: toConfigDTO(feed.state()) };
}
/**
* Full edit: apply the patch, recompute expiry, and reject expired feeds. Fields
* left undefined are preserved. Mirrors title/description/expiry into the list.
*/
export async function editFeed(
env: Env,
feedId: FeedId,
input: UpdateFeedInput,
): Promise<UpdateFeedResult> {
const repo = FeedRepository.from(env);
const feed = await repo.load(feedId);
if (!feed) return { status: "not_found" };
// Recompute expiry only when a server TTL or a client lifetime applies;
// otherwise pass no lifetime so the aggregate preserves the current expiry.
const lifetime =
Boolean(env.FEED_TTL_HOURS) || input.lifetimeHours !== undefined
? resolveLifetime(env, input.lifetimeHours)
: undefined;
if (feed.edit(input, { lifetime }).status === "expired") {
return { status: "expired" };
}
await repo.saveConfig(feed);
return { status: "ok", config: toConfigDTO(feed.state()) };
}
type DeleteFeedFastResult = {
ok: boolean;
configDeleted: boolean;
metadataDeleted: boolean;
errors: string[];
};
/**
* Delete a feed's config + metadata keys, reporting per-key outcomes. The
* larger email/attachment cleanup is handled separately via purgeFeedKeysStep.
*/
export async function deleteFeedFastDetailed(
emailStorage: KVNamespace,
feedId: FeedId,
): Promise<DeleteFeedFastResult> {
const repo = new FeedRepository(emailStorage);
const errors: string[] = [];
let configDeleted = false;
let metadataDeleted = false;
try {
await repo.deleteConfig(feedId);
configDeleted = true;
} catch (error) {
errors.push(`config delete failed: ${String(error)}`);
}
try {
await repo.deleteMetadata(feedId);
metadataDeleted = true;
} catch (error) {
errors.push(`metadata delete failed: ${String(error)}`);
}
return { ok: configDeleted, configDeleted, metadataDeleted, errors };
}
/**
* Delete a single feed end-to-end: capture unsubscribe URLs, drop its config +
* metadata, remove it from the list, bump the counter, and hand the background
* unsubscribe requests + key purge to the supplied scheduler. Returns whether
* the feed was present in the global list.
*/
export async function deleteFeedRecord(
env: Env,
feedId: FeedId,
schedule: BackgroundScheduler,
): Promise<boolean> {
const emailStorage = env.EMAIL_STORAGE;
const repo = new FeedRepository(emailStorage);
// Read unsubscribe URLs before the metadata is deleted below.
const unsubscribeUrls = await collectUnsubscribeUrls(emailStorage, feedId);
await deleteFeedFastDetailed(emailStorage, feedId);
const removed = await repo.removeFromList(feedId);
if (removed) {
await bumpCounters(emailStorage, { feeds_deleted: 1 });
}
if (unsubscribeUrls.length > 0) {
schedule(sendUnsubscribes(unsubscribeUrls, env));
}
schedule(
purgeFeedKeysStep(emailStorage, feedId, {
bucket: getAttachmentBucket(env),
}),
);
return removed;
}
+217
View File
@@ -0,0 +1,217 @@
import { describe, it, expect } from "vitest";
import { createMockEnv, MockR2 } from "../test/setup";
import {
getCounters,
bumpCounters,
countKeysByPrefix,
getStats,
scanR2Usage,
scanKvUsage,
setStorageSnapshot,
} from "./stats";
import { getAttachmentBucket } from "../infrastructure/attachments";
import { STATS_KEY, FEEDS_LIST_KEY } from "../config/constants";
import { Env } from "../types";
describe("stats helper", () => {
it("returns zeroed counters when nothing is stored", async () => {
const env = createMockEnv() as unknown as Env;
const counters = await getCounters(env.EMAIL_STORAGE);
expect(counters).toMatchObject({
feeds_created: 0,
feeds_deleted: 0,
emails_received: 0,
emails_rejected: 0,
});
expect(counters.first_seen).toBeUndefined();
});
it("accumulates numeric deltas across bumps", async () => {
const env = createMockEnv() as unknown as Env;
const kv = env.EMAIL_STORAGE;
await bumpCounters(kv, { emails_received: 1 });
await bumpCounters(kv, { emails_received: 2, emails_rejected: 1 });
await bumpCounters(kv, { feeds_created: 1, feeds_deleted: 3 });
const counters = await getCounters(kv);
expect(counters.emails_received).toBe(3);
expect(counters.emails_rejected).toBe(1);
expect(counters.feeds_created).toBe(1);
expect(counters.feeds_deleted).toBe(3);
});
it("overwrites date-time fields and sets first_seen once", async () => {
const env = createMockEnv() as unknown as Env;
const kv = env.EMAIL_STORAGE;
await bumpCounters(kv, {
emails_received: 1,
last_email_at: "2026-01-01T00:00:00.000Z",
});
const first = await getCounters(kv);
const firstSeen = first.first_seen;
expect(firstSeen).toBeDefined();
expect(first.last_email_at).toBe("2026-01-01T00:00:00.000Z");
await bumpCounters(kv, {
emails_received: 1,
last_email_at: "2026-02-02T00:00:00.000Z",
});
const second = await getCounters(kv);
expect(second.last_email_at).toBe("2026-02-02T00:00:00.000Z");
expect(second.first_seen).toBe(firstSeen);
});
it("counts keys by prefix", async () => {
const env = createMockEnv() as unknown as Env;
const kv = env.EMAIL_STORAGE;
await kv.put("websub:a:1", "{}");
await kv.put("websub:a:2", "{}");
await kv.put("feed:x:config", "{}");
expect(await countKeysByPrefix(kv, "websub:")).toBe(2);
expect(await countKeysByPrefix(kv, "missing:")).toBe(0);
});
it("getStats combines persisted counters with live values", async () => {
const env = createMockEnv() as unknown as Env;
const kv = env.EMAIL_STORAGE;
await kv.put(
FEEDS_LIST_KEY,
JSON.stringify({
feeds: [
{ id: "a", title: "A" },
{ id: "b", title: "B" },
],
}),
);
await kv.put("websub:subs:a", "{}");
await bumpCounters(kv, { emails_received: 5, feeds_created: 2 });
const stats = await getStats(env);
expect(stats.active_feeds).toBe(2);
expect(stats.websub_subscriptions_active).toBe(1);
expect(stats.emails_received).toBe(5);
expect(stats.feeds_created).toBe(2);
});
it("never throws on a failing KV (counters are best-effort)", async () => {
const brokenKv = {
get: async () => {
throw new Error("kv down");
},
put: async () => {
throw new Error("kv down");
},
} as unknown as KVNamespace;
await expect(
bumpCounters(brokenKv, { emails_received: 1 }),
).resolves.toBeUndefined();
expect(await getCounters(brokenKv)).toMatchObject({ emails_received: 0 });
expect(await countKeysByPrefix(brokenKv, "websub:")).toBe(0);
});
it("persists under the stats KV key", async () => {
const env = createMockEnv() as unknown as Env;
const kv = env.EMAIL_STORAGE;
await bumpCounters(kv, { feeds_created: 1 });
const raw = (await kv.get(STATS_KEY, { type: "json" })) as {
feeds_created: number;
};
expect(raw.feeds_created).toBe(1);
});
it("getStats reports attachments_enabled based on the toggle", async () => {
const off = createMockEnv() as unknown as Env;
expect((await getStats(off)).attachments_enabled).toBe(false);
const on = createMockEnv({ withR2: true }) as unknown as Env;
expect((await getStats(on)).attachments_enabled).toBe(true);
const disabled = createMockEnv({ withR2: true }) as unknown as Env;
(disabled as any).ATTACHMENTS_ENABLED = "false";
expect((await getStats(disabled)).attachments_enabled).toBe(false);
});
});
describe("getAttachmentBucket", () => {
it("returns the bucket when bound and not disabled", () => {
const env = createMockEnv({ withR2: true }) as unknown as Env;
expect(getAttachmentBucket(env)).toBeDefined();
});
it("returns undefined when no bucket is bound", () => {
const env = createMockEnv() as unknown as Env;
expect(getAttachmentBucket(env)).toBeUndefined();
});
it("returns undefined when explicitly disabled", () => {
const env = createMockEnv({ withR2: true }) as unknown as Env;
(env as any).ATTACHMENTS_ENABLED = "false";
expect(getAttachmentBucket(env)).toBeUndefined();
});
});
describe("storage usage scans", () => {
it("scanR2Usage sums object sizes and counts", async () => {
const bucket = new MockR2();
await bucket.put("a", new Uint8Array(100));
await bucket.put("b", new Uint8Array(250));
const usage = await scanR2Usage(bucket as unknown as R2Bucket);
expect(usage.count).toBe(2);
expect(usage.bytes).toBe(350);
});
it("scanR2Usage returns zeros for an empty bucket", async () => {
const usage = await scanR2Usage(new MockR2() as unknown as R2Bucket);
expect(usage).toEqual({ bytes: 0, count: 0 });
});
it("scanKvUsage estimates KV bytes from stored email sizes", async () => {
const env = createMockEnv() as unknown as Env;
const kv = env.EMAIL_STORAGE;
await kv.put(
FEEDS_LIST_KEY,
JSON.stringify({
feeds: [
{ id: "a", title: "A" },
{ id: "b", title: "B" },
],
}),
);
await kv.put(
"feed:a:metadata",
JSON.stringify({
emails: [
{ key: "k1", size: 100 },
{ key: "k2", size: 50 },
],
}),
);
await kv.put(
"feed:b:metadata",
JSON.stringify({ emails: [{ key: "k3", size: 25 }] }),
);
const usage = await scanKvUsage(kv);
expect(usage.bytes).toBe(175);
});
it("setStorageSnapshot writes the snapshot fields", async () => {
const env = createMockEnv() as unknown as Env;
const kv = env.EMAIL_STORAGE;
await setStorageSnapshot(kv, {
attachments_bytes: 1234,
attachments_count: 5,
kv_bytes_estimated: 678,
});
const counters = await getCounters(kv);
expect(counters.attachments_bytes).toBe(1234);
expect(counters.attachments_count).toBe(5);
expect(counters.kv_bytes_estimated).toBe(678);
expect(counters.storage_scanned_at).toBeDefined();
});
});
+150
View File
@@ -0,0 +1,150 @@
import { Counters, Env, StatsResponse } from "../types";
import { logger } from "../infrastructure/logger";
import { FeedRepository } from "../infrastructure/feed-repository";
import { CountersRepository } from "../infrastructure/counters-repository";
import { WebSubSubscriptionRepository } from "../infrastructure/websub-subscription-repository";
import { FeedId } from "../domain/value-objects/feed-id";
import { getAttachmentBucket } from "../infrastructure/attachments";
const EMPTY_COUNTERS: Counters = {
feeds_created: 0,
feeds_deleted: 0,
emails_received: 0,
emails_rejected: 0,
emails_forwarded: 0,
unsubscribes_sent: 0,
};
export async function getCounters(kv: KVNamespace): Promise<Counters> {
try {
const stored = await new CountersRepository(kv).getRaw();
return { ...EMPTY_COUNTERS, ...(stored || {}) };
} catch (error) {
logger.error("Error reading counters", { error: String(error) });
return { ...EMPTY_COUNTERS };
}
}
/**
* Read-modify-write the counters singleton. KV has no atomic increment, so
* concurrent invocations can lose updates — accepted given KV's eventual
* consistency and this app's low volume (see email-processor.ts storeEmail).
* Never throws: counter failures must not break ingestion or admin flows.
*/
export async function bumpCounters(
kv: KVNamespace,
changes: Partial<Omit<Counters, "first_seen">>,
): Promise<void> {
try {
const current = await getCounters(kv);
current.feeds_created += changes.feeds_created ?? 0;
current.feeds_deleted += changes.feeds_deleted ?? 0;
current.emails_received += changes.emails_received ?? 0;
current.emails_rejected += changes.emails_rejected ?? 0;
current.emails_forwarded += changes.emails_forwarded ?? 0;
current.unsubscribes_sent += changes.unsubscribes_sent ?? 0;
if (changes.last_email_at) current.last_email_at = changes.last_email_at;
if (changes.last_feed_created_at)
current.last_feed_created_at = changes.last_feed_created_at;
if (!current.first_seen) current.first_seen = new Date().toISOString();
await new CountersRepository(kv).put(current);
} catch (error) {
logger.error("Error updating counters", { error: String(error) });
}
}
export async function countKeysByPrefix(
kv: KVNamespace,
prefix: string,
): Promise<number> {
return new FeedRepository(kv).countKeysByPrefix(prefix);
}
export async function getStats(env: Env): Promise<StatsResponse> {
const repo = FeedRepository.from(env);
const [counters, feeds, websubCount] = await Promise.all([
getCounters(env.EMAIL_STORAGE),
repo.listFeeds(),
WebSubSubscriptionRepository.from(env).countKeys(),
]);
return {
...counters,
active_feeds: feeds.length,
websub_subscriptions_active: websubCount,
attachments_enabled: !!getAttachmentBucket(env),
};
}
/** Sum the byte size and object count of every attachment stored in R2. */
export async function scanR2Usage(
bucket: R2Bucket,
): Promise<{ bytes: number; count: number }> {
let bytes = 0;
let count = 0;
let cursor: string | undefined;
try {
do {
const listed = await bucket.list({ cursor });
for (const obj of listed.objects) {
bytes += obj.size;
count += 1;
}
cursor = listed.truncated ? listed.cursor : undefined;
} while (cursor);
} catch (error) {
logger.error("Error scanning R2 usage", { error: String(error) });
}
return { bytes, count };
}
/**
* Estimate KV storage used. KV exposes no size API, so we sum the per-email
* sizes already recorded in each feed's metadata — email bodies dominate KV
* usage. Feed config/websub/stats keys are excluded, so this is a lower-bound
* estimate.
*/
export async function scanKvUsage(kv: KVNamespace): Promise<{ bytes: number }> {
let bytes = 0;
try {
const repo = new FeedRepository(kv);
const feeds = await repo.listFeeds();
for (const feed of feeds) {
const metadata = await repo.getMetadata(FeedId.unchecked(feed.id));
if (!metadata) continue;
for (const email of metadata.emails) {
bytes += email.size ?? 0;
}
}
} catch (error) {
logger.error("Error estimating KV usage", { error: String(error) });
}
return { bytes };
}
/**
* Overwrite the storage-usage snapshot fields on the counters singleton.
* Unlike bumpCounters these are set (not incremented). Never throws.
*/
export async function setStorageSnapshot(
kv: KVNamespace,
snapshot: {
attachments_bytes: number;
attachments_count: number;
kv_bytes_estimated: number;
},
): Promise<void> {
try {
const current = await getCounters(kv);
current.attachments_bytes = snapshot.attachments_bytes;
current.attachments_count = snapshot.attachments_count;
current.kv_bytes_estimated = snapshot.kv_bytes_estimated;
current.storage_scanned_at = new Date().toISOString();
if (!current.first_seen) current.first_seen = new Date().toISOString();
await new CountersRepository(kv).put(current);
} catch (error) {
logger.error("Error writing storage snapshot", { error: String(error) });
}
}
+41
View File
@@ -0,0 +1,41 @@
/** Maximum total size of emails stored per feed (bytes). */
export const FEED_MAX_BYTES = 524288; // 512 KB
/** Cloudflare R2 free tier storage allowance (bytes). */
export const R2_FREE_TIER_BYTES = 10 * 1024 ** 3; // 10 GB
/** Cloudflare KV free tier storage allowance (bytes). */
export const KV_FREE_TIER_BYTES = 1 * 1024 ** 3; // 1 GB
/** Cache TTL for ForwardEmail.net IP list (milliseconds). */
export const FORWARD_EMAIL_IPS_CACHE_TTL_MS = 24 * 60 * 60 * 1000; // 24 hours
/** Admin session cookie max-age (seconds). */
export const ADMIN_COOKIE_MAX_AGE = 60 * 60 * 24 * 7; // 1 week
/** Maximum number of feed items exposed in RSS/Atom responses. */
export const MAX_FEED_ITEMS = 20;
/** Default WebSub lease duration (seconds). */
export const DEFAULT_LEASE_SECONDS = 86400; // 24 hours
/** Maximum WebSub lease duration (seconds). */
export const MAX_LEASE_SECONDS = 30 * 24 * 3600; // 30 days
/** KV key for the global feed list. */
export const FEEDS_LIST_KEY = "feeds:list";
/** KV key for the monitoring counters singleton. */
export const STATS_KEY = "stats:counters";
/** Default TTL for a cached per-domain favicon (seconds). */
export const ICON_TTL_SECONDS = 7 * 24 * 60 * 60; // 1 week
/** Maximum accepted favicon size (bytes); larger responses are rejected. */
export const MAX_ICON_BYTES = 100 * 1024; // 100 KB
/** Timeout for an outbound favicon fetch (milliseconds). */
export const ICON_FETCH_TIMEOUT_MS = 5000;
/** Timeout for an outbound RFC 8058 one-click unsubscribe request (milliseconds). */
export const UNSUBSCRIBE_TIMEOUT_MS = 5000;
+575 -75
View File
@@ -2,79 +2,579 @@
* Collection of common nouns for feed ID generation
*/
export const nouns = [
'actor', 'almond', 'amber', 'anchor', 'angel', 'animal', 'answer', 'apple',
'autumn', 'avenue', 'badge', 'bagel', 'baker', 'ballet', 'bamboo',
'banana', 'basket', 'beach', 'beard', 'beauty', 'beetle', 'berry', 'bicycle',
'bird', 'blanket', 'blossom', 'boat', 'bottle', 'bowl', 'breeze', 'bubble',
'bucket', 'button', 'cabin', 'cactus', 'cafe', 'camera', 'candle', 'candy',
'canvas', 'canyon', 'captain', 'carpet', 'carrot', 'castle', 'cave', 'cellar',
'chair', 'chalk', 'cheese', 'cherry', 'chest', 'chicken', 'chimney',
'circus', 'cliff', 'clock', 'cloud', 'clover', 'coast', 'cobalt', 'cocoa',
'coffee', 'coin', 'comet', 'compass', 'cookie', 'copper', 'coral', 'corner',
'cotton', 'cradle', 'craft', 'creek', 'cricket', 'crown', 'crystal', 'cube',
'cupboard', 'curtain', 'cushion', 'daisy', 'dance', 'date', 'dawn', 'deer',
'desert', 'dew', 'diamond', 'dinner', 'dish', 'doctor', 'dolphin',
'donut', 'door', 'dream', 'dress', 'drink', 'drum', 'duck', 'dusk',
'eagle', 'earth', 'echo', 'emerald', 'engine', 'evening', 'face', 'fairy',
'fall', 'family', 'fan', 'farm', 'feather', 'fence', 'ferry', 'field',
'finger', 'fire', 'fish', 'flag', 'flame', 'flash', 'flavor', 'flight',
'floor', 'flour', 'flower', 'flute', 'fog', 'foil', 'forest', 'fork',
'fox', 'frame', 'friend', 'frog', 'frost', 'fruit', 'garden', 'garlic',
'gate', 'gem', 'gift', 'ginger', 'giraffe', 'glacier', 'glass',
'glitter', 'glove', 'glow', 'goat', 'gold', 'grape', 'grass', 'gravel',
'gravity', 'guitar', 'gum', 'hair', 'hammer', 'hand', 'harbor', 'harp',
'hat', 'hawk', 'heart', 'heath', 'heaven', 'helmet', 'herb', 'hill',
'hippo', 'honey', 'hood', 'horn', 'horse', 'hotel', 'hour', 'house',
'hunter', 'ice', 'icicle', 'idea', 'ink', 'insect', 'iron', 'island',
'ivy', 'jacket', 'jade', 'jam', 'jasmine', 'jelly', 'jewel',
'joke', 'journal', 'journey', 'joy', 'judge', 'jungle', 'kettle', 'key',
'kid', 'kingdom', 'kitchen', 'kite', 'kitten', 'knight',
'lab', 'ladder', 'lake', 'lamb', 'lamp', 'land', 'lantern',
'laptop', 'laugh', 'lava', 'lawn', 'leaf', 'legend', 'lemon', 'letter',
'library', 'light', 'lily', 'lime', 'lion', 'lip', 'lobby', 'lock',
'locket', 'lodge', 'lotus', 'love', 'lunch', 'lyric', 'magic', 'magnet',
'mango', 'maple', 'marble', 'market', 'mask', 'meadow', 'melody', 'melon',
'memory', 'metal', 'meteor', 'milk', 'mint', 'mirror', 'mist', 'mitten',
'moon', 'morning', 'moth', 'motor', 'mountain', 'mouse',
'movie', 'muffin', 'museum', 'music', 'myth', 'napkin', 'nectar', 'needle',
'nest', 'net', 'nickel', 'night', 'nose', 'note', 'novel', 'number',
'nurse', 'nutmeg', 'oasis', 'ocean', 'olive', 'onion', 'opera', 'orange',
'orbit', 'orchard', 'orchid', 'ostrich', 'otter', 'oven', 'owl', 'oxygen',
'oyster', 'page', 'paint', 'palace', 'palm', 'pan', 'pancake', 'panda',
'paper', 'parade', 'parcel', 'park', 'parrot', 'party', 'pasta', 'patch',
'path', 'peach', 'peanut', 'pear', 'pearl', 'pebble', 'pencil', 'penny',
'people', 'pepper', 'petal', 'phone', 'photo', 'piano', 'pickle', 'picture',
'pie', 'pillow', 'pine', 'pink', 'pirate', 'pizza', 'planet',
'plant', 'plum', 'pocket', 'poem', 'poet', 'point', 'pony', 'pool',
'popcorn', 'porch', 'port', 'potato', 'powder', 'prairie', 'pretzel', 'prism',
'prose', 'puppet', 'puppy', 'puzzle', 'quail', 'quartz', 'queen', 'quilt',
'rabbit', 'raccoon', 'radio', 'raft', 'rain', 'rainbow', 'raisin',
'ranch', 'rapids', 'raven', 'ray', 'record', 'reef', 'ribbon', 'rice',
'ring', 'river', 'road', 'robin', 'robot', 'rock', 'rocket', 'rodeo',
'roof', 'room', 'root', 'rope', 'rose', 'ruby', 'rug', 'ruler', 'sage',
'sail', 'salad', 'salmon', 'salt', 'sand', 'sandal', 'sauce', 'saucer',
'scale', 'scarf', 'school', 'sea', 'seed', 'shadow', 'shell', 'ship',
'shirt', 'shoe', 'shop', 'shower', 'shrimp', 'side', 'sign', 'silk',
'silver', 'singer', 'sink', 'sky', 'sled', 'sleet', 'sleigh', 'slice',
'slide', 'slipper', 'slope', 'smoke', 'snail', 'snake', 'snow', 'soap',
'sock', 'soda', 'sofa', 'soil', 'song', 'soup', 'spade', 'spark', 'sparrow',
'spice', 'spider', 'spoon', 'spot', 'spring', 'sprout', 'square', 'squirrel',
'stable', 'stage', 'stair', 'stamp', 'star', 'station', 'steam', 'steel',
'stem', 'stick', 'stone', 'stork', 'storm', 'story', 'stove', 'straw',
'stream', 'street', 'string', 'studio', 'sugar', 'summer', 'sun', 'sunset',
'swan', 'sweater', 'sweets', 'sword', 'table', 'tablet', 'tail', 'talent',
'tangerine', 'tank', 'tea', 'team', 'teapot', 'tear', 'temple', 'tennis',
'tent', 'theater', 'thistle', 'thought', 'thread', 'thunder',
'ticket', 'tide', 'tiger', 'tile', 'time', 'toast', 'toffee', 'tomato',
'tooth', 'top', 'torch', 'tower', 'town', 'toy', 'track',
'train', 'tree', 'triangle', 'trick', 'truck', 'trumpet', 'tulip', 'tunnel',
'turkey', 'turtle', 'twig', 'uncle', 'unicorn', 'universe', 'vacuum', 'valley',
'vanilla', 'vase', 'velvet', 'vessel', 'village', 'vine', 'violin',
'voice', 'volcano', 'voyage', 'wagon', 'walnut', 'waltz', 'water',
'wave', 'wax', 'weather', 'web', 'wedding', 'whale', 'wheat',
'wheel', 'whistle', 'whisper', 'willow', 'wind', 'window', 'wine', 'wing',
'winter', 'wire', 'wish', 'wizard', 'wood', 'wool',
'word', 'world', 'wreath', 'wrist', 'writer', 'xylophone', 'yacht',
'yard', 'yarn', 'year', 'yolk', 'zebra', 'zephyr', 'zinc', 'zipper',
'zone', 'zoo'
"actor",
"almond",
"amber",
"anchor",
"angel",
"animal",
"answer",
"apple",
"autumn",
"avenue",
"badge",
"bagel",
"baker",
"ballet",
"bamboo",
"banana",
"basket",
"beach",
"beard",
"beauty",
"beetle",
"berry",
"bicycle",
"bird",
"blanket",
"blossom",
"boat",
"bottle",
"bowl",
"breeze",
"bubble",
"bucket",
"button",
"cabin",
"cactus",
"cafe",
"camera",
"candle",
"candy",
"canvas",
"canyon",
"captain",
"carpet",
"carrot",
"castle",
"cave",
"cellar",
"chair",
"chalk",
"cheese",
"cherry",
"chest",
"chicken",
"chimney",
"circus",
"cliff",
"clock",
"cloud",
"clover",
"coast",
"cobalt",
"cocoa",
"coffee",
"coin",
"comet",
"compass",
"cookie",
"copper",
"coral",
"corner",
"cotton",
"cradle",
"craft",
"creek",
"cricket",
"crown",
"crystal",
"cube",
"cupboard",
"curtain",
"cushion",
"daisy",
"dance",
"date",
"dawn",
"deer",
"desert",
"dew",
"diamond",
"dinner",
"dish",
"doctor",
"dolphin",
"donut",
"door",
"dream",
"dress",
"drink",
"drum",
"duck",
"dusk",
"eagle",
"earth",
"echo",
"emerald",
"engine",
"evening",
"face",
"fairy",
"fall",
"family",
"fan",
"farm",
"feather",
"fence",
"ferry",
"field",
"finger",
"fire",
"fish",
"flag",
"flame",
"flash",
"flavor",
"flight",
"floor",
"flour",
"flower",
"flute",
"fog",
"foil",
"forest",
"fork",
"fox",
"frame",
"friend",
"frog",
"frost",
"fruit",
"garden",
"garlic",
"gate",
"gem",
"gift",
"ginger",
"giraffe",
"glacier",
"glass",
"glitter",
"glove",
"glow",
"goat",
"gold",
"grape",
"grass",
"gravel",
"gravity",
"guitar",
"gum",
"hair",
"hammer",
"hand",
"harbor",
"harp",
"hat",
"hawk",
"heart",
"heath",
"heaven",
"helmet",
"herb",
"hill",
"hippo",
"honey",
"hood",
"horn",
"horse",
"hotel",
"hour",
"house",
"hunter",
"ice",
"icicle",
"idea",
"ink",
"insect",
"iron",
"island",
"ivy",
"jacket",
"jade",
"jam",
"jasmine",
"jelly",
"jewel",
"joke",
"journal",
"journey",
"joy",
"judge",
"jungle",
"kettle",
"key",
"kid",
"kingdom",
"kitchen",
"kite",
"kitten",
"knight",
"lab",
"ladder",
"lake",
"lamb",
"lamp",
"land",
"lantern",
"laptop",
"laugh",
"lava",
"lawn",
"leaf",
"legend",
"lemon",
"letter",
"library",
"light",
"lily",
"lime",
"lion",
"lip",
"lobby",
"lock",
"locket",
"lodge",
"lotus",
"love",
"lunch",
"lyric",
"magic",
"magnet",
"mango",
"maple",
"marble",
"market",
"mask",
"meadow",
"melody",
"melon",
"memory",
"metal",
"meteor",
"milk",
"mint",
"mirror",
"mist",
"mitten",
"moon",
"morning",
"moth",
"motor",
"mountain",
"mouse",
"movie",
"muffin",
"museum",
"music",
"myth",
"napkin",
"nectar",
"needle",
"nest",
"net",
"nickel",
"night",
"nose",
"note",
"novel",
"number",
"nurse",
"nutmeg",
"oasis",
"ocean",
"olive",
"onion",
"opera",
"orange",
"orbit",
"orchard",
"orchid",
"ostrich",
"otter",
"oven",
"owl",
"oxygen",
"oyster",
"page",
"paint",
"palace",
"palm",
"pan",
"pancake",
"panda",
"paper",
"parade",
"parcel",
"park",
"parrot",
"party",
"pasta",
"patch",
"path",
"peach",
"peanut",
"pear",
"pearl",
"pebble",
"pencil",
"penny",
"people",
"pepper",
"petal",
"phone",
"photo",
"piano",
"pickle",
"picture",
"pie",
"pillow",
"pine",
"pink",
"pirate",
"pizza",
"planet",
"plant",
"plum",
"pocket",
"poem",
"poet",
"point",
"pony",
"pool",
"popcorn",
"porch",
"port",
"potato",
"powder",
"prairie",
"pretzel",
"prism",
"prose",
"puppet",
"puppy",
"puzzle",
"quail",
"quartz",
"queen",
"quilt",
"rabbit",
"raccoon",
"radio",
"raft",
"rain",
"rainbow",
"raisin",
"ranch",
"rapids",
"raven",
"ray",
"record",
"reef",
"ribbon",
"rice",
"ring",
"river",
"road",
"robin",
"robot",
"rock",
"rocket",
"rodeo",
"roof",
"room",
"root",
"rope",
"rose",
"ruby",
"rug",
"ruler",
"sage",
"sail",
"salad",
"salmon",
"salt",
"sand",
"sandal",
"sauce",
"saucer",
"scale",
"scarf",
"school",
"sea",
"seed",
"shadow",
"shell",
"ship",
"shirt",
"shoe",
"shop",
"shower",
"shrimp",
"side",
"sign",
"silk",
"silver",
"singer",
"sink",
"sky",
"sled",
"sleet",
"sleigh",
"slice",
"slide",
"slipper",
"slope",
"smoke",
"snail",
"snake",
"snow",
"soap",
"sock",
"soda",
"sofa",
"soil",
"song",
"soup",
"spade",
"spark",
"sparrow",
"spice",
"spider",
"spoon",
"spot",
"spring",
"sprout",
"square",
"squirrel",
"stable",
"stage",
"stair",
"stamp",
"star",
"station",
"steam",
"steel",
"stem",
"stick",
"stone",
"stork",
"storm",
"story",
"stove",
"straw",
"stream",
"street",
"string",
"studio",
"sugar",
"summer",
"sun",
"sunset",
"swan",
"sweater",
"sweets",
"sword",
"table",
"tablet",
"tail",
"talent",
"tangerine",
"tank",
"tea",
"team",
"teapot",
"tear",
"temple",
"tennis",
"tent",
"theater",
"thistle",
"thought",
"thread",
"thunder",
"ticket",
"tide",
"tiger",
"tile",
"time",
"toast",
"toffee",
"tomato",
"tooth",
"top",
"torch",
"tower",
"town",
"toy",
"track",
"train",
"tree",
"triangle",
"trick",
"truck",
"trumpet",
"tulip",
"tunnel",
"turkey",
"turtle",
"twig",
"uncle",
"unicorn",
"universe",
"vacuum",
"valley",
"vanilla",
"vase",
"velvet",
"vessel",
"village",
"vine",
"violin",
"voice",
"volcano",
"voyage",
"wagon",
"walnut",
"waltz",
"water",
"wave",
"wax",
"weather",
"web",
"wedding",
"whale",
"wheat",
"wheel",
"whistle",
"whisper",
"willow",
"wind",
"window",
"wine",
"wing",
"winter",
"wire",
"wish",
"wizard",
"wood",
"wool",
"word",
"world",
"wreath",
"wrist",
"writer",
"xylophone",
"yacht",
"yard",
"yarn",
"year",
"yolk",
"zebra",
"zephyr",
"zinc",
"zipper",
"zone",
"zoo",
];
+12
View File
@@ -0,0 +1,12 @@
/**
* A source of "now", injected into the domain so aggregates never reach for
* ambient `Date.now()`. Production wires `systemClock`; tests can supply a fixed
* clock for deterministic expiry/timestamp assertions.
*/
export interface Clock {
now(): number;
}
export const systemClock: Clock = {
now: () => Date.now(),
};
+185
View File
@@ -0,0 +1,185 @@
import { describe, it, expect } from "vitest";
import { EmailParser } from "./email-parser";
describe("EmailParser.extractFeedId", () => {
it("extracts a valid feed ID from an email address", () => {
expect(
EmailParser.extractFeedId("[email protected]")?.value,
).toBe("river.castle.42");
});
it("is case-insensitive for the local part", () => {
expect(
EmailParser.extractFeedId("[email protected]")?.value,
).toBe("River.Castle.42");
});
it("returns null for an address with no feed ID format", () => {
expect(EmailParser.extractFeedId("[email protected]")).toBeNull();
});
it("returns null for a plain string without @", () => {
expect(EmailParser.extractFeedId("notanemail")).toBeNull();
});
it("returns null when the numeric suffix is only one digit", () => {
expect(EmailParser.extractFeedId("[email protected]")).toBeNull();
});
it("returns null when the numeric suffix has more than two digits", () => {
expect(
EmailParser.extractFeedId("[email protected]"),
).toBeNull();
});
});
describe("EmailParser.decodeEncodedWords", () => {
it("returns plain text unchanged", () => {
expect(EmailParser.decodeEncodedWords("Hello World")).toBe("Hello World");
});
it("returns empty string for empty input", () => {
expect(EmailParser.decodeEncodedWords("")).toBe("");
});
it("decodes a Base64-encoded word (UTF-8 subject)", () => {
// =?UTF-8?B?SGVsbG8=?= → "Hello"
expect(EmailParser.decodeEncodedWords("=?UTF-8?B?SGVsbG8=?=")).toBe(
"Hello",
);
});
it("decodes a quoted-printable encoded word", () => {
// =?UTF-8?Q?caf=C3=A9?= → "café" (but decodeQuotedPrintable works byte-by-byte)
// Use a simple ASCII QP sequence to stay charset-agnostic in tests
// =?US-ASCII?Q?Hello=20World?= → "Hello World" (=20 → space, _ → space)
expect(EmailParser.decodeEncodedWords("=?US-ASCII?Q?Hello=20World?=")).toBe(
"Hello World",
);
});
it("decodes underscores as spaces in QP encoding", () => {
expect(EmailParser.decodeEncodedWords("=?US-ASCII?Q?Hello_World?=")).toBe(
"Hello World",
);
});
it("leaves unrecognised encoded-word syntax unchanged", () => {
expect(EmailParser.decodeEncodedWords("=?UTF-8?X?something?=")).toBe(
"=?UTF-8?X?something?=",
);
});
});
describe("EmailParser.parseForwardEmailPayload", () => {
it("throws on null payload", () => {
expect(() => EmailParser.parseForwardEmailPayload(null)).toThrow(
"Missing or invalid webhook payload",
);
});
it("throws on undefined payload", () => {
expect(() => EmailParser.parseForwardEmailPayload(undefined)).toThrow();
});
it("parses subject, from, and HTML content", () => {
const payload = {
subject: "Test Subject",
from: { text: "[email protected]" },
html: "<p>Hello</p>",
date: "2024-01-15T10:00:00.000Z",
};
const result = EmailParser.parseForwardEmailPayload(payload);
expect(result.subject).toBe("Test Subject");
expect(result.from).toBe("[email protected]");
expect(result.content).toBe("<p>Hello</p>");
expect(result.receivedAt).toBe(
new Date("2024-01-15T10:00:00.000Z").getTime(),
);
});
it("prefers HTML content over plain text", () => {
const payload = {
from: { text: "[email protected]" },
html: "<b>HTML</b>",
text: "Plain",
};
expect(EmailParser.parseForwardEmailPayload(payload).content).toBe(
"<b>HTML</b>",
);
});
it("falls back to plain text when HTML is absent", () => {
const payload = {
from: { text: "[email protected]" },
text: "Plain text",
};
expect(EmailParser.parseForwardEmailPayload(payload).content).toBe(
"Plain text",
);
});
it("uses structured from.value when from.text is absent", () => {
const payload = {
from: {
value: [{ name: "Alice", address: "[email protected]" }],
},
html: "",
};
const result = EmailParser.parseForwardEmailPayload(payload);
expect(result.from).toBe("Alice <[email protected]>");
});
it("falls back to Unknown Sender when from is absent", () => {
const result = EmailParser.parseForwardEmailPayload({ html: "" });
expect(result.from).toBe("Unknown Sender");
});
it("uses Date.now() when date field is absent", () => {
const before = Date.now();
const result = EmailParser.parseForwardEmailPayload({
from: { text: "[email protected]" },
});
const after = Date.now();
expect(result.receivedAt).toBeGreaterThanOrEqual(before);
expect(result.receivedAt).toBeLessThanOrEqual(after);
});
it("defaults subject to 'No Subject' when absent", () => {
const result = EmailParser.parseForwardEmailPayload({
from: { text: "[email protected]" },
});
expect(result.subject).toBe("No Subject");
});
it("extracts headers from headerLines array", () => {
const payload = {
from: { text: "[email protected]" },
headerLines: [
{ key: "X-Custom", line: "X-Custom: my-value" },
{ key: "List-ID", line: "List-ID: <list.example.com>" },
],
};
const result = EmailParser.parseForwardEmailPayload(payload);
expect(result.headers["x-custom"]).toBe("my-value");
expect(result.headers["list-id"]).toBe("<list.example.com>");
});
it("extracts headers from raw headers string", () => {
const payload = {
from: { text: "[email protected]" },
headers: "X-Foo: bar\r\nX-Baz: qux",
};
const result = EmailParser.parseForwardEmailPayload(payload);
expect(result.headers["x-foo"]).toBe("bar");
expect(result.headers["x-baz"]).toBe("qux");
});
it("decodes RFC 2047 encoded-word subjects", () => {
const payload = {
from: { text: "[email protected]" },
subject: "=?UTF-8?B?SGVsbG8=?=",
};
expect(EmailParser.parseForwardEmailPayload(payload).subject).toBe("Hello");
});
});
+88
View File
@@ -0,0 +1,88 @@
import { EmailData } from "../types";
import { FeedId } from "./value-objects/feed-id";
export class EmailParser {
/**
* Extract the feed id from an inbound recipient address. Returns a validated
* `FeedId` value object (not a raw string) so the most untrusted input in the
* system — an address typed by a sender — is guarded at the parse boundary and
* never needs `FeedId.unchecked` downstream.
*/
static extractFeedId(emailAddress: string): FeedId | null {
return FeedId.parse(emailAddress);
}
// eslint-disable-next-line @typescript-eslint/no-explicit-any
static parseForwardEmailPayload(payload: any): EmailData {
if (!payload) {
throw new Error("Missing or invalid webhook payload");
}
const fromAddress =
payload.from?.text ||
(payload.from?.value?.[0]?.address
? `${payload.from.value[0].name || ""} <${payload.from.value[0].address}>`
: "Unknown Sender");
const subject = this.decodeEncodedWords(payload.subject || "No Subject");
const content = payload.html || payload.text || "";
return {
subject,
from: fromAddress,
content,
receivedAt: payload.date ? new Date(payload.date).getTime() : Date.now(),
headers: this.extractHeaders(payload),
};
}
// eslint-disable-next-line @typescript-eslint/no-explicit-any
private static extractHeaders(payload: any): Record<string, string> {
const headers: Record<string, string> = {};
if (payload.headerLines && Array.isArray(payload.headerLines)) {
payload.headerLines.forEach((h: { key: string; line: string }) => {
const key = h.key.toLowerCase();
const value = h.line
.replace(new RegExp(`^${h.key}:\\s*`, "i"), "")
.trim();
headers[key] = value;
});
} else if (typeof payload.headers === "string") {
payload.headers.split(/\r?\n/).forEach((line: string) => {
const match = line.match(/^([^:]+):\s*(.*)$/);
if (match) {
headers[match[1].toLowerCase()] = match[2];
}
});
}
return headers;
}
static decodeEncodedWords(text: string): string {
if (!text) return "";
return text.replace(
/=\?([^?]+)\?([BQ])\?([^?]+)\?=/gi,
(_, charset, encoding, text) => {
if (encoding.toUpperCase() === "B") {
try {
return atob(text);
} catch {
return text;
}
} else if (encoding.toUpperCase() === "Q") {
return this.decodeQuotedPrintable(text.replace(/_/g, " "));
}
return text;
},
);
}
private static decodeQuotedPrintable(text: string): string {
return text.replace(/=([0-9A-F]{2})/gi, (_, hex) => {
return String.fromCharCode(parseInt(hex, 16));
});
}
}
+18
View File
@@ -0,0 +1,18 @@
import { FeedId } from "./value-objects/feed-id";
/**
* Domain events the Feed aggregate records when it mutates. They describe *what
* happened* in business terms and carry their own `feedId`, so the application
* dispatcher can route side effects (counters, WebSub pings, favicon caching)
* without the caller threading the id back in. This keeps the aggregate ignorant
* of infrastructure and the orchestration code free of scattered, inline effects.
*
* Only mutations that currently have side effects emit events — feed creation
* and email ingestion. Edits and removals carry no side effect, so they emit
* nothing. Side effects that don't flow through the aggregate (a rejected email,
* a feed deletion that bypasses the aggregate, bulk admin operations) stay
* outside this mechanism by design — they have no aggregate event to ride on.
*/
export type FeedEvent =
| { type: "FeedCreated"; feedId: FeedId }
| { type: "EmailIngested"; feedId: FeedId; iconDomain?: string };
+40
View File
@@ -0,0 +1,40 @@
import { FEEDS_LIST_KEY, STATS_KEY } from "../config/constants";
/**
* The KV key schema, in one pure place. Every repository builds its keys here so
* the wire format lives in a single module — never inline a `feed:`/`icon:`/
* `websub:` string elsewhere. Strings are byte-identical to the original schema;
* changing them would require migrating live KV data.
*/
const WEBSUB_PREFIX = "websub:subs:";
export const feedKeys = {
config: (feedId: string): string => `feed:${feedId}:config`,
metadata: (feedId: string): string => `feed:${feedId}:metadata`,
/** Prefix covering every key owned by a feed (config, metadata, emails). */
feedPrefix: (feedId: string): string => `feed:${feedId}:`,
/** Mint a fresh, time-ordered email key. Call once and reuse the result. */
newEmail: (feedId: string): string => `feed:${feedId}:${Date.now()}`,
/** KV key for a domain's cached favicon (shared across feeds). */
icon: (domain: string): string => `icon:${domain}`,
websub: (feedId: string): string => `${WEBSUB_PREFIX}${feedId}`,
/** Prefix matching every per-feed WebSub subscription key. */
websubPrefix: (): string => WEBSUB_PREFIX,
/** True when `key` is an email entry (not the feed's config/metadata key). */
isEmail: (feedId: string, key: string): boolean => {
const suffix = key.slice(feedKeys.feedPrefix(feedId).length);
return suffix !== "config" && suffix !== "metadata";
},
/** Recover the feed id embedded in an email key (`feed:<id>:<ts>`). */
feedIdFromEmail: (key: string): string => key.split(":")[1],
} as const;
export { FEEDS_LIST_KEY, STATS_KEY };
+20
View File
@@ -0,0 +1,20 @@
/**
* The Feed aggregate's internal config state, in domain (camelCase) vocabulary.
* This is deliberately NOT the persistence shape: the snake_case `FeedConfig`
* DTO is an infrastructure concern, and the translation between the two lives in
* `infrastructure/feed-mapper.ts`. The domain never speaks the storage dialect.
*
* `expiresAt` is an absolute instant (epoch ms) already resolved from a
* `Lifetime`; the aggregate stores the resolved value, not the policy.
*/
export interface FeedState {
title: string;
description?: string;
language: string;
author?: string;
allowedSenders: string[];
blockedSenders: string[];
createdAt: number;
updatedAt?: number;
expiresAt?: number;
}
+246
View File
@@ -0,0 +1,246 @@
import { describe, it, expect } from "vitest";
import { createMockEnv } from "../test/setup";
import { Feed, CreateFeedInput } from "./feed.aggregate";
import { FeedRepository } from "../infrastructure/feed-repository";
import { FeedId } from "./value-objects/feed-id";
import { Lifetime } from "./value-objects/lifetime";
import { FeedState } from "./feed-state";
import { Clock } from "./clock";
import type { Env, EmailMetadata } from "../types";
const FID = FeedId.unchecked("a.b.42");
const mockEnv = () => createMockEnv() as unknown as Env;
const fixedClock = (now: number): Clock => ({ now: () => now });
const createInput = (
overrides: Partial<CreateFeedInput> = {},
): CreateFeedInput => ({
title: "News",
language: "en",
allowedSenders: [],
blockedSenders: [],
...overrides,
});
const state = (overrides: Partial<FeedState> = {}): FeedState => ({
title: "T",
language: "en",
allowedSenders: [],
blockedSenders: [],
createdAt: 0,
...overrides,
});
const entry = (overrides: Partial<EmailMetadata> = {}): EmailMetadata => ({
key: "feed:a.b.42:1",
subject: "Hello",
receivedAt: 1,
size: 10,
...overrides,
});
describe("Feed.create", () => {
it("builds a config with an empty email index and no expiry by default", () => {
const feed = Feed.create(FID, createInput());
expect(feed.id.value).toBe("a.b.42");
expect(feed.title).toBe("News");
expect(feed.expiresAt).toBeUndefined();
expect(feed.emails).toEqual([]);
});
it("resolves expiry from the supplied lifetime using the injected clock", () => {
const NOW = 1_000_000;
const feed = Feed.create(FID, createInput(), {
clock: fixedClock(NOW),
lifetime: Lifetime.ofHours(2),
});
expect(feed.createdAt).toBe(NOW);
expect(feed.updatedAt).toBe(NOW);
expect(feed.expiresAt).toBe(NOW + 2 * 3_600_000);
});
it("trusts only deps.lifetime, not the client lifetimeHours field", () => {
// The aggregate no longer parses lifetime policy: the application resolves
// the effective Lifetime (env override etc.) and hands it in.
const feed = Feed.create(FID, createInput({ lifetimeHours: 9999 }));
expect(feed.expiresAt).toBeUndefined();
});
it("treats a non-positive lifetime as no expiry", () => {
expect(
Feed.create(FID, createInput(), { lifetime: Lifetime.ofHours(0) })
.expiresAt,
).toBeUndefined();
expect(
Feed.create(FID, createInput(), { lifetime: Lifetime.ofHours(-5) })
.expiresAt,
).toBeUndefined();
});
});
describe("Feed.isExpired / accepts", () => {
it("reports expiry against the configured instant", () => {
const feed = Feed.reconstitute(FID, state({ expiresAt: 100 }), {
emails: [],
});
expect(feed.isExpired(50)).toBe(false);
expect(feed.isExpired(150)).toBe(true);
});
it("uses the injected clock when no instant is supplied", () => {
const feed = Feed.reconstitute(
FID,
state({ expiresAt: 100 }),
{ emails: [] },
fixedClock(150),
);
expect(feed.isExpired()).toBe(true);
});
it("applies the sender policy", () => {
const feed = Feed.reconstitute(
FID,
state({ allowedSenders: ["[email protected]"] }),
{ emails: [] },
);
expect(feed.accepts(["[email protected]"])).toBe("accepted");
expect(feed.accepts(["[email protected]"])).toBe("blocked");
});
});
describe("Feed.edit", () => {
it("recomputes expiry only when a lifetime is supplied", () => {
const NOW = 5_000_000;
const FUTURE = NOW + 10 * 3_600_000;
const feed = Feed.reconstitute(
FID,
state({ expiresAt: FUTURE }),
{ emails: [] },
fixedClock(NOW),
);
feed.edit({ title: "T2" }); // no lifetime ⇒ expiry preserved
expect(feed.expiresAt).toBe(FUTURE);
expect(feed.updatedAt).toBe(NOW);
feed.edit({ title: "T3" }, { lifetime: Lifetime.ofHours(1) });
expect(feed.expiresAt).toBe(NOW + 3_600_000);
});
it("refuses to edit an already-expired feed", () => {
const feed = Feed.reconstitute(
FID,
state({ expiresAt: 100 }),
{ emails: [] },
fixedClock(200),
);
expect(feed.edit({ title: "X" }).status).toBe("expired");
});
});
describe("Feed.ingest", () => {
it("prepends the entry, tracks icon/unsub and trims to the byte budget", () => {
const feed = Feed.reconstitute(FID, state(), {
emails: [entry({ key: "old", size: 400 })],
});
const { dropped } = feed.ingest(entry({ key: "new", size: 400 }), {
maxBytes: 500,
iconDomain: "example.com",
unsub: { senderKey: "[email protected]", url: "https://u/1" },
});
expect(feed.emails[0].key).toBe("new");
expect(feed.iconDomain).toBe("example.com");
expect(feed.unsubscribeUrls()).toEqual({
"[email protected]": "https://u/1",
});
expect(dropped.map((e) => e.key)).toEqual(["old"]);
expect(feed.emails.map((e) => e.key)).toEqual(["new"]);
});
it("always keeps the just-ingested entry, even when it alone is oversized", () => {
const feed = Feed.reconstitute(FID, state(), { emails: [] });
const { dropped } = feed.ingest(entry({ key: "huge", size: 999 }), {
maxBytes: 1,
});
expect(dropped).toEqual([]);
expect(feed.emails.map((e) => e.key)).toEqual(["huge"]);
});
});
describe("Feed.removeEmails", () => {
it("drops matching keys and returns the removed entries", () => {
const feed = Feed.reconstitute(FID, state(), {
emails: [
entry({ key: "k1" }),
entry({ key: "k2" }),
entry({ key: "k3" }),
],
});
const { removed } = feed.removeEmails(["k1", "k3", "missing"]);
expect(removed.map((e) => e.key).sort()).toEqual(["k1", "k3"]);
expect(feed.emails.map((e) => e.key)).toEqual(["k2"]);
});
});
describe("Feed events", () => {
it("records FeedCreated on create and drains it once", () => {
const feed = Feed.create(FID, createInput());
expect(feed.pullEvents()).toEqual([{ type: "FeedCreated", feedId: FID }]);
// Draining clears: a second pull is empty.
expect(feed.pullEvents()).toEqual([]);
});
it("records EmailIngested (with icon domain) on ingest", () => {
const feed = Feed.reconstitute(FID, state(), { emails: [] });
feed.ingest(entry({ key: "k" }), {
maxBytes: 1_000_000,
iconDomain: "example.com",
});
expect(feed.pullEvents()).toEqual([
{ type: "EmailIngested", feedId: FID, iconDomain: "example.com" },
]);
});
it("emits no events for edit / removeEmails", () => {
const feed = Feed.reconstitute(
FID,
state({ expiresAt: 9_999_999_999 }),
{ emails: [entry({ key: "k1" })] },
fixedClock(1000),
);
feed.edit({ title: "X" });
feed.edit({ description: "Y" });
feed.removeEmails(["k1"]);
expect(feed.pullEvents()).toEqual([]);
});
});
describe("FeedRepository.load / save round-trip", () => {
it("persists a created feed and reflects later mutations", async () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
const created = Feed.create(FID, createInput({ title: "Round" }));
await repo.save(created);
const loaded = await repo.load(FID);
expect(loaded).not.toBeNull();
expect(loaded!.title).toBe("Round");
loaded!.ingest(entry({ key: "feed:a.b.42:1" }), { maxBytes: 1_000_000 });
await repo.saveMetadata(loaded!);
const reloaded = await repo.load(FID);
expect(reloaded!.emails.map((e) => e.key)).toEqual(["feed:a.b.42:1"]);
});
it("returns null when the feed has no config", async () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
expect(await repo.load(FeedId.unchecked("missing"))).toBeNull();
});
});
+303
View File
@@ -0,0 +1,303 @@
import { FeedMetadata, EmailMetadata } from "../types";
import { FeedState } from "./feed-state";
import { FeedId } from "./value-objects/feed-id";
import { Lifetime } from "./value-objects/lifetime";
import { SenderPolicy, SenderDecision } from "./value-objects/sender-policy";
import { Clock, systemClock } from "./clock";
import { FeedEvent } from "./events";
export interface CreateFeedInput {
title: string;
description?: string;
language: string;
allowedSenders: string[];
blockedSenders: string[];
/** Raw client-requested lifetime; the application resolves it into a `Lifetime`. */
lifetimeHours?: number;
}
export interface UpdateFeedInput {
title?: string;
description?: string;
language?: string;
allowedSenders?: string[];
blockedSenders?: string[];
lifetimeHours?: number;
}
/**
* Dependencies the aggregate needs from the outside but must not reach for
* itself: a clock (never ambient `Date.now()`) and an already-resolved
* `Lifetime`. The application layer decides the lifetime — parsing env config and
* applying any server-side `FEED_TTL_HOURS` override — and hands the VO in.
*/
export interface CreateFeedDeps {
clock?: Clock;
/** Effective lifetime, already resolved by the application. */
lifetime?: Lifetime;
}
export interface EditFeedDeps {
/**
* Effective lifetime, already resolved by the application. Its *presence* means
* "recompute expiry"; its absence preserves the current expiry — which covers
* the dashboard's title/description quick-edit.
*/
lifetime?: Lifetime;
}
export interface IngestOptions {
maxBytes: number;
iconDomain?: string;
/** RFC 8058 one-click unsubscribe link, keyed by the sending newsletter. */
unsub?: { senderKey: string; url: string };
}
/**
* The Feed aggregate: the consistency boundary around a feed's config and the
* metadata index of its emails. All mutations to either go through a method
* here so the invariants (expiry policy, sender policy, byte budget) live in one
* place. Email bodies are large blobs referenced by `metadata.emails[].key` and
* deliberately sit *outside* the aggregate — the caller flushes them alongside
* `FeedRepository.save`/`saveMetadata`.
*
* Its config is held as domain `FeedState` (camelCase), never the snake_case
* persistence DTO — `FeedRepository` translates via `feed-mapper.ts`. I/O-free
* and time-free: load and persist through the repository; time comes from an
* injected `Clock`. KV has no multi-key transaction, so a future Durable Object
* keyed by feed id would wrap load→mutate→save to serialise concurrent writers
* (see email-processor.ts).
*/
export class Feed {
private readonly _events: FeedEvent[] = [];
private constructor(
readonly id: FeedId,
private _state: FeedState,
private _metadata: FeedMetadata,
private readonly clock: Clock,
) {}
/** Mint a brand-new feed with an empty email index. */
static create(
id: FeedId,
input: CreateFeedInput,
deps: CreateFeedDeps = {},
): Feed {
const clock = deps.clock ?? systemClock;
const now = clock.now();
const expiresAt = (deps.lifetime ?? Lifetime.never).resolveExpiry(now);
const state: FeedState = {
title: input.title,
description: input.description,
language: input.language,
allowedSenders: input.allowedSenders,
blockedSenders: input.blockedSenders,
createdAt: now,
updatedAt: now,
expiresAt,
};
const feed = new Feed(id, state, { emails: [] }, clock);
feed._events.push({ type: "FeedCreated", feedId: id });
return feed;
}
/** Rebuild an aggregate from persisted (already-mapped) domain state. */
static reconstitute(
id: FeedId,
state: FeedState,
metadata: FeedMetadata,
clock: Clock = systemClock,
): Feed {
return new Feed(id, state, metadata, clock);
}
// ── Intention-revealing reads ─────────────────────────────────────────────
// The aggregate exposes named fields and copies of its collections, never the
// raw `state`/`metadata` objects — a shallow `Readonly<…>` would still let a
// caller mutate the arrays inside. Persistence reads `state()` /
// `toMetadataSnapshot()`; the mapper derives the DTOs.
get title(): string {
return this._state.title;
}
get description(): string | undefined {
return this._state.description;
}
get language(): string {
return this._state.language;
}
get createdAt(): number {
return this._state.createdAt;
}
get updatedAt(): number | undefined {
return this._state.updatedAt;
}
get expiresAt(): number | undefined {
return this._state.expiresAt;
}
get iconDomain(): string | undefined {
return this._metadata.iconDomain;
}
allowedSenders(): string[] {
return [...this._state.allowedSenders];
}
blockedSenders(): string[] {
return [...this._state.blockedSenders];
}
/** A copy of the email index — mutating it never touches aggregate state. */
get emails(): readonly EmailMetadata[] {
return [...this._metadata.emails];
}
/** Per-sender one-click unsubscribe links (copy). */
unsubscribeUrls(): Record<string, string> {
return { ...(this._metadata.unsubscribe ?? {}) };
}
// ── Persistence snapshots (repository-only) ───────────────────────────────
/** A copy of the domain config state for the repository to map + persist. */
state(): FeedState {
return {
...this._state,
allowedSenders: [...this._state.allowedSenders],
blockedSenders: [...this._state.blockedSenders],
};
}
/** A serialisable copy of the email index for the repository to persist. */
toMetadataSnapshot(): FeedMetadata {
return { ...this._metadata, emails: [...this._metadata.emails] };
}
/**
* Drain the domain events recorded since the last pull. The application layer
* calls this after persisting and feeds them to a dispatcher that runs the
* side effects (counters, WebSub, favicon). Clearing on read keeps a long-lived
* aggregate from re-emitting.
*/
pullEvents(): FeedEvent[] {
return this._events.splice(0, this._events.length);
}
isExpired(now: number = this.clock.now()): boolean {
// The shared `isExpired` predicate (domain/feed.ts) lives on the read path
// and speaks the persistence DTO; the aggregate checks its own domain state.
return this._state.expiresAt !== undefined && this._state.expiresAt <= now;
}
accepts(senders: string[]): SenderDecision {
return SenderPolicy.fromLists(
this._state.allowedSenders,
this._state.blockedSenders,
).decide(senders);
}
/**
* Add an email to the front of the index, refresh the icon domain and the
* per-sender unsubscribe link, then trim the oldest entries back under the
* byte budget. Returns the dropped entries so the caller can purge their
* bodies/attachments.
*/
ingest(
entry: EmailMetadata,
opts: IngestOptions,
): { dropped: EmailMetadata[] } {
this._metadata.emails.unshift(entry);
if (opts.iconDomain) {
this._metadata.iconDomain = opts.iconDomain;
}
if (opts.unsub) {
this._metadata.unsubscribe = {
...(this._metadata.unsubscribe ?? {}),
[opts.unsub.senderKey]: opts.unsub.url,
};
}
this._events.push({
type: "EmailIngested",
feedId: this.id,
iconDomain: opts.iconDomain,
});
return this.trimToByteBudget(opts.maxBytes);
}
/**
* Enforce the per-feed byte budget by dropping the oldest emails (from the
* tail of the index) until the total fits, always keeping at least one entry.
* Returns the dropped entries so the caller can purge their KV/R2 storage.
*/
private trimToByteBudget(maxBytes: number): { dropped: EmailMetadata[] } {
const emails = this._metadata.emails;
let totalSize = emails.reduce((sum, e) => sum + (e.size ?? 0), 0);
const dropped: EmailMetadata[] = [];
while (totalSize > maxBytes && emails.length > 1) {
const entry = emails.pop()!;
totalSize -= entry.size ?? 0;
dropped.push(entry);
}
return { dropped };
}
/**
* Drop the given email keys from the index. Returns the removed entries so the
* caller can purge their bodies/attachments.
*/
removeEmails(keys: string[]): { removed: EmailMetadata[] } {
const target = new Set(keys);
const removed: EmailMetadata[] = [];
const kept: EmailMetadata[] = [];
for (const entry of this._metadata.emails) {
(target.has(entry.key) ? removed : kept).push(entry);
}
this._metadata.emails = kept;
return { removed };
}
/**
* The single edit path. Apply the patch (only the fields it carries) and
* recompute expiry when the application supplies a `Lifetime` — an absent
* lifetime preserves the current expiry, which covers the dashboard's
* title/description quick-edit. Rejects an already-expired feed without
* mutating it, so a quick-edit can no more touch an expired feed than a full
* edit can.
*/
edit(
patch: UpdateFeedInput,
deps: EditFeedDeps = {},
): { status: "ok" | "expired" } {
if (this.isExpired()) return { status: "expired" };
const now = this.clock.now();
const expiresAt = deps.lifetime
? deps.lifetime.resolveExpiry(now)
: this._state.expiresAt;
if (patch.title !== undefined) this._state.title = patch.title;
if (patch.description !== undefined) {
this._state.description = patch.description;
}
if (patch.language !== undefined) this._state.language = patch.language;
if (patch.allowedSenders !== undefined) {
this._state.allowedSenders = patch.allowedSenders;
}
if (patch.blockedSenders !== undefined) {
this._state.blockedSenders = patch.blockedSenders;
}
this._state.updatedAt = now;
this._state.expiresAt = expiresAt;
return { status: "ok" };
}
}
+14
View File
@@ -0,0 +1,14 @@
import { describe, it, expect } from "vitest";
import { isExpired } from "./feed";
describe("isExpired", () => {
it("is false when no expiry is set", () => {
expect(isExpired({ expires_at: undefined }, 1000)).toBe(false);
});
it("is true at or past the expiry instant", () => {
expect(isExpired({ expires_at: 1000 }, 1000)).toBe(true);
expect(isExpired({ expires_at: 1000 }, 1001)).toBe(true);
expect(isExpired({ expires_at: 1000 }, 999)).toBe(false);
});
});
+18
View File
@@ -0,0 +1,18 @@
import { FeedConfig } from "../types";
/**
* The expiry predicate, shared between the Feed aggregate and the read-model
* routes (rss/atom/entries) that render from a config snapshot without loading
* the aggregate. This is the *only* feed invariant that lives outside the
* aggregate, precisely because the hot read path bypasses it.
*
* `now` defaults to the wall clock for convenience at the HTTP edge; the
* aggregate always passes its injected clock so its own behaviour stays
* deterministic.
*/
export function isExpired(
config: Pick<FeedConfig, "expires_at">,
now: number = Date.now(),
): boolean {
return config.expires_at !== undefined && config.expires_at <= now;
}
+8
View File
@@ -0,0 +1,8 @@
/** Human-readable byte size (B / KB / MB / GB). */
export function formatBytes(bytes: number): string {
if (bytes < 1024) return `${bytes} B`;
if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
if (bytes < 1024 * 1024 * 1024)
return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
return `${(bytes / (1024 * 1024 * 1024)).toFixed(2)} GB`;
}
+25
View File
@@ -0,0 +1,25 @@
import { describe, it, expect } from "vitest";
import { Domain } from "./domain";
describe("Domain", () => {
it("normalises case and whitespace", () => {
expect(Domain.parse(" Example.COM ")?.value).toBe("example.com");
});
it("strips a leading @ and trailing dots", () => {
expect(Domain.parse("@example.com")?.value).toBe("example.com");
expect(Domain.parse("example.com.")?.value).toBe("example.com");
});
it("returns null for empty input", () => {
expect(Domain.parse("")).toBeNull();
expect(Domain.parse("@")).toBeNull();
});
it("compares by normalised value", () => {
expect(
Domain.parse("Example.com")!.matches(Domain.parse("example.com")!),
).toBe(true);
expect(Domain.parse("a.com")!.matches(Domain.parse("b.com")!)).toBe(false);
});
});
+24
View File
@@ -0,0 +1,24 @@
/**
* A normalised DNS domain (lowercased, no leading `@`, no trailing dots).
* Accepts both bare (`example.com`) and allowlist-style (`@example.com`) input.
*/
export class Domain {
private constructor(readonly value: string) {}
static parse(raw: string): Domain | null {
const normalized = raw
.trim()
.toLowerCase()
.replace(/^@+/, "")
.replace(/\.+$/, "");
return normalized ? new Domain(normalized) : null;
}
matches(other: Domain): boolean {
return this.value === other.value;
}
toString(): string {
return this.value;
}
}
@@ -0,0 +1,33 @@
import { describe, it, expect } from "vitest";
import { EmailAddress } from "./email-address";
describe("EmailAddress", () => {
it("parses a bare address and normalises it", () => {
const email = EmailAddress.parse("[email protected]")!;
expect(email.normalized).toBe("[email protected]");
expect(email.domain.value).toBe("example.com");
});
it("parses a display form (Name <addr>)", () => {
const email = EmailAddress.parse("GitHub <[email protected]>")!;
expect(email.normalized).toBe("[email protected]");
expect(email.domain.value).toBe("github.com");
});
it("strips a trailing dot from the domain", () => {
expect(EmailAddress.parse("[email protected].")?.domain.value).toBe(
"example.com",
);
});
it("returns null when there is no address", () => {
expect(EmailAddress.parse("not an email")).toBeNull();
expect(EmailAddress.parse("")).toBeNull();
});
it("derives the sender site base URL from the domain", () => {
expect(EmailAddress.parse("News <[email protected]>")?.siteBaseUrl()).toBe(
"https://example.com/",
);
});
});
+35
View File
@@ -0,0 +1,35 @@
import { Domain } from "./domain";
/**
* A normalised email address. `parse` accepts a bare address (`[email protected]`) or a
* display form (`Name <[email protected]>`), lowercasing the local part and normalising
* the domain. Returns null when no plausible address can be found.
*/
export class EmailAddress {
private constructor(
readonly normalized: string,
readonly domain: Domain,
) {}
static parse(raw: string): EmailAddress | null {
const match = raw.match(/([^\s<>@]+)@([^\s<>@]+)/);
if (!match) return null;
const domain = Domain.parse(match[2]);
if (!domain) return null;
const local = match[1].trim().toLowerCase();
return new EmailAddress(`${local}@${domain.value}`, domain);
}
/**
* Best-effort website origin implied by the sender's domain
* (e.g. `https://example.com/`). Used to absolutize relative links in the
* email body — the sender's site is the only base we can infer.
*/
siteBaseUrl(): string {
return `https://${this.domain.value}/`;
}
toString(): string {
return this.normalized;
}
}
+36
View File
@@ -0,0 +1,36 @@
import { describe, it, expect } from "vitest";
import { FeedId } from "./feed-id";
describe("FeedId.parse", () => {
it("extracts the feed id from an inbound address", () => {
expect(FeedId.parse("[email protected]")?.value).toBe(
"river.castle.42",
);
});
it("preserves the original casing of the local part", () => {
expect(FeedId.parse("[email protected]")?.value).toBe(
"River.Castle.42",
);
});
it("rejects malformed feed ids", () => {
expect(FeedId.parse("[email protected]")).toBeNull();
expect(FeedId.parse("notanemail")).toBeNull();
expect(FeedId.parse("[email protected]")).toBeNull();
expect(FeedId.parse("[email protected]")).toBeNull();
});
});
describe("FeedId.generate", () => {
it("produces the noun.noun.NN format", () => {
for (let i = 0; i < 50; i++) {
expect(FeedId.generate().value).toMatch(/^[a-z]+\.[a-z]+\.\d{2}$/);
}
});
it("round-trips through parse from an address", () => {
const id = FeedId.generate();
expect(FeedId.parse(`${id.value}@example.com`)?.value).toBe(id.value);
});
});
+40
View File
@@ -0,0 +1,40 @@
import { nouns } from "../../data/nouns";
// Feed IDs are noun1.noun2.XY (two lowercase nouns + a 2-digit suffix).
const FEED_ID_IN_ADDRESS = /^([a-z]+\.[a-z]+\.\d{2})@/i;
/**
* A feed identifier. `parse` pulls it from the local part of an inbound email
* address; `generate` mints a fresh one. The original casing is preserved.
*/
export class FeedId {
private constructor(readonly value: string) {}
/** Extract the feed id from an inbound address (`noun.noun.NN@domain`). */
static parse(emailAddress: string): FeedId | null {
const match = emailAddress.match(FEED_ID_IN_ADDRESS);
return match ? new FeedId(match[1]) : null;
}
/**
* Wrap a string as a FeedId WITHOUT revalidating it. The caller asserts the id
* originated from our own minting — a route param echoing a stored id, a
* `feeds:list` entry, or an email/KV key. The name is deliberately blunt: a
* wrong id is not rejected here, it simply misses in KV and 404s downstream.
* Untrusted external input (an inbound address) must go through `parse` instead.
*/
static unchecked(value: string): FeedId {
return new FeedId(value);
}
static generate(): FeedId {
const noun1 = nouns[Math.floor(Math.random() * nouns.length)];
const noun2 = nouns[Math.floor(Math.random() * nouns.length)];
const number = Math.floor(Math.random() * 90) + 10;
return new FeedId(`${noun1}.${noun2}.${number}`);
}
toString(): string {
return this.value;
}
}
+22
View File
@@ -0,0 +1,22 @@
import { describe, it, expect } from "vitest";
import { Lifetime } from "./lifetime";
const NOW = 1_000_000;
const HOUR = 3_600_000;
describe("Lifetime", () => {
it("resolves a positive lifetime to an absolute expiry", () => {
expect(Lifetime.ofHours(2).resolveExpiry(NOW)).toBe(NOW + 2 * HOUR);
});
it("never expires for Lifetime.never", () => {
expect(Lifetime.never.resolveExpiry(NOW)).toBeUndefined();
});
it("treats non-positive or non-finite hours as no expiry", () => {
expect(Lifetime.ofHours(0).resolveExpiry(NOW)).toBeUndefined();
expect(Lifetime.ofHours(-5).resolveExpiry(NOW)).toBeUndefined();
expect(Lifetime.ofHours(NaN).resolveExpiry(NOW)).toBeUndefined();
expect(Lifetime.ofHours(Infinity).resolveExpiry(NOW)).toBeUndefined();
});
});
+32
View File
@@ -0,0 +1,32 @@
const HOUR_MS = 3_600_000;
/**
* A feed's lifetime as a value object: either a positive number of hours or
* "never". `resolveExpiry(now)` turns it into an absolute `expires_at` instant
* (or undefined for a feed that never expires).
*
* Which lifetime applies — client request vs. server-side `FEED_TTL_HOURS`
* override — is the application layer's policy; it builds the VO and hands it to
* the aggregate. The aggregate never parses env config or reaches for a clock to
* compute expiry itself.
*/
export class Lifetime {
private constructor(private readonly hours: number | undefined) {}
/** A finite, positive lifetime. Non-positive/non-finite inputs collapse to never. */
static ofHours(hours: number): Lifetime {
return new Lifetime(hours);
}
/** A feed that never expires. */
static readonly never = new Lifetime(undefined);
/** The absolute expiry instant for this lifetime, or undefined if it never expires. */
resolveExpiry(now: number): number | undefined {
return this.hours !== undefined &&
Number.isFinite(this.hours) &&
this.hours > 0
? now + this.hours * HOUR_MS
: undefined;
}
}
+84
View File
@@ -0,0 +1,84 @@
import { EmailAddress } from "./email-address";
import { Domain } from "./domain";
export type SenderDecision = "accepted" | "blocked";
type SenderMatch = "blocked" | "allowed" | "neutral";
function normalizeEmail(value: string): string {
return value.trim().toLowerCase();
}
function toDomains(entries: string[]): Domain[] {
return entries
.map((e) => Domain.parse(e))
.filter((d): d is Domain => d !== null);
}
/**
* The sender allow/block policy as a value object, built ONCE from a feed's
* lists. The exact-vs-domain split is pre-computed here so `decide` is a cheap
* lookup per candidate sender instead of re-parsing both lists for every one
* (the previous `applySenderPolicy` was O(senders × lists)).
*
* Semantics (unchanged): no lists ⇒ everything accepted; a blocklist hit always
* rejects; an allowlist (when present) must be matched by at least one sender.
*/
export class SenderPolicy {
private constructor(
private readonly exactAllowed: string[],
private readonly exactBlocked: string[],
private readonly domainAllowed: Domain[],
private readonly domainBlocked: Domain[],
private readonly hasAllowlist: boolean,
private readonly hasAnyRule: boolean,
) {}
static fromLists(
allowed: string[] = [],
blocked: string[] = [],
): SenderPolicy {
const allowedSenders = allowed.map(normalizeEmail).filter(Boolean);
const blockedSenders = blocked.map(normalizeEmail).filter(Boolean);
return new SenderPolicy(
allowedSenders.filter((e) => e.includes("@")),
blockedSenders.filter((e) => e.includes("@")),
toDomains(allowedSenders.filter((e) => !e.includes("@"))),
toDomains(blockedSenders.filter((e) => !e.includes("@"))),
allowedSenders.length > 0,
allowedSenders.length > 0 || blockedSenders.length > 0,
);
}
private evaluate(sender: string): SenderMatch {
const parsed = EmailAddress.parse(sender);
const normalized = parsed ? parsed.normalized : normalizeEmail(sender);
const senderDomain = parsed?.domain ?? null;
if (this.exactBlocked.includes(normalized)) return "blocked";
if (this.exactAllowed.includes(normalized)) return "allowed";
if (senderDomain && this.domainBlocked.some((d) => d.matches(senderDomain)))
return "blocked";
if (senderDomain && this.domainAllowed.some((d) => d.matches(senderDomain)))
return "allowed";
return "neutral";
}
/**
* Decide whether an inbound email is accepted, given its candidate sender
* addresses. A blocklist hit on any sender rejects; with an allowlist set, at
* least one sender must match it.
*/
decide(senders: string[]): SenderDecision {
if (!this.hasAnyRule) return "accepted";
const accepted = senders.some((sender) => {
const decision = this.evaluate(sender);
if (decision === "allowed") return true;
if (decision === "blocked") return false;
return !this.hasAllowlist;
});
return accepted ? "accepted" : "blocked";
}
}
+70
View File
@@ -0,0 +1,70 @@
import { describe, it, expect } from "vitest";
import worker from "./index";
import { createMockEnv } from "./test/setup";
import type { Env } from "./types";
const env = createMockEnv();
function req(path: string, init: RequestInit = {}): Request {
return new Request(`https://test.getmynews.app${path}`, init);
}
describe("CORS middleware", () => {
it("adds CORS headers for an allowed origin", async () => {
const res = await worker.fetch(
req("/rss/some-feed", { headers: { Origin: "https://kill-the.news" } }),
env as unknown as Env,
);
expect(res.headers.get("Access-Control-Allow-Origin")).toBe(
"https://kill-the.news",
);
});
it("omits CORS headers for an unknown origin", async () => {
const res = await worker.fetch(
req("/rss/some-feed", { headers: { Origin: "https://evil.com" } }),
env as unknown as Env,
);
expect(res.headers.get("Access-Control-Allow-Origin")).toBeNull();
});
it("handles OPTIONS preflight for an allowed origin with 204", async () => {
const res = await worker.fetch(
req("/rss/some-feed", {
method: "OPTIONS",
headers: {
Origin: "https://kill-the.news",
"Access-Control-Request-Method": "GET",
},
}),
env as unknown as Env,
);
expect(res.status).toBe(204);
expect(res.headers.get("Access-Control-Allow-Origin")).toBe(
"https://kill-the.news",
);
});
it("makes /api/v1/stats readable from any origin", async () => {
const res = await worker.fetch(
req("/api/v1/stats", { headers: { Origin: "https://example.com" } }),
env as unknown as Env,
);
expect(res.status).toBe(200);
expect(res.headers.get("Access-Control-Allow-Origin")).toBe("*");
});
});
describe("GET /robots.txt", () => {
it("returns 200 and disallows the private feed/entry paths", async () => {
const res = await worker.fetch(req("/robots.txt"), env as unknown as Env);
expect(res.status).toBe(200);
const body = await res.text();
expect(body).toContain("User-agent: *");
expect(body).toContain("Disallow: /rss/");
expect(body).toContain("Disallow: /atom/");
expect(body).toContain("Disallow: /entries/");
expect(body).toContain("Disallow: /files/");
expect(body).toContain("Disallow: /admin/");
});
});
+164 -61
View File
@@ -1,21 +1,43 @@
import { Hono } from 'hono';
import { handle as handleInbound } from './routes/inbound';
import { handle as handleRSS } from './routes/rss';
import { handle as handleAdmin } from './routes/admin';
import { Env } from './types';
import { Hono } from "hono";
import { cors } from "hono/cors";
import { handle as handleInbound } from "./routes/inbound";
import { handle as handleRSS } from "./routes/rss";
import { handle as handleAtom } from "./routes/atom";
import { handle as handleAdmin } from "./routes/admin";
import { handle as handleEntry } from "./routes/entries";
import { handle as handleFiles } from "./routes/files";
import { handle as handleHome } from "./routes/home";
import { handle as handleFavicon, handleFeedFavicon } from "./routes/favicon";
import { hubRouter } from "./routes/hub";
import { apiApp } from "./routes/api";
import { handleCloudflareEmail } from "./infrastructure/cloudflare-email";
import { Env } from "./types";
import { logger } from "./infrastructure/logger";
import { FeedRepository } from "./infrastructure/feed-repository";
import { purgeExpiredFeeds } from "./application/feed-cleanup";
import { FeedId } from "./domain/value-objects/feed-id";
import {
bumpCounters,
scanR2Usage,
scanKvUsage,
setStorageSnapshot,
} from "./application/stats";
import { getAttachmentBucket } from "./infrastructure/attachments";
import { FORWARD_EMAIL_IPS_CACHE_TTL_MS } from "./config/constants";
// Define allowed origins for CORS
const ALLOWED_ORIGINS = ['https://getmynews.app', 'https://www.getmynews.app'];
type AppEnv = { Bindings: Env };
const ALLOWED_ORIGINS = ["https://kill-the.news", "https://www.kill-the.news"];
// Fallback ForwardEmail.net IP addresses in case API fetch fails
const FALLBACK_FORWARD_EMAIL_IPS = [
'138.197.213.185', // mx1.forwardemail.net
'121.127.44.56', // mx1.forwardemail.net (alternate)
'104.248.224.170' // mx2.forwardemail.net
"138.197.213.185", // mx1.forwardemail.net
"121.127.44.56", // mx1.forwardemail.net (alternate)
"104.248.224.170", // mx2.forwardemail.net
];
// Create the main Hono app
const app = new Hono();
const app = new Hono<AppEnv>();
// Cache for ForwardEmail.net IPs with expiration
let forwardEmailIpsCache: {
@@ -32,9 +54,9 @@ async function getForwardEmailIps(): Promise<string[]> {
}
// Fetch the latest IPs from ForwardEmail.net
const response = await fetch('https://forwardemail.net/ips/v4.json', {
const response = await fetch("https://forwardemail.net/ips/v4.json", {
headers: {
'User-Agent': 'Email-to-RSS/1.0',
"User-Agent": "Email-to-RSS/1.0",
},
cf: {
cacheTtl: 3600, // Cache for 1 hour in Cloudflare's cache
@@ -53,94 +75,175 @@ async function getForwardEmailIps(): Promise<string[]> {
updated: string;
}
const data = await response.json() as IpEntry[];
const data = (await response.json()) as IpEntry[];
// Extract IPs for mx1 and mx2 servers
const mxIps = data
.filter(entry =>
entry.hostname === 'mx1.forwardemail.net' ||
entry.hostname === 'mx2.forwardemail.net'
.filter(
(entry) =>
entry.hostname === "mx1.forwardemail.net" ||
entry.hostname === "mx2.forwardemail.net",
)
.flatMap(entry => entry.ipv4);
.flatMap((entry) => entry.ipv4);
// Store in cache for 24 hours
forwardEmailIpsCache = {
ips: mxIps,
expiresAt: Date.now() + (24 * 60 * 60 * 1000) // 24 hours
expiresAt: Date.now() + FORWARD_EMAIL_IPS_CACHE_TTL_MS,
};
console.log('Fetched ForwardEmail.net IPs:', mxIps);
logger.info("Fetched ForwardEmail.net IPs", { count: mxIps.length });
return mxIps;
} catch (error) {
console.error('Error fetching ForwardEmail.net IPs:', error);
logger.error("Failed to fetch ForwardEmail.net IPs", {
error: String(error),
});
// Return fallback IPs if fetch fails
return FALLBACK_FORWARD_EMAIL_IPS;
}
}
// CORS middleware
app.use('*', async (c, next) => {
const origin = c.req.header('Origin');
if (origin && ALLOWED_ORIGINS.includes(origin)) {
c.header('Access-Control-Allow-Origin', origin);
c.header('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');
c.header('Access-Control-Allow-Headers', 'Content-Type, Authorization');
c.header('Access-Control-Max-Age', '86400');
}
// Handle preflight requests
if (c.req.method === 'OPTIONS') {
return c.text('', 204);
}
await next();
});
app.use(
"*",
cors({
origin: ALLOWED_ORIGINS,
allowMethods: ["GET", "POST", "PUT", "DELETE", "OPTIONS"],
allowHeaders: ["Content-Type", "Authorization"],
maxAge: 86400,
}),
);
// Group routes by functionality
const api = new Hono();
const rss = new Hono();
const admin = new Hono();
const api = new Hono<AppEnv>();
const rss = new Hono<AppEnv>();
const atom = new Hono<AppEnv>();
const entries = new Hono<AppEnv>();
const files = new Hono<AppEnv>();
const admin = new Hono<AppEnv>();
// Webhook security middleware for /inbound - verify ForwardEmail.net IP
api.use('/inbound', async (c, next) => {
api.use("/inbound", async (c, next) => {
// Get the client IP
const clientIP = c.req.header('CF-Connecting-IP') || // Cloudflare-specific header
c.req.header('X-Forwarded-For')?.split(',')[0].trim() ||
c.req.raw.headers.get('x-real-ip') ||
'0.0.0.0';
const clientIP =
c.req.header("CF-Connecting-IP") || // Cloudflare-specific header
c.req.header("X-Forwarded-For")?.split(",")[0].trim() ||
c.req.raw.headers.get("x-real-ip") ||
"0.0.0.0";
// Get the latest ForwardEmail.net IPs
const allowedIps = await getForwardEmailIps();
// Check if the request is coming from ForwardEmail.net
if (!allowedIps.includes(clientIP)) {
console.error(`Unauthorized webhook request from IP: ${clientIP}`);
return c.text('Unauthorized', 401);
logger.warn("Unauthorized webhook request", { clientIP });
return c.text("Unauthorized", 401);
}
console.log(`Authorized webhook request from ForwardEmail.net (${clientIP})`);
logger.info("Authorized webhook request", { clientIP });
await next();
});
// API routes (inbound webhook)
api.post('/inbound', handleInbound);
api.post("/inbound", handleInbound);
// RSS feed routes (public)
rss.get('/:feedId', handleRSS);
rss.get("/:feedId", handleRSS);
// Atom feed routes (public)
atom.get("/:feedId", handleAtom);
// Email entry HTML view (public)
entries.get("/:feedId/:entryId", handleEntry);
// Attachment file serving (public)
files.get("/:attachmentId/:filename", handleFiles);
// Admin routes (protected)
admin.route('/', handleAdmin);
admin.route("/", handleAdmin);
// Mount the route groups
app.route('/api', api);
app.route('/rss', rss);
app.route('/admin', admin);
app.route("/api", api);
// Versioned REST API + OpenAPI spec/docs (/api/v1/*, /api/openapi.json, /api/docs)
app.route("/api", apiApp);
app.route("/rss", rss);
app.route("/atom", atom);
app.route("/entries", entries);
app.route("/files", files);
app.route("/admin", admin);
app.route("/hub", hubRouter);
// Root path redirects to admin dashboard
app.get('/', (c) => c.redirect('/admin'));
// Project favicon (also the fallback for the per-feed favicon)
app.get("/favicon.svg", handleFavicon);
app.get("/favicon.ico", handleFavicon); // readers/browsers that hardcode .ico
// Per-feed favicon derived from the last sender's domain
app.get("/favicon/:feedId", handleFeedFavicon);
// Health check endpoint for monitoring
app.get("/health", (c) => c.json({ status: "ok", timestamp: Date.now() }));
// Public status page (counters + link to admin)
app.get("/", handleHome);
// Keep private feeds/emails out of search engines (defense in depth alongside
// the X-Robots-Tag headers on the feed/entry/file responses).
app.get("/robots.txt", (c) =>
c.text(
"User-agent: *\nDisallow: /rss/\nDisallow: /atom/\nDisallow: /entries/\nDisallow: /files/\nDisallow: /admin/\n",
),
);
// Catch-all for 404s
app.all('*', (c) => c.text('Not Found', 404));
app.all("*", (c) => c.text("Not Found", 404));
// Export the worker handler
export default app;
// Export both the HTTP fetch handler and the Cloudflare Email handler
export default {
fetch: app.fetch.bind(app),
async email(
message: ForwardableEmailMessage,
env: Env,
ctx: ExecutionContext,
) {
await handleCloudflareEmail(message, env, ctx);
},
async scheduled(_event: ScheduledEvent, env: Env, _ctx: ExecutionContext) {
const attachmentBucket = getAttachmentBucket(env);
const repo = FeedRepository.from(env);
const feeds = await repo.listFeeds();
const now = Date.now();
const expiredIds = feeds
.filter((f) => f.expires_at !== undefined && f.expires_at <= now)
.map((f) => f.id);
for (const feedId of expiredIds) {
await purgeExpiredFeeds(
env.EMAIL_STORAGE,
FeedId.unchecked(feedId),
attachmentBucket,
);
}
if (expiredIds.length > 0) {
await repo.removeFromListBulk(expiredIds);
await bumpCounters(env.EMAIL_STORAGE, {
feeds_deleted: expiredIds.length,
});
logger.info("Feed TTL cleanup", { deleted: expiredIds.length });
}
// Refresh the cached storage-usage snapshot for the status page / /api/v1/stats.
try {
const r2 = attachmentBucket
? await scanR2Usage(attachmentBucket)
: { bytes: 0, count: 0 };
const kv = await scanKvUsage(env.EMAIL_STORAGE);
await setStorageSnapshot(env.EMAIL_STORAGE, {
attachments_bytes: r2.bytes,
attachments_count: r2.count,
kv_bytes_estimated: kv.bytes,
});
} catch (error) {
logger.error("Error refreshing storage snapshot", {
error: String(error),
});
}
},
};
+9
View File
@@ -0,0 +1,9 @@
import { Env } from "../types";
// Returns the attachment bucket only when the feature is enabled, so callers can
// narrow cleanly. Attachments are on whenever R2 is bound, unless explicitly
// turned off with ATTACHMENTS_ENABLED="false".
export function getAttachmentBucket(env: Env): R2Bucket | undefined {
if (env.ATTACHMENTS_ENABLED === "false") return undefined;
return env.ATTACHMENT_BUCKET;
}
+78
View File
@@ -0,0 +1,78 @@
import { Context } from "hono";
import { Env } from "../types";
/**
* Constant-time string comparison. Prefers the runtime's native
* `crypto.subtle.timingSafeEqual` (Cloudflare Workers) and falls back to a
* manual constant-time loop in environments that lack it (Node test runtime).
*/
export function timingSafeEqual(a: string, b: string): boolean {
const enc = new TextEncoder();
const aBytes = enc.encode(a);
const bBytes = enc.encode(b);
// Try native timing-safe implementation first (Cloudflare Workers runtime)
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const subtle = crypto.subtle as any;
if (typeof subtle.timingSafeEqual === "function") {
if (aBytes.length !== bBytes.length) return false;
return subtle.timingSafeEqual(aBytes, bBytes);
}
// Constant-time fallback for Node (test environment): encode length
// mismatch into `diff` so the loop always runs over the full length.
const len = Math.max(aBytes.length, bBytes.length);
let diff = aBytes.length ^ bBytes.length;
for (let i = 0; i < len; i++) {
diff |= (aBytes[i] ?? 0) ^ (bBytes[i] ?? 0);
}
return diff === 0;
}
/**
* Reverse-proxy authentication: trusted only when both `PROXY_AUTH_SECRET` and
* `PROXY_TRUSTED_IPS` are configured, the request comes from a trusted IP, the
* shared secret matches, and a `Remote-User`/`X-Forwarded-User` is present.
*/
export function checkProxyAuth(c: Context, env: Env): boolean {
if (!env.PROXY_AUTH_SECRET || !env.PROXY_TRUSTED_IPS) return false;
const trustedIps = env.PROXY_TRUSTED_IPS.split(",")
.map((s: string) => s.trim())
.filter(Boolean);
const clientIp = c.req.header("CF-Connecting-IP") ?? "";
const providedSecret = c.req.header("X-Auth-Proxy-Secret") ?? "";
const remoteUser =
c.req.header("Remote-User") || c.req.header("X-Forwarded-User") || "";
return (
trustedIps.includes(clientIp) &&
timingSafeEqual(providedSecret, env.PROXY_AUTH_SECRET) &&
remoteUser.length > 0
);
}
/**
* Authentication for the machine-facing REST API (`/api/v1/*`).
* Grants access when proxy auth passes OR the request carries a valid
* `Authorization: Bearer <ADMIN_PASSWORD>`. No cookie, no CSRF — token only.
*/
export async function apiAuthMiddleware(
c: Context<{ Bindings: Env }>,
next: () => Promise<void>,
): Promise<Response | void> {
const env = c.env;
if (checkProxyAuth(c, env)) {
return next();
}
const authHeader = c.req.header("Authorization") ?? "";
const token = authHeader.startsWith("Bearer ")
? authHeader.slice("Bearer ".length)
: "";
if (token && timingSafeEqual(token, env.ADMIN_PASSWORD)) {
return next();
}
return c.json({ error: "Unauthorized" }, 401);
}
+298
View File
@@ -0,0 +1,298 @@
import { describe, it, expect, beforeEach } from "vitest";
import "../test/setup";
import { createMockEnv } from "../test/setup";
import { handleCloudflareEmail } from "./cloudflare-email";
import { getCounters } from "../application/stats";
const VALID_FEED_ID = "apple.mountain.42";
const DOMAIN = "test.getmynews.app";
const RAW_EMAIL = [
"From: Sender Name <[email protected]>",
`To: ${VALID_FEED_ID}@${DOMAIN}`,
"Subject: Hello World",
"Date: Thu, 01 Jan 2026 12:00:00 +0000",
"MIME-Version: 1.0",
"Content-Type: text/plain; charset=utf-8",
"",
"This is the email body.",
].join("\r\n");
function makeMessage(
overrides: Partial<{
from: string;
to: string;
rawText: string;
forward: (rcptTo: string, headers?: Headers) => Promise<void>;
}> = {},
): ForwardableEmailMessage {
const rawText = overrides.rawText ?? RAW_EMAIL;
const encoder = new TextEncoder();
const bytes = encoder.encode(rawText);
const stream = new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(bytes);
controller.close();
},
});
return {
from: overrides.from ?? "[email protected]",
to: overrides.to ?? `${VALID_FEED_ID}@${DOMAIN}`,
headers: new Headers(),
raw: stream,
rawSize: bytes.length,
forward: overrides.forward ?? (async () => {}),
reply: async () => {},
setReject: () => {},
} as unknown as ForwardableEmailMessage;
}
/** Records every message.forward() call so tests can assert on routing. */
function spyForward() {
const calls: string[] = [];
const forward = async (rcptTo: string) => {
calls.push(rcptTo);
};
return { calls, forward };
}
const FALLBACK = "[email protected]";
describe("handleCloudflareEmail", () => {
let env: ReturnType<typeof createMockEnv>;
beforeEach(() => {
env = createMockEnv();
});
it("stores email in KV when feed exists", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
await handleCloudflareEmail(
makeMessage(),
env as any,
{ waitUntil: () => {} } as any,
);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
expect(metadata.emails).toHaveLength(1);
expect(metadata.emails[0].subject).toBe("Hello World");
});
it("does not throw when feed does not exist", async () => {
await expect(
handleCloudflareEmail(
makeMessage(),
env as any,
{ waitUntil: () => {} } as any,
),
).resolves.toBeUndefined();
});
it("does not throw when email is malformed", async () => {
const msg = makeMessage({ rawText: "not a valid email" });
await expect(
handleCloudflareEmail(msg, env as any, { waitUntil: () => {} } as any),
).resolves.toBeUndefined();
});
it("uses sender from message.from for allowlist check", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ allowed_senders: ["[email protected]"] }),
);
await handleCloudflareEmail(
makeMessage(),
env as any,
{ waitUntil: () => {} } as any,
);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
expect(metadata.emails).toHaveLength(1);
});
it("rejects email when sender is not in allowlist (stored nothing)", async () => {
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ allowed_senders: ["[email protected]"] }),
);
await handleCloudflareEmail(
makeMessage(),
env as any,
{ waitUntil: () => {} } as any,
);
const metadata = await env.EMAIL_STORAGE.get(
`feed:${VALID_FEED_ID}:metadata`,
"json",
);
expect(metadata).toBeNull();
});
describe("FALLBACK_FORWARD_ADDRESS catch-all fallback", () => {
it("forwards to the fallback when the feed does not exist", async () => {
const { calls, forward } = spyForward();
env.FALLBACK_FORWARD_ADDRESS = FALLBACK;
await handleCloudflareEmail(
makeMessage({ forward }),
env as any,
{ waitUntil: () => {} } as any,
);
expect(calls).toEqual([FALLBACK]);
});
it("forwards to the fallback when the address is not a feed", async () => {
const { calls, forward } = spyForward();
env.FALLBACK_FORWARD_ADDRESS = FALLBACK;
await handleCloudflareEmail(
makeMessage({ to: `not-a-feed@${DOMAIN}`, forward }),
env as any,
{ waitUntil: () => {} } as any,
);
expect(calls).toEqual([FALLBACK]);
});
it("does NOT forward an expired feed's mail (no newsletter leak)", async () => {
const { calls, forward } = spyForward();
env.FALLBACK_FORWARD_ADDRESS = FALLBACK;
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ expires_at: Date.now() - 1000 }),
);
await handleCloudflareEmail(
makeMessage({ forward }),
env as any,
{ waitUntil: () => {} } as any,
);
expect(calls).toEqual([]);
});
it("does NOT forward when the sender is blocked", async () => {
const { calls, forward } = spyForward();
env.FALLBACK_FORWARD_ADDRESS = FALLBACK;
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ allowed_senders: ["[email protected]"] }),
);
await handleCloudflareEmail(
makeMessage({ forward }),
env as any,
{ waitUntil: () => {} } as any,
);
expect(calls).toEqual([]);
});
it("does NOT forward when the email was ingested", async () => {
const { calls, forward } = spyForward();
env.FALLBACK_FORWARD_ADDRESS = FALLBACK;
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({}),
);
await handleCloudflareEmail(
makeMessage({ forward }),
env as any,
{ waitUntil: () => {} } as any,
);
expect(calls).toEqual([]);
});
it("does NOT forward when the env var is unset (current drop behavior)", async () => {
const { calls, forward } = spyForward();
// env.FALLBACK_FORWARD_ADDRESS intentionally left unset.
await handleCloudflareEmail(
makeMessage({ forward }),
env as any,
{ waitUntil: () => {} } as any,
);
expect(calls).toEqual([]);
});
it("does not throw when the fallback forward fails (unverified address)", async () => {
env.FALLBACK_FORWARD_ADDRESS = FALLBACK;
const forward = async () => {
throw new Error("destination address not verified");
};
await expect(
handleCloudflareEmail(
makeMessage({ forward }),
env as any,
{ waitUntil: () => {} } as any,
),
).resolves.toBeUndefined();
});
it("increments the emails_forwarded counter on a successful forward", async () => {
const { forward } = spyForward();
env.FALLBACK_FORWARD_ADDRESS = FALLBACK;
await handleCloudflareEmail(
makeMessage({ forward }),
env as any,
{ waitUntil: () => {} } as any,
);
const counters = await getCounters(env.EMAIL_STORAGE as any);
expect(counters.emails_forwarded).toBe(1);
});
it("does not increment emails_forwarded when the forward fails", async () => {
env.FALLBACK_FORWARD_ADDRESS = FALLBACK;
const forward = async () => {
throw new Error("destination address not verified");
};
await handleCloudflareEmail(
makeMessage({ forward }),
env as any,
{ waitUntil: () => {} } as any,
);
const counters = await getCounters(env.EMAIL_STORAGE as any);
expect(counters.emails_forwarded).toBe(0);
});
it("does not increment emails_forwarded for dropped reasons", async () => {
const { forward } = spyForward();
env.FALLBACK_FORWARD_ADDRESS = FALLBACK;
await env.EMAIL_STORAGE.put(
`feed:${VALID_FEED_ID}:config`,
JSON.stringify({ expires_at: Date.now() - 1000 }),
);
await handleCloudflareEmail(
makeMessage({ forward }),
env as any,
{ waitUntil: () => {} } as any,
);
const counters = await getCounters(env.EMAIL_STORAGE as any);
expect(counters.emails_forwarded).toBe(0);
});
});
});
+94
View File
@@ -0,0 +1,94 @@
import PostalMime from "postal-mime";
import { Env } from "../types";
import {
processEmail,
RawAttachment,
IngestRejectionReason,
} from "../application/email-processor";
import { bumpCounters } from "../application/stats";
import { normalizeCid } from "../infrastructure/html-processor";
import { logger } from "./logger";
export async function handleCloudflareEmail(
message: ForwardableEmailMessage,
env: Env,
ctx: ExecutionContext,
): Promise<void> {
try {
const email = await PostalMime.parse(message.raw);
const fromAddress = email.from?.address ?? message.from;
const from =
email.from?.name && email.from.address
? `${email.from.name} <${email.from.address}>`
: fromAddress;
const headers: Record<string, string> = {};
for (const h of email.headers) {
headers[h.key] = h.value;
}
const rawAttachments: RawAttachment[] = (email.attachments ?? [])
.filter((a) => a.content instanceof ArrayBuffer)
.map((a) => ({
filename: a.filename || "attachment",
contentType: a.mimeType || "application/octet-stream",
content: a.content as ArrayBuffer,
contentId: normalizeCid(a.contentId),
}));
const result = await processEmail(
{
toAddress: message.to,
from,
senders: [message.from],
subject: email.subject ?? "(no subject)",
content: email.html ?? email.text ?? "",
receivedAt: email.date ? new Date(email.date).getTime() : Date.now(),
headers,
attachments: rawAttachments,
},
env,
ctx,
);
if (!result.ok) {
logger.warn("Inbound email rejected", {
to: message.to,
reason: result.reason,
});
await maybeForwardFallback(message, env, result.reason);
}
} catch (error) {
console.error("Error processing Cloudflare email:", error);
}
}
// Reasons safe to forward to the catch-all fallback: the mail was never a feed's
// (wrong address shape, or no such feed). Expired feeds and blocked senders are
// dropped so a real newsletter never leaks into the fallback inbox.
const FORWARDABLE_REASONS = new Set<IngestRejectionReason>([
"invalid_address",
"feed_not_found",
]);
async function maybeForwardFallback(
message: ForwardableEmailMessage,
env: Env,
reason: IngestRejectionReason,
): Promise<void> {
const fallback = env.FALLBACK_FORWARD_ADDRESS;
if (!fallback || !FORWARDABLE_REASONS.has(reason)) return;
try {
await message.forward(fallback);
// Counted as a subset of emails_rejected (already bumped in processEmail);
// the dropped count is derived as emails_rejected − emails_forwarded.
await bumpCounters(env.EMAIL_STORAGE, { emails_forwarded: 1 });
} catch (error) {
logger.warn("Fallback forward failed", {
to: message.to,
fallback,
error: error instanceof Error ? error.message : String(error),
});
}
}
@@ -0,0 +1,22 @@
import { describe, it, expect } from "vitest";
import { createMockEnv } from "../test/setup";
import { CountersRepository } from "./counters-repository";
import type { Env } from "../types";
const mockEnv = () => createMockEnv() as unknown as Env;
describe("CountersRepository", () => {
it("round-trips the counters singleton", async () => {
const repo = new CountersRepository(mockEnv().EMAIL_STORAGE);
expect(await repo.getRaw()).toBeNull();
await repo.put({
feeds_created: 1,
feeds_deleted: 0,
emails_received: 2,
emails_rejected: 0,
emails_forwarded: 0,
unsubscribes_sent: 0,
});
expect(await repo.getRaw()).toMatchObject({ emails_received: 2 });
});
});
+23
View File
@@ -0,0 +1,23 @@
import { Counters, Env } from "../types";
import { STATS_KEY } from "../domain/feed-keys";
/**
* KV access for the monitoring counters singleton (`stats:counters`). The
* increment policy lives in the application layer (utils/stats.ts); this
* repository owns only the raw read/write of the blob.
*/
export class CountersRepository {
constructor(private readonly kv: KVNamespace) {}
static from(env: Env): CountersRepository {
return new CountersRepository(env.EMAIL_STORAGE);
}
async getRaw(): Promise<Counters | null> {
return (await this.kv.get(STATS_KEY, { type: "json" })) as Counters | null;
}
async put(counters: Counters): Promise<void> {
await this.kv.put(STATS_KEY, JSON.stringify(counters));
}
}
+149
View File
@@ -0,0 +1,149 @@
import { describe, it, expect } from "vitest";
import { http, HttpResponse } from "msw";
import { server, createMockEnv } from "../test/setup";
import {
cacheFaviconForDomain,
extractEmailDomain,
getCachedIcon,
} from "./favicon-fetcher";
import { MAX_ICON_BYTES } from "../config/constants";
const iconKey = (domain: string) => `icon:${domain}`;
import type { Env } from "../types";
const PNG = new Uint8Array([0x89, 0x50, 0x4e, 0x47, 1, 2, 3, 4]);
function imageResponse(bytes: Uint8Array, contentType = "image/png") {
return new HttpResponse(bytes, { headers: { "Content-Type": contentType } });
}
describe("extractEmailDomain", () => {
it("parses a bare address", () => {
expect(extractEmailDomain("[email protected]")).toBe("github.com");
});
it("parses a display-form address", () => {
expect(extractEmailDomain("GitHub <[email protected]>")).toBe("github.com");
});
it("strips a trailing dot and lowercases", () => {
expect(extractEmailDomain("[email protected].")).toBe("example.com");
});
it("returns null when there is no address", () => {
expect(extractEmailDomain("not an email")).toBeNull();
});
});
describe("cacheFaviconForDomain", () => {
it("caches the direct /favicon.ico when available", async () => {
const env = createMockEnv() as unknown as Env;
server.use(
http.get("https://github.com/favicon.ico", () => imageResponse(PNG)),
);
await cacheFaviconForDomain("github.com", env);
const record = await env.EMAIL_STORAGE.get(iconKey("github.com"), "json");
expect(record).toMatchObject({ contentType: "image/png" });
expect((record as { data: string }).data).toBeTruthy();
expect(record).not.toHaveProperty("fetchedAt");
const icon = await getCachedIcon("github.com", env);
expect(icon?.contentType).toBe("image/png");
expect(new Uint8Array(icon!.bytes)).toEqual(PNG);
});
it("falls back to DuckDuckGo when the direct icon 404s", async () => {
const env = createMockEnv() as unknown as Env;
server.use(
http.get("https://acme.test/favicon.ico", () =>
HttpResponse.text("nope", { status: 404 }),
),
http.get("https://icons.duckduckgo.com/ip3/acme.test.ico", () =>
imageResponse(PNG, "image/x-icon"),
),
);
await cacheFaviconForDomain("acme.test", env);
const icon = await getCachedIcon("acme.test", env);
expect(icon?.contentType).toBe("image/x-icon");
});
it("writes a negative entry when no icon is found", async () => {
const env = createMockEnv() as unknown as Env;
server.use(
http.get("https://nope.test/favicon.ico", () =>
HttpResponse.text("", { status: 404 }),
),
http.get("https://icons.duckduckgo.com/ip3/nope.test.ico", () =>
HttpResponse.text("", { status: 404 }),
),
);
await cacheFaviconForDomain("nope.test", env);
const record = await env.EMAIL_STORAGE.get(iconKey("nope.test"), "json");
expect(record).toEqual({ data: null, contentType: "" });
expect(await getCachedIcon("nope.test", env)).toBeNull();
});
it("rejects oversized responses as negative", async () => {
const env = createMockEnv() as unknown as Env;
const big = new Uint8Array(MAX_ICON_BYTES + 1);
server.use(
http.get("https://big.test/favicon.ico", () => imageResponse(big)),
http.get("https://icons.duckduckgo.com/ip3/big.test.ico", () =>
HttpResponse.text("", { status: 404 }),
),
);
await cacheFaviconForDomain("big.test", env);
expect(await getCachedIcon("big.test", env)).toBeNull();
});
it("rejects non-image content types as negative", async () => {
const env = createMockEnv() as unknown as Env;
server.use(
http.get("https://html.test/favicon.ico", () =>
HttpResponse.text("<html>", {
headers: { "Content-Type": "text/html" },
}),
),
http.get("https://icons.duckduckgo.com/ip3/html.test.ico", () =>
HttpResponse.text("", { status: 404 }),
),
);
await cacheFaviconForDomain("html.test", env);
expect(await getCachedIcon("html.test", env)).toBeNull();
});
it("short-circuits when an entry already exists (no outbound fetch)", async () => {
const env = createMockEnv() as unknown as Env;
// Pre-seed a record; with MSW onUnhandledRequest:"error", any fetch fails.
await env.EMAIL_STORAGE.put(
iconKey("cached.test"),
JSON.stringify({ data: null, contentType: "" }),
);
await expect(
cacheFaviconForDomain("cached.test", env),
).resolves.toBeUndefined();
});
it("never throws on network errors", async () => {
const env = createMockEnv() as unknown as Env;
server.use(
http.get("https://err.test/favicon.ico", () => HttpResponse.error()),
http.get("https://icons.duckduckgo.com/ip3/err.test.ico", () =>
HttpResponse.error(),
),
);
await expect(
cacheFaviconForDomain("err.test", env),
).resolves.toBeUndefined();
});
});
+124
View File
@@ -0,0 +1,124 @@
import { Env } from "../types";
import {
ICON_FETCH_TIMEOUT_MS,
ICON_TTL_SECONDS,
MAX_ICON_BYTES,
} from "../config/constants";
import { IconRepository } from "./icon-repository";
import { EmailAddress } from "../domain/value-objects/email-address";
import { logger } from "./logger";
interface IconRecord {
data: string | null; // base64 icon bytes, or null for a negative cache entry
contentType: string;
}
/**
* Extract the lowercased domain from a `from` value, accepting either a bare
* address (`[email protected]`) or a display form (`Name <[email protected]>`). Returns null when
* no plausible address can be parsed.
*/
export function extractEmailDomain(from: string): string | null {
return EmailAddress.parse(from)?.domain.value ?? null;
}
function arrayBufferToBase64(buffer: ArrayBuffer): string {
const bytes = new Uint8Array(buffer);
let binary = "";
const chunkSize = 0x8000;
for (let i = 0; i < bytes.length; i += chunkSize) {
binary += String.fromCharCode(...bytes.subarray(i, i + chunkSize));
}
return btoa(binary);
}
function base64ToArrayBuffer(base64: string): ArrayBuffer {
const binary = atob(base64);
const bytes = new Uint8Array(binary.length);
for (let i = 0; i < binary.length; i++) {
bytes[i] = binary.charCodeAt(i);
}
return bytes.buffer;
}
async function fetchIconFrom(
url: string,
): Promise<{ buffer: ArrayBuffer; contentType: string } | null> {
const res = await fetch(url, {
redirect: "follow",
signal: AbortSignal.timeout(ICON_FETCH_TIMEOUT_MS),
headers: { "User-Agent": "kill-the-news/1.0" },
});
if (!res.ok) return null;
const contentType = res.headers.get("content-type") ?? "";
if (!contentType.startsWith("image/")) return null;
const buffer = await res.arrayBuffer();
if (buffer.byteLength === 0 || buffer.byteLength > MAX_ICON_BYTES)
return null;
return { buffer, contentType: contentType.split(";")[0].trim() };
}
async function resolveIcon(
domain: string,
): Promise<{ buffer: ArrayBuffer; contentType: string } | null> {
const candidates = [
`https://${domain}/favicon.ico`,
`https://icons.duckduckgo.com/ip3/${domain}.ico`,
];
for (const url of candidates) {
try {
const icon = await fetchIconFrom(url);
if (icon) return icon;
} catch {
// Try the next candidate; network/timeout errors must never propagate.
}
}
return null;
}
/**
* Resolve and cache the favicon for a sender domain. Idempotent and never
* throws: if a (success or negative) cache entry already exists it returns
* immediately, so callers can fire this on every email without refetching.
* The KV TTL is the sole expiry mechanism.
*/
export async function cacheFaviconForDomain(
domain: string,
env: Env,
): Promise<void> {
try {
const repo = IconRepository.from(env);
const existing = await repo.getText(domain);
if (existing !== null) return; // present (incl. negative) → nothing to do
const icon = await resolveIcon(domain);
const record: IconRecord = icon
? {
data: arrayBufferToBase64(icon.buffer),
contentType: icon.contentType,
}
: { data: null, contentType: "" };
await repo.put(domain, JSON.stringify(record), ICON_TTL_SECONDS);
} catch (error) {
logger.warn("Favicon cache failed", { domain, error: String(error) });
}
}
/**
* Read a cached icon for a domain. Returns null on a miss or a negative entry.
*/
export async function getCachedIcon(
domain: string,
env: Env,
): Promise<{ bytes: ArrayBuffer; contentType: string } | null> {
const record = await IconRepository.from(env).getJson<IconRecord>(domain);
if (!record || record.data === null) return null;
return {
bytes: base64ToArrayBuffer(record.data),
contentType: record.contentType,
};
}
+397
View File
@@ -0,0 +1,397 @@
import { describe, it, expect } from "vitest";
import {
generateRssFeed,
generateAtomFeed,
extractBodyContent,
} from "./feed-generator";
import { FeedConfig, EmailData } from "../types";
const mockFeedConfig: FeedConfig = {
title: "Test Newsletter",
description: "A test feed",
language: "en",
created_at: 1700000000000,
};
const mockEmails: EmailData[] = [
{
subject: "Hello World",
from: "Alice <[email protected]>",
content: "<p>Hello from Alice</p>",
receivedAt: 1700000001000,
headers: {},
},
];
const mockEmailWithAttachment: EmailData = {
...mockEmails[0],
attachments: [
{
id: "550e8400-e29b-41d4-a716-446655440000",
filename: "report.pdf",
contentType: "application/pdf",
size: 12345,
},
],
};
const BASE_URL = "https://test.getmynews.app";
const FEED_ID = "abc123";
describe("extractBodyContent", () => {
it("extracts content inside <body> tags", () => {
const html = "<html><head></head><body><p>Hello</p></body></html>";
expect(extractBodyContent(html)).toBe("<p>Hello</p>");
});
it("handles body tag with attributes", () => {
const html = '<html><body style="margin:0"><p>Hi</p></body></html>';
expect(extractBodyContent(html)).toBe("<p>Hi</p>");
});
it("returns html unchanged when no body tags present", () => {
const fragment = "<p>Already a fragment</p>";
expect(extractBodyContent(fragment)).toBe(fragment);
});
it("is case-insensitive for body tag matching", () => {
const html = "<HTML><BODY><p>content</p></BODY></HTML>";
expect(extractBodyContent(html)).toBe("<p>content</p>");
});
});
describe("generateRssFeed", () => {
it("returns RSS 2.0 with channel element", () => {
const result = generateRssFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain("<channel>");
expect(result).toContain("<title>Test Newsletter</title>");
});
it("includes the per-feed icon as the channel <image>", () => {
const result = generateRssFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain("<image>");
expect(result).toContain(`${BASE_URL}/favicon/${FEED_ID}`);
});
it("includes <enclosure> element for email with attachment", () => {
const result = generateRssFeed(
mockFeedConfig,
[mockEmailWithAttachment],
BASE_URL,
FEED_ID,
);
expect(result).toContain("<enclosure");
expect(result).toContain("550e8400-e29b-41d4-a716-446655440000");
expect(result).toContain("report.pdf");
expect(result).toContain("application/pdf");
expect(result).toContain("12345");
});
it("does not include <enclosure> for email without attachments", () => {
const result = generateRssFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).not.toContain("<enclosure");
});
it("enclosure URL uses /files/{id}/{filename} scheme", () => {
const result = generateRssFeed(
mockFeedConfig,
[mockEmailWithAttachment],
BASE_URL,
FEED_ID,
);
expect(result).toContain(
`${BASE_URL}/files/550e8400-e29b-41d4-a716-446655440000/report.pdf`,
);
});
it("includes rss self-link in RSS output", () => {
const result = generateRssFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain(`${BASE_URL}/rss/${FEED_ID}`);
});
it("includes email entries as <item> elements", () => {
const result = generateRssFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain("<item>");
expect(result).toContain("Hello World");
});
it("works with empty emails array", () => {
const result = generateRssFeed(mockFeedConfig, [], BASE_URL, FEED_ID);
expect(result).toContain("<channel>");
expect(result).not.toContain("<item>");
});
it("feed link points to admin emails page", () => {
const result = generateRssFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain(`${BASE_URL}/admin/feeds/${FEED_ID}/emails`);
});
it("strips html/head/body wrapper from item description", () => {
const emailWithFullHtml: EmailData = {
...mockEmails[0],
content: "<html><head></head><body><p>Body only</p></body></html>",
};
const result = generateRssFeed(
mockFeedConfig,
[emailWithFullHtml],
BASE_URL,
FEED_ID,
);
expect(result).toContain("<p>Body only</p>");
expect(result).not.toContain("<html>");
});
});
describe("generateAtomFeed", () => {
it("returns Atom 1.0 namespace", () => {
const result = generateAtomFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain('xmlns="http://www.w3.org/2005/Atom"');
});
it("includes the per-feed icon as <icon> and <logo>", () => {
const result = generateAtomFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
const iconUrl = `${BASE_URL}/favicon/${FEED_ID}`;
expect(result).toContain(`<icon>${iconUrl}</icon>`);
expect(result).toContain(`<logo>${iconUrl}</logo>`);
});
it("contains <feed> root element", () => {
const result = generateAtomFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain("<feed");
expect(result).toContain("</feed>");
});
it("includes feed title", () => {
const result = generateAtomFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain("Test Newsletter");
});
it("includes <entry> elements for each email", () => {
const result = generateAtomFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain("<entry>");
expect(result).toContain("Hello World");
});
it("includes author information", () => {
const result = generateAtomFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain("Alice");
});
it("self-link points to atom URL", () => {
const result = generateAtomFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain(`${BASE_URL}/atom/${FEED_ID}`);
});
it("includes rss alternate link", () => {
const result = generateAtomFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain(`${BASE_URL}/rss/${FEED_ID}`);
});
it("works with empty emails array", () => {
const result = generateAtomFeed(mockFeedConfig, [], BASE_URL, FEED_ID);
expect(result).toContain("<feed");
expect(result).not.toContain("<entry>");
});
it("feed link points to admin emails page", () => {
const result = generateAtomFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain(`${BASE_URL}/admin/feeds/${FEED_ID}/emails`);
});
it("strips html/head/body wrapper from entry content", () => {
const emailWithFullHtml: EmailData = {
...mockEmails[0],
content: "<html><head></head><body><p>Body only</p></body></html>",
};
const result = generateAtomFeed(
mockFeedConfig,
[emailWithFullHtml],
BASE_URL,
FEED_ID,
);
expect(result).toContain("<p>Body only</p>");
expect(result).not.toContain("<html>");
});
it("handles config without description", () => {
const configNoDesc: FeedConfig = {
...mockFeedConfig,
description: undefined,
};
const result = generateAtomFeed(
configNoDesc,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain('xmlns="http://www.w3.org/2005/Atom"');
});
it("handles config with author field", () => {
const configWithAuthor: FeedConfig = { ...mockFeedConfig, author: "Bob" };
const result = generateAtomFeed(
configWithAuthor,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).toContain("Bob");
});
it("renders the subject as plain text in <title> (strips tags, decodes entities)", () => {
const emailWithHtmlSubject: EmailData = {
...mockEmails[0],
subject: "<b>Sale</b> Tom &amp; Jerry",
};
const result = generateAtomFeed(
mockFeedConfig,
[emailWithHtmlSubject],
BASE_URL,
FEED_ID,
);
// Tags are stripped and entities decoded; markup must not survive.
expect(result).toContain("Sale Tom & Jerry");
expect(result).not.toContain("<b>Sale</b>");
});
it("strips XML-illegal control characters from the output", () => {
const emailWithControlChar: EmailData = {
...mockEmails[0],
subject: "Bad\x00\x1Fchar",
content: "<p>body\x0Bhere</p>",
};
const result = generateAtomFeed(
mockFeedConfig,
[emailWithControlChar],
BASE_URL,
FEED_ID,
);
expect(result).not.toMatch(/[\x00\x0B\x1F]/);
});
it("preserves emoji (surrogate pairs) in the output", () => {
const emailWithEmoji: EmailData = {
...mockEmails[0],
subject: "Launch 🚀 today",
};
const result = generateAtomFeed(
mockFeedConfig,
[emailWithEmoji],
BASE_URL,
FEED_ID,
);
expect(result).toContain("🚀");
});
it("absolutizes relative content URLs against the sender domain", () => {
const emailWithRelative: EmailData = {
...mockEmails[0],
from: "News <[email protected]>",
content: '<body><a href="/article">read</a></body>',
};
const result = generateAtomFeed(
mockFeedConfig,
[emailWithRelative],
BASE_URL,
FEED_ID,
);
expect(result).toContain("https://acme.com/article");
});
it("includes enclosure link for email with attachment in Atom feed", () => {
const result = generateAtomFeed(
mockFeedConfig,
[mockEmailWithAttachment],
BASE_URL,
FEED_ID,
);
expect(result).toContain('rel="enclosure"');
expect(result).toContain("550e8400-e29b-41d4-a716-446655440000");
expect(result).toContain("report.pdf");
});
it("does not include enclosure link for email without attachments in Atom feed", () => {
const result = generateAtomFeed(
mockFeedConfig,
mockEmails,
BASE_URL,
FEED_ID,
);
expect(result).not.toContain('rel="enclosure"');
});
});
+129
View File
@@ -0,0 +1,129 @@
import { Feed } from "feed";
import { FeedConfig, EmailData } from "../types";
import { processEmailContent, htmlToText } from "./html-processor";
import { EmailAddress } from "../domain/value-objects/email-address";
export { processEmailContent as extractBodyContent };
// XML 1.0 valid chars: #x9 #xA #xD #x20-#xD7FF #xE000-#xFFFD #x10000-#x10FFFF.
// A single illegal codepoint fails the whole feed parse in strict readers, so
// strip the complement before returning. The `u` flag iterates by code point, so
// valid surrogate pairs (emoji, …) survive while lone surrogates are removed.
function stripInvalidXmlChars(xml: string): string {
return xml.replace(/[^\x09\x0A\x0D\x20-퟿-�\u{10000}-\u{10FFFF}]/gu, "");
}
function parseFromAddress(from: string): { name: string; email?: string } {
const match = from.match(/^(.*?)\s*<([^>]+)>\s*$/);
if (match) {
return { name: match[1].trim() || match[2], email: match[2].trim() };
}
const emailOnly = from.match(/^[^\s@]+@[^\s@]+\.[^\s@]+$/);
if (emailOnly) {
return { email: from.trim(), name: from.trim() };
}
return { name: from.trim() };
}
function buildFeed(
feedConfig: FeedConfig,
emails: EmailData[],
baseUrl: string,
feedId: string,
selfUrl?: { rss?: string; atom?: string },
): Feed {
const iconUrl = `${baseUrl}/favicon/${feedId}`;
const feed = new Feed({
title: feedConfig.title,
description: feedConfig.description || "",
// Per-feed icon derived from the last sender's domain (self-falls-back to
// the project icon). image → RSS <image>/Atom <logo>; favicon → Atom <icon>.
image: iconUrl,
favicon: iconUrl,
// Computed dynamically so the id is always canonical regardless of what
// was stored in KV at feed-creation time (which may have used a stale domain).
id: `${baseUrl}/rss/${feedId}`,
// Link points to the admin emails page — the "website" this feed represents.
link: `${baseUrl}/admin/feeds/${feedId}/emails`,
language: feedConfig.language,
updated: new Date(),
generator: "kill-the-news",
copyright: `Copyright © ${new Date().getFullYear()} ${feedConfig.title}`,
feedLinks: {
rss: selfUrl?.rss ?? `${baseUrl}/rss/${feedId}`,
atom: selfUrl?.atom ?? `${baseUrl}/atom/${feedId}`,
},
author: feedConfig.author
? {
name: feedConfig.author,
email: `noreply@${new URL(baseUrl).hostname}`,
}
: undefined,
});
for (const email of emails) {
const entryUrl = `${baseUrl}/entries/${feedId}/${email.receivedAt}`;
// Inline images are rendered in the body, not surfaced as an enclosure.
const firstAttachment = email.attachments?.find((a) => !a.inline);
const bodyContent = processEmailContent(
email.content,
email.attachments,
baseUrl,
EmailAddress.parse(email.from)?.siteBaseUrl() ?? "",
);
feed.addItem({
title: htmlToText(email.subject),
id: entryUrl,
link: entryUrl,
description: bodyContent,
content: bodyContent,
author: [parseFromAddress(email.from)],
date: new Date(email.receivedAt),
enclosure: firstAttachment
? {
url: `${baseUrl}/files/${firstAttachment.id}/${encodeURIComponent(firstAttachment.filename)}`,
type: firstAttachment.contentType,
length: firstAttachment.size,
}
: undefined,
});
}
return feed;
}
export function generateRssFeed(
feedConfig: FeedConfig,
emails: EmailData[],
baseUrl: string,
feedId: string,
selfUrl?: string,
): string {
return stripInvalidXmlChars(
buildFeed(
feedConfig,
emails,
baseUrl,
feedId,
selfUrl ? { rss: selfUrl } : undefined,
).rss2(),
);
}
export function generateAtomFeed(
feedConfig: FeedConfig,
emails: EmailData[],
baseUrl: string,
feedId: string,
selfUrl?: string,
): string {
return stripInvalidXmlChars(
buildFeed(
feedConfig,
emails,
baseUrl,
feedId,
selfUrl ? { atom: selfUrl } : undefined,
).atom1(),
);
}
+45
View File
@@ -0,0 +1,45 @@
import { describe, it, expect } from "vitest";
import { fromConfigDTO, toConfigDTO, toListItemDTO } from "./feed-mapper";
import { FeedId } from "../domain/value-objects/feed-id";
import type { FeedConfig } from "../types";
const fullConfig: FeedConfig = {
title: "News",
description: "desc",
language: "en",
author: "Jane",
allowed_senders: ["[email protected]"],
blocked_senders: ["[email protected]"],
created_at: 1000,
updated_at: 2000,
expires_at: 3000,
};
describe("feed-mapper", () => {
it("round-trips a full config DTO through domain state unchanged", () => {
expect(toConfigDTO(fromConfigDTO(fullConfig))).toEqual(fullConfig);
});
it("defaults absent sender lists to empty arrays on the domain side", () => {
const state = fromConfigDTO({
title: "T",
language: "en",
created_at: 1,
});
expect(state.allowedSenders).toEqual([]);
expect(state.blockedSenders).toEqual([]);
});
it("projects the feeds:list item from domain state", () => {
const item = toListItemDTO(
FeedId.unchecked("a.b.42"),
fromConfigDTO(fullConfig),
);
expect(item).toEqual({
id: "a.b.42",
title: "News",
description: "desc",
expires_at: 3000,
});
});
});
+51
View File
@@ -0,0 +1,51 @@
import { FeedConfig, FeedListItem } from "../types";
import { FeedState } from "../domain/feed-state";
import { FeedId } from "../domain/value-objects/feed-id";
/**
* The translation seam between the Feed aggregate's domain state (camelCase) and
* the persistence/edge DTOs (`FeedConfig`/`FeedListItem`, snake_case). This is
* the ONLY place outside the HTTP edge that knows the stored field names — the
* domain stays free of the storage dialect, and the repository round-trips
* through here on every load/save.
*/
/** Persisted config DTO → domain state (used by `FeedRepository.load`). */
export function fromConfigDTO(dto: FeedConfig): FeedState {
return {
title: dto.title,
description: dto.description,
language: dto.language,
author: dto.author,
allowedSenders: dto.allowed_senders ?? [],
blockedSenders: dto.blocked_senders ?? [],
createdAt: dto.created_at,
updatedAt: dto.updated_at,
expiresAt: dto.expires_at,
};
}
/** Domain state → persisted config DTO (used by `FeedRepository.save`). */
export function toConfigDTO(state: FeedState): FeedConfig {
return {
title: state.title,
description: state.description,
language: state.language,
author: state.author,
allowed_senders: state.allowedSenders,
blocked_senders: state.blockedSenders,
created_at: state.createdAt,
updated_at: state.updatedAt,
expires_at: state.expiresAt,
};
}
/** Domain state → the projection cached in the global `feeds:list` registry. */
export function toListItemDTO(id: FeedId, state: FeedState): FeedListItem {
return {
id: id.value,
title: state.title,
description: state.description,
expires_at: state.expiresAt,
};
}
+156
View File
@@ -0,0 +1,156 @@
import { describe, it, expect } from "vitest";
import { createMockEnv } from "../test/setup";
import { FeedRepository } from "./feed-repository";
import { Feed } from "../domain/feed.aggregate";
import { FeedId } from "../domain/value-objects/feed-id";
import type { Env, FeedConfig, FeedMetadata, EmailData } from "../types";
const mockEnv = () => createMockEnv() as unknown as Env;
const fid = (value: string) => FeedId.unchecked(value);
const sampleConfig = (overrides: Partial<FeedConfig> = {}): FeedConfig => ({
title: "Test Feed",
language: "en",
created_at: 1000,
...overrides,
});
const sampleEmail = (overrides: Partial<EmailData> = {}): EmailData => ({
subject: "Hello",
from: "[email protected]",
content: "<p>hi</p>",
receivedAt: 1234,
headers: {},
...overrides,
});
describe("FeedRepository key schema", () => {
it("builds the canonical KV keys via the public API", () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
expect(repo.feedKeyPrefix(fid("a.b.42"))).toBe("feed:a.b.42:");
expect(repo.newEmailKey(fid("a.b.42"))).toMatch(/^feed:a\.b\.42:\d+$/);
});
it("recognises email keys vs config/metadata keys", () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
expect(repo.isEmailKey(fid("a.b.42"), "feed:a.b.42:config")).toBe(false);
expect(repo.isEmailKey(fid("a.b.42"), "feed:a.b.42:metadata")).toBe(false);
expect(repo.isEmailKey(fid("a.b.42"), "feed:a.b.42:1700000000000")).toBe(
true,
);
});
it("recovers the feed id from an email key", () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
expect(repo.feedIdFromEmailKey("feed:a.b.42:1700000000000")).toBe("a.b.42");
});
});
describe("FeedRepository config & metadata", () => {
it("round-trips and deletes a feed config", async () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
expect(await repo.getConfig(fid("a.b.42"))).toBeNull();
await repo.putConfig(fid("a.b.42"), sampleConfig());
expect(await repo.getConfig(fid("a.b.42"))).toMatchObject({
title: "Test Feed",
});
await repo.deleteConfig(fid("a.b.42"));
expect(await repo.getConfig(fid("a.b.42"))).toBeNull();
});
it("round-trips and deletes feed metadata", async () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
const meta: FeedMetadata = { emails: [] };
await repo.putMetadata(fid("a.b.42"), meta);
expect(await repo.getMetadata(fid("a.b.42"))).toEqual(meta);
await repo.deleteMetadata(fid("a.b.42"));
expect(await repo.getMetadata(fid("a.b.42"))).toBeNull();
});
});
describe("FeedRepository emails", () => {
it("stores and reads an email under a minted key", async () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
const key = repo.newEmailKey(fid("a.b.42"));
await repo.putEmail(key, sampleEmail());
expect(await repo.getEmail(key)).toMatchObject({ subject: "Hello" });
await repo.deleteEmail(key);
expect(await repo.getEmail(key)).toBeNull();
});
it("lists every key under a feed prefix", async () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
await repo.putConfig(fid("a.b.42"), sampleConfig());
await repo.putMetadata(fid("a.b.42"), { emails: [] });
const emailKey = repo.newEmailKey(fid("a.b.42"));
await repo.putEmail(emailKey, sampleEmail());
const listed = await repo.listFeedKeys(fid("a.b.42"));
expect(listed.names).toContain("feed:a.b.42:config");
expect(listed.names).toContain("feed:a.b.42:metadata");
expect(listed.names).toContain(emailKey);
expect(
listed.names.filter((k) => repo.isEmailKey(fid("a.b.42"), k)),
).toEqual([emailKey]);
});
});
describe("FeedRepository feed list", () => {
const feedWith = (
id: string,
title: string,
opts: { description?: string; expires_at?: number } = {},
) =>
Feed.reconstitute(
fid(id),
{
title,
language: "en",
allowedSenders: [],
blockedSenders: [],
createdAt: 1000,
description: opts.description,
expiresAt: opts.expires_at,
},
{ emails: [] },
);
it("upserts the list entry from the aggregate on save/saveConfig", async () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
await repo.save(
feedWith("a.b.42", "One", { description: "desc", expires_at: 5000 }),
);
await repo.save(feedWith("c.d.99", "Two"));
let feeds = await repo.listFeeds();
expect(feeds).toHaveLength(2);
expect(feeds.find((f) => f.id === "a.b.42")).toMatchObject({
title: "One",
expires_at: 5000,
});
// saveConfig refreshes the same entry in place (no duplicate, expiry cleared).
await repo.saveConfig(feedWith("a.b.42", "One-updated"));
feeds = await repo.listFeeds();
expect(feeds.filter((f) => f.id === "a.b.42")).toHaveLength(1);
const updated = feeds.find((f) => f.id === "a.b.42");
expect(updated).toMatchObject({ title: "One-updated" });
expect(updated?.expires_at).toBeUndefined();
expect(await repo.removeFromList(fid("a.b.42"))).toBe(true);
expect(await repo.removeFromList(fid("missing"))).toBe(false);
feeds = await repo.listFeeds();
expect(feeds.map((f) => f.id)).toEqual(["c.d.99"]);
});
it("bulk-removes only the matching ids", async () => {
const repo = new FeedRepository(mockEnv().EMAIL_STORAGE);
await repo.save(feedWith("a.b.42", "One"));
await repo.save(feedWith("c.d.99", "Two"));
await repo.save(feedWith("e.f.10", "Three"));
const removed = await repo.removeFromListBulk(["a.b.42", "e.f.10", "nope"]);
expect(removed.sort()).toEqual(["a.b.42", "e.f.10"]);
expect((await repo.listFeeds()).map((f) => f.id)).toEqual(["c.d.99"]);
});
});
+270
View File
@@ -0,0 +1,270 @@
import {
EmailData,
Env,
FeedConfig,
FeedList,
FeedListItem,
FeedMetadata,
} from "../types";
import { FEEDS_LIST_KEY } from "../config/constants";
import { feedKeys } from "../domain/feed-keys";
import { Feed } from "../domain/feed.aggregate";
import { FeedId } from "../domain/value-objects/feed-id";
import { fromConfigDTO, toConfigDTO, toListItemDTO } from "./feed-mapper";
import { logger } from "./logger";
/**
* Single source of truth for KV access to the Feed aggregate. The key schema
* itself lives in `feed-keys.ts`; this repository owns the get/put operations.
* No other module should build a `feed:`/`feeds:list`/`websub:`/`icon:`/
* `stats:counters` key string — go through `feed-keys` or a repository method.
*
* Wraps one `KVNamespace`; construct per request via `FeedRepository.from(env)`.
*/
export class FeedRepository {
constructor(private readonly kv: KVNamespace) {}
static from(env: Env): FeedRepository {
return new FeedRepository(env.EMAIL_STORAGE);
}
// ── Key schema (delegates to feed-keys) ───────────────────────────────────
private configKey(feedId: FeedId): string {
return feedKeys.config(feedId.value);
}
private metadataKey(feedId: FeedId): string {
return feedKeys.metadata(feedId.value);
}
/** Prefix covering every key owned by a feed (config, metadata, emails). */
feedKeyPrefix(feedId: FeedId): string {
return feedKeys.feedPrefix(feedId.value);
}
/** Mint a fresh, time-ordered email key. Call once and reuse the result. */
newEmailKey(feedId: FeedId): string {
return feedKeys.newEmail(feedId.value);
}
/** True when `key` is an email entry (not the feed's config/metadata key). */
isEmailKey(feedId: FeedId, key: string): boolean {
return feedKeys.isEmail(feedId.value, key);
}
/** Recover the feed id embedded in an email key (`feed:<id>:<ts>`). */
feedIdFromEmailKey(key: string): string {
return feedKeys.feedIdFromEmail(key);
}
// ── Feed aggregate ────────────────────────────────────────────────────────
/**
* Load the aggregate (config + email index). A feed exists iff it has a
* config; metadata defaults to empty so a freshly-created feed still loads.
*/
async load(feedId: FeedId): Promise<Feed | null> {
const [config, metadata] = await Promise.all([
this.getConfig(feedId),
this.getMetadata(feedId),
]);
if (!config) return null;
return Feed.reconstitute(
feedId,
fromConfigDTO(config),
metadata ?? { emails: [] },
);
}
/**
* Persist both keys the aggregate owns (config + metadata) and keep the global
* `feeds:list` entry in sync. Config/list DTOs are derived from the aggregate's
* domain `state()` via `feed-mapper`, so no caller has to mirror snake_case.
*/
async save(feed: Feed): Promise<void> {
await Promise.all([
this.putConfig(feed.id, toConfigDTO(feed.state())),
this.putMetadata(feed.id, feed.toMetadataSnapshot()),
this.upsertListEntry(toListItemDTO(feed.id, feed.state())),
]);
}
/**
* Persist only the email index. Used by the ingest/delete paths where config
* is unchanged — avoids a redundant config write on the hot path. The list
* projection (title/description/expiry) is untouched, so it is not rewritten.
*/
async saveMetadata(feed: Feed): Promise<void> {
await this.putMetadata(feed.id, feed.toMetadataSnapshot());
}
/**
* Persist only the config and refresh the `feeds:list` entry from it. Used by
* the rename/edit paths where metadata is unchanged — avoids re-writing (and
* risking clobbering) the email index.
*/
async saveConfig(feed: Feed): Promise<void> {
await Promise.all([
this.putConfig(feed.id, toConfigDTO(feed.state())),
this.upsertListEntry(toListItemDTO(feed.id, feed.state())),
]);
}
// ── Feed config ───────────────────────────────────────────────────────────
async getConfig(feedId: FeedId): Promise<FeedConfig | null> {
return (await this.kv.get(this.configKey(feedId), {
type: "json",
})) as FeedConfig | null;
}
async putConfig(feedId: FeedId, config: FeedConfig): Promise<void> {
await this.kv.put(this.configKey(feedId), JSON.stringify(config));
}
async deleteConfig(feedId: FeedId): Promise<void> {
await this.kv.delete(this.configKey(feedId));
}
// ── Feed metadata ─────────────────────────────────────────────────────────
async getMetadata(feedId: FeedId): Promise<FeedMetadata | null> {
return (await this.kv.get(this.metadataKey(feedId), {
type: "json",
})) as FeedMetadata | null;
}
async putMetadata(feedId: FeedId, metadata: FeedMetadata): Promise<void> {
await this.kv.put(this.metadataKey(feedId), JSON.stringify(metadata));
}
async deleteMetadata(feedId: FeedId): Promise<void> {
await this.kv.delete(this.metadataKey(feedId));
}
// ── Emails ────────────────────────────────────────────────────────────────
async putEmail(key: string, data: EmailData): Promise<void> {
await this.kv.put(key, JSON.stringify(data));
}
async getEmail(key: string): Promise<EmailData | null> {
return (await this.kv.get(key, { type: "json" })) as EmailData | null;
}
async deleteEmail(key: string): Promise<void> {
await this.kv.delete(key);
}
// ── Global feed list ──────────────────────────────────────────────────────
async listFeeds(): Promise<FeedListItem[]> {
try {
const feedList = (await this.kv.get(FEEDS_LIST_KEY, {
type: "json",
})) as FeedList | null;
return feedList?.feeds || [];
} catch (error) {
logger.error("Error listing feeds", { error: String(error) });
return [];
}
}
/**
* Insert-or-update a feed's entry in the global `feeds:list` registry from its
* aggregate summary. Idempotent by feed id. Private: callers persist a `Feed`
* via `save`/`saveConfig`, which keep the projection in sync — never mirror the
* list by hand. (Read-modify-write is not atomic under KV, unchanged from the
* prior add/update split.)
*/
private async upsertListEntry(summary: FeedListItem): Promise<void> {
try {
const feedList = ((await this.kv.get(FEEDS_LIST_KEY, {
type: "json",
})) as FeedList | null) || { feeds: [] };
const index = feedList.feeds.findIndex((feed) => feed.id === summary.id);
if (index === -1) {
feedList.feeds.push(summary);
} else {
feedList.feeds[index] = summary;
}
await this.kv.put(FEEDS_LIST_KEY, JSON.stringify(feedList));
} catch (error) {
logger.error("Error upserting feed in list", {
feedId: summary.id,
error: String(error),
});
}
}
async removeFromListBulk(feedIds: string[]): Promise<string[]> {
try {
const feedList = ((await this.kv.get(FEEDS_LIST_KEY, {
type: "json",
})) as FeedList | null) || { feeds: [] };
const toRemove = new Set(feedIds.filter(Boolean));
if (toRemove.size === 0) return [];
const removed: string[] = [];
const nextFeeds: FeedListItem[] = [];
for (const feed of feedList.feeds) {
if (toRemove.has(feed.id)) {
removed.push(feed.id);
continue;
}
nextFeeds.push(feed);
}
if (removed.length === 0) return [];
feedList.feeds = nextFeeds;
await this.kv.put(FEEDS_LIST_KEY, JSON.stringify(feedList));
return removed;
} catch (error) {
logger.error("Error removing feeds from list", { error: String(error) });
return [];
}
}
async removeFromList(feedId: FeedId): Promise<boolean> {
const removed = await this.removeFromListBulk([feedId.value]);
return removed.includes(feedId.value);
}
// ── Key listing / counting ────────────────────────────────────────────────
async listFeedKeys(
feedId: FeedId,
options: { cursor?: string; limit?: number } = {},
): Promise<{ names: string[]; cursor: string; listComplete: boolean }> {
const prefix = this.feedKeyPrefix(feedId);
const limit = Math.min(1000, Math.max(1, Math.floor(options.limit || 100)));
const cursor = options.cursor || undefined;
const listed = await this.kv.list({ prefix, cursor, limit });
return {
names: (listed.keys || []).map((k) => k.name),
cursor: listed.cursor || "",
listComplete: !!listed.list_complete,
};
}
async countKeysByPrefix(prefix: string): Promise<number> {
let total = 0;
let cursor: string | undefined;
try {
do {
const listed = await this.kv.list({ prefix, cursor, limit: 1000 });
total += listed.keys.length;
cursor = listed.list_complete ? undefined : listed.cursor;
} while (cursor);
} catch (error) {
logger.error("Error counting keys", { prefix, error: String(error) });
}
return total;
}
}
+127
View File
@@ -0,0 +1,127 @@
import { EmailParser } from "../domain/email-parser";
import { Env } from "../types";
import {
processEmail,
IngestResult,
RawAttachment,
} from "../application/email-processor";
import { normalizeCid } from "../infrastructure/html-processor";
/** Map an ingestion result to the HTTP response ForwardEmail expects. */
export function ingestResultToResponse(result: IngestResult): Response {
if (result.ok) {
return new Response("Email processed successfully", { status: 200 });
}
switch (result.reason) {
case "invalid_address":
return new Response("Invalid email address format", { status: 400 });
case "feed_not_found":
return new Response("Feed does not exist", { status: 404 });
case "feed_expired":
return new Response("Feed has expired", { status: 410 });
case "sender_blocked":
return new Response("Sender not allowed for this feed", { status: 403 });
}
}
export interface ForwardEmailAttachment {
filename?: string;
contentType?: string;
size?: number;
cid?: string;
contentId?: string;
content?: { type: "Buffer"; data: number[] } | ArrayBuffer | ArrayBufferView;
}
export interface ForwardEmailPayload {
recipients?: string[];
from?: {
value?: Array<{ address?: string; name?: string }>;
text?: string;
html?: string;
};
subject?: string;
text?: string;
html?: string;
date?: string;
messageId?: string;
headerLines?: Array<{ key: string; line: string }>;
headers?: string;
raw?: string;
attachments?: ForwardEmailAttachment[];
}
function normalizeEmail(value: string): string {
return value.trim().toLowerCase();
}
function extractSenderAddresses(payload: ForwardEmailPayload): string[] {
const valueEntries = payload.from?.value || [];
const structuredAddresses = valueEntries
.map((entry) => entry.address || "")
.map(normalizeEmail)
.filter(Boolean);
if (structuredAddresses.length > 0) {
return Array.from(new Set(structuredAddresses));
}
const fromText = payload.from?.text || "";
const matches =
fromText.match(/[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,}/gi) || [];
return Array.from(new Set(matches.map(normalizeEmail)));
}
function toArrayBuffer(
content: ForwardEmailAttachment["content"],
): ArrayBuffer | null {
if (!content) return null;
if (content instanceof ArrayBuffer) return content;
if (ArrayBuffer.isView(content))
return (content as ArrayBufferView).buffer as ArrayBuffer;
if (
typeof content === "object" &&
content.type === "Buffer" &&
Array.isArray(content.data)
) {
return Uint8Array.from(content.data).buffer as ArrayBuffer;
}
return null;
}
export async function handleForwardEmail(
payload: ForwardEmailPayload,
env: Env,
ctx?: ExecutionContext,
): Promise<Response> {
const emailData = EmailParser.parseForwardEmailPayload(payload);
const rawAttachments: RawAttachment[] = (payload.attachments ?? [])
.map((a): RawAttachment | null => {
const buffer = toArrayBuffer(a.content);
if (!buffer) return null;
return {
filename: a.filename || "attachment",
contentType: a.contentType || "application/octet-stream",
content: buffer,
contentId: normalizeCid(a.cid ?? a.contentId),
};
})
.filter((a): a is RawAttachment => a !== null);
const result = await processEmail(
{
toAddress: payload.recipients?.[0] || "",
from: emailData.from,
senders: extractSenderAddresses(payload),
subject: emailData.subject,
content: emailData.content,
receivedAt: emailData.receivedAt,
headers: emailData.headers,
attachments: rawAttachments,
},
env,
ctx,
);
return ingestResultToResponse(result);
}
+321
View File
@@ -0,0 +1,321 @@
import { describe, it, expect } from "vitest";
import {
processEmailContent,
extractInlineCids,
htmlToText,
} from "./html-processor";
import type { AttachmentData } from "../types";
describe("processEmailContent — body extraction", () => {
it("extracts content inside <body> tags", () => {
const html = "<html><head></head><body><p>Hello</p></body></html>";
expect(processEmailContent(html)).toBe("<p>Hello</p>");
});
it("handles body tag with attributes", () => {
const html = '<html><body style="margin:0"><p>Hi</p></body></html>';
expect(processEmailContent(html)).toBe("<p>Hi</p>");
});
it("returns fragment unchanged when no body tags present", () => {
const fragment = "<p>Already a fragment</p>";
expect(processEmailContent(fragment)).toBe("<p>Already a fragment</p>");
});
it("is case-insensitive for body tag matching", () => {
const html = "<HTML><BODY><p>content</p></BODY></HTML>";
expect(processEmailContent(html)).toBe("<p>content</p>");
});
});
describe("processEmailContent — plain text", () => {
it("wraps plain text in <pre>", () => {
const text = "Hello world\nSecond line";
const result = processEmailContent(text);
expect(result).toMatch(/^<pre /);
expect(result).toContain("Hello world\nSecond line");
});
it("escapes < and > in plain text", () => {
const text = "Price < 10 & size > 5";
const result = processEmailContent(text);
expect(result).toContain("&lt;");
expect(result).toContain("&gt;");
expect(result).toContain("&amp;");
expect(result).not.toContain("<10");
});
it("returns empty string for empty input", () => {
expect(processEmailContent("")).toBe("");
});
});
describe("processEmailContent — dangerous element removal", () => {
it("removes <script> tags", () => {
const html = "<body><p>Hello</p><script>alert('xss')</script></body>";
const result = processEmailContent(html);
expect(result).not.toContain("<script");
expect(result).not.toContain("alert");
expect(result).toContain("<p>Hello</p>");
});
it("removes <iframe> tags", () => {
const html =
"<body><iframe src='https://evil.com'></iframe><p>ok</p></body>";
const result = processEmailContent(html);
expect(result).not.toContain("<iframe");
expect(result).toContain("<p>ok</p>");
});
it("removes <object> and <embed> tags", () => {
const html = "<body><object></object><embed src='x'/><p>ok</p></body>";
const result = processEmailContent(html);
expect(result).not.toContain("<object");
expect(result).not.toContain("<embed");
});
});
describe("processEmailContent — attribute sanitization", () => {
it("removes event handler attributes", () => {
const html =
"<body><a href='https://x.com' onclick='evil()'>link</a></body>";
const result = processEmailContent(html);
expect(result).not.toContain("onclick");
expect(result).toContain('href="https://x.com"');
});
it("removes onerror on images", () => {
const html = "<body><img src='x' onerror='evil()' /></body>";
const result = processEmailContent(html);
expect(result).not.toContain("onerror");
});
it("removes javascript: hrefs", () => {
const html = "<body><a href='javascript:evil()'>click</a></body>";
const result = processEmailContent(html);
expect(result).not.toContain("javascript:");
});
it("preserves legitimate href and src attributes", () => {
const html =
"<body><a href='https://example.com'>link</a><img src='https://example.com/img.png'/></body>";
const result = processEmailContent(html);
expect(result).toContain("https://example.com");
});
});
describe("processEmailContent — mso style cleanup", () => {
it("strips mso-* properties from inline styles", () => {
const html =
'<body><p style="mso-margin-top: 0; color: red;">text</p></body>';
const result = processEmailContent(html);
expect(result).not.toContain("mso-margin-top");
expect(result).toContain("color: red");
});
it("removes style attribute entirely when only mso properties remain", () => {
const html =
'<body><p style="mso-line-height-rule: exactly;">text</p></body>';
const result = processEmailContent(html);
expect(result).not.toContain("style=");
});
it("preserves style attribute when non-mso properties remain", () => {
const html =
'<body><p style="mso-font-size: 12pt; font-weight: bold;">text</p></body>';
const result = processEmailContent(html);
expect(result).toContain("font-weight");
expect(result).not.toContain("mso-font-size");
});
});
describe("processEmailContent — inline cid: rewriting", () => {
const attachment = (
overrides: Partial<AttachmentData> = {},
): AttachmentData => ({
id: "att-123",
filename: "chicken big.png",
contentType: "image/png",
size: 100,
contentId: "ii_mpi85rqy0",
...overrides,
});
it("rewrites cid: src to a relative /files URL when no baseUrl", () => {
const html = '<body><img src="cid:ii_mpi85rqy0" alt="x"/></body>';
const result = processEmailContent(html, [attachment()]);
expect(result).toContain('src="/files/att-123/chicken%20big.png"');
expect(result).not.toContain("cid:");
});
it("rewrites cid: src to an absolute URL when baseUrl is given", () => {
const html = '<body><img src="cid:ii_mpi85rqy0"/></body>';
const result = processEmailContent(
html,
[attachment()],
"https://feed.example",
);
expect(result).toContain(
'src="https://feed.example/files/att-123/chicken%20big.png"',
);
});
it("matches a stored Content-ID that has angle brackets", () => {
const html = '<body><img src="cid:ii_mpi85rqy0"/></body>';
const result = processEmailContent(html, [
attachment({ contentId: "<ii_mpi85rqy0>" }),
]);
expect(result).toContain('src="/files/att-123/chicken%20big.png"');
});
it("is case-insensitive on the cid: scheme", () => {
const html = '<body><img src="CID:ii_mpi85rqy0"/></body>';
const result = processEmailContent(html, [attachment()]);
expect(result).toContain('src="/files/att-123/chicken%20big.png"');
});
it("leaves unknown cid references unchanged", () => {
const html = '<body><img src="cid:unknown"/></body>';
const result = processEmailContent(html, [attachment()]);
expect(result).toContain('src="cid:unknown"');
});
it("leaves cid references unchanged when no attachments are provided", () => {
const html = '<body><img src="cid:ii_mpi85rqy0"/></body>';
const result = processEmailContent(html);
expect(result).toContain('src="cid:ii_mpi85rqy0"');
});
it("ignores attachments without a contentId", () => {
const html = '<body><img src="cid:ii_mpi85rqy0"/></body>';
const result = processEmailContent(html, [
attachment({ contentId: undefined }),
]);
expect(result).toContain('src="cid:ii_mpi85rqy0"');
});
it("does not touch normal http image sources", () => {
const html = '<body><img src="https://example.com/a.png"/></body>';
const result = processEmailContent(html, [attachment()]);
expect(result).toContain('src="https://example.com/a.png"');
});
});
describe("processEmailContent — lazy image promotion", () => {
it("promotes data-src to src when src is missing", () => {
const html = '<body><img data-src="https://x.com/a.png"/></body>';
const result = processEmailContent(html);
expect(result).toContain('src="https://x.com/a.png"');
});
it("promotes data-src over a data: placeholder src", () => {
const html =
'<body><img src="data:image/gif;base64,AAAA" data-src="https://x.com/a.png"/></body>';
const result = processEmailContent(html);
expect(result).toContain('src="https://x.com/a.png"');
expect(result).not.toContain("data:image/gif");
});
it("does not clobber a real src with data-src", () => {
const html =
'<body><img src="https://real.com/a.png" data-src="https://lazy.com/b.png"/></body>';
const result = processEmailContent(html);
expect(result).toContain('src="https://real.com/a.png"');
});
it("promotes data-srcset when srcset is absent", () => {
const html = '<body><img data-srcset="https://x.com/a.png 2x"/></body>';
const result = processEmailContent(html);
expect(result).toContain('srcset="https://x.com/a.png 2x"');
});
it("strips loading=lazy", () => {
const html = '<body><img src="https://x.com/a.png" loading="lazy"/></body>';
const result = processEmailContent(html);
expect(result).not.toContain("loading");
});
});
describe("processEmailContent — relative URL absolutization", () => {
const base = "https://news.example.com/";
it("absolutizes a root-relative href against the sender base", () => {
const html = '<body><a href="/path">link</a></body>';
const result = processEmailContent(html, undefined, "", base);
expect(result).toContain('href="https://news.example.com/path"');
});
it("absolutizes a relative img src against the sender base", () => {
const html = '<body><img src="img/a.png"/></body>';
const result = processEmailContent(html, undefined, "", base);
expect(result).toContain('src="https://news.example.com/img/a.png"');
});
it("resolves protocol-relative URLs using https", () => {
const html = '<body><img src="//cdn.example.com/a.png"/></body>';
const result = processEmailContent(html, undefined, "", base);
expect(result).toContain('src="https://cdn.example.com/a.png"');
});
it("leaves absolute URLs unchanged", () => {
const html = '<body><a href="https://other.com/x">l</a></body>';
const result = processEmailContent(html, undefined, "", base);
expect(result).toContain('href="https://other.com/x"');
});
it("does not touch relative URLs when no sender base is given", () => {
const html = '<body><a href="/path">link</a></body>';
const result = processEmailContent(html);
expect(result).toContain('href="/path"');
});
it("does not absolutize mailto: or anchors", () => {
const html =
'<body><a href="mailto:[email protected]">m</a><a href="#top">t</a></body>';
const result = processEmailContent(html, undefined, "", base);
expect(result).toContain('href="mailto:[email protected]"');
expect(result).toContain('href="#top"');
});
});
describe("htmlToText", () => {
it("strips HTML tags", () => {
expect(htmlToText("<b>Bold</b> text")).toBe("Bold text");
});
it("decodes HTML entities", () => {
expect(htmlToText("Tom &amp; Jerry &lt;3")).toBe("Tom & Jerry <3");
});
it("collapses whitespace and trims", () => {
expect(htmlToText(" a\n\n b ")).toBe("a b");
});
it("returns empty string for empty input", () => {
expect(htmlToText("")).toBe("");
});
it("leaves plain text untouched", () => {
expect(htmlToText("Just a subject")).toBe("Just a subject");
});
});
describe("extractInlineCids", () => {
it("collects normalized cids referenced by cid: image sources", () => {
const html = '<body><img src="cid:ii_abc"/><img src="CID:ii_def"/></body>';
expect(extractInlineCids(html)).toEqual(new Set(["ii_abc", "ii_def"]));
});
it("ignores non-cid sources", () => {
const html = '<body><img src="https://example.com/a.png"/></body>';
expect(extractInlineCids(html).size).toBe(0);
});
it("returns an empty set for plain text", () => {
expect(extractInlineCids("just text, no html").size).toBe(0);
});
it("returns an empty set for empty input", () => {
expect(extractInlineCids("").size).toBe(0);
});
});
+208
View File
@@ -0,0 +1,208 @@
import { parseHTML } from "linkedom";
import escapeHtml from "escape-html";
import type { AttachmentData } from "../types";
type ParsedDocument = ReturnType<typeof parseHTML>["document"];
// Strip surrounding angle brackets and whitespace from a Content-ID so that a
// stored value like "<ii_mpi85rqy0>" matches an HTML reference "cid:ii_mpi85rqy0".
export function normalizeCid(
cid: string | null | undefined,
): string | undefined {
if (!cid) return undefined;
const trimmed = cid.trim().replace(/^<|>$/g, "").trim();
return trimmed || undefined;
}
// Collect the normalized Content-IDs referenced by `cid:` image sources in the
// email body — exactly the set rewriteCidSrc would turn into inline <img> URLs.
// Used at ingest to flag those attachments as inline (rendered in place, hidden
// from the downloadable attachment lists).
export function extractInlineCids(content: string): Set<string> {
const cids = new Set<string>();
if (!content || isPlainText(content)) return cids;
const { document } = parseHTML(content);
document.querySelectorAll("[src]").forEach((el: Element) => {
const match = (el.getAttribute("src") ?? "").match(/^\s*cid:(.+)$/i);
const cid = match ? normalizeCid(match[1]) : undefined;
if (cid) cids.add(cid);
});
return cids;
}
// Render an HTML fragment (or already-plain string) down to plain text: strips
// tags and decodes entities. Used for feed <title>s, which must be plain text —
// raw markup/entities show literally in readers.
export function htmlToText(value: string): string {
if (!value) return "";
const { document } = parseHTML(`<body>${value}</body>`);
return (document.documentElement?.textContent ?? "")
.replace(/\s+/g, " ")
.trim();
}
// Newsletters frequently defer images via data-src/loading="lazy"; readers don't
// run the lazy-loader, so the image renders blank. Promote the real source.
function promoteLazyImages(document: ParsedDocument): void {
document.querySelectorAll("img").forEach((img: Element) => {
const lazySrc =
img.getAttribute("data-src") ||
img.getAttribute("data-original") ||
img.getAttribute("data-lazy-src");
if (lazySrc) {
const current = (img.getAttribute("src") ?? "").trim();
if (!current || /^data:/i.test(current)) {
img.setAttribute("src", lazySrc);
}
}
const lazySrcset = img.getAttribute("data-srcset");
if (lazySrcset && !img.getAttribute("srcset")) {
img.setAttribute("srcset", lazySrcset);
}
img.removeAttribute("loading");
});
}
// Resolve a single URL against the sender base. Returns null for values that are
// already absolute or should never be rewritten (mailto:, data:, cid:, anchors).
function toAbsolute(value: string, base: string): string | null {
const v = value.trim();
if (!v || /^(https?:|mailto:|tel:|data:|cid:|#)/i.test(v)) return null;
try {
return new URL(v, base).href;
} catch {
return null;
}
}
// Most readers ignore xml:base, so relative href/src in content break. Absolutize
// them against the sender's site (best-effort, derived from its email domain).
// Protocol-relative //host/x are resolved too (they pick up the base's https:).
function absolutizeUrls(document: ParsedDocument, base: string): void {
if (!base) return;
document.querySelectorAll("a[href], area[href]").forEach((el: Element) => {
const abs = toAbsolute(el.getAttribute("href") ?? "", base);
if (abs) el.setAttribute("href", abs);
});
document.querySelectorAll("img[src]").forEach((el: Element) => {
const abs = toAbsolute(el.getAttribute("src") ?? "", base);
if (abs) el.setAttribute("src", abs);
});
}
function cleanMsoStyles(style: string): string {
return style
.split(";")
.map((p) => p.trim())
.filter((p) => p && !/^mso-/i.test(p))
.join("; ");
}
function isPlainText(content: string): boolean {
return !/<[a-z][\s\S]*>/i.test(content);
}
function rewriteCidSrc(
el: Element,
cidMap: Map<string, AttachmentData>,
baseUrl: string,
): void {
const src = el.getAttribute("src") ?? "";
const match = src.match(/^\s*cid:(.+)$/i);
if (!match) return;
const attachment = cidMap.get(normalizeCid(match[1]) ?? "");
if (!attachment) return;
el.setAttribute(
"src",
`${baseUrl}/files/${attachment.id}/${encodeURIComponent(attachment.filename)}`,
);
}
function sanitizeElement(el: Element): void {
// Snapshot attribute names before mutating (linkedom attributes is array-like)
const attrs = Array.from(
el.attributes as unknown as ArrayLike<{ name: string }>,
).map((a) => a.name);
for (const attr of attrs) {
// Remove event handlers (onclick, onerror, onload, …)
if (/^on/i.test(attr)) {
el.removeAttribute(attr);
continue;
}
// Remove javascript: URLs
if (["href", "src", "action"].includes(attr.toLowerCase())) {
const val = el.getAttribute(attr) ?? "";
if (/^\s*javascript:/i.test(val)) {
el.removeAttribute(attr);
continue;
}
}
}
// Strip mso-* inline style properties (Office HTML noise)
const style = el.getAttribute("style");
if (style !== null) {
const cleaned = cleanMsoStyles(style);
if (cleaned) {
el.setAttribute("style", cleaned);
} else {
el.removeAttribute("style");
}
}
}
/**
* Processes email content for safe display in feeds and entry pages:
* - Detects plain text and wraps it in a <pre> block
* - Extracts the <body> fragment from full HTML documents
* - Removes dangerous elements: <script>, <iframe>, <object>, <embed>
* - Removes event handler attributes and javascript: URLs
* - Strips mso-* inline style properties (Office HTML)
* - Rewrites inline cid: image refs to the stored attachment URL. baseUrl=""
* yields relative URLs (entry page, same origin); a baseUrl yields absolute
* URLs (feeds, for external RSS readers).
* - Promotes lazy-loaded images (data-src → src, strips loading="lazy").
* - Absolutizes relative href/src against senderBaseUrl (the sender's site,
* best-effort) so links/images don't break in readers that ignore xml:base.
*/
export function processEmailContent(
content: string,
attachments?: AttachmentData[],
baseUrl = "",
senderBaseUrl = "",
): string {
if (!content) return "";
if (isPlainText(content)) {
return `<pre style="white-space: pre-wrap; word-break: break-word;">${escapeHtml(content)}</pre>`;
}
const cidMap = new Map<string, AttachmentData>();
for (const att of attachments ?? []) {
const cid = normalizeCid(att.contentId);
if (cid) cidMap.set(cid, att);
}
const { document } = parseHTML(content);
document
.querySelectorAll("script, object, embed, iframe, frame, frameset")
.forEach((el: Element) => el.remove());
document.querySelectorAll("*").forEach((el: Element) => sanitizeElement(el));
promoteLazyImages(document);
// Absolutize first: cid: refs are skipped here (not http(s)), then rewritten
// below to our /files/ URL — which must NOT be absolutized to the sender.
absolutizeUrls(document, senderBaseUrl);
if (cidMap.size > 0) {
document
.querySelectorAll("[src]")
.forEach((el: Element) => rewriteCidSrc(el, cidMap, baseUrl));
}
// Full documents expose a <body>; bodyless fragments are serialized directly
// so that sanitization and cid rewriting still apply to their nodes.
const body = document.querySelector("body");
return body ? body.innerHTML : document.toString();
}
@@ -0,0 +1,18 @@
import { describe, it, expect } from "vitest";
import { createMockEnv } from "../test/setup";
import { IconRepository } from "./icon-repository";
import type { Env } from "../types";
const mockEnv = () => createMockEnv() as unknown as Env;
describe("IconRepository", () => {
it("stores and reads favicons as text or json under the icon: key", async () => {
const repo = new IconRepository(mockEnv().EMAIL_STORAGE);
expect(await repo.getText("example.com")).toBeNull();
await repo.put("example.com", JSON.stringify({ data: null }), 60);
expect(await repo.getText("example.com")).toBe('{"data":null}');
expect(await repo.getJson<{ data: null }>("example.com")).toEqual({
data: null,
});
});
});
+31
View File
@@ -0,0 +1,31 @@
import { Env } from "../types";
import { feedKeys } from "../domain/feed-keys";
/**
* KV access for cached per-domain favicons (`icon:<domain>`). Entries may be
* positive (base64 bytes) or negative (a sentinel marking a failed fetch), and
* always carry a TTL — the cache's sole expiry mechanism.
*/
export class IconRepository {
constructor(private readonly kv: KVNamespace) {}
static from(env: Env): IconRepository {
return new IconRepository(env.EMAIL_STORAGE);
}
getText(domain: string): Promise<string | null> {
return this.kv.get(feedKeys.icon(domain), "text");
}
async getJson<T>(domain: string): Promise<T | null> {
return (await this.kv.get(feedKeys.icon(domain), {
type: "json",
})) as T | null;
}
async put(domain: string, value: string, ttlSeconds: number): Promise<void> {
await this.kv.put(feedKeys.icon(domain), value, {
expirationTtl: ttlSeconds,
});
}
}
+26
View File
@@ -0,0 +1,26 @@
type LogLevel = "info" | "warn" | "error" | "debug";
function log(
level: LogLevel,
message: string,
data?: Record<string, unknown>,
): void {
const entry = data ? { level, message, ...data } : { level, message };
const line = JSON.stringify(entry);
if (level === "error" || level === "warn") {
console.error(line);
} else {
console.log(line);
}
}
export const logger = {
info: (message: string, data?: Record<string, unknown>) =>
log("info", message, data),
warn: (message: string, data?: Record<string, unknown>) =>
log("warn", message, data),
error: (message: string, data?: Record<string, unknown>) =>
log("error", message, data),
debug: (message: string, data?: Record<string, unknown>) =>
log("debug", message, data),
};
+184
View File
@@ -0,0 +1,184 @@
import { describe, it, expect } from "vitest";
import { http, HttpResponse } from "msw";
import { server, createMockEnv } from "../test/setup";
import {
parseOneClickUnsubscribe,
sendOneClickUnsubscribe,
sendUnsubscribes,
} from "./unsubscribe";
import { getCounters } from "../application/stats";
import type { Env } from "../types";
const POST_HEADER = "List-Unsubscribe=One-Click";
describe("parseOneClickUnsubscribe", () => {
it("returns the https URL when the one-click Post header is present", () => {
expect(
parseOneClickUnsubscribe({
"list-unsubscribe": "<https://news.example.com/u?t=abc>",
"list-unsubscribe-post": POST_HEADER,
}),
).toBe("https://news.example.com/u?t=abc");
});
it("prefers the https URL when both https and mailto are present", () => {
expect(
parseOneClickUnsubscribe({
"list-unsubscribe":
"<mailto:[email protected]>, <https://example.com/u/1>",
"list-unsubscribe-post": POST_HEADER,
}),
).toBe("https://example.com/u/1");
});
it("returns null for a mailto-only header", () => {
expect(
parseOneClickUnsubscribe({
"list-unsubscribe": "<mailto:[email protected]>",
"list-unsubscribe-post": POST_HEADER,
}),
).toBeNull();
});
it("returns null when the Post header is missing", () => {
expect(
parseOneClickUnsubscribe({
"list-unsubscribe": "<https://example.com/u/1>",
}),
).toBeNull();
});
it("returns null when the Post header has the wrong value", () => {
expect(
parseOneClickUnsubscribe({
"list-unsubscribe": "<https://example.com/u/1>",
"list-unsubscribe-post": "List-Unsubscribe=Something",
}),
).toBeNull();
});
it("matches headers and Post value case-insensitively", () => {
expect(
parseOneClickUnsubscribe({
"List-Unsubscribe": "<https://example.com/u/1>",
"List-Unsubscribe-Post": "list-unsubscribe=ONE-CLICK",
}),
).toBe("https://example.com/u/1");
});
it("ignores plaintext http URLs", () => {
expect(
parseOneClickUnsubscribe({
"list-unsubscribe": "<http://example.com/u/1>",
"list-unsubscribe-post": POST_HEADER,
}),
).toBeNull();
});
it("returns null when there are no headers", () => {
expect(parseOneClickUnsubscribe({})).toBeNull();
});
});
describe("sendOneClickUnsubscribe", () => {
it("POSTs the one-click body and returns true on success", async () => {
let captured: { method: string; contentType: string; body: string } | null =
null;
server.use(
http.post("https://example.com/u/1", async ({ request }) => {
captured = {
method: request.method,
contentType: request.headers.get("content-type") ?? "",
body: await request.text(),
};
return HttpResponse.text("ok");
}),
);
const ok = await sendOneClickUnsubscribe("https://example.com/u/1");
expect(ok).toBe(true);
expect(captured).toEqual({
method: "POST",
contentType: "application/x-www-form-urlencoded",
body: POST_HEADER,
});
});
it("returns false on a non-ok response", async () => {
server.use(
http.post("https://example.com/u/1", () =>
HttpResponse.text("nope", { status: 404 }),
),
);
expect(await sendOneClickUnsubscribe("https://example.com/u/1")).toBe(
false,
);
});
it("returns false (no throw) on a network error", async () => {
server.use(
http.post("https://example.com/u/1", () => HttpResponse.error()),
);
expect(await sendOneClickUnsubscribe("https://example.com/u/1")).toBe(
false,
);
});
});
describe("sendUnsubscribes", () => {
it("de-dupes URLs and bumps unsubscribes_sent by the success count", async () => {
const env = createMockEnv() as unknown as Env;
let hitsOne = 0;
let hitsTwo = 0;
server.use(
http.post("https://example.com/a", () => {
hitsOne += 1;
return HttpResponse.text("ok");
}),
http.post("https://example.com/b", () => {
hitsTwo += 1;
return HttpResponse.text("ok");
}),
);
await sendUnsubscribes(
[
"https://example.com/a",
"https://example.com/a",
"https://example.com/b",
],
env,
);
expect(hitsOne).toBe(1);
expect(hitsTwo).toBe(1);
const counters = await getCounters(env.EMAIL_STORAGE);
expect(counters.unsubscribes_sent).toBe(2);
});
it("only counts successful requests", async () => {
const env = createMockEnv() as unknown as Env;
server.use(
http.post("https://example.com/ok", () => HttpResponse.text("ok")),
http.post("https://example.com/bad", () =>
HttpResponse.text("no", { status: 500 }),
),
);
await sendUnsubscribes(
["https://example.com/ok", "https://example.com/bad"],
env,
);
const counters = await getCounters(env.EMAIL_STORAGE);
expect(counters.unsubscribes_sent).toBe(1);
});
it("does nothing for an empty list", async () => {
const env = createMockEnv() as unknown as Env;
await sendUnsubscribes([], env);
const counters = await getCounters(env.EMAIL_STORAGE);
expect(counters.unsubscribes_sent).toBe(0);
});
});
+83
View File
@@ -0,0 +1,83 @@
import { Env } from "../types";
import { UNSUBSCRIBE_TIMEOUT_MS } from "../config/constants";
import { bumpCounters } from "../application/stats";
import { logger } from "../infrastructure/logger";
/**
* Extract a one-click unsubscribe URL from a stored email's headers per
* RFC 8058. Returns the first `https:` URL in `List-Unsubscribe` only when
* `List-Unsubscribe-Post: List-Unsubscribe=One-Click` is also present — that
* Post header is what authorises an unattended one-click POST. `mailto:` and
* plaintext `http:` links are ignored (Workers cannot send SMTP and we never
* unsubscribe over plaintext). Header keys are matched case-insensitively;
* `EmailData.headers` already lowercases them, but we don't rely on it.
*/
export function parseOneClickUnsubscribe(
headers: Record<string, string>,
): string | null {
let listUnsubscribe = "";
let post = "";
for (const [key, value] of Object.entries(headers)) {
const k = key.toLowerCase();
if (k === "list-unsubscribe") listUnsubscribe = value;
else if (k === "list-unsubscribe-post") post = value;
}
if (post.trim().toLowerCase() !== "list-unsubscribe=one-click") return null;
const matches = listUnsubscribe.match(/<([^>]+)>/g);
if (!matches) return null;
for (const token of matches) {
const url = token.slice(1, -1).trim();
if (/^https:\/\//i.test(url)) return url;
}
return null;
}
/**
* Fire a single RFC 8058 one-click unsubscribe POST. Returns whether the
* endpoint accepted it. Never throws: network/timeout errors are logged and
* reported as a failure so callers can keep going.
*/
export async function sendOneClickUnsubscribe(url: string): Promise<boolean> {
try {
const res = await fetch(url, {
method: "POST",
redirect: "follow",
signal: AbortSignal.timeout(UNSUBSCRIBE_TIMEOUT_MS),
headers: {
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "kill-the-news/1.0",
},
body: "List-Unsubscribe=One-Click",
});
return res.ok;
} catch (error) {
logger.warn("One-click unsubscribe failed", { url, error: String(error) });
return false;
}
}
/**
* Send one-click unsubscribe requests for a batch of URLs (de-duplicated) and
* record the number that succeeded in the `unsubscribes_sent` counter. Never
* throws — intended to run in the background via ctx.waitUntil on feed deletion.
*/
export async function sendUnsubscribes(
urls: string[],
env: Env,
): Promise<void> {
const unique = Array.from(new Set(urls.filter(Boolean)));
if (unique.length === 0) return;
const results = await Promise.allSettled(
unique.map((url) => sendOneClickUnsubscribe(url)),
);
const succeeded = results.filter(
(r) => r.status === "fulfilled" && r.value,
).length;
if (succeeded > 0) {
await bumpCounters(env.EMAIL_STORAGE, { unsubscribes_sent: succeeded });
}
}
+30
View File
@@ -0,0 +1,30 @@
import { Env } from "../types";
export function baseUrl(env: Env): string {
return `https://${env.DOMAIN}`;
}
export function feedRssUrl(feedId: string, env: Env): string {
return `${baseUrl(env)}/rss/${feedId}`;
}
export function feedAtomUrl(feedId: string, env: Env): string {
return `${baseUrl(env)}/atom/${feedId}`;
}
export function feedUrl(
format: "rss" | "atom",
feedId: string,
env: Env,
): string {
return format === "rss" ? feedRssUrl(feedId, env) : feedAtomUrl(feedId, env);
}
export function feedEmailAddress(feedId: string, env: Env): string {
return `${feedId}@${env.EMAIL_DOMAIN ?? env.DOMAIN}`;
}
export function feedTopicPattern(env: Env): RegExp {
const escaped = env.DOMAIN.replaceAll(".", "\\.");
return new RegExp(`^https://${escaped}/(rss|atom)/([^/]+)$`);
}
@@ -0,0 +1,21 @@
import { describe, it, expect } from "vitest";
import { createMockEnv } from "../test/setup";
import { WebSubSubscriptionRepository } from "./websub-subscription-repository";
import { FeedId } from "../domain/value-objects/feed-id";
import type { Env, WebSubSubscription } from "../types";
const mockEnv = () => createMockEnv() as unknown as Env;
const fid = FeedId.unchecked("a.b.42");
describe("WebSubSubscriptionRepository", () => {
it("round-trips subscriptions and counts feeds with subscribers", async () => {
const repo = new WebSubSubscriptionRepository(mockEnv().EMAIL_STORAGE);
expect(await repo.get(fid)).toEqual([]);
const subs: WebSubSubscription[] = [
{ callbackUrl: "https://r.example/cb", expiresAt: 9999 },
];
await repo.save(fid, subs);
expect(await repo.get(fid)).toEqual(subs);
expect(await repo.countKeys()).toBe(1);
});
});
@@ -0,0 +1,49 @@
import { Env, WebSubSubscription } from "../types";
import { feedKeys } from "../domain/feed-keys";
import { FeedId } from "../domain/value-objects/feed-id";
import { logger } from "./logger";
/**
* KV access for per-feed WebSub subscriber lists (`websub:subs:<feedId>`).
*/
export class WebSubSubscriptionRepository {
constructor(private readonly kv: KVNamespace) {}
static from(env: Env): WebSubSubscriptionRepository {
return new WebSubSubscriptionRepository(env.EMAIL_STORAGE);
}
async get(feedId: FeedId): Promise<WebSubSubscription[]> {
const raw = await this.kv.get(feedKeys.websub(feedId.value), "json");
return (raw as WebSubSubscription[] | null) ?? [];
}
async save(
feedId: FeedId,
subscriptions: WebSubSubscription[],
): Promise<void> {
await this.kv.put(
feedKeys.websub(feedId.value),
JSON.stringify(subscriptions),
);
}
/** Number of feeds that currently hold at least one WebSub subscription. */
async countKeys(): Promise<number> {
const prefix = feedKeys.websubPrefix();
let total = 0;
let cursor: string | undefined;
try {
do {
const listed = await this.kv.list({ prefix, cursor, limit: 1000 });
total += listed.keys.length;
cursor = listed.list_complete ? undefined : listed.cursor;
} while (cursor);
} catch (error) {
logger.error("Error counting subscription keys", {
error: String(error),
});
}
return total;
}
}
+543
View File
@@ -0,0 +1,543 @@
import { describe, it, expect } from "vitest";
import { http, HttpResponse } from "msw";
import { server, createMockEnv } from "../test/setup";
import {
buildHmacSignature,
getSubscriptions,
saveSubscriptions,
notifySubscribers,
verifyAndStoreSubscription,
verifyAndDeleteSubscription,
} from "./websub";
import { FeedId } from "../domain/value-objects/feed-id";
import type { Env, WebSubSubscription } from "../types";
const mockEnv = () => createMockEnv() as unknown as Env;
const fid = (value: string) => FeedId.unchecked(value);
describe("buildHmacSignature", () => {
it("returns sha256= prefixed hex", async () => {
const sig = await buildHmacSignature("hello", "secret");
expect(sig).toMatch(/^sha256=[0-9a-f]{64}$/);
});
it("produces different sigs for different secrets", async () => {
const a = await buildHmacSignature("body", "secret1");
const b = await buildHmacSignature("body", "secret2");
expect(a).not.toBe(b);
});
it("produces the same sig for same inputs", async () => {
const a = await buildHmacSignature("body", "secret");
const b = await buildHmacSignature("body", "secret");
expect(a).toBe(b);
});
});
describe("getSubscriptions / saveSubscriptions", () => {
it("returns empty array when no subs exist", async () => {
const env = mockEnv();
expect(await getSubscriptions(fid("feed1"), env)).toEqual([]);
});
it("round-trips stored subscriptions", async () => {
const env = mockEnv();
const subs: WebSubSubscription[] = [
{
callbackUrl: "https://reader.example/sub",
expiresAt: Date.now() + 60000,
},
];
await saveSubscriptions(fid("feed1"), subs, env);
expect(await getSubscriptions(fid("feed1"), env)).toEqual(subs);
});
it("uses the correct KV key", async () => {
const env = mockEnv();
await saveSubscriptions(fid("abc"), [], env);
expect(
await env.EMAIL_STORAGE.get("websub:subs:abc", { type: "json" }),
).toEqual([]);
});
});
describe("notifySubscribers", () => {
it("does nothing when no subscriptions exist", async () => {
const env = mockEnv();
let called = false;
server.use(
http.post("https://reader.example/callback", () => {
called = true;
return HttpResponse.text("ok");
}),
);
await notifySubscribers(fid("feed1"), env);
expect(called).toBe(false);
});
it("does nothing when feed metadata missing", async () => {
const env = mockEnv();
const subs: WebSubSubscription[] = [
{
callbackUrl: "https://reader.example/callback",
expiresAt: Date.now() + 60000,
},
];
await saveSubscriptions(fid("feed1"), subs, env);
let called = false;
server.use(
http.post("https://reader.example/callback", () => {
called = true;
return HttpResponse.text("ok");
}),
);
await notifySubscribers(fid("feed1"), env);
expect(called).toBe(false);
});
it("POSTs feed XML to subscriber callback", async () => {
const env = mockEnv();
await env.EMAIL_STORAGE.put(
"feed:feed1:metadata",
JSON.stringify({ emails: [] }),
);
await env.EMAIL_STORAGE.put(
"feed:feed1:config",
JSON.stringify({
title: "Test Feed",
language: "en",
created_at: Date.now(),
}),
);
const subs: WebSubSubscription[] = [
{
callbackUrl: "https://reader.example/callback",
expiresAt: Date.now() + 60000,
},
];
await saveSubscriptions(fid("feed1"), subs, env);
let receivedBody = "";
let receivedContentType = "";
server.use(
http.post("https://reader.example/callback", async ({ request }) => {
receivedBody = await request.text();
receivedContentType = request.headers.get("Content-Type") ?? "";
return HttpResponse.text("ok");
}),
);
await notifySubscribers(fid("feed1"), env);
expect(receivedBody).toContain("<?xml");
expect(receivedContentType).toContain("application/rss+xml");
});
it("includes X-Hub-Signature-256 header when secret set (no X-Hub-Signature)", async () => {
const env = mockEnv();
await env.EMAIL_STORAGE.put(
"feed:feed1:metadata",
JSON.stringify({ emails: [] }),
);
await env.EMAIL_STORAGE.put(
"feed:feed1:config",
JSON.stringify({
title: "Test Feed",
language: "en",
created_at: Date.now(),
}),
);
const subs: WebSubSubscription[] = [
{
callbackUrl: "https://reader.example/callback",
expiresAt: Date.now() + 60000,
secret: "mysecret",
},
];
await saveSubscriptions(fid("feed1"), subs, env);
let receivedSig256 = "";
let receivedSig = "";
server.use(
http.post("https://reader.example/callback", async ({ request }) => {
receivedSig256 = request.headers.get("X-Hub-Signature-256") ?? "";
receivedSig = request.headers.get("X-Hub-Signature") ?? "";
return HttpResponse.text("ok");
}),
);
await notifySubscribers(fid("feed1"), env);
expect(receivedSig256).toMatch(/^sha256=[0-9a-f]{64}$/);
expect(receivedSig).toBe(""); // legacy header should NOT be sent
});
it("POSTs Atom feed XML with correct Content-Type to Atom subscriber", async () => {
const env = mockEnv();
await env.EMAIL_STORAGE.put(
"feed:feed1:metadata",
JSON.stringify({ emails: [] }),
);
await env.EMAIL_STORAGE.put(
"feed:feed1:config",
JSON.stringify({
title: "Test Feed",
language: "en",
created_at: Date.now(),
}),
);
const subs: WebSubSubscription[] = [
{
callbackUrl: "https://atom-reader.example/callback",
expiresAt: Date.now() + 60000,
format: "atom",
},
];
await saveSubscriptions(fid("feed1"), subs, env);
let receivedContentType = "";
let receivedLink = "";
server.use(
http.post("https://atom-reader.example/callback", async ({ request }) => {
receivedContentType = request.headers.get("Content-Type") ?? "";
receivedLink = request.headers.get("Link") ?? "";
return HttpResponse.text("ok");
}),
);
await notifySubscribers(fid("feed1"), env);
expect(receivedContentType).toContain("application/atom+xml");
expect(receivedLink).toContain(`/atom/feed1`);
expect(receivedLink).toContain(`rel="self"`);
});
it("notifies RSS and Atom subscribers independently with correct formats", async () => {
const env = mockEnv();
await env.EMAIL_STORAGE.put(
"feed:feed1:metadata",
JSON.stringify({ emails: [] }),
);
await env.EMAIL_STORAGE.put(
"feed:feed1:config",
JSON.stringify({
title: "Test Feed",
language: "en",
created_at: Date.now(),
}),
);
const subs: WebSubSubscription[] = [
{
callbackUrl: "https://rss-reader.example/callback",
expiresAt: Date.now() + 60000,
format: "rss",
},
{
callbackUrl: "https://atom-reader.example/callback",
expiresAt: Date.now() + 60000,
format: "atom",
},
];
await saveSubscriptions(fid("feed1"), subs, env);
const received: Record<string, string> = {};
server.use(
http.post("https://rss-reader.example/callback", async ({ request }) => {
received.rss = request.headers.get("Content-Type") ?? "";
return HttpResponse.text("ok");
}),
http.post("https://atom-reader.example/callback", async ({ request }) => {
received.atom = request.headers.get("Content-Type") ?? "";
return HttpResponse.text("ok");
}),
);
await notifySubscribers(fid("feed1"), env);
expect(received.rss).toContain("application/rss+xml");
expect(received.atom).toContain("application/atom+xml");
});
it("prunes expired subscriptions and does not notify them", async () => {
const env = mockEnv();
await env.EMAIL_STORAGE.put(
"feed:feed1:metadata",
JSON.stringify({ emails: [] }),
);
await env.EMAIL_STORAGE.put(
"feed:feed1:config",
JSON.stringify({
title: "Test Feed",
language: "en",
created_at: Date.now(),
}),
);
const subs: WebSubSubscription[] = [
{
callbackUrl: "https://expired.example/callback",
expiresAt: Date.now() - 1000,
},
{
callbackUrl: "https://active.example/callback",
expiresAt: Date.now() + 60000,
},
];
await saveSubscriptions(fid("feed1"), subs, env);
const notified: string[] = [];
server.use(
http.post("https://expired.example/callback", () => {
notified.push("expired");
return HttpResponse.text("ok");
}),
http.post("https://active.example/callback", () => {
notified.push("active");
return HttpResponse.text("ok");
}),
);
await notifySubscribers(fid("feed1"), env);
expect(notified).toEqual(["active"]);
const remaining = await getSubscriptions(fid("feed1"), env);
expect(remaining).toHaveLength(1);
expect(remaining[0].callbackUrl).toBe("https://active.example/callback");
});
});
describe("verifyAndStoreSubscription", () => {
it("stores subscription and returns true when callback echoes challenge", async () => {
const env = mockEnv();
server.use(
http.get("https://reader.example/callback", ({ request }) => {
const url = new URL(request.url);
const challenge = url.searchParams.get("hub.challenge") ?? "";
return HttpResponse.text(challenge);
}),
);
const result = await verifyAndStoreSubscription(
fid("feed1"),
"https://reader.example/callback",
undefined,
86400,
"rss",
env,
);
expect(result).toBe(true);
const subs = await getSubscriptions(fid("feed1"), env);
expect(subs).toHaveLength(1);
expect(subs[0].callbackUrl).toBe("https://reader.example/callback");
expect(subs[0].expiresAt).toBeGreaterThan(Date.now());
});
it("stores format=atom and sends atom topic URL in verification request", async () => {
const env = mockEnv();
let receivedTopic = "";
server.use(
http.get("https://reader.example/callback", ({ request }) => {
const url = new URL(request.url);
receivedTopic = url.searchParams.get("hub.topic") ?? "";
const challenge = url.searchParams.get("hub.challenge") ?? "";
return HttpResponse.text(challenge);
}),
);
const result = await verifyAndStoreSubscription(
fid("feed1"),
"https://reader.example/callback",
undefined,
86400,
"atom",
env,
);
expect(result).toBe(true);
expect(receivedTopic).toContain("/atom/feed1");
const subs = await getSubscriptions(fid("feed1"), env);
expect(subs[0].format).toBe("atom");
});
it("returns false and does not store when callback returns wrong challenge", async () => {
const env = mockEnv();
server.use(
http.get("https://reader.example/callback", () =>
HttpResponse.text("wrong"),
),
);
const result = await verifyAndStoreSubscription(
fid("feed1"),
"https://reader.example/callback",
undefined,
86400,
"rss",
env,
);
expect(result).toBe(false);
const subs = await getSubscriptions(fid("feed1"), env);
expect(subs).toHaveLength(0);
});
it("updates existing subscription with same callback", async () => {
const env = mockEnv();
const existing: WebSubSubscription[] = [
{ callbackUrl: "https://reader.example/callback", expiresAt: 1000 },
];
await saveSubscriptions(fid("feed1"), existing, env);
server.use(
http.get("https://reader.example/callback", ({ request }) => {
const challenge =
new URL(request.url).searchParams.get("hub.challenge") ?? "";
return HttpResponse.text(challenge);
}),
);
const result = await verifyAndStoreSubscription(
fid("feed1"),
"https://reader.example/callback",
"newsecret",
3600,
"rss",
env,
);
expect(result).toBe(true);
const subs = await getSubscriptions(fid("feed1"), env);
expect(subs).toHaveLength(1);
expect(subs[0].secret).toBe("newsecret");
});
it("returns false when callback fetch fails", async () => {
const env = mockEnv();
server.use(
http.get("https://reader.example/callback", () => HttpResponse.error()),
);
const result = await verifyAndStoreSubscription(
fid("feed1"),
"https://reader.example/callback",
undefined,
86400,
"rss",
env,
);
expect(result).toBe(false);
const subs = await getSubscriptions(fid("feed1"), env);
expect(subs).toHaveLength(0);
});
it("returns false when callback returns non-ok HTTP status", async () => {
const env = mockEnv();
server.use(
http.get("https://reader.example/callback", ({ request }) => {
const challenge =
new URL(request.url).searchParams.get("hub.challenge") ?? "";
return HttpResponse.text(challenge, { status: 500 });
}),
);
const result = await verifyAndStoreSubscription(
fid("feed1"),
"https://reader.example/callback",
undefined,
86400,
"rss",
env,
);
expect(result).toBe(false);
const subs = await getSubscriptions(fid("feed1"), env);
expect(subs).toHaveLength(0);
});
});
describe("verifyAndDeleteSubscription", () => {
it("removes subscription and returns true when callback echoes challenge", async () => {
const env = mockEnv();
await saveSubscriptions(
fid("feed1"),
[
{
callbackUrl: "https://reader.example/callback",
expiresAt: Date.now() + 60000,
},
],
env,
);
server.use(
http.get("https://reader.example/callback", ({ request }) => {
const challenge =
new URL(request.url).searchParams.get("hub.challenge") ?? "";
return HttpResponse.text(challenge);
}),
);
const result = await verifyAndDeleteSubscription(
fid("feed1"),
"https://reader.example/callback",
env,
);
expect(result).toBe(true);
const subs = await getSubscriptions(fid("feed1"), env);
expect(subs).toHaveLength(0);
});
it("returns false and leaves subscription intact when callback returns wrong challenge", async () => {
const env = mockEnv();
await saveSubscriptions(
fid("feed1"),
[
{
callbackUrl: "https://reader.example/callback",
expiresAt: Date.now() + 60000,
},
],
env,
);
server.use(
http.get("https://reader.example/callback", () =>
HttpResponse.text("nope"),
),
);
const result = await verifyAndDeleteSubscription(
fid("feed1"),
"https://reader.example/callback",
env,
);
expect(result).toBe(false);
const subs = await getSubscriptions(fid("feed1"), env);
expect(subs).toHaveLength(1);
});
it("returns false and leaves subscription intact when callback fetch fails", async () => {
const env = mockEnv();
await saveSubscriptions(
fid("feed1"),
[
{
callbackUrl: "https://reader.example/callback",
expiresAt: Date.now() + 60000,
},
],
env,
);
server.use(
http.get("https://reader.example/callback", () => HttpResponse.error()),
);
const result = await verifyAndDeleteSubscription(
fid("feed1"),
"https://reader.example/callback",
env,
);
expect(result).toBe(false);
const subs = await getSubscriptions(fid("feed1"), env);
expect(subs).toHaveLength(1);
});
});
+232
View File
@@ -0,0 +1,232 @@
import { Env, FeedConfig, EmailData, WebSubSubscription } from "../types";
import { generateRssFeed, generateAtomFeed } from "./feed-generator";
import { baseUrl, feedRssUrl, feedAtomUrl, feedUrl } from "./urls";
import { FeedRepository } from "./feed-repository";
import { WebSubSubscriptionRepository } from "./websub-subscription-repository";
import { FeedId } from "../domain/value-objects/feed-id";
export async function getSubscriptions(
feedId: FeedId,
env: Env,
): Promise<WebSubSubscription[]> {
return WebSubSubscriptionRepository.from(env).get(feedId);
}
export async function saveSubscriptions(
feedId: FeedId,
subscriptions: WebSubSubscription[],
env: Env,
): Promise<void> {
await WebSubSubscriptionRepository.from(env).save(feedId, subscriptions);
}
export async function buildHmacSignature(
body: string,
secret: string,
): Promise<string> {
const key = await crypto.subtle.importKey(
"raw",
new TextEncoder().encode(secret),
{ name: "HMAC", hash: "SHA-256" },
false,
["sign"],
);
const sig = await crypto.subtle.sign(
"HMAC",
key,
new TextEncoder().encode(body),
);
const hex = Array.from(new Uint8Array(sig))
.map((b) => b.toString(16).padStart(2, "0"))
.join("");
return `sha256=${hex}`;
}
async function buildFeedXml(
feedId: FeedId,
env: Env,
format: "rss" | "atom" = "rss",
): Promise<string | null> {
const repo = FeedRepository.from(env);
const [feedMetadata, rawConfig] = await Promise.all([
repo.getMetadata(feedId),
repo.getConfig(feedId),
]);
if (!feedMetadata) return null;
const base = baseUrl(env);
const feedConfig: FeedConfig = rawConfig ?? {
title: `Newsletter Feed ${feedId.value}`,
description: "Converted email newsletter",
language: "en",
created_at: Date.now(),
};
const emails = feedMetadata.emails.slice(0, 20);
const emailsData = (
await Promise.all(emails.map((m) => repo.getEmail(m.key)))
).filter((d): d is EmailData => d !== null);
if (format === "atom") {
return generateAtomFeed(
feedConfig,
emailsData,
base,
feedId.value,
feedAtomUrl(feedId.value, env),
);
}
return generateRssFeed(feedConfig, emailsData, base, feedId.value);
}
export async function notifySubscribers(
feedId: FeedId,
env: Env,
): Promise<void> {
const subs = await getSubscriptions(feedId, env);
const now = Date.now();
const active = subs.filter((s) => s.expiresAt > now);
if (active.length === 0) {
if (active.length < subs.length) {
await saveSubscriptions(feedId, active, env);
}
return;
}
const rssSubs = active.filter((s) => (s.format ?? "rss") === "rss");
const atomSubs = active.filter((s) => s.format === "atom");
const [rssFeed, atomFeed] = await Promise.all([
rssSubs.length > 0 ? buildFeedXml(feedId, env, "rss") : null,
atomSubs.length > 0 ? buildFeedXml(feedId, env, "atom") : null,
]);
if (!rssFeed && !atomFeed) return;
const base = baseUrl(env);
const deliver = async (
sub: WebSubSubscription,
feedXml: string,
contentType: string,
selfPath: string,
) => {
const linkHeader = `<${base}/hub>; rel="hub", <${base}${selfPath}>; rel="self"`;
const headers: Record<string, string> = {
"Content-Type": contentType,
Link: linkHeader,
};
if (sub.secret) {
headers["X-Hub-Signature-256"] = await buildHmacSignature(
feedXml,
sub.secret,
);
}
const res = await fetch(sub.callbackUrl, {
method: "POST",
headers,
body: feedXml,
});
if (!res.ok) {
console.error(
`WebSub: delivery failed ${sub.callbackUrl}: ${res.status}`,
);
}
};
await Promise.allSettled([
...(rssFeed
? rssSubs.map((sub) =>
deliver(sub, rssFeed, "application/rss+xml", `/rss/${feedId.value}`),
)
: []),
...(atomFeed
? atomSubs.map((sub) =>
deliver(
sub,
atomFeed,
"application/atom+xml",
`/atom/${feedId.value}`,
),
)
: []),
]);
if (active.length < subs.length) {
await saveSubscriptions(feedId, active, env);
}
}
async function verifyCallback(
callbackUrl: string,
params: Record<string, string>,
): Promise<boolean> {
const challenge = crypto.randomUUID().replace(/-/g, "");
const url = new URL(callbackUrl);
for (const [k, v] of Object.entries(params)) url.searchParams.set(k, v);
url.searchParams.set("hub.challenge", challenge);
let res: Response;
try {
res = await fetch(url.toString());
} catch {
return false;
}
if (!res.ok) return false;
return (await res.text()).trim() === challenge;
}
export async function verifyAndStoreSubscription(
feedId: FeedId,
callbackUrl: string,
secret: string | undefined,
leaseSeconds: number,
format: "rss" | "atom",
env: Env,
): Promise<boolean> {
const verified = await verifyCallback(callbackUrl, {
"hub.mode": "subscribe",
"hub.topic": feedUrl(format, feedId.value, env),
"hub.lease_seconds": String(leaseSeconds),
});
if (!verified) return false;
const subs = await getSubscriptions(feedId, env);
const idx = subs.findIndex((s) => s.callbackUrl === callbackUrl);
const entry: WebSubSubscription = {
callbackUrl,
expiresAt: Date.now() + leaseSeconds * 1000,
format,
...(secret ? { secret } : {}),
};
if (idx >= 0) {
subs[idx] = entry;
} else {
subs.push(entry);
}
await saveSubscriptions(feedId, subs, env);
return true;
}
export async function verifyAndDeleteSubscription(
feedId: FeedId,
callbackUrl: string,
env: Env,
): Promise<boolean> {
const verified = await verifyCallback(callbackUrl, {
"hub.mode": "unsubscribe",
"hub.topic": feedRssUrl(feedId.value, env),
});
if (!verified) return false;
const subs = await getSubscriptions(feedId, env);
await saveSubscriptions(
feedId,
subs.filter((s) => s.callbackUrl !== callbackUrl),
env,
);
return true;
}
+17
View File
@@ -0,0 +1,17 @@
import { Context } from "hono";
/**
* Schedules a fire-and-forget background task. The HTTP edge adapts this over
* `ctx.waitUntil`; the application/domain layers depend on this plain function
* type instead of Hono's `Context`.
*/
export type BackgroundScheduler = (task: Promise<unknown>) => void;
/** Calls ctx.waitUntil() without throwing when the ExecutionContext is absent (e.g. Node tests). */
export function waitUntilSafe(c: Context, promise: Promise<unknown>): void {
try {
c.executionCtx.waitUntil(promise);
} catch {
// ExecutionContext unavailable in Node test environment — ignore.
}
}
+634 -18
View File
@@ -1,7 +1,9 @@
import { describe, it, expect, beforeEach } from "vitest";
import { http, HttpResponse } from "msw";
import { Hono } from "hono";
import app from "./admin";
import { createMockEnv } from "../test/setup";
import { createMockEnv, server } from "../test/setup";
import { getCounters } from "../application/stats";
import { Env } from "../types";
describe("Admin Routes", () => {
@@ -11,10 +13,11 @@ describe("Admin Routes", () => {
let loginAndGetCookie: () => Promise<string>;
beforeEach(() => {
mockEnv = createMockEnv();
mockEnv = createMockEnv() as unknown as Env;
testApp = new Hono();
testApp.route("/admin", app);
request = (path, init = {}) => testApp.request(path, init, mockEnv);
request = (path, init = {}) =>
Promise.resolve(testApp.request(path, init, mockEnv));
loginAndGetCookie = async () => {
const formData = new FormData();
formData.append("password", "test-password");
@@ -93,6 +96,7 @@ describe("Admin Routes", () => {
const res = await request("/admin", {
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
});
expect(res.status).toBe(200);
@@ -138,12 +142,13 @@ describe("Admin Routes", () => {
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
body: formData,
});
expect(res.status).toBe(302); // Redirects back to dashboard
expect(res.headers.get("Location")).toBe("/admin?view=list");
expect(res.headers.get("Location")).toBe("/admin?view=list#your-feeds");
// Verify feed was created in KV
const feedList = (await mockEnv.EMAIL_STORAGE.get(
@@ -161,8 +166,8 @@ describe("Admin Routes", () => {
"json",
);
expect(feedConfig).toBeTruthy();
expect(feedConfig.title).toBe("Test Feed");
expect(feedConfig.description).toBe("Test Description");
expect((feedConfig as any).title).toBe("Test Feed");
expect((feedConfig as any).description).toBe("Test Description");
});
it("should reject feed creation with missing title", async () => {
@@ -174,6 +179,7 @@ describe("Admin Routes", () => {
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
body: formData,
});
@@ -186,6 +192,24 @@ describe("Admin Routes", () => {
});
});
describe("API Feed Update", () => {
it("returns 400 with structured validation error for empty title", async () => {
const authCookie = await loginAndGetCookie();
const res = await request("/admin/api/feeds/test-feed/update", {
method: "POST",
headers: {
Cookie: authCookie,
"Content-Type": "application/json",
Origin: "https://test.getmynews.app",
},
body: JSON.stringify({ title: "", description: "desc" }),
});
expect(res.status).toBe(400);
const body = await res.json<{ success: boolean }>();
expect(body.success).toBe(false);
});
});
describe("Feed Management", () => {
it("should prevent feed deletion without authentication", async () => {
const res = await request("/admin/feeds/test-feed/delete", {
@@ -223,6 +247,7 @@ describe("Admin Routes", () => {
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
body: formData,
});
@@ -241,6 +266,7 @@ describe("Admin Routes", () => {
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
});
@@ -273,6 +299,7 @@ describe("Admin Routes", () => {
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
body: formData,
});
@@ -285,20 +312,137 @@ describe("Admin Routes", () => {
)) as { feeds: Array<{ id: string; title: string }> } | null;
const feedId = feedList?.feeds[0].id as string;
const deleteRes = await request(`/admin/feeds/${feedId}/delete?view=list`, {
const deleteRes = await request(
`/admin/feeds/${feedId}/delete?view=list`,
{
method: "POST",
headers: {
Cookie: authCookie,
Accept: "application/json",
Origin: "https://test.getmynews.app",
},
});
},
);
expect(deleteRes.status).toBe(200);
const payload = await deleteRes.json();
const payload = (await deleteRes.json()) as any;
expect(payload.ok).toBe(true);
expect(payload.feedId).toBe(feedId);
});
it("fires one-click unsubscribe requests on feed deletion and bumps the counter", async () => {
const authCookie = await loginAndGetCookie();
const formData = new FormData();
formData.append("title", "Unsub Feed");
await request("/admin/feeds/create", {
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
body: formData,
});
const feedList = (await mockEnv.EMAIL_STORAGE.get(
"feeds:list",
"json",
)) as {
feeds: Array<{ id: string }>;
} | null;
const feedId = feedList?.feeds[0].id as string;
// Simulate an ingested email having captured an unsubscribe URL.
await mockEnv.EMAIL_STORAGE.put(
`feed:${feedId}:metadata`,
JSON.stringify({
emails: [],
unsubscribe: { "[email protected]": "https://example.com/u/1" },
}),
);
let unsubHit = false;
server.use(
http.post("https://example.com/u/1", () => {
unsubHit = true;
return HttpResponse.text("ok");
}),
);
const pending: Promise<unknown>[] = [];
const ctx = {
waitUntil: (p: Promise<unknown>) => pending.push(p),
passThroughOnException: () => {},
} as unknown as ExecutionContext;
const deleteRes = await testApp.request(
`/admin/feeds/${feedId}/delete`,
{
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
},
mockEnv,
ctx,
);
expect(deleteRes.status).toBe(302);
await Promise.all(pending);
expect(unsubHit).toBe(true);
const counters = await getCounters(mockEnv.EMAIL_STORAGE);
expect(counters.unsubscribes_sent).toBe(1);
});
it("sends no unsubscribe requests when the feed has none", async () => {
const authCookie = await loginAndGetCookie();
const formData = new FormData();
formData.append("title", "No Unsub Feed");
await request("/admin/feeds/create", {
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
body: formData,
});
const feedList = (await mockEnv.EMAIL_STORAGE.get(
"feeds:list",
"json",
)) as {
feeds: Array<{ id: string }>;
} | null;
const feedId = feedList?.feeds[0].id as string;
const pending: Promise<unknown>[] = [];
const ctx = {
waitUntil: (p: Promise<unknown>) => pending.push(p),
passThroughOnException: () => {},
} as unknown as ExecutionContext;
await testApp.request(
`/admin/feeds/${feedId}/delete`,
{
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
},
mockEnv,
ctx,
);
await Promise.all(pending);
const counters = await getCounters(mockEnv.EMAIL_STORAGE);
expect(counters.unsubscribes_sent).toBe(0);
});
it("should allow bulk feed deletion with valid authentication", async () => {
const authCookie = await loginAndGetCookie();
@@ -308,7 +452,10 @@ describe("Admin Routes", () => {
formData.append("description", "Test");
const createRes = await request("/admin/feeds/create", {
method: "POST",
headers: { Cookie: authCookie },
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
body: formData,
});
expect(createRes.status).toBe(302);
@@ -329,13 +476,17 @@ describe("Admin Routes", () => {
const bulkDeleteRes = await request("/admin/feeds/bulk-delete", {
method: "POST",
headers: { Cookie: authCookie },
headers: { Cookie: authCookie, Origin: "https://test.getmynews.app" },
body: bulkForm,
});
expect(bulkDeleteRes.status).toBe(302);
expect(bulkDeleteRes.headers.get("Location")).toContain("/admin?view=list");
expect(bulkDeleteRes.headers.get("Location")).toContain("message=bulkDeleted");
expect(bulkDeleteRes.headers.get("Location")).toContain(
"/admin?view=list",
);
expect(bulkDeleteRes.headers.get("Location")).toContain(
"message=bulkDeleted",
);
const feedListAfter = (await mockEnv.EMAIL_STORAGE.get(
"feeds:list",
@@ -347,6 +498,102 @@ describe("Admin Routes", () => {
});
});
describe("Proxy authentication", () => {
const TRUSTED_IP = "10.0.0.1";
const PROXY_SECRET = "my-proxy-secret";
function proxyEnv() {
return {
...createMockEnv(),
PROXY_TRUSTED_IPS: TRUSTED_IP,
PROXY_AUTH_SECRET: PROXY_SECRET,
} as unknown as Env;
}
function makeProxyRequest(
path: string,
headers: Record<string, string> = {},
) {
const proxyApp = new Hono();
proxyApp.route("/admin", app);
return proxyApp.request(path, { headers }, proxyEnv());
}
it("grants access when IP, secret and Remote-User are all valid", async () => {
const res = await makeProxyRequest("/admin", {
"CF-Connecting-IP": TRUSTED_IP,
"X-Auth-Proxy-Secret": PROXY_SECRET,
"Remote-User": "alice",
});
expect(res.status).toBe(200);
});
it("grants access using X-Forwarded-User instead of Remote-User", async () => {
const res = await makeProxyRequest("/admin", {
"CF-Connecting-IP": TRUSTED_IP,
"X-Auth-Proxy-Secret": PROXY_SECRET,
"X-Forwarded-User": "bob",
});
expect(res.status).toBe(200);
});
it("rejects when IP is not in trusted list", async () => {
const res = await makeProxyRequest("/admin", {
"CF-Connecting-IP": "1.2.3.4",
"X-Auth-Proxy-Secret": PROXY_SECRET,
"Remote-User": "alice",
});
expect(res.status).toBe(302);
});
it("rejects when secret is wrong", async () => {
const res = await makeProxyRequest("/admin", {
"CF-Connecting-IP": TRUSTED_IP,
"X-Auth-Proxy-Secret": "wrong-secret",
"Remote-User": "alice",
});
expect(res.status).toBe(302);
});
it("rejects when Remote-User is missing", async () => {
const res = await makeProxyRequest("/admin", {
"CF-Connecting-IP": TRUSTED_IP,
"X-Auth-Proxy-Secret": PROXY_SECRET,
});
expect(res.status).toBe(302);
});
it("falls back to cookie auth when proxy env vars are not configured", async () => {
const res = await request("/admin");
expect(res.status).toBe(302);
expect(res.headers.get("Location")).toBe("/admin/login");
});
it("falls back to cookie auth when only one proxy env var is configured", async () => {
const partialProxyApp = new Hono();
partialProxyApp.route("/admin", app);
const partialEnv = {
...createMockEnv(),
PROXY_TRUSTED_IPS: "10.0.0.1",
// PROXY_AUTH_SECRET intentionally absent
} as unknown as Env;
const res = await partialProxyApp.request(
"/admin",
{
headers: {
"CF-Connecting-IP": "10.0.0.1",
"Remote-User": "alice",
// No X-Auth-Proxy-Secret — proxy auth should NOT activate
},
},
partialEnv,
);
expect(res.status).toBe(302);
expect(res.headers.get("Location")).toBe("/admin/login");
});
});
describe("Email Management", () => {
it("should return JSON for email deletion when requested", async () => {
const authCookie = await loginAndGetCookie();
@@ -358,6 +605,7 @@ describe("Admin Routes", () => {
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
body: formData,
});
@@ -386,7 +634,9 @@ describe("Admin Routes", () => {
const feedMetadata = (await mockEnv.EMAIL_STORAGE.get(
feedMetadataKey,
"json",
)) as { emails: Array<{ key: string; subject: string; receivedAt: number }> } | null;
)) as {
emails: Array<{ key: string; subject: string; receivedAt: number }>;
} | null;
const updatedMetadata = {
emails: [
...(feedMetadata?.emails || []),
@@ -398,16 +648,20 @@ describe("Admin Routes", () => {
JSON.stringify(updatedMetadata),
);
const deleteRes = await request(`/admin/emails/${emailKey}/delete?feedId=${feedId}`, {
const deleteRes = await request(
`/admin/emails/${emailKey}/delete?feedId=${feedId}`,
{
method: "POST",
headers: {
Cookie: authCookie,
Accept: "application/json",
Origin: "https://test.getmynews.app",
},
});
},
);
expect(deleteRes.status).toBe(200);
const payload = await deleteRes.json();
const payload = (await deleteRes.json()) as any;
expect(payload.ok).toBe(true);
expect(payload.emailKey).toBe(emailKey);
@@ -417,9 +671,371 @@ describe("Admin Routes", () => {
const metadataAfter = (await mockEnv.EMAIL_STORAGE.get(
feedMetadataKey,
"json",
)) as { emails: Array<{ key: string; subject: string; receivedAt: number }> } | null;
)) as {
emails: Array<{ key: string; subject: string; receivedAt: number }>;
} | null;
expect(metadataAfter?.emails.length).toBe(0);
});
it("should show a paperclip indicator only for emails with attachments", async () => {
const authCookie = await loginAndGetCookie();
const formData = new FormData();
formData.append("title", "Email Feed");
const createRes = await request("/admin/feeds/create", {
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
body: formData,
});
expect(createRes.status).toBe(302);
const feedList = (await mockEnv.EMAIL_STORAGE.get(
"feeds:list",
"json",
)) as { feeds: Array<{ id: string; title: string }> } | null;
const feedId = feedList?.feeds[0].id as string;
await mockEnv.EMAIL_STORAGE.put(
`feed:${feedId}:metadata`,
JSON.stringify({
emails: [
{
key: `feed:${feedId}:1`,
subject: "With attachments",
receivedAt: 2,
attachmentIds: ["att-1", "att-2"],
},
{
key: `feed:${feedId}:2`,
subject: "No attachments",
receivedAt: 1,
},
],
}),
);
const res = await request(`/admin/feeds/${feedId}/emails`, {
headers: { Cookie: authCookie },
});
expect(res.status).toBe(200);
const body = await res.text();
expect(body).toContain("2 attachments");
const indicatorCount = (body.match(/attachment-indicator/g) || [])
.length;
expect(indicatorCount).toBe(1);
});
it("lists attachments with download links on the email detail page", async () => {
const authCookie = await loginAndGetCookie();
const feedId = "detail-feed";
const emailKey = `feed:${feedId}:1`;
await mockEnv.EMAIL_STORAGE.put(
emailKey,
JSON.stringify({
subject: "With attachments",
from: "[email protected]",
content: "<p>hello</p>",
receivedAt: 1,
headers: {},
attachments: [
{
id: "att-123",
filename: "report final.pdf",
contentType: "application/pdf",
size: 2048,
},
],
}),
);
const res = await request(`/admin/emails/${emailKey}`, {
headers: { Cookie: authCookie },
});
expect(res.status).toBe(200);
const body = await res.text();
expect(body).toContain("Attachments");
expect(body).toContain(
`/files/att-123/${encodeURIComponent("report final.pdf")}`,
);
expect(body).toContain("report final.pdf");
expect(body).toContain("2.0 KB");
});
it("renders inline cid images in place and hides them from the attachments list", async () => {
const authCookie = await loginAndGetCookie();
const feedId = "detail-feed";
const emailKey = `feed:${feedId}:3`;
await mockEnv.EMAIL_STORAGE.put(
emailKey,
JSON.stringify({
subject: "With inline image",
from: "[email protected]",
content: '<p>hello</p><img src="cid:logo123"/>',
receivedAt: 3,
headers: {},
attachments: [
{
id: "img-1",
filename: "logo.png",
contentType: "image/png",
size: 512,
contentId: "logo123",
inline: true,
},
],
}),
);
const res = await request(`/admin/emails/${emailKey}`, {
headers: { Cookie: authCookie },
});
expect(res.status).toBe(200);
const body = await res.text();
// The rendered preview is a base64 data: iframe; decode and inspect it.
const match = body.match(/data:text\/html;base64,([A-Za-z0-9+/=]+)/);
expect(match).not.toBeNull();
const decoded = Buffer.from(match![1], "base64").toString("utf-8");
// cid: is rewritten to an absolute /files URL so it resolves in the iframe.
expect(decoded).toContain(
"https://test.getmynews.app/files/img-1/logo.png",
);
expect(decoded).not.toContain("cid:logo123");
// Inline image is not surfaced as a downloadable attachment.
expect(body).not.toContain("Attachments");
});
it("does not render an attachments section when the email has none", async () => {
const authCookie = await loginAndGetCookie();
const feedId = "detail-feed";
const emailKey = `feed:${feedId}:2`;
await mockEnv.EMAIL_STORAGE.put(
emailKey,
JSON.stringify({
subject: "No attachments",
from: "[email protected]",
content: "<p>hello</p>",
receivedAt: 2,
headers: {},
}),
);
const res = await request(`/admin/emails/${emailKey}`, {
headers: { Cookie: authCookie },
});
expect(res.status).toBe(200);
const body = await res.text();
expect(body).not.toContain("Attachments");
});
it("form-based bulk-delete also removes R2 attachments", async () => {
const r2Env = createMockEnv({ withR2: true }) as unknown as Env;
const bucket = r2Env.ATTACHMENT_BUCKET as unknown as {
put: (k: string, v: string) => Promise<void>;
_has: (k: string) => boolean;
};
await bucket.put("att-1", "data1");
await bucket.put("att-2", "data2");
const loginForm = new FormData();
loginForm.append("password", "test-password");
const loginRes = await testApp.request(
"/admin/login",
{ method: "POST", body: loginForm },
r2Env,
);
const authCookie = (loginRes.headers.get("Set-Cookie") as string).split(
";",
)[0];
const feedId = "bulk-r2-feed";
await r2Env.EMAIL_STORAGE.put(
"feeds:list",
JSON.stringify({ feeds: [{ id: feedId, title: "F" }] }),
);
await r2Env.EMAIL_STORAGE.put(
`feed:${feedId}:config`,
JSON.stringify({ title: "F", language: "en", created_at: 1 }),
);
const emailKey = `feed:${feedId}:1`;
await r2Env.EMAIL_STORAGE.put(
emailKey,
JSON.stringify({
subject: "x",
from: "[email protected]",
content: "<p>x</p>",
receivedAt: 1,
headers: {},
attachments: [{ id: "att-1" }, { id: "att-2" }],
}),
);
await r2Env.EMAIL_STORAGE.put(
`feed:${feedId}:metadata`,
JSON.stringify({
emails: [
{
key: emailKey,
subject: "x",
receivedAt: 1,
attachmentIds: ["att-1", "att-2"],
},
],
}),
);
const form = new FormData();
form.append("emailKeys", emailKey);
const res = await testApp.request(
`/admin/feeds/${feedId}/emails/bulk-delete`,
{
method: "POST",
headers: {
Cookie: authCookie,
Origin: "https://test.getmynews.app",
},
body: form,
},
r2Env,
);
expect(res.status).toBe(302);
expect(res.headers.get("Location")).toContain("bulkDeleted");
expect(await r2Env.EMAIL_STORAGE.get(emailKey, "json")).toBeNull();
expect(bucket._has("att-1")).toBe(false);
expect(bucket._has("att-2")).toBe(false);
});
});
});
describe("Sender filter", () => {
let authCookie: string;
let feedId: string;
beforeEach(async () => {
authCookie = await loginAndGetCookie();
const createRes = await request("/admin/feeds/create", {
method: "POST",
headers: {
Cookie: authCookie,
"Content-Type": "application/json",
Origin: "https://test.getmynews.app",
},
body: JSON.stringify({ title: "Filter Test Feed" }),
});
const payload = (await createRes.json()) as { feedId: string };
feedId = payload.feedId;
});
const post = (body: object, cookie = authCookie) =>
request(`/admin/feeds/${feedId}/sender-filter`, {
method: "POST",
headers: {
Cookie: cookie,
"Content-Type": "application/json",
Origin: "https://test.getmynews.app",
},
body: JSON.stringify(body),
});
it("adds exact email to allowlist", async () => {
const res = await post({
action: "allow_sender",
value: "[email protected]",
});
expect(res.status).toBe(200);
expect(((await res.json()) as any).ok).toBe(true);
const cfg = (await mockEnv.EMAIL_STORAGE.get(
`feed:${feedId}:config`,
"json",
)) as any;
expect(cfg.allowed_senders).toContain("[email protected]");
});
it("adds domain to allowlist", async () => {
const res = await post({ action: "allow_domain", value: "example.com" });
expect(res.status).toBe(200);
const cfg = (await mockEnv.EMAIL_STORAGE.get(
`feed:${feedId}:config`,
"json",
)) as any;
expect(cfg.allowed_senders).toContain("example.com");
});
it("adds exact email to blocklist", async () => {
const res = await post({ action: "block_sender", value: "[email protected]" });
expect(res.status).toBe(200);
const cfg = (await mockEnv.EMAIL_STORAGE.get(
`feed:${feedId}:config`,
"json",
)) as any;
expect(cfg.blocked_senders).toContain("[email protected]");
});
it("adds domain to blocklist", async () => {
const res = await post({ action: "block_domain", value: "bad.com" });
expect(res.status).toBe(200);
const cfg = (await mockEnv.EMAIL_STORAGE.get(
`feed:${feedId}:config`,
"json",
)) as any;
expect(cfg.blocked_senders).toContain("bad.com");
});
it("returns 409 when value already exists in the opposite list", async () => {
await post({ action: "block_sender", value: "[email protected]" });
const res = await post({
action: "allow_sender",
value: "[email protected]",
});
expect(res.status).toBe(409);
const data = (await res.json()) as any;
expect(data.ok).toBe(false);
expect(data.error).toMatch(/blocklist/);
});
it("is idempotent when value already in target list", async () => {
await post({ action: "allow_sender", value: "[email protected]" });
const res = await post({
action: "allow_sender",
value: "[email protected]",
});
expect(res.status).toBe(200);
const cfg = (await mockEnv.EMAIL_STORAGE.get(
`feed:${feedId}:config`,
"json",
)) as any;
expect(
cfg.allowed_senders.filter((s: string) => s === "[email protected]")
.length,
).toBe(1);
});
it("returns 400 for invalid action", async () => {
const res = await post({
action: "invalid_action",
value: "[email protected]",
});
expect(res.status).toBe(400);
});
it("normalizes value to lowercase", async () => {
const res = await post({
action: "allow_sender",
value: "[email protected]",
});
expect(res.status).toBe(200);
const cfg = (await mockEnv.EMAIL_STORAGE.get(
`feed:${feedId}:config`,
"json",
)) as any;
expect(cfg.allowed_senders).toContain("[email protected]");
});
});
});
-3554
View File
File diff suppressed because it is too large. Load diff
+1025
View File
File diff suppressed because it is too large. Load diff
+796
View File
@@ -0,0 +1,796 @@
import { Hono } from "hono";
import { Env, EmailMetadata } from "../../types";
import { logger } from "../../infrastructure/logger";
import { Layout, clampText } from "./ui";
import {
deleteAttachmentsForEmails,
deleteKeysWithConcurrency,
} from "../../application/feed-cleanup";
import { FeedRepository } from "../../infrastructure/feed-repository";
import { FeedId } from "../../domain/value-objects/feed-id";
import {
feedRssUrl,
feedAtomUrl,
feedEmailAddress,
baseUrl,
} from "../../infrastructure/urls";
import { processEmailContent } from "../../infrastructure/html-processor";
import { formatBytes } from "../../domain/format";
import { EmailAddress } from "../../domain/value-objects/email-address";
import { emailsPageScript } from "../../scripts/generated/emails-page";
import emailPreviewCss from "../../styles/email-preview.css";
type AppEnv = { Bindings: Env };
export const emailsRouter = new Hono<AppEnv>();
// ── Shared SVG icons ──────────────────────────────────────────────────────────
const CopyIcon = () => (
<svg
class="copy-icon copy-icon-original"
width="16"
height="16"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<rect x="9" y="9" width="13" height="13" rx="2" ry="2"></rect>
<path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"></path>
</svg>
);
const CheckIcon = () => (
<svg
class="copy-icon copy-icon-success"
width="16"
height="16"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<path d="M20 6L9 17l-5-5"></path>
</svg>
);
type CopyFieldProps = {
label: string;
value: string;
display?: string;
};
const CopyField = ({ label, value, display }: CopyFieldProps) => (
<div class="copyable">
<span class="copyable-label">{label}</span>
<div class="copyable-content">
<span class="copyable-value" data-copy={value}>
{display ?? value}
</span>
<div class="copy-icon-container">
<CopyIcon />
<CheckIcon />
</div>
</div>
</div>
);
type SenderFieldProps = {
from: string;
feedId: string;
};
const SenderField = ({ from, feedId }: SenderFieldProps) => {
const parsed = EmailAddress.parse(from);
const senderEmail = parsed?.normalized ?? from.trim().toLowerCase();
const senderDomain = parsed?.domain.value ?? "";
return (
<div class="copyable">
<span class="copyable-label">From:</span>
<div class="copyable-content">
<span class="copyable-value" data-copy={from}>
{from}
</span>
<div class="copy-icon-container">
<CopyIcon />
<CheckIcon />
</div>
</div>
<div
class="sender-filter-dropdown"
data-feed-id={feedId}
data-email={senderEmail}
data-domain={senderDomain}
>
<button
type="button"
class="sender-filter-btn"
aria-label="Sender filter options"
onclick="toggleSenderFilter(this)"
>
⋮
</button>
<div class="sender-filter-menu" role="menu">
<button
type="button"
class="sender-filter-item sender-filter-allow"
data-action="allow_sender"
>
Allow {senderEmail}
</button>
<button
type="button"
class="sender-filter-item sender-filter-allow"
data-action="allow_domain"
>
Allow @{senderDomain}
</button>
<button
type="button"
class="sender-filter-item sender-filter-block"
data-action="block_sender"
>
Block {senderEmail}
</button>
<button
type="button"
class="sender-filter-item sender-filter-block"
data-action="block_domain"
>
Block @{senderDomain}
</button>
</div>
<span class="sender-filter-feedback" aria-live="polite"></span>
</div>
</div>
);
};
// ── View all emails for a feed ────────────────────────────────────────────────
emailsRouter.get("/feeds/:feedId/emails", async (c) => {
const env = c.env;
const repo = FeedRepository.from(env);
const feedId = c.req.param("feedId");
const message = c.req.query("message");
const count = Number(c.req.query("count") || "0");
const id = FeedId.unchecked(feedId);
const feedConfig = await repo.getConfig(id);
const feedMetadata = await repo.getMetadata(id);
if (!feedConfig || !feedMetadata) {
return c.text("Feed not found", 404);
}
const emailAddress = feedEmailAddress(feedId, env);
const rssUrl = feedRssUrl(feedId, env);
const atomUrl = feedAtomUrl(feedId, env);
return c.html(
<Layout title={`${feedConfig.title} - Emails`}>
<div class="container container-wide fade-in">
<div class="header-with-actions">
<div class="header-title">
<h1>{feedConfig.title} - Emails</h1>
</div>
<div class="header-actions">
<a href="/admin" class="button button-secondary button-back">
Back to Dashboard
</a>
</div>
</div>
<div class="card">
<h2>Feed Details</h2>
<div>
<CopyField label="Email Address:" value={emailAddress} />
<CopyField label="RSS Feed:" value={rssUrl} />
<CopyField label="Atom Feed:" value={atomUrl} />
</div>
<div class="feed-validate">
<a
href={`https://validator.w3.org/feed/check.cgi?url=${encodeURIComponent(atomUrl)}`}
target="_blank"
rel="noopener noreferrer"
>
<img
src="https://validator.w3.org/feed/images/valid-atom.png"
alt="[Valid Atom 1.0]"
title="Validate my Atom 1.0 feed"
/>
</a>
<a
href={`https://validator.w3.org/feed/check.cgi?url=${encodeURIComponent(rssUrl)}`}
target="_blank"
rel="noopener noreferrer"
>
<img
src="https://validator.w3.org/feed/images/valid-rss-rogers.png"
alt="[Valid RSS]"
title="Validate my RSS feed"
/>
</a>
</div>
</div>
<h2>
Emails (
<span id="email-total-count">{feedMetadata.emails.length}</span>)
</h2>
{message === "bulkDeleted" && (
<div class="card">
<p>Deleted {Number.isFinite(count) ? count : 0} email(s).</p>
</div>
)}
{message === "bulkDeleteNoop" && (
<div class="card">
<p>No emails were selected.</p>
</div>
)}
{feedMetadata.emails.length > 0 ? (
<form
action={`/admin/feeds/${feedId}/emails/bulk-delete`}
method="post"
onsubmit="return onBulkEmailDeleteSubmit(event)"
>
<div class="toolbar">
<div class="toolbar-group toolbar-group-fill">
<input
type="search"
id="email-search"
class="search"
placeholder="Search email subjects"
oninput="scheduleEmailFilter()"
/>
<span class="pill" id="email-match-count">
Showing {feedMetadata.emails.length}
</span>
<span class="pill" id="selected-email-count">
0 selected
</span>
<button
type="button"
class="button button-small button-secondary"
onclick="selectMatchingEmails()"
>
Select Results
</button>
<button
type="button"
class="button button-small button-secondary"
onclick="clearEmailSelection()"
>
Clear Selection
</button>
<button
id="bulk-delete-emails-button"
type="submit"
class="button button-small button-danger"
disabled
>
Delete Selected
</button>
</div>
</div>
<div class="table-wrap">
<table class="table table-emails">
<colgroup>
<col data-col="select" style="width: 44px;" />
<col data-col="subject" style="width: 520px;" />
<col data-col="receivedAt" style="width: 220px;" />
<col data-col="actions" style="width: 200px;" />
</colgroup>
<thead>
<tr>
<th>
<input
type="checkbox"
id="select-all-emails"
onchange="toggleAllEmails(this.checked)"
/>
</th>
<th
class="th-resizable"
data-sort-key="subject"
aria-sort="none"
>
<button
type="button"
class="th-button"
data-sort-key="subject"
>
Subject
<span class="sort-indicator" aria-hidden="true"></span>
</button>
<div
class="col-resizer"
data-col="subject"
title="Resize"
></div>
</th>
<th
class="th-resizable"
data-sort-key="receivedAt"
aria-sort="none"
>
<button
type="button"
class="th-button"
data-sort-key="receivedAt"
>
Received
<span class="sort-indicator" aria-hidden="true"></span>
</button>
<div
class="col-resizer"
data-col="receivedAt"
title="Resize"
></div>
</th>
<th class="th-resizable">
<span>Actions</span>
<div
class="col-resizer"
data-col="actions"
title="Resize"
></div>
</th>
</tr>
</thead>
<tbody>
{feedMetadata.emails.map((email: EmailMetadata) => {
const subjectDisplay = clampText(email.subject, 180);
const subjectHover = clampText(email.subject, 1000);
const attachmentCount = email.attachmentIds?.length ?? 0;
const attachmentLabel = `${attachmentCount} attachment${
attachmentCount > 1 ? "s" : ""
}`;
const sortSubject = subjectHover.toLowerCase();
const sortReceivedAt = String(email.receivedAt);
const searchHaystack = clampText(
email.subject,
320,
).toLowerCase();
return (
<tr
class="email-row"
data-email-key={email.key}
data-search={searchHaystack}
data-sort-subject={sortSubject}
data-sort-received-at={sortReceivedAt}
>
<td>
<input
type="checkbox"
class="email-select"
name="emailKeys"
value={email.key}
onchange="updateEmailSelectionState()"
/>
</td>
<td>
<div class="subject-cell">
{attachmentCount > 0 ? (
<span
class="attachment-indicator"
title={attachmentLabel}
aria-label={attachmentLabel}
>
<svg
width="14"
height="14"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
aria-hidden="true"
>
<path d="M21.44 11.05l-9.19 9.19a6 6 0 0 1-8.49-8.49l9.19-9.19a4 4 0 0 1 5.66 5.66l-9.2 9.19a2 2 0 0 1-2.83-2.83l8.49-8.48" />
</svg>
</span>
) : null}
<span class="truncate" title={subjectHover}>
{subjectDisplay}
</span>
</div>
</td>
<td>{new Date(email.receivedAt).toLocaleString()}</td>
<td>
<div class="row-actions">
<a
href={`/admin/emails/${email.key}`}
class="button button-small"
>
View
</a>
<button
type="button"
class="button button-small button-danger button-delete"
data-delete-kind="email"
data-email-key={email.key}
data-feed-id={feedId}
>
Delete
</button>
</div>
</td>
</tr>
);
})}
</tbody>
</table>
</div>
</form>
) : (
<div class="card">
<p>
No emails received yet. Subscribe to newsletters using the email
address above.
</p>
</div>
)}
</div>
{/* Config bootstrap — injects dynamic server-side data before the static compiled script */}
<script
dangerouslySetInnerHTML={{
__html: `window.__APP_CONFIG__=${JSON.stringify({ feedId })}`,
}}
/>
{/* Emails page logic compiled from src/scripts/client/emails-page.ts */}
<script dangerouslySetInnerHTML={{ __html: emailsPageScript }} />
</Layout>,
);
});
// ── View single email ─────────────────────────────────────────────────────────
emailsRouter.get("/emails/:emailKey", async (c) => {
const env = c.env;
const repo = FeedRepository.from(env);
const emailKey = c.req.param("emailKey");
const emailData = await repo.getEmail(emailKey);
if (!emailData) return c.text("Email not found", 404);
const feedId = repo.feedIdFromEmailKey(emailKey);
// Inline images render in place; only downloadable attachments go in the list.
const attachments = (emailData.attachments ?? []).filter((a) => !a.inline);
// The rendered preview lives in a `data:` iframe, which has no origin to
// resolve relative URLs against — so cid: refs must be rewritten to absolute
// /files URLs (and the content sanitized) before embedding.
const renderedBody = processEmailContent(
emailData.content,
emailData.attachments,
baseUrl(env),
);
const htmlContent = `<!DOCTYPE html><html><head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><style>${emailPreviewCss}</style></head><body>${renderedBody}</body></html>`;
const encodedHtmlContent = (() => {
const encoder = new TextEncoder();
const bytes = encoder.encode(htmlContent);
return btoa(String.fromCharCode(...new Uint8Array(bytes)));
})();
const rawHtml = emailData.content.replace(/</g, "&lt;").replace(/>/g, "&gt;");
const viewScript = `
function showRendered() {
document.getElementById('rendered-view').style.display = 'block';
document.getElementById('raw-view').style.display = 'none';
document.getElementById('rendered-button').classList.add('active');
document.getElementById('raw-button').classList.remove('active');
}
function showRaw() {
document.getElementById('rendered-view').style.display = 'none';
document.getElementById('raw-view').style.display = 'block';
document.getElementById('rendered-button').classList.remove('active');
document.getElementById('raw-button').classList.add('active');
}
window.addEventListener('load', function() {
const iframe = document.querySelector('.email-iframe');
if (!iframe) return;
iframe.style.height = '500px';
try {
iframe.onload = function() {
const doc = iframe.contentDocument || iframe.contentWindow.document;
if (doc) iframe.style.height = Math.min(800, Math.max(500, doc.body.scrollHeight)) + 'px';
};
} catch (e) { /* cross-origin */ }
});
function toggleSenderFilter(btn) {
var dropdown = btn.closest('.sender-filter-dropdown');
var isOpen = dropdown.hasAttribute('data-open');
document.querySelectorAll('.sender-filter-dropdown[data-open]').forEach(function(d) {
d.removeAttribute('data-open');
});
if (!isOpen) dropdown.setAttribute('data-open', '');
}
function showSenderFeedback(feedback, ok, msg) {
feedback.textContent = msg;
feedback.className = 'sender-filter-feedback ' + (ok ? 'sender-filter-feedback-ok' : 'sender-filter-feedback-error');
setTimeout(function() {
feedback.textContent = '';
feedback.className = 'sender-filter-feedback';
}, 3000);
}
document.addEventListener('click', function(e) {
var item = e.target.closest('.sender-filter-item');
if (item) {
var dropdown = item.closest('.sender-filter-dropdown');
var action = item.dataset.action;
var value = (action === 'allow_sender' || action === 'block_sender')
? dropdown.dataset.email
: dropdown.dataset.domain;
dropdown.removeAttribute('data-open');
var feedback = dropdown.querySelector('.sender-filter-feedback');
fetch('/admin/feeds/' + dropdown.dataset.feedId + '/sender-filter', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ action: action, value: value })
}).then(function(res) {
return res.json();
}).then(function(data) {
showSenderFeedback(feedback, data.ok, data.ok ? 'Saved' : (data.error || 'Error'));
}).catch(function() {
showSenderFeedback(feedback, false, 'Network error');
});
return;
}
if (!e.target.closest('.sender-filter-dropdown')) {
document.querySelectorAll('.sender-filter-dropdown[data-open]').forEach(function(d) {
d.removeAttribute('data-open');
});
}
});
`;
return c.html(
<Layout title={`Email: ${emailData.subject}`}>
<div class="container fade-in">
<div class="header-with-actions">
<div class="header-title">
<h1>Email Content</h1>
</div>
<div class="header-actions">
<a
href={`/admin/feeds/${feedId}/emails`}
class="button button-secondary button-back"
>
Back to Emails
</a>
</div>
</div>
<div class="card">
<div class="email-meta">
<div class="email-metadata-grid">
<CopyField label="Subject:" value={emailData.subject} />
<CopyField
label="Received:"
value={new Date(emailData.receivedAt).toLocaleString()}
/>
<SenderField from={emailData.from} feedId={feedId} />
<CopyField label="To:" value={feedEmailAddress(feedId, env)} />
</div>
</div>
<div class="toggle-view">
<button
id="rendered-button"
class="toggle-button active"
onclick="showRendered()"
>
Rendered View
</button>
<button id="raw-button" class="toggle-button" onclick="showRaw()">
Raw HTML
</button>
</div>
<div class="email-content">
<div id="rendered-view" class="email-iframe-container">
<iframe
class="email-iframe"
src={`data:text/html;base64,${encodedHtmlContent}`}
></iframe>
</div>
<div id="raw-view" class="email-raw" style="display: none;">
<pre dangerouslySetInnerHTML={{ __html: rawHtml }}></pre>
</div>
</div>
{attachments.length > 0 && (
<div class="attachments">
<h2>Attachments</h2>
<ul class="attachment-list">
{attachments.map((a) => (
<li>
<svg
width="14"
height="14"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
aria-hidden="true"
>
<path d="M21.44 11.05l-9.19 9.19a6 6 0 0 1-8.49-8.49l9.19-9.19a4 4 0 0 1 5.66 5.66l-9.2 9.19a2 2 0 0 1-2.83-2.83l8.49-8.48" />
</svg>
<a
href={`/files/${a.id}/${encodeURIComponent(a.filename)}`}
download
>
{a.filename}
</a>
<span class="attachment-size">{formatBytes(a.size)}</span>
</li>
))}
</ul>
</div>
)}
</div>
</div>
<script dangerouslySetInnerHTML={{ __html: viewScript }} />
</Layout>,
);
});
// ── Delete single email ───────────────────────────────────────────────────────
emailsRouter.post("/emails/:emailKey/delete", async (c) => {
const env = c.env;
const repo = FeedRepository.from(env);
const emailKey = c.req.param("emailKey");
const wantsJson = (c.req.header("Accept") || "").includes("application/json");
try {
const feedId = c.req.query("feedId");
if (!feedId) {
if (wantsJson)
return c.json({ ok: false, error: "Feed ID is required" }, 400);
return c.text("Feed ID is required", 400);
}
const feed = await repo.load(FeedId.unchecked(feedId));
await repo.deleteEmail(emailKey);
if (feed) {
const { removed } = feed.removeEmails([emailKey]);
await deleteAttachmentsForEmails(env, removed, [emailKey]);
await repo.saveMetadata(feed);
}
if (wantsJson) return c.json({ ok: true, emailKey, feedId });
return c.redirect(`/admin/feeds/${feedId}/emails`);
} catch (error) {
logger.error("Error deleting email", { emailKey, error: String(error) });
if (wantsJson)
return c.json(
{ ok: false, error: "Error deleting email. Please try again." },
400,
);
return c.text("Error deleting email. Please try again.", 400);
}
});
// ── Bulk delete emails ────────────────────────────────────────────────────────
emailsRouter.post("/feeds/:feedId/emails/bulk-delete", async (c) => {
const env = c.env;
const emailStorage = env.EMAIL_STORAGE;
const repo = new FeedRepository(emailStorage);
const feedId = c.req.param("feedId");
const contentType = c.req.header("Content-Type") || "";
const wantsJson =
contentType.includes("application/json") ||
(c.req.header("Accept") || "").includes("application/json");
try {
const feed = await repo.load(FeedId.unchecked(feedId));
if (!feed) {
return wantsJson
? c.json({ ok: false, error: "Feed not found" }, 404)
: c.text("Feed not found", 404);
}
const allowedKeys = new Set(feed.emails.map((email) => email.key));
if (wantsJson) {
const body = (await c.req.json().catch(() => null)) as {
emailKeys?: unknown;
} | null;
const rawEmailKeys = Array.isArray(body?.emailKeys)
? body?.emailKeys
: [];
const emailKeys = Array.from(
new Set(rawEmailKeys.map((value) => String(value)).filter(Boolean)),
);
if (emailKeys.length === 0)
return c.json({ ok: false, error: "No emails were selected." }, 400);
if (emailKeys.length > 250) {
return c.json(
{
ok: false,
error:
"Too many emailKeys for a single request. Please delete in smaller batches.",
},
413,
);
}
const candidates = emailKeys.filter((key) => allowedKeys.has(key));
const { ok: deletedOk, failed: failedEmailKeys } =
await deleteKeysWithConcurrency(emailStorage, candidates, 35);
await deleteAttachmentsForEmails(env, feed.emails, candidates);
feed.removeEmails(deletedOk);
await repo.saveMetadata(feed);
return c.json({
ok: failedEmailKeys.length === 0,
deletedEmailKeys: deletedOk,
failedEmailKeys,
});
}
const formData = await c.req.formData();
const rawEmailKeys = formData
.getAll("emailKeys")
.map((value) => value.toString());
const emailKeys = Array.from(new Set(rawEmailKeys.filter(Boolean)));
if (emailKeys.length === 0)
return c.redirect(`/admin/feeds/${feedId}/emails?message=bulkDeleteNoop`);
const candidates = emailKeys.filter((key) => allowedKeys.has(key));
const { ok: deletedOk } = await deleteKeysWithConcurrency(
emailStorage,
candidates,
35,
);
await deleteAttachmentsForEmails(env, feed.emails, candidates);
feed.removeEmails(deletedOk);
await repo.saveMetadata(feed);
return c.redirect(
`/admin/feeds/${feedId}/emails?message=bulkDeleted&count=${deletedOk.length}`,
);
} catch (error) {
logger.error("Error bulk deleting emails", {
feedId,
error: String(error),
});
return wantsJson
? c.json(
{
ok: false,
error:
"Server error while deleting emails. This can happen if Cloudflare is rate-limiting requests or if the Worker hit a plan quota. Please try again.",
},
500,
)
: c.text("Error bulk deleting emails. Please try again.", 500);
}
});
+654
View File
@@ -0,0 +1,654 @@
import { Hono } from "hono";
import { z } from "zod";
import { Env } from "../../types";
import { bumpCounters } from "../../application/stats";
import { waitUntilSafe } from "../../infrastructure/worker";
import { feedRssUrl, feedEmailAddress } from "../../infrastructure/urls";
import { logger } from "../../infrastructure/logger";
import { sendUnsubscribes } from "../../infrastructure/unsubscribe";
import { getAttachmentBucket } from "../../infrastructure/attachments";
import { Layout } from "./ui";
import {
purgeFeedKeysStep,
collectUnsubscribeUrls,
} from "../../application/feed-cleanup";
import { FeedRepository } from "../../infrastructure/feed-repository";
import { FeedId } from "../../domain/value-objects/feed-id";
import {
createFeedRecord,
editFeed,
deleteFeedRecord,
deleteFeedFastDetailed,
} from "../../application/feed-service";
type AppEnv = { Bindings: Env };
export const feedsRouter = new Hono<AppEnv>();
function normalizeAllowedSenders(senders: string[]): string[] {
return senders.map((s) => s.trim().toLowerCase()).filter(Boolean);
}
function parseAllowedSenders(rawAllowedSenders: string): string[] {
return normalizeAllowedSenders(rawAllowedSenders.split(/[\n,]+/));
}
const createFeedSchema = z.object({
title: z.string().min(1, "Title is required"),
description: z.string().optional(),
language: z.string().optional().default("en"),
allowedSenders: z.array(z.string()).optional().default([]),
blockedSenders: z.array(z.string()).optional().default([]),
});
const updateFeedSchema = z.object({
title: z.string().min(1, "Title is required"),
description: z.string().optional(),
language: z.string().optional().default("en"),
allowedSenders: z.array(z.string()).optional().default([]),
blockedSenders: z.array(z.string()).optional().default([]),
});
const senderFilterSchema = z.object({
action: z.enum([
"allow_sender",
"allow_domain",
"block_sender",
"block_domain",
]),
value: z.string().min(1),
});
// ── Routes ────────────────────────────────────────────────────────────────────
feedsRouter.post("/create", async (c) => {
const env = c.env;
const isJson =
c.req.header("Content-Type")?.includes("application/json") ?? false;
try {
let title: string;
let description: string | undefined;
let language: string;
let view: string;
let allowedSenders: string[];
let blockedSenders: string[];
let lifetimeHoursRaw: string | undefined;
if (isJson) {
const body = await c.req.json<Record<string, unknown>>();
title = String(body.title ?? "");
description =
body.description != null ? String(body.description) : undefined;
language = String(body.language ?? "en");
view = "list";
allowedSenders = Array.isArray(body.allowedSenders)
? normalizeAllowedSenders(
(body.allowedSenders as unknown[]).map(String),
)
: [];
blockedSenders = Array.isArray(body.blockedSenders)
? normalizeAllowedSenders(
(body.blockedSenders as unknown[]).map(String),
)
: [];
lifetimeHoursRaw =
body.lifetimeHours != null ? String(body.lifetimeHours) : undefined;
} else {
const formData = await c.req.formData();
title = formData.get("title")?.toString() || "";
description = formData.get("description")?.toString();
language = formData.get("language")?.toString() || "en";
view = formData.get("view")?.toString() === "table" ? "table" : "list";
allowedSenders = parseAllowedSenders(
formData.get("allowed_senders")?.toString() || "",
);
blockedSenders = parseAllowedSenders(
formData.get("blocked_senders")?.toString() || "",
);
lifetimeHoursRaw = formData.get("lifetime_hours")?.toString();
}
const parsedData = createFeedSchema.parse({
title,
description,
language,
allowedSenders,
blockedSenders,
});
const lifetimeHours = lifetimeHoursRaw
? parseInt(lifetimeHoursRaw, 10)
: undefined;
const { feedId } = await createFeedRecord(env, {
title: parsedData.title,
description: parsedData.description,
language: parsedData.language,
allowedSenders: parsedData.allowedSenders,
blockedSenders: parsedData.blockedSenders,
lifetimeHours,
});
if (isJson) {
return c.json({
feedId,
email: feedEmailAddress(feedId, env),
feedUrl: feedRssUrl(feedId, env),
});
}
return c.redirect(`/admin?view=${view}#your-feeds`);
} catch (error) {
logger.error("Error creating feed", { error: String(error) });
if (c.req.header("Content-Type")?.includes("application/json")) {
return c.json({ error: "Error creating feed." }, 400);
}
return c.text("Error creating feed. Please try again.", 400);
}
});
feedsRouter.get("/:feedId/edit", async (c) => {
const env = c.env;
const feedId = c.req.param("feedId");
const feedConfig = await FeedRepository.from(env).getConfig(
FeedId.unchecked(feedId),
);
if (!feedConfig) {
return c.text("Feed not found", 404);
}
const now = Date.now();
const isExpired =
feedConfig.expires_at !== undefined && feedConfig.expires_at <= now;
const ttlLocked = !!env.FEED_TTL_HOURS;
// Remaining hours: ceil so we don't show 0 when there's still time left
const remainingHours =
feedConfig.expires_at !== undefined && feedConfig.expires_at > now
? Math.ceil((feedConfig.expires_at - now) / 3_600_000)
: undefined;
const lifetimeFieldValue =
ttlLocked && !isExpired
? (env.FEED_TTL_HOURS ?? "")
: (remainingHours?.toString() ?? "");
return c.html(
<Layout title="Edit Feed">
<div class="container fade-in">
<div class="header-with-actions">
<div class="header-title">
<h1>{feedConfig.title} - Edit Feed</h1>
</div>
<div class="header-actions">
<a href="/admin" class="button button-secondary button-back">
Back to Dashboard
</a>
</div>
</div>
{isExpired && (
<div class="card card-warning">
<p>
<strong>This feed has expired.</strong> It no longer accepts
emails and its content is no longer publicly accessible.
</p>
<form
action={`/admin/feeds/${feedId}/delete`}
method="post"
style="margin-top: 0.75rem;"
>
<button type="submit" class="button button-danger">
Delete this feed
</button>
</form>
</div>
)}
<div class={`card${isExpired ? " card-disabled" : ""}`}>
<form action={`/admin/feeds/${feedId}/edit`} method="post">
<div class="form-group">
<label for="title">Feed Title</label>
<input
type="text"
id="title"
name="title"
value={feedConfig.title}
required
disabled={isExpired}
/>
</div>
<div class="form-group">
<label for="description">Description</label>
<textarea
id="description"
name="description"
rows={3}
disabled={isExpired}
>
{feedConfig.description || ""}
</textarea>
</div>
<div class="form-group">
<label for="allowed_senders">
Allowed senders (optional, one email or domain per line)
</label>
<textarea
id="allowed_senders"
name="allowed_senders"
rows={3}
placeholder={"[email protected]\ntechmeme.com"}
disabled={isExpired}
>
{(feedConfig.allowed_senders || []).join("\n")}
</textarea>
<small>
When set, inbound emails are only accepted from these
senders/domains.
</small>
</div>
<div class="form-group">
<label for="blocked_senders">
Blocked senders (optional, one email or domain per line)
</label>
<textarea
id="blocked_senders"
name="blocked_senders"
rows={3}
placeholder={"[email protected]\nunwanted.com"}
disabled={isExpired}
>
{(feedConfig.blocked_senders || []).join("\n")}
</textarea>
<small>
Emails from these senders/domains are always rejected, even if
they match the allowlist.
</small>
</div>
<div class="form-group">
<label for="lifetime_hours">Lifetime (hours)</label>
<input
type="number"
id="lifetime_hours"
name="lifetime_hours"
min="1"
value={lifetimeFieldValue}
disabled={isExpired || ttlLocked}
placeholder={feedConfig.expires_at ? undefined : "No expiry"}
/>
{ttlLocked ? (
<small>
Feed lifetime is fixed to {env.FEED_TTL_HOURS}h by server
configuration.
</small>
) : (
<small>
Hours from now until this feed expires. Leave empty to keep
the current expiry (or no expiry).
</small>
)}
</div>
<input type="hidden" id="language" name="language" value="en" />
{!isExpired && (
<button type="submit" class="button">
Update Feed
</button>
)}
</form>
</div>
</div>
</Layout>,
);
});
feedsRouter.post("/:feedId/edit", async (c) => {
const env = c.env;
const feedId = c.req.param("feedId");
try {
const formData = await c.req.formData();
const title = formData.get("title")?.toString() || "";
const description = formData.get("description")?.toString();
const language = formData.get("language")?.toString() || "en";
const allowedSenders = parseAllowedSenders(
formData.get("allowed_senders")?.toString() || "",
);
const blockedSenders = parseAllowedSenders(
formData.get("blocked_senders")?.toString() || "",
);
const lifetimeHoursRaw = formData.get("lifetime_hours")?.toString();
const parsedData = updateFeedSchema.parse({
title,
description,
language,
allowedSenders,
blockedSenders,
});
const result = await editFeed(env, FeedId.unchecked(feedId), {
title: parsedData.title,
description: parsedData.description,
language: parsedData.language,
allowedSenders: parsedData.allowedSenders,
blockedSenders: parsedData.blockedSenders,
lifetimeHours: lifetimeHoursRaw
? parseInt(lifetimeHoursRaw, 10)
: undefined,
});
if (result.status === "not_found") {
return c.text("Feed not found", 404);
}
if (result.status === "expired") {
return c.text("Feed has expired and cannot be modified.", 403);
}
return c.redirect("/admin");
} catch (error) {
logger.error("Error updating feed", { feedId, error: String(error) });
return c.text("Error updating feed. Please try again.", 400);
}
});
// ── Sender filter quick-add ───────────────────────────────────────────────────
feedsRouter.post("/:feedId/sender-filter", async (c) => {
const env = c.env;
const feedId = c.req.param("feedId");
const id = FeedId.unchecked(feedId);
const repo = FeedRepository.from(env);
const body = await c.req.json().catch(() => null);
const parsed = senderFilterSchema.safeParse(body);
if (!parsed.success) {
return c.json({ ok: false, error: "Invalid request" }, 400);
}
const { action, value } = parsed.data;
const normalized = value.trim().toLowerCase();
const feedConfig = await repo.getConfig(id);
if (!feedConfig) return c.json({ ok: false, error: "Feed not found" }, 404);
const allowedSenders = (feedConfig.allowed_senders || []).map((s) =>
s.trim().toLowerCase(),
);
const blockedSenders = (feedConfig.blocked_senders || []).map((s) =>
s.trim().toLowerCase(),
);
const isAllowAction = action === "allow_sender" || action === "allow_domain";
const targetList = isAllowAction ? allowedSenders : blockedSenders;
const oppositeList = isAllowAction ? blockedSenders : allowedSenders;
const oppositeLabel = isAllowAction ? "blocklist" : "allowlist";
if (oppositeList.includes(normalized)) {
return c.json(
{
ok: false,
error: `"${normalized}" is already in the ${oppositeLabel}`,
},
409,
);
}
if (!targetList.includes(normalized)) {
targetList.push(normalized);
await repo.putConfig(id, {
...feedConfig,
allowed_senders: allowedSenders,
blocked_senders: blockedSenders,
updated_at: Date.now(),
});
}
return c.json({ ok: true });
});
feedsRouter.post("/:feedId/delete", async (c) => {
const env = c.env;
const feedId = c.req.param("feedId");
const view = c.req.query("view") === "table" ? "table" : "list";
const wantsJson = (c.req.header("Accept") || "").includes("application/json");
try {
await deleteFeedRecord(env, FeedId.unchecked(feedId), (p) =>
waitUntilSafe(c, p),
);
if (wantsJson) {
return c.json({ ok: true, feedId });
}
return c.redirect(`/admin?view=${view}`);
} catch (error) {
logger.error("Error deleting feed", { feedId, error: String(error) });
if (wantsJson) {
return c.json(
{ ok: false, error: "Error deleting feed. Please try again." },
400,
);
}
return c.text("Error deleting feed. Please try again.", 400);
}
});
feedsRouter.post("/:feedId/purge", async (c) => {
const env = c.env;
const emailStorage = env.EMAIL_STORAGE;
const feedId = c.req.param("feedId");
try {
const body = (await c.req.json().catch(() => null)) as {
cursor?: unknown;
limit?: unknown;
} | null;
const cursor = body?.cursor ? String(body.cursor) : undefined;
const limit = Number.isFinite(Number(body?.limit))
? Number(body?.limit)
: 100;
const step = await purgeFeedKeysStep(
emailStorage,
FeedId.unchecked(feedId),
{
cursor,
limit,
bucket: getAttachmentBucket(env),
},
);
return c.json({
ok: step.failedKeys.length === 0,
deletedCount: step.deletedKeys.length,
failedCount: step.failedKeys.length,
cursor: step.cursor,
listComplete: step.listComplete,
});
} catch (error) {
logger.error("Error purging feed keys", { feedId, error: String(error) });
return c.json({ ok: false, error: "Error purging feed keys" }, 500);
}
});
feedsRouter.post("/bulk-delete", async (c) => {
const env = c.env;
const emailStorage = env.EMAIL_STORAGE;
const contentType = c.req.header("Content-Type") || "";
const wantsJson =
contentType.includes("application/json") ||
(c.req.header("Accept") || "").includes("application/json");
try {
if (wantsJson) {
const body = (await c.req.json().catch(() => null)) as {
feedIds?: unknown;
} | null;
const rawIds = Array.isArray(body?.feedIds) ? body?.feedIds : [];
const parsedFeedIds = Array.from(
new Set(rawIds.map((value) => String(value)).filter(Boolean)),
);
if (parsedFeedIds.length === 0) {
return c.json({ ok: false, error: "No feeds were selected." }, 400);
}
if (parsedFeedIds.length > 50) {
return c.json(
{
ok: false,
error:
"Too many feedIds for a single request. Please delete in smaller batches.",
},
413,
);
}
const okIds: string[] = [];
const failures: Array<{ feedId: string; error: string }> = [];
const warnings: Array<{ feedId: string; warning: string }> = [];
const unsubscribeUrls: string[] = [];
for (const feedId of parsedFeedIds) {
try {
const id = FeedId.unchecked(feedId);
// Read unsubscribe URLs before the feed metadata is deleted.
const urls = await collectUnsubscribeUrls(emailStorage, id);
const result = await deleteFeedFastDetailed(emailStorage, id);
if (!result.ok) {
failures.push({
feedId,
error:
result.errors.join("; ") ||
"Failed to delete feed config (feed may still be active).",
});
continue;
}
if (!result.metadataDeleted) {
warnings.push({
feedId,
warning:
"Feed config deleted, but metadata cleanup failed. This is usually safe, but storage cleanup may be incomplete.",
});
}
unsubscribeUrls.push(...urls);
okIds.push(feedId);
} catch (error) {
logger.error("Error bulk deleting feed", {
feedId,
error: String(error),
});
failures.push({ feedId, error: String(error) });
}
}
const deletedFeedIds = await new FeedRepository(
emailStorage,
).removeFromListBulk(okIds);
if (deletedFeedIds.length > 0) {
await bumpCounters(emailStorage, {
feeds_deleted: deletedFeedIds.length,
});
}
if (unsubscribeUrls.length > 0) {
waitUntilSafe(c, sendUnsubscribes(unsubscribeUrls, env));
}
const removed = new Set(deletedFeedIds);
okIds.forEach((feedId) => {
if (!removed.has(feedId)) {
failures.push({
feedId,
error:
"Feed config deleted, but failed to remove it from feeds:list. Refresh and try again.",
});
}
});
const failedFeedIds = Array.from(new Set(failures.map((f) => f.feedId)));
return c.json({
ok: failedFeedIds.length === 0,
deletedFeedIds,
failedFeedIds,
failures,
warnings,
});
}
const formData = await c.req.formData();
const view =
formData.get("view")?.toString() === "table" ? "table" : "list";
const redirectBase = `/admin?view=${view}`;
const rawIds = formData.getAll("feedIds").map((value) => value.toString());
const parsedFeedIds = Array.from(new Set(rawIds.filter(Boolean)));
if (parsedFeedIds.length === 0) {
return c.redirect(`${redirectBase}&message=bulkDeleteNoop`);
}
const okIds: string[] = [];
const unsubscribeUrls: string[] = [];
for (const feedId of parsedFeedIds) {
try {
const id = FeedId.unchecked(feedId);
// Read unsubscribe URLs before the feed metadata is deleted.
const urls = await collectUnsubscribeUrls(emailStorage, id);
const result = await deleteFeedFastDetailed(emailStorage, id);
if (result.ok) {
unsubscribeUrls.push(...urls);
okIds.push(feedId);
}
} catch (error) {
logger.error("Error bulk deleting feed", {
feedId,
error: String(error),
});
}
}
const deletedFeedIds = await new FeedRepository(
emailStorage,
).removeFromListBulk(okIds);
if (deletedFeedIds.length > 0) {
await bumpCounters(emailStorage, {
feeds_deleted: deletedFeedIds.length,
});
}
if (unsubscribeUrls.length > 0) {
waitUntilSafe(c, sendUnsubscribes(unsubscribeUrls, env));
}
return c.redirect(
`${redirectBase}&message=bulkDeleted&count=${deletedFeedIds.length}`,
);
} catch (error) {
logger.error("Error bulk deleting feeds", { error: String(error) });
return wantsJson
? c.json(
{
ok: false,
error:
"Server error while deleting feeds. This can happen if Cloudflare is rate-limiting requests or if the Worker hit a plan quota. Please try again.",
},
500,
)
: c.text("Error bulk deleting feeds. Please try again.", 500);
}
});
+91
View File
@@ -0,0 +1,91 @@
import variablesCss from "../../styles/variables.css";
import layoutCss from "../../styles/layout.css";
import componentsCss from "../../styles/components.css";
import utilitiesCss from "../../styles/utilities.css";
import { interactiveScripts } from "../../scripts/index";
import { FAVICON_PATH } from "../favicon";
const designSystem = [
variablesCss,
layoutCss,
componentsCss,
utilitiesCss,
].join("\n");
type LayoutProps = {
title: string;
label?: string;
children: import("hono/jsx").Child;
};
export const Layout = ({ title, label = "admin", children }: LayoutProps) => {
return (
<html>
<head>
<title>{title} — kill-the-news</title>
<link rel="icon" type="image/svg+xml" href={FAVICON_PATH} />
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="color-scheme" content="dark light" />
<link rel="preconnect" href="https://fonts.googleapis.com" />
<link
rel="preconnect"
href="https://fonts.gstatic.com"
crossorigin="anonymous"
/>
<link
href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap"
rel="stylesheet"
/>
{/* designSystem and interactiveScripts are static trusted strings, not user input */}
<style dangerouslySetInnerHTML={{ __html: designSystem }} />
<script
dangerouslySetInnerHTML={{ __html: interactiveScripts + ";" }}
/>
</head>
<body class="page">
<header class="site-header">
<a
href="https://kill-the.news/"
class="site-header-logo"
target="_blank"
rel="noopener"
>
kill-the-news
</a>
<span class="site-header-label">{label}</span>
</header>
{children}
<footer class="site-footer">
<a href="https://kill-the.news/" target="_blank" rel="noopener">
kill-the.news
</a>
<span class="site-footer-sep" aria-hidden="true">
·
</span>
<a
href="https://github.com/sponsors/juherr"
target="_blank"
rel="noopener"
class="site-footer-sponsor"
>
♥ Sponsor
</a>
</footer>
</body>
</html>
);
};
export { Layout as layout };
export function clampText(value: string, maxLen: number): string {
const raw = `${value || ""}`;
if (raw.length <= maxLen) {
return raw.trim();
}
if (maxLen <= 3) {
return raw.slice(0, maxLen).trim();
}
return `${raw.slice(0, maxLen - 3).trimEnd()}...`;
}
+288
View File
@@ -0,0 +1,288 @@
import { describe, it, expect, beforeEach } from "vitest";
import { Hono } from "hono";
import { apiApp } from "./index";
import { createMockEnv } from "../../test/setup";
import { Env } from "../../types";
const PASSWORD = "test-password";
const authHeaders = { Authorization: `Bearer ${PASSWORD}` };
describe("REST API (/api/v1)", () => {
let testApp: Hono;
let mockEnv: Env;
let request: (path: string, init?: RequestInit) => Promise<Response>;
beforeEach(() => {
mockEnv = createMockEnv() as unknown as Env;
testApp = new Hono();
testApp.route("/api", apiApp);
request = (path, init = {}) =>
Promise.resolve(testApp.request(path, init, mockEnv));
});
async function createFeed(title = "Test Feed"): Promise<string> {
const res = await request("/api/v1/feeds", {
method: "POST",
headers: { ...authHeaders, "Content-Type": "application/json" },
body: JSON.stringify({ title }),
});
expect(res.status).toBe(201);
const body = (await res.json()) as { id: string };
return body.id;
}
describe("Authentication", () => {
it("rejects requests without a token", async () => {
const res = await request("/api/v1/feeds");
expect(res.status).toBe(401);
expect((await res.json()) as { error: string }).toEqual({
error: "Unauthorized",
});
});
it("rejects requests with a wrong token", async () => {
const res = await request("/api/v1/feeds", {
headers: { Authorization: "Bearer nope" },
});
expect(res.status).toBe(401);
});
it("accepts a valid Bearer token", async () => {
const res = await request("/api/v1/feeds", { headers: authHeaders });
expect(res.status).toBe(200);
});
it("accepts proxy auth headers", async () => {
const proxyApp = new Hono();
proxyApp.route("/api", apiApp);
const proxyEnv = {
...createMockEnv(),
PROXY_TRUSTED_IPS: "10.0.0.1",
PROXY_AUTH_SECRET: "proxy-secret",
} as unknown as Env;
const res = await proxyApp.request(
"/api/v1/feeds",
{
headers: {
"CF-Connecting-IP": "10.0.0.1",
"X-Auth-Proxy-Secret": "proxy-secret",
"Remote-User": "alice",
},
},
proxyEnv,
);
expect(res.status).toBe(200);
});
});
describe("Feeds CRUD", () => {
it("creates, reads, lists, updates and deletes a feed", async () => {
// Create
const createRes = await request("/api/v1/feeds", {
method: "POST",
headers: { ...authHeaders, "Content-Type": "application/json" },
body: JSON.stringify({
title: "Daily Digest",
description: "news",
allowedSenders: ["[email protected]"],
}),
});
expect(createRes.status).toBe(201);
const created = (await createRes.json()) as {
id: string;
title: string;
allowedSenders: string[];
emailAddress: string;
rssUrl: string;
atomUrl: string;
emailCount: number;
};
expect(created.title).toBe("Daily Digest");
// senders are normalized to lowercase
expect(created.allowedSenders).toEqual(["[email protected]"]);
expect(created.emailCount).toBe(0);
expect(created.rssUrl).toContain(`/rss/${created.id}`);
// Get
const getRes = await request(`/api/v1/feeds/${created.id}`, {
headers: authHeaders,
});
expect(getRes.status).toBe(200);
expect((await getRes.json()) as { id: string }).toMatchObject({
id: created.id,
title: "Daily Digest",
});
// List
const listRes = await request("/api/v1/feeds", { headers: authHeaders });
const list = (await listRes.json()) as { feeds: { id: string }[] };
expect(list.feeds.map((f) => f.id)).toContain(created.id);
// Update
const patchRes = await request(`/api/v1/feeds/${created.id}`, {
method: "PATCH",
headers: { ...authHeaders, "Content-Type": "application/json" },
body: JSON.stringify({ title: "Renamed" }),
});
expect(patchRes.status).toBe(200);
expect((await patchRes.json()) as { title: string }).toMatchObject({
title: "Renamed",
});
// Delete
const delRes = await request(`/api/v1/feeds/${created.id}`, {
method: "DELETE",
headers: authHeaders,
});
expect(delRes.status).toBe(200);
expect((await delRes.json()) as { ok: boolean }).toEqual({ ok: true });
// Gone from the list
const after = await request("/api/v1/feeds", { headers: authHeaders });
const afterList = (await after.json()) as { feeds: { id: string }[] };
expect(afterList.feeds.map((f) => f.id)).not.toContain(created.id);
});
it("returns 400 for an invalid create body", async () => {
const res = await request("/api/v1/feeds", {
method: "POST",
headers: { ...authHeaders, "Content-Type": "application/json" },
body: JSON.stringify({ title: "" }),
});
expect(res.status).toBe(400);
expect((await res.json()) as { error: string }).toHaveProperty("error");
});
it("returns 404 when getting a missing feed", async () => {
const res = await request("/api/v1/feeds/does-not-exist", {
headers: authHeaders,
});
expect(res.status).toBe(404);
});
it("returns 404 when deleting a missing feed", async () => {
const res = await request("/api/v1/feeds/does-not-exist", {
method: "DELETE",
headers: authHeaders,
});
expect(res.status).toBe(404);
});
it("returns 404 when updating a missing feed", async () => {
const res = await request("/api/v1/feeds/does-not-exist", {
method: "PATCH",
headers: { ...authHeaders, "Content-Type": "application/json" },
body: JSON.stringify({ title: "x" }),
});
expect(res.status).toBe(404);
});
});
describe("Emails", () => {
it("lists, reads and deletes an email", async () => {
const feedId = await createFeed();
// Seed an email directly into KV (mirrors storeEmail's key shape).
const receivedAt = 1737000000000;
const key = `feed:${feedId}:email:${receivedAt}`;
await mockEnv.EMAIL_STORAGE.put(
key,
JSON.stringify({
subject: "Hello",
from: "[email protected]",
content: "<p>hi</p>",
receivedAt,
headers: {},
}),
);
await mockEnv.EMAIL_STORAGE.put(
`feed:${feedId}:metadata`,
JSON.stringify({
emails: [{ key, subject: "Hello", receivedAt }],
}),
);
// List
const listRes = await request(`/api/v1/feeds/${feedId}/emails`, {
headers: authHeaders,
});
expect(listRes.status).toBe(200);
const list = (await listRes.json()) as {
emails: { entryId: number; subject: string }[];
};
expect(list.emails).toHaveLength(1);
expect(list.emails[0]).toMatchObject({
entryId: receivedAt,
subject: "Hello",
});
// Get single
const getRes = await request(
`/api/v1/feeds/${feedId}/emails/${receivedAt}`,
{ headers: authHeaders },
);
expect(getRes.status).toBe(200);
expect((await getRes.json()) as { content: string }).toMatchObject({
from: "[email protected]",
content: "<p>hi</p>",
});
// Delete
const delRes = await request(
`/api/v1/feeds/${feedId}/emails/${receivedAt}`,
{ method: "DELETE", headers: authHeaders },
);
expect(delRes.status).toBe(200);
expect(await mockEnv.EMAIL_STORAGE.get(key)).toBeNull();
// Gone
const after = await request(
`/api/v1/feeds/${feedId}/emails/${receivedAt}`,
{ headers: authHeaders },
);
expect(after.status).toBe(404);
});
it("returns 404 listing emails for a missing feed", async () => {
const res = await request("/api/v1/feeds/missing/emails", {
headers: authHeaders,
});
expect(res.status).toBe(404);
});
});
describe("Stats", () => {
it("returns monitoring counters without a token (public)", async () => {
await createFeed();
const res = await request("/api/v1/stats");
expect(res.status).toBe(200);
expect(res.headers.get("Access-Control-Allow-Origin")).toBe("*");
const stats = (await res.json()) as {
feeds_created: number;
active_feeds: number;
attachments_enabled: boolean;
};
expect(stats.feeds_created).toBeGreaterThanOrEqual(1);
expect(stats.active_feeds).toBeGreaterThanOrEqual(1);
expect(typeof stats.attachments_enabled).toBe("boolean");
});
});
describe("OpenAPI document", () => {
it("serves a public OpenAPI 3.1 spec", async () => {
const res = await request("/api/openapi.json");
expect(res.status).toBe(200);
const doc = (await res.json()) as {
openapi: string;
paths: Record<string, { get?: { security?: unknown[] } }>;
};
expect(doc.openapi).toBe("3.1.0");
expect(doc.paths).toHaveProperty("/v1/feeds");
expect(doc.paths).toHaveProperty("/v1/feeds/{feedId}");
expect(doc.paths).toHaveProperty("/v1/stats");
// Feed routes are secured; stats is public.
expect(doc.paths["/v1/feeds"].get?.security).toBeTruthy();
expect(doc.paths["/v1/stats"].get?.security).toBeUndefined();
});
});
});
+411
View File
@@ -0,0 +1,411 @@
import { OpenAPIHono, createRoute, z } from "@hono/zod-openapi";
import { cors } from "hono/cors";
import { Scalar } from "@scalar/hono-api-reference";
import { Env, FeedConfig } from "../../types";
import { apiAuthMiddleware } from "../../infrastructure/auth";
import {
createFeedRecord,
editFeed,
deleteFeedRecord,
} from "../../application/feed-service";
import { deleteAttachmentsForEmails } from "../../application/feed-cleanup";
import { waitUntilSafe } from "../../infrastructure/worker";
import { FeedRepository } from "../../infrastructure/feed-repository";
import { FeedId } from "../../domain/value-objects/feed-id";
import { getStats } from "../../application/stats";
import {
feedEmailAddress,
feedRssUrl,
feedAtomUrl,
} from "../../infrastructure/urls";
import {
ErrorSchema,
FeedIdParam,
EntryIdParam,
FeedCreateSchema,
FeedUpdateSchema,
FeedSchema,
FeedListSchema,
EmailListSchema,
EmailSchema,
StatsSchema,
} from "./schemas";
type AppEnv = { Bindings: Env };
const OkSchema = z.object({ ok: z.boolean() }).openapi("Ok");
const jsonContent = <T>(schema: T, description: string) => ({
content: { "application/json": { schema } },
description,
});
const bearer = [{ bearerAuth: [] }];
function normalizeSenders(senders?: string[]): string[] | undefined {
return senders?.map((s) => s.trim().toLowerCase()).filter(Boolean);
}
function toFeed(
id: string,
config: FeedConfig,
emailCount: number,
env: Env,
): z.infer<typeof FeedSchema> {
return {
id,
title: config.title,
description: config.description,
language: config.language,
allowedSenders: config.allowed_senders ?? [],
blockedSenders: config.blocked_senders ?? [],
createdAt: config.created_at,
updatedAt: config.updated_at,
expiresAt: config.expires_at,
emailCount,
emailAddress: feedEmailAddress(id, env),
rssUrl: feedRssUrl(id, env),
atomUrl: feedAtomUrl(id, env),
};
}
export const apiApp = new OpenAPIHono<AppEnv>({
defaultHook: (result, c) => {
if (!result.success) {
const message = result.error.issues
.map((i) => `${i.path.join(".")}: ${i.message}`)
.join("; ");
return c.json({ error: `Validation failed: ${message}` }, 400);
}
},
});
// Token auth on the feed/email routes. The spec, docs, and /v1/stats stay public.
apiApp.use("/v1/feeds", apiAuthMiddleware);
apiApp.use("/v1/feeds/*", apiAuthMiddleware);
// Public monitoring stats — readable from any origin (landing page, embeds).
apiApp.use("/v1/stats", cors({ origin: "*" }));
apiApp.openAPIRegistry.registerComponent("securitySchemes", "bearerAuth", {
type: "http",
scheme: "bearer",
description: "Use the admin password as the bearer token.",
});
// ── Feeds ─────────────────────────────────────────────────────────────────────
apiApp.openapi(
createRoute({
method: "get",
path: "/v1/feeds",
tags: ["Feeds"],
summary: "List all feeds",
security: bearer,
responses: {
200: jsonContent(FeedListSchema, "The list of feeds"),
401: jsonContent(ErrorSchema, "Unauthorized"),
},
}),
async (c) => {
const env = c.env;
const feeds = await FeedRepository.from(env).listFeeds();
return c.json(
{
feeds: feeds.map((f) => ({
id: f.id,
title: f.title,
description: f.description,
expiresAt: f.expires_at,
emailAddress: feedEmailAddress(f.id, env),
rssUrl: feedRssUrl(f.id, env),
atomUrl: feedAtomUrl(f.id, env),
})),
},
200,
);
},
);
apiApp.openapi(
createRoute({
method: "post",
path: "/v1/feeds",
tags: ["Feeds"],
summary: "Create a feed",
security: bearer,
request: {
body: { content: { "application/json": { schema: FeedCreateSchema } } },
},
responses: {
201: jsonContent(FeedSchema, "The created feed"),
400: jsonContent(ErrorSchema, "Invalid request body"),
401: jsonContent(ErrorSchema, "Unauthorized"),
},
}),
async (c) => {
const env = c.env;
const body = c.req.valid("json");
const { feedId, config } = await createFeedRecord(env, {
title: body.title,
description: body.description,
language: body.language,
allowedSenders: normalizeSenders(body.allowedSenders) ?? [],
blockedSenders: normalizeSenders(body.blockedSenders) ?? [],
lifetimeHours: body.lifetimeHours,
});
return c.json(toFeed(feedId, config, 0, env), 201);
},
);
apiApp.openapi(
createRoute({
method: "get",
path: "/v1/feeds/{feedId}",
tags: ["Feeds"],
summary: "Get a feed",
security: bearer,
request: { params: FeedIdParam },
responses: {
200: jsonContent(FeedSchema, "The feed"),
401: jsonContent(ErrorSchema, "Unauthorized"),
404: jsonContent(ErrorSchema, "Feed not found"),
},
}),
async (c) => {
const env = c.env;
const { feedId } = c.req.valid("param");
const repo = FeedRepository.from(env);
const id = FeedId.unchecked(feedId);
const config = await repo.getConfig(id);
if (!config) return c.json({ error: "Feed not found" }, 404);
const metadata = await repo.getMetadata(id);
return c.json(
toFeed(feedId, config, metadata?.emails.length ?? 0, env),
200,
);
},
);
apiApp.openapi(
createRoute({
method: "patch",
path: "/v1/feeds/{feedId}",
tags: ["Feeds"],
summary: "Update a feed",
security: bearer,
request: {
params: FeedIdParam,
body: { content: { "application/json": { schema: FeedUpdateSchema } } },
},
responses: {
200: jsonContent(FeedSchema, "The updated feed"),
400: jsonContent(ErrorSchema, "Invalid request body"),
401: jsonContent(ErrorSchema, "Unauthorized"),
404: jsonContent(ErrorSchema, "Feed not found"),
409: jsonContent(ErrorSchema, "Feed has expired"),
},
}),
async (c) => {
const env = c.env;
const { feedId } = c.req.valid("param");
const id = FeedId.unchecked(feedId);
const body = c.req.valid("json");
const result = await editFeed(env, id, {
title: body.title,
description: body.description,
language: body.language,
allowedSenders: normalizeSenders(body.allowedSenders),
blockedSenders: normalizeSenders(body.blockedSenders),
lifetimeHours: body.lifetimeHours,
});
if (result.status === "not_found")
return c.json({ error: "Feed not found" }, 404);
if (result.status === "expired")
return c.json({ error: "Feed has expired and cannot be modified" }, 409);
const metadata = await FeedRepository.from(env).getMetadata(id);
return c.json(
toFeed(feedId, result.config, metadata?.emails.length ?? 0, env),
200,
);
},
);
apiApp.openapi(
createRoute({
method: "delete",
path: "/v1/feeds/{feedId}",
tags: ["Feeds"],
summary: "Delete a feed",
security: bearer,
request: { params: FeedIdParam },
responses: {
200: jsonContent(OkSchema, "The feed was deleted"),
401: jsonContent(ErrorSchema, "Unauthorized"),
404: jsonContent(ErrorSchema, "Feed not found"),
},
}),
async (c) => {
const env = c.env;
const { feedId } = c.req.valid("param");
const removed = await deleteFeedRecord(env, FeedId.unchecked(feedId), (p) =>
waitUntilSafe(c, p),
);
if (!removed) return c.json({ error: "Feed not found" }, 404);
return c.json({ ok: true }, 200);
},
);
// ── Emails ──────────────────────────────────────────────────────────────────
apiApp.openapi(
createRoute({
method: "get",
path: "/v1/feeds/{feedId}/emails",
tags: ["Emails"],
summary: "List a feed's emails",
security: bearer,
request: { params: FeedIdParam },
responses: {
200: jsonContent(EmailListSchema, "The feed's emails"),
401: jsonContent(ErrorSchema, "Unauthorized"),
404: jsonContent(ErrorSchema, "Feed not found"),
},
}),
async (c) => {
const env = c.env;
const { feedId } = c.req.valid("param");
const metadata = await FeedRepository.from(env).getMetadata(
FeedId.unchecked(feedId),
);
if (!metadata) return c.json({ error: "Feed not found" }, 404);
return c.json(
{
emails: metadata.emails.map((e) => ({
entryId: e.receivedAt,
subject: e.subject,
receivedAt: e.receivedAt,
size: e.size,
attachmentIds: e.attachmentIds,
})),
},
200,
);
},
);
apiApp.openapi(
createRoute({
method: "get",
path: "/v1/feeds/{feedId}/emails/{entryId}",
tags: ["Emails"],
summary: "Get a single email",
security: bearer,
request: { params: EntryIdParam },
responses: {
200: jsonContent(EmailSchema, "The email"),
401: jsonContent(ErrorSchema, "Unauthorized"),
404: jsonContent(ErrorSchema, "Email not found"),
},
}),
async (c) => {
const env = c.env;
const { feedId, entryId } = c.req.valid("param");
const receivedAt = parseInt(entryId, 10);
const repo = FeedRepository.from(env);
const metadata = await repo.getMetadata(FeedId.unchecked(feedId));
const metaEntry = metadata?.emails.find((e) => e.receivedAt === receivedAt);
if (!metaEntry) return c.json({ error: "Email not found" }, 404);
const data = await repo.getEmail(metaEntry.key);
if (!data) return c.json({ error: "Email not found" }, 404);
return c.json(
{
entryId: receivedAt,
subject: data.subject,
from: data.from,
receivedAt: data.receivedAt,
content: data.content,
attachments: (data.attachments ?? [])
.filter((a) => !a.inline)
.map((a) => ({
id: a.id,
filename: a.filename,
contentType: a.contentType,
size: a.size,
url: `/files/${a.id}/${encodeURIComponent(a.filename)}`,
})),
},
200,
);
},
);
apiApp.openapi(
createRoute({
method: "delete",
path: "/v1/feeds/{feedId}/emails/{entryId}",
tags: ["Emails"],
summary: "Delete a single email",
security: bearer,
request: { params: EntryIdParam },
responses: {
200: jsonContent(OkSchema, "The email was deleted"),
401: jsonContent(ErrorSchema, "Unauthorized"),
404: jsonContent(ErrorSchema, "Email not found"),
},
}),
async (c) => {
const env = c.env;
const repo = FeedRepository.from(env);
const { feedId, entryId } = c.req.valid("param");
const receivedAt = parseInt(entryId, 10);
const feed = await repo.load(FeedId.unchecked(feedId));
const metaEntry = feed?.emails.find((e) => e.receivedAt === receivedAt);
if (!feed || !metaEntry) return c.json({ error: "Email not found" }, 404);
await repo.deleteEmail(metaEntry.key);
const { removed } = feed.removeEmails([metaEntry.key]);
await deleteAttachmentsForEmails(env, removed, [metaEntry.key]);
await repo.saveMetadata(feed);
return c.json({ ok: true }, 200);
},
);
// ── Stats ─────────────────────────────────────────────────────────────────────
apiApp.openapi(
createRoute({
method: "get",
path: "/v1/stats",
tags: ["Stats"],
summary: "Read monitoring counters (public)",
responses: {
200: jsonContent(StatsSchema, "Monitoring counters"),
},
}),
async (c) => {
return c.json(await getStats(c.env), 200);
},
);
// ── OpenAPI document + docs (public) ───────────────────────────────────────────
apiApp.doc31("/openapi.json", {
openapi: "3.1.0",
info: {
title: "kill-the-news API",
version: "1.0.0",
description:
"REST API for managing email-to-RSS feeds, their emails, and monitoring counters.",
},
servers: [{ url: "/api" }],
});
apiApp.get(
"/docs",
Scalar({
url: "/api/openapi.json",
pageTitle: "kill-the-news API",
}),
);
+154
View File
@@ -0,0 +1,154 @@
import { z } from "@hono/zod-openapi";
// ── Shared ──────────────────────────────────────────────────────────────────
export const ErrorSchema = z
.object({
error: z.string().openapi({ example: "Feed not found" }),
})
.openapi("Error");
export const FeedIdParam = z.object({
feedId: z
.string()
.min(1)
.openapi({
param: { name: "feedId", in: "path" },
example: "happy-otter-1234",
}),
});
export const EntryIdParam = FeedIdParam.extend({
entryId: z
.string()
.regex(/^\d+$/, "entryId must be the email's receivedAt timestamp")
.openapi({
param: { name: "entryId", in: "path" },
example: "1737000000000",
}),
});
// ── Feeds ───────────────────────────────────────────────────────────────────
export const FeedCreateSchema = z
.object({
title: z.string().min(1).openapi({ example: "Daily Tech Digest" }),
description: z.string().optional(),
language: z.string().optional().default("en"),
allowedSenders: z.array(z.string()).optional().default([]),
blockedSenders: z.array(z.string()).optional().default([]),
lifetimeHours: z.number().int().positive().optional().openapi({
description:
"Hours until the feed expires. Ignored when the server enforces a fixed FEED_TTL_HOURS.",
}),
})
.openapi("FeedCreate");
export const FeedUpdateSchema = z
.object({
title: z.string().min(1).optional(),
description: z.string().optional(),
language: z.string().optional(),
allowedSenders: z.array(z.string()).optional(),
blockedSenders: z.array(z.string()).optional(),
lifetimeHours: z.number().int().positive().optional().openapi({
description: "Reset the feed's lifetime to this many hours from now.",
}),
})
.openapi("FeedUpdate");
export const FeedSummarySchema = z
.object({
id: z.string(),
title: z.string(),
description: z.string().optional(),
expiresAt: z.number().optional(),
emailAddress: z.string(),
rssUrl: z.string(),
atomUrl: z.string(),
})
.openapi("FeedSummary");
export const FeedListSchema = z
.object({ feeds: z.array(FeedSummarySchema) })
.openapi("FeedList");
export const FeedSchema = z
.object({
id: z.string(),
title: z.string(),
description: z.string().optional(),
language: z.string(),
allowedSenders: z.array(z.string()),
blockedSenders: z.array(z.string()),
createdAt: z.number(),
updatedAt: z.number().optional(),
expiresAt: z.number().optional(),
emailCount: z.number(),
emailAddress: z.string(),
rssUrl: z.string(),
atomUrl: z.string(),
})
.openapi("Feed");
// ── Emails ──────────────────────────────────────────────────────────────────
export const EmailSummarySchema = z
.object({
entryId: z.number().openapi({
description: "Email receivedAt timestamp; used as the path id.",
}),
subject: z.string(),
receivedAt: z.number(),
size: z.number().optional(),
attachmentIds: z.array(z.string()).optional(),
})
.openapi("EmailSummary");
export const EmailListSchema = z
.object({ emails: z.array(EmailSummarySchema) })
.openapi("EmailList");
export const AttachmentSchema = z
.object({
id: z.string(),
filename: z.string(),
contentType: z.string(),
size: z.number(),
url: z.string(),
})
.openapi("Attachment");
export const EmailSchema = z
.object({
entryId: z.number(),
subject: z.string(),
from: z.string(),
receivedAt: z.number(),
content: z.string(),
attachments: z.array(AttachmentSchema),
})
.openapi("Email");
// ── Stats ───────────────────────────────────────────────────────────────────
export const StatsSchema = z
.object({
feeds_created: z.number(),
feeds_deleted: z.number(),
emails_received: z.number(),
emails_rejected: z.number(),
emails_forwarded: z.number(),
unsubscribes_sent: z.number(),
active_feeds: z.number(),
websub_subscriptions_active: z.number(),
attachments_enabled: z.boolean(),
last_email_at: z.string().optional(),
last_feed_created_at: z.string().optional(),
first_seen: z.string().optional(),
attachments_bytes: z.number().optional(),
attachments_count: z.number().optional(),
kv_bytes_estimated: z.number().optional(),
storage_scanned_at: z.string().optional(),
})
.openapi("Stats");
+147
View File
@@ -0,0 +1,147 @@
import { describe, it, expect, beforeEach } from "vitest";
import { Hono } from "hono";
import { handle } from "./atom";
import { createMockEnv } from "../test/setup";
import { Env } from "../types";
describe("Atom Feed Route", () => {
let testApp: Hono;
let mockEnv: Env;
beforeEach(() => {
mockEnv = createMockEnv() as unknown as Env;
testApp = new Hono();
testApp.get("/:feedId", handle);
});
describe("unknown feed", () => {
it("returns 404 when no metadata exists in KV", async () => {
const res = await testApp.request("/nonexistent-feed", {}, mockEnv);
expect(res.status).toBe(404);
expect(await res.text()).toBe("Feed not found");
});
});
describe("valid feed with no emails", () => {
beforeEach(async () => {
await mockEnv.EMAIL_STORAGE.put(
"feed:empty-feed:metadata",
JSON.stringify({ emails: [] }),
);
});
it("returns 200 with application/atom+xml content type", async () => {
const res = await testApp.request("/empty-feed", {}, mockEnv);
expect(res.status).toBe(200);
expect(res.headers.get("Content-Type")).toContain("application/atom+xml");
});
it("returns valid Atom XML with no <entry> elements", async () => {
const res = await testApp.request("/empty-feed", {}, mockEnv);
const body = await res.text();
expect(body).toContain('xmlns="http://www.w3.org/2005/Atom"');
expect(body).not.toContain("<entry>");
});
it("includes Cache-Control header", async () => {
const res = await testApp.request("/empty-feed", {}, mockEnv);
expect(res.headers.get("Cache-Control")).toBe("max-age=1800");
});
it("sets X-Robots-Tag: noindex", async () => {
const res = await testApp.request("/empty-feed", {}, mockEnv);
expect(res.headers.get("X-Robots-Tag")).toBe("noindex");
});
});
describe("valid feed with emails", () => {
const FEED_ID = "test-feed-atom";
beforeEach(async () => {
const emailKey = `feed:${FEED_ID}:1700000001000`;
await mockEnv.EMAIL_STORAGE.put(
emailKey,
JSON.stringify({
subject: "Atom Entry Subject",
from: "Sender <[email protected]>",
content: "<p>Email body</p>",
receivedAt: 1700000001000,
headers: {},
}),
);
await mockEnv.EMAIL_STORAGE.put(
`feed:${FEED_ID}:metadata`,
JSON.stringify({
emails: [
{
key: emailKey,
subject: "Atom Entry Subject",
receivedAt: 1700000001000,
},
],
}),
);
await mockEnv.EMAIL_STORAGE.put(
`feed:${FEED_ID}:config`,
JSON.stringify({
title: "Atom Test Feed",
description: "Integration test",
language: "en",
created_at: 1700000000000,
}),
);
});
it("returns 200 with application/atom+xml", async () => {
const res = await testApp.request(`/${FEED_ID}`, {}, mockEnv);
expect(res.status).toBe(200);
expect(res.headers.get("Content-Type")).toContain("application/atom+xml");
});
it("contains Atom namespace declaration", async () => {
const res = await testApp.request(`/${FEED_ID}`, {}, mockEnv);
const body = await res.text();
expect(body).toContain('xmlns="http://www.w3.org/2005/Atom"');
});
it("contains <entry> for the email", async () => {
const res = await testApp.request(`/${FEED_ID}`, {}, mockEnv);
const body = await res.text();
expect(body).toContain("<entry>");
expect(body).toContain("Atom Entry Subject");
});
it("contains feed title", async () => {
const res = await testApp.request(`/${FEED_ID}`, {}, mockEnv);
const body = await res.text();
expect(body).toContain("Atom Test Feed");
});
it("self-link points to atom URL", async () => {
const res = await testApp.request(`/${FEED_ID}`, {}, mockEnv);
const body = await res.text();
expect(body).toContain(`/atom/${FEED_ID}`);
});
it("Link header advertises hub and self for WebSub discovery", async () => {
const res = await testApp.request(`/${FEED_ID}`, {}, mockEnv);
const link = res.headers.get("Link") ?? "";
expect(link).toContain(`rel="hub"`);
expect(link).toContain(`/atom/${FEED_ID}`);
expect(link).toContain(`rel="self"`);
});
});
describe("fallback config when no config in KV", () => {
it("uses atom path in fallback and returns 200", async () => {
await mockEnv.EMAIL_STORAGE.put(
"feed:no-config-feed:metadata",
JSON.stringify({ emails: [] }),
);
const res = await testApp.request("/no-config-feed", {}, mockEnv);
expect(res.status).toBe(200);
const body = await res.text();
expect(body).toContain('xmlns="http://www.w3.org/2005/Atom"');
});
});
});
+51
View File
@@ -0,0 +1,51 @@
import { Context } from "hono";
import { Env } from "../types";
import { generateAtomFeed } from "../infrastructure/feed-generator";
import { fetchFeedData } from "../application/feed-fetcher";
import { baseUrl, feedAtomUrl } from "../infrastructure/urls";
import { isExpired } from "../domain/feed";
import { FeedId } from "../domain/value-objects/feed-id";
export async function handle(c: Context<{ Bindings: Env }>): Promise<Response> {
try {
const feedId = c.req.param("feedId");
if (!feedId) {
return new Response("Feed ID is required", { status: 400 });
}
const feedData = await fetchFeedData(FeedId.unchecked(feedId), c.env);
if (!feedData) {
return new Response("Feed not found", { status: 404 });
}
if (isExpired(feedData.feedConfig)) {
return new Response("Feed has expired", { status: 410 });
}
const base = baseUrl(c.env);
const selfUrl = new URL(c.req.url).origin + `/atom/${feedId}`;
const atomXml = generateAtomFeed(
feedData.feedConfig,
feedData.emails,
base,
feedId,
selfUrl,
);
const linkHeader = [
`<${base}/hub>; rel="hub"`,
`<${feedAtomUrl(feedId, c.env)}>; rel="self"`,
].join(", ");
return new Response(atomXml, {
status: 200,
headers: {
"Content-Type": "application/atom+xml",
"Cache-Control": "max-age=1800",
"X-Robots-Tag": "noindex",
Link: linkHeader,
},
});
} catch (error) {
console.error("Error generating Atom feed:", error);
return new Response("Internal Server Error", { status: 500 });
}
}
+180
View File
@@ -0,0 +1,180 @@
import { describe, it, expect, beforeEach } from "vitest";
import { Hono } from "hono";
import { handle } from "./entries";
import { createMockEnv } from "../test/setup";
const FEED_ID = "test-feed";
const RECEIVED_AT = 1700000001000;
const EMAIL_KEY = `feed:${FEED_ID}:${RECEIVED_AT}`;
function makeApp() {
const app = new Hono();
app.get("/:feedId/:entryId", handle);
return app;
}
async function seedFeed(
env: ReturnType<typeof createMockEnv>,
attachments?: {
id: string;
filename: string;
contentType: string;
size: number;
contentId?: string;
inline?: boolean;
}[],
content = "<p>Email body</p>",
) {
await env.EMAIL_STORAGE.put(
EMAIL_KEY,
JSON.stringify({
subject: "Test Subject",
from: "[email protected]",
content,
receivedAt: RECEIVED_AT,
headers: {},
...(attachments ? { attachments } : {}),
}),
);
await env.EMAIL_STORAGE.put(
`feed:${FEED_ID}:metadata`,
JSON.stringify({
emails: [
{ key: EMAIL_KEY, subject: "Test Subject", receivedAt: RECEIVED_AT },
],
}),
);
}
describe("GET /entries/:feedId/:entryId", () => {
let env: ReturnType<typeof createMockEnv>;
beforeEach(() => {
env = createMockEnv();
});
it("returns 404 when feed does not exist", async () => {
const app = makeApp();
const res = await app.request(`/${FEED_ID}/999`, {}, env as any);
expect(res.status).toBe(404);
expect(await res.text()).toContain("Feed not found");
});
it("returns 404 when entry does not exist in metadata", async () => {
const app = makeApp();
await env.EMAIL_STORAGE.put(
`feed:${FEED_ID}:metadata`,
JSON.stringify({ emails: [] }),
);
const res = await app.request(`/${FEED_ID}/999`, {}, env as any);
expect(res.status).toBe(404);
expect(await res.text()).toContain("Entry not found");
});
it("returns 404 when entryId is not a number", async () => {
const app = makeApp();
const res = await app.request(`/${FEED_ID}/not-a-number`, {}, env as any);
expect(res.status).toBe(404);
});
it("returns 200 with HTML for valid entry", async () => {
await seedFeed(env);
const app = makeApp();
const res = await app.request(`/${FEED_ID}/${RECEIVED_AT}`, {}, env as any);
expect(res.status).toBe(200);
expect(res.headers.get("Content-Type")).toContain("text/html");
});
it("includes email subject in HTML title", async () => {
await seedFeed(env);
const app = makeApp();
const res = await app.request(`/${FEED_ID}/${RECEIVED_AT}`, {}, env as any);
const body = await res.text();
expect(body).toContain("Test Subject");
});
it("includes email content in HTML body", async () => {
await seedFeed(env);
const app = makeApp();
const res = await app.request(`/${FEED_ID}/${RECEIVED_AT}`, {}, env as any);
const body = await res.text();
expect(body).toContain("<p>Email body</p>");
});
it("includes sender in HTML", async () => {
await seedFeed(env);
const app = makeApp();
const res = await app.request(`/${FEED_ID}/${RECEIVED_AT}`, {}, env as any);
const body = await res.text();
expect(body).toContain("[email protected]");
});
it("lists attachments with download links when present", async () => {
await seedFeed(env, [
{
id: "att-123",
filename: "report final.pdf",
contentType: "application/pdf",
size: 2048,
},
]);
const app = makeApp();
const res = await app.request(`/${FEED_ID}/${RECEIVED_AT}`, {}, env as any);
const body = await res.text();
expect(body).toContain("Attachments");
expect(body).toContain("report final.pdf");
expect(body).toContain(
`/files/att-123/${encodeURIComponent("report final.pdf")}`,
);
expect(body).toContain("2.0 KB");
});
it("renders inline images in place and omits them from the attachments list", async () => {
await seedFeed(
env,
[
{
id: "img-1",
filename: "logo.png",
contentType: "image/png",
size: 512,
contentId: "logo123",
inline: true,
},
],
'<p>Body</p><img src="cid:logo123"/>',
);
const app = makeApp();
const res = await app.request(`/${FEED_ID}/${RECEIVED_AT}`, {}, env as any);
const body = await res.text();
// The cid: ref is rewritten to the stored file URL (rendered in place)…
expect(body).toContain('src="/files/img-1/logo.png"');
expect(body).not.toContain("cid:logo123");
// …and the image is not listed as a downloadable attachment.
expect(body).not.toContain("Attachments");
});
it("does not render an attachments section when there are none", async () => {
await seedFeed(env);
const app = makeApp();
const res = await app.request(`/${FEED_ID}/${RECEIVED_AT}`, {}, env as any);
const body = await res.text();
expect(body).not.toContain("Attachments");
});
it("sets Content-Security-Policy header", async () => {
await seedFeed(env);
const app = makeApp();
const res = await app.request(`/${FEED_ID}/${RECEIVED_AT}`, {}, env as any);
expect(res.headers.get("Content-Security-Policy")).toContain(
"default-src 'none'",
);
});
it("sets X-Robots-Tag: noindex", async () => {
await seedFeed(env);
const app = makeApp();
const res = await app.request(`/${FEED_ID}/${RECEIVED_AT}`, {}, env as any);
expect(res.headers.get("X-Robots-Tag")).toBe("noindex");
});
});
+109
View File
@@ -0,0 +1,109 @@
import { Context } from "hono";
import { html, raw } from "hono/html";
import { Env } from "../types";
import { processEmailContent } from "../infrastructure/html-processor";
import { EmailAddress } from "../domain/value-objects/email-address";
import { formatBytes } from "../domain/format";
import { FeedRepository } from "../infrastructure/feed-repository";
import { FeedId } from "../domain/value-objects/feed-id";
import { isExpired } from "../domain/feed";
import entryCss from "../styles/entry.css";
export async function handle(c: Context<{ Bindings: Env }>): Promise<Response> {
const feedId = c.req.param("feedId");
const receivedAt = parseInt(c.req.param("entryId") ?? "", 10);
if (!feedId || isNaN(receivedAt)) {
return new Response("Not Found", { status: 404 });
}
const repo = FeedRepository.from(c.env);
const id = FeedId.unchecked(feedId);
const [feedMetadata, feedConfig] = await Promise.all([
repo.getMetadata(id),
repo.getConfig(id),
]);
if (!feedMetadata) {
return new Response("Feed not found", { status: 404 });
}
if (feedConfig && isExpired(feedConfig)) {
return new Response("Feed has expired", { status: 410 });
}
const metaEntry = feedMetadata.emails.find(
(e) => e.receivedAt === receivedAt,
);
if (!metaEntry) {
return new Response("Entry not found", { status: 404 });
}
const emailData = await repo.getEmail(metaEntry.key);
if (!emailData) {
return new Response("Entry not found", { status: 404 });
}
c.header(
"Content-Security-Policy",
"default-src 'none'; style-src 'unsafe-inline'; img-src *; frame-src 'none'",
);
c.header("X-Robots-Tag", "noindex");
const bodyContent = processEmailContent(
emailData.content,
emailData.attachments,
"",
EmailAddress.parse(emailData.from)?.siteBaseUrl() ?? "",
);
// Inline images render in place (cid: refs are rewritten by processEmailContent);
// only genuine, downloadable attachments belong in the list below.
const attachments = (emailData.attachments ?? []).filter((a) => !a.inline);
const attachmentsSection = attachments.length
? html`<section class="attachments">
<h2>Attachments</h2>
<ul>
${attachments.map(
(a) =>
html`<li>
<a
href="/files/${a.id}/${encodeURIComponent(a.filename)}"
download
>${a.filename}</a
>
<span class="size">${formatBytes(a.size)}</span>
</li>`,
)}
</ul>
</section>`
: "";
return c.html(
html`<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8" />
<meta
name="viewport"
content="width=device-width, initial-scale=1.0"
/>
<title>${emailData.subject}</title>
<link rel="icon" type="image/svg+xml" href="/favicon.svg" />
<style>
${raw(entryCss)}
</style>
</head>
<body>
<h1>${emailData.subject}</h1>
<dl class="meta">
<dt>From:</dt>
<dd>${emailData.from}</dd>
<dt>Date:</dt>
<dd>${new Date(emailData.receivedAt).toUTCString()}</dd>
</dl>
<div class="content">${raw(bodyContent)}</div>
${attachmentsSection}
</body>
</html>`,
);
}
Loaded 100 of 140 files, more files were not shown because too many files have changed in this diff. Show more