Compare commits

...
Author SHA1 Message Date
neo773andClaude Opus 4.8 0216a13f3e feat(emailing-domain): paywall provisioning + sending behind active billing
The GraphQL surface was already flag-gated (IS_EMAIL_GROUP_ENABLED), but
provisioning a domain — the costly AWS step — was reachable on any plan once
an admin enabled the flag. Twenty has no entitlement primitive, so mirror how
workflows/AI are paywalled: gate on BillingUsageService.canFeatureBeUsed
(active, non-canceled subscription; permissive when billing is disabled).

- assertBillingAllowsEmailGroup now guards both createEmailingDomain and
  sendEmail.
- Self-host / billing-disabled is unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 14:14:09 +05:30
neo773andClaude Opus 4.8 347669dcaa feat(emailing-domain): rate limits + per-recipient credit metering
Input caps:
- @MaxLength on subject/text/html, @ArrayMaxSize on to/cc/bcc/replyTo, and a
  hard EMAIL_MAX_TOTAL_RECIPIENTS guard on the deliverable total (SES's
  50-recipient limit).

Throttle:
- Reuse ThrottlerService token bucket keyed emailing-domain-send:<workspaceId>,
  one token per deliverable recipient, checked before the driver send.

Credit metering (3x AWS cost; 1 credit = $1):
- New EMAIL_SEND UsageOperationType (+ exhaustive USAGE_UNIT map entry).
- Emit USAGE_RECORDED via the global WorkspaceEventEmitter, quantity = accepted
  recipients, creditsUsedMicro = 0.0001 * 3 * DOLLAR_TO_CREDIT_MULTIPLIER per
  recipient (300 micro/recipient). Enforcement rides the existing
  USAGE_RECORDED -> ClickHouse -> cap pipeline, as WORKFLOW_EXECUTION does.
- Pre-send BillingUsageService.canFeatureBeUsed gate (rejects inactive plans;
  permissive when billing disabled).

Order in sendEmail: verified -> tenant-status -> from-domain -> suppression
filter -> recipient cap -> billing gate -> throttle -> send -> meter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 02:43:09 +05:30
neo773andClaude Opus 4.8 e350c391fd feat(emailing-domain): SES bounce/complaint suppression + app-owned unsubscribe
Consume SES message events and own the unsubscribe flow, replacing the
broken per-workspace SES contact list.

Events:
- Extend the existing /outbound EventBridge->SNS router to dispatch on
  detail-type; new SesSuppressionEventHandlerService suppresses on
  'Email Bounced' (only bounceType Permanent -> HARD_BOUNCE) and
  'Email Complaint Received' (-> COMPLAINT). feedbackId is stored in
  providerEventId. Workspace resolved from the workspace EmailTag, falling
  back to the resource ARN.

Rip out broken SES list management (one contact list per AWS account):
- Remove CreateContactList/DeleteContactList, ListManagementOptions, the
  contactListName plumbing, and the marketing-topic constant.

App-owned unsubscribe:
- EmailGroupUnsubscribeService builds RFC 8058 List-Unsubscribe +
  List-Unsubscribe-Post headers (via Content.Simple.Headers, no Raw MIME),
  signed with a self-contained HMAC-SHA256 APP_SECRET token.
- Public EmailingDomainUnsubscribeController (POST one-click + GET fallback)
  writes a MARKETING/UNSUBSCRIBE suppression row on a valid token.
- Headers attach only for single-recipient sends with includeUnsubscribe,
  so transactional mail stays untouched until the channel type lands.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 02:30:54 +05:30
neo773andClaude Opus 4.8 c78e45aa04 feat(emailing-domain): per-workspace email suppression
SES suppression is account-level only, so a bounce/complaint in one
workspace blocked sends in every other workspace. Replace it with an
app-owned, per-workspace suppression list.

- EmailGroupSuppressedRecipientEntity (core.emailGroupSuppressedRecipient)
  keyed UNIQUE(workspaceId, emailAddress, scope). scope separates
  permanent all-channel reasons (HARD_BOUNCE/COMPLAINT -> GLOBAL) from the
  reversible marketing unsubscribe (-> MARKETING); status ACTIVE/RELEASED
  makes resubscribe non-destructive; providerEventId reserved for SES
  event-grain dedupe.
- EmailGroupSuppressionService: reactivating per-scope upsert, marketing-
  only release, ACTIVE-only read filter.
- EmailingDomainService.sendEmail filters suppressed to/cc/bcc and throws
  ALL_RECIPIENTS_SUPPRESSED when every primary recipient is suppressed.
- Opt the config set out of the shared account list
  (SuppressionOptions.SuppressedReasons = []).

Schema chosen via a durability simulation (docs/EMAILING_DOMAIN_
SUPPRESSION_DESIGN_REVIEW.md): the naive single-row design diverged from
correct behavior in 22.8% of the event space (resending to dead/complained
mailboxes on resubscribe; blocking transactional mail on a marketing
unsubscribe); this schema verifies at 0%.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-05-29 02:14:34 +05:30
neo773 0273c1a8d5 fix: harden email-group SES provisioning, cleanup, and inbound replay
- workspace deletion captures emailing domains before the row is removed and
  passes them into the cleanup job; cleanup now propagates errors so the queue
  retries instead of silently leaking per-domain SES resources
- SES workspace provisioning ensures each resource independently and ignores
  AlreadyExistsException, so a retry after a partial failure completes the
  remaining steps (removes the single config-set existence gate)
- inbound mail enqueue dedupes on the SNS MessageId so retries/replays import once
2026-05-29 01:24:48 +05:30
35 changed files with 1475 additions and 200 deletions
@@ -0,0 +1,263 @@
# Email-Group (SES): Per-Workspace Suppression, Events, Rate Limits, Enterprise Gate
## Problem
SES suppression list is **account-level only** — one shared list per AWS account/region. Config-set
`SuppressionOptions.SuppressedReasons` only toggles whether a reason feeds/honors that one shared list.
So a hard bounce in workspace A silently blocks sends to the same address in workspace B. SES "tenants"
isolate reputation and sending-pause, **not** suppression.
Second defect: SES allows only **one contact list per account** (max 20 topics). The current
per-workspace `CreateContactListCommand` works for workspace #1 and fails for #2. Unsubscribe must be
app-owned, not SES-list-owned.
## Scope (v1)
- Per-workspace suppression, **GLOBAL scope only**. No campaign/topic-level scope — no campaign entity
exists and transactional-vs-marketing split is coming via a workspace transactional channel type, not
via suppression scope.
- Three actionable SES event types: `HARD_BOUNCE` (Permanent bounce only) + `COMPLAINT` → suppress;
`SUBSCRIPTION` → unsubscribe. All other 7 event types are telemetry.
- Soft bounces arrive as `DeliveryDelay`, not `Bounce` — never suppressed.
---
## 1. Per-workspace suppression
### Entity
`EmailGroupSuppressedRecipientEntity` → table `core.emailGroupSuppressedRecipient`. Schema was
revised after a durability simulation — see
`EMAILING_DOMAIN_SUPPRESSION_DESIGN_REVIEW.md` for why the naive single-row design fails (22.8% of
the event space) and why this one is verified correct (0%).
| column | type | notes |
|---|---|---|
| `id` | uuid | `@PrimaryGeneratedColumn('uuid')` |
| `workspaceId` | uuid | FK → workspace, `ON DELETE CASCADE` (via `WorkspaceRelatedEntity`) |
| `emailAddress` | varchar | normalized (trim + lowercase) |
| `scope` | enum | `GLOBAL` (bounce/complaint, all channels) \| `MARKETING` (unsubscribe, reversible) |
| `reason` | enum | `HARD_BOUNCE \| COMPLAINT \| UNSUBSCRIBE` |
| `status` | enum | `ACTIVE \| RELEASED` — resubscribe sets `RELEASED`, never deletes |
| `providerEventId` | varchar null | SES feedbackId/messageId for event-grain dedupe |
| `createdBySource` | enum | `FieldActorSource``WEBHOOK` for SES events, `API` for self-serve unsubscribe |
| `createdAt` | timestamptz | `@CreateDateColumn` |
| `updatedAt` | timestamptz | `@UpdateDateColumn` |
Constraint: `UNIQUE(workspaceId, emailAddress, scope)`.
Extend `WorkspaceRelatedEntity` (gives `workspaceId` + cascade `@ManyToOne` to `WorkspaceEntity`),
matching `EmailingDomainEntity`. Enum columns match the local `EmailingDomainTenantStatus` pattern
(TypeORM `enum` type), not string-literal columns.
Write rules (in `EmailGroupSuppressionService`):
- `suppress`: scope derived from reason (`UNSUBSCRIBE → MARKETING`, else `GLOBAL`); reactivating
upsert on `(workspaceId, emailAddress, scope)` with `status = ACTIVE`. Scope separation keeps
bounce/complaint and unsubscribe in distinct rows, so they never overwrite each other.
- `releaseMarketingSuppression`: sets `status = RELEASED` on the `MARKETING` row only; `GLOBAL`
(permanent bounce/complaint) is never touched by a resubscribe.
- Read path (`getSuppressedAddresses`): returns addresses with any `ACTIVE` row. v1 is
channel-agnostic (any active scope blocks); add a scope filter when the transactional channel
type lands.
### Migration
Generate via CLI, never hand-write:
```
npx nx run twenty-server:database:migrate:generate --name addEmailGroupSuppressedRecipient --type fast
```
Auto-registers in `instance-commands.constant.ts`.
### Opt out of account-level list
In `aws-ses-register-domain.service.ts`, `CreateConfigurationSetCommand` currently sends
`SuppressionOptions: { SuppressedReasons: ['BOUNCE', 'COMPLAINT'] }`. Change to
`SuppressedReasons: []` so this config set neither feeds nor honors the shared account list. Suppression
becomes entirely app-owned.
Existing provisioned config sets need a one-time `PutConfigurationSetSuppressionOptionsCommand` backfill —
fold into the same instance command (workspace-scoped) or a follow-up.
### Enforce at send time
In `EmailingDomainService.sendEmail` (after tenant-status check, before driver send): query
`emailGroupSuppressedRecipient` for `workspaceId` where `emailAddress IN (lowercased to/cc/bcc)`. Filter
suppressed addresses out of each list. If **all primary `to` recipients** are suppressed → throw
(`ALL_RECIPIENTS_SUPPRESSED`). Partial suppression → send to survivors.
---
## 2. Handle actionable events
### Rip out broken contact-list code
- Remove `CreateContactListCommand` from `aws-ses-register-domain.service.ts` (breaks at workspace #2).
- Remove `ListManagementOptions: { ContactListName, TopicName }` from `SendEmailCommand` in
`aws-ses-send-email.service.ts`.
- Drop the `AWS_SES_MARKETING_TOPIC_NAME` contact-list plumbing.
### Routing (extend existing path — do NOT build a new webhook)
EventBridge → SNS → `POST /webhooks/messaging/ses/outbound``SesOutboundWebhookRouterService` already
exists and consumes `SesEventBridgeNotification`, switching on `detail-type`. Today it handles only
`'Sending Status Enabled' | 'Sending Status Disabled'` → tenant pause.
As built:
1. Widened `SesEventBridgeNotification['detail-type']` to include `'Email Bounced'` and
`'Email Complaint Received'` (the actual EventBridge envelope strings — verified against AWS docs;
the SNS-style `detail.eventType` is `'Bounce'`/`'Complaint'`) plus the `detail.mail` / `detail.bounce`
/ `detail.complaint` payload fields.
2. Router dispatches by `detail-type`: sending-status → `SesOutboundSendingStateHandlerService`;
`'Email Bounced'`/`'Email Complaint Received'` → new `SesSuppressionEventHandlerService`.
3. Handler suppresses via the Item-1 reactivating upsert. Suppress **only** `bounce.bounceType ===
'Permanent'` (HARD_BOUNCE) and all complaints (COMPLAINT). `feedbackId` stored in `providerEventId`.
4. Workspace ID resolved from the `workspace` EmailTag (`detail.mail.tags.workspace[0]`, the exact id we
set at send time), falling back to parsing the resource ARN.
5. `Subscription` events are moot once SES list management is removed (we no longer own an SES list);
unsubscribe is handled entirely by the app endpoint below.
### twenty-infra reconciliation
Dangling uncommitted files (`ses-suppression-events.tf`, `sns-ses-suppression-events.tf`, per-route
`SES_SNS_TOPIC_*_ARN` env vars in `charts/*/values.yaml`) point at a route that never existed. Since
events ride the existing outbound topic/route, **delete** them rather than wire them up — unless the
existing outbound topic does not also carry Bounce/Complaint (verify the EventBridge rule → SNS topic
fan-out in twenty-infra before deleting).
### Unsubscribe (app-owned) — as built
SES list management is unusable (fact: one list/account). Own it:
- `EmailGroupUnsubscribeService` builds `List-Unsubscribe` + `List-Unsubscribe-Post:
List-Unsubscribe=One-Click` (RFC 8058) headers, attached via `Content.Simple.Headers`
(`MessageHeader[]` — supported in `@aws-sdk/client-sesv2@3.1001.0`; **no Raw MIME**).
- Token is a self-contained HMAC-SHA256 over `APP_SECRET` of `{workspaceId, emailAddress}`
(base64url payload + signature, constant-time compare). No JWT key/token-type coupling, stateless
verify. Never trusts raw query params.
- `EmailingDomainUnsubscribeController` exposes public `POST /emailing/unsubscribe` (one-click) and
`GET /emailing/unsubscribe` (browser fallback); a valid token writes a `MARKETING`/`UNSUBSCRIBE`
suppression row (`FieldActorSource.API`). Always returns 200 (no recipient-existence leak).
- Headers are attached **only** when the caller sets `includeUnsubscribe: true` **and** the message
resolves to exactly one deliverable recipient (one-click targets a single address). v1 read path is
channel-agnostic, so an unsubscribe blocks all sends for that address until the transactional channel
type lands — transactional callers must leave `includeUnsubscribe` unset.
---
## 3. Rate limits + billing units
### Input caps (`send-email-via-domain.input.ts`)
Add `class-validator` decorators:
- `@MaxLength` on `subject`, `text`, `html`.
- `@ArrayMaxSize` on `to`, `cc`, `bcc`, `replyTo` (existing: only `@ArrayMinSize(1)` on `to`).
### Throttle — as built
- Reuses the existing `ThrottlerService.tokenBucketThrottleOrThrow`, keyed
`emailing-domain-send:<workspaceId>`, consuming one token per deliverable recipient
(`EMAIL_SEND_THROTTLE_MAX_RECIPIENTS` per `EMAIL_SEND_THROTTLE_WINDOW_MS`). Checked after the
suppression filter, before the driver send.
- Hard per-message recipient cap (`EMAIL_MAX_TOTAL_RECIPIENTS`, SES's 50-recipient limit) on the
deliverable total; DTO also caps each field (`@ArrayMaxSize`) and body/subject lengths
(`@MaxLength`).
### Credit metering (Felix)
Charge ~**3× the AWS cost**. The existing billing system already defines **1 credit = $1**:
`DOLLAR_TO_CREDIT_MULTIPLIER = 1_000_000` stores micro-credits, so
`creditsUsedMicro = dollarCost × 1_000_000`. No new credit unit needed.
Cost basis:
- AWS SES outbound = **$0.10 / 1000 emails = $0.0001 per recipient** (SES bills per recipient, each
To/Cc/Bcc counts).
- 3× markup → **$0.0003 per recipient** → `creditsUsedMicro = 300` per recipient.
Define both pieces as constants so the markup is auditable and AWS repricing is one edit:
```
SES_AWS_COST_PER_RECIPIENT_USD = 0.0001
EMAIL_SEND_CREDIT_MARKUP = 3
```
Meter **per recipient, post-suppression-filter** (charge only what is actually accepted for sending —
matches how SES itself bills). Bcc/Cc count.
Integration (existing path):
1. Add `EMAIL_SEND` to `UsageOperationType`
(`src/engine/core-modules/usage/enums/usage-operation-type.enum.ts`).
2. Create/map a Stripe meter with eventName `'EMAIL_SEND'` (mirror `WORKFLOW_NODE_RUN`).
3. After a successful driver send, emit:
```
workspaceEventEmitter.emitCustomBatchEvent(USAGE_RECORDED, [{
resourceType: UsageResourceType.API,
operationType: UsageOperationType.EMAIL_SEND,
quantity: acceptedRecipientCount,
creditsUsedMicro: SES_AWS_COST_PER_RECIPIENT_USD * EMAIL_SEND_CREDIT_MARKUP
* DOLLAR_TO_CREDIT_MULTIPLIER * acceptedRecipientCount,
unit: UsageUnit.<email unit>,
periodStart: subscriptionPeriodStart,
}], workspaceId)
```
4. Pre-send gate: `BillingUsageService.canFeatureBeUsed(workspaceId)` (rejects when the workspace has
no active subscription); when billing is disabled it returns true so dev/self-host still sends.
Order in `sendEmail`: verified → tenant-status → from-domain → suppression filter → recipient cap →
billing gate → throttle → send → meter.
As built: metering emits via the **global `WorkspaceEventEmitter`** only (no `BillingModule` coupling
for the meter itself); the existing `USAGE_RECORDED` → ClickHouse → cap pipeline performs credit
**enforcement** asynchronously (pausing the workspace at the cap), matching how
`WORKFLOW_EXECUTION` is metered. A hard pre-send balance reject (vs the coarse subscription gate) can
layer on later via `BillingUsageService` if product wants it. Adding `EMAIL_SEND` to
`UsageOperationType` also required an entry in the exhaustive
`USAGE_UNIT_BY_OPERATION_TYPE` map (`app-billing.service.ts`).
Attachment data ($0.12/GB on SES) not metered in v1 — note as follow-up. A Stripe meter with
eventName `EMAIL_SEND` must be provisioned for revenue to land (no reverse op-type→meter mapping
exists in code yet).
---
## 4. Enterprise gate / paywall — as built
Intent: the whole feature is paywalled, especially emailing-domain provisioning.
Twenty has **no `entitlement` primitive**; billed features (workflows, AI) are paywalled with
`BillingUsageService.canFeatureBeUsed` (active, non-canceled subscription) plus the resource-credit
cap. This feature follows that same pattern:
- **Visibility / admin gate:** every GraphQL operation already carries
`@RequireFeatureFlag(FeatureFlagKey.IS_EMAIL_GROUP_ENABLED)` + `FeatureFlagGuard`.
- **Paywall gate:** `assertBillingAllowsEmailGroup` (`canFeatureBeUsed`) now guards **both**
`createEmailingDomain` (provisioning — the costly AWS step) **and** `sendEmail`. A workspace cannot
provision a domain or send without an active plan, even if an admin flips the feature flag.
`canFeatureBeUsed` returns true when billing is disabled, so dev/self-host is unaffected.
- The public SES webhook + unsubscribe endpoints are intentionally not flag-gated: they are
event/token-driven and only become reachable after a (now paywalled) provisioning.
Follow-ups for a true per-plan paywall (billing config, not code): provision a billing product/price
for the email feature and scope `IS_EMAIL_GROUP_ENABLED` to paid plans; a frontend upsell for the
blocked state.
---
## Conventions
- No code comments — self-documenting names.
- Swallow expected SDK errors via `.send(cmd).catch((e) => { if (!(e instanceof X)) throw e; })`, not
try/catch or helper wrappers.
- `isDefined` from `twenty-shared/utils`; `isNonEmptyString`/`isNonEmptyArray` from `@sniptt/guards`.
- String-literal unions over enums. Named exports. Functional. No `any`.
- After changes: `npx nx typecheck twenty-server`; `oxlint --type-aware` on changed files; jest scoped
to the file/test, never the full package suite.
## Verified facts (do not re-litigate)
- SES suppression list is account-level only; config-set options only toggle feed/honor of that one list.
- SES tenants isolate reputation + sending-pause, not suppression.
- One contact list per account (max 20 topics) → per-workspace `CreateContactList` is broken.
- Only 3 of 10 SES event types actionable: Permanent Bounce + Complaint (suppress), Subscription
(unsubscribe). Soft bounce = `DeliveryDelay`.
- `@aws-sdk/client-sesv2@3.1001.0` `Content.Simple` (`Message`) supports `Headers?: MessageHeader[]`.
- Outbound webhook route + `SesOutboundWebhookRouterService` already consume EventBridge→SNS; extend it.
@@ -0,0 +1,129 @@
# Suppression Entity — Durability Review (simulation-backed)
## Question
Is `EmailGroupSuppressedRecipientEntity` as shipped in Item 1 the final design, or does it
encode mistakes that bite later?
**Answer: it is not final.** A simulation over the full event space shows the single-row schema
produces incorrect send decisions in **22.8%** of cases, of which the dominant class is
**resending to dead-mailbox / complained addresses** (140 critical cells), plus silently dropping
**transactional** mail on a marketing unsubscribe (15 cells).
## Model
Each `(workspaceId, emailAddress)` receives a sequence of events
`{B = hard bounce, C = complaint, U = unsubscribe, R = resubscribe}` followed by a `SEND` on a
channel `{T = transactional, M = marketing}`.
Correct spec (the lattice we actually need):
| reason | strength | scope | reversible |
|---|---|---|---|
| COMPLAINT | 3 | all channels | no (legal record) |
| HARD_BOUNCE | 2 | all channels | no (dead mailbox) |
| UNSUBSCRIBE | 1 | marketing only | yes (resubscribe) |
`R` releases an active `U` only; it must never revive `B`/`C`. Transactional mail is exempt from
unsubscribe (CAN-SPAM / GDPR both treat it separately).
Current design under test: one row, `UNIQUE(workspaceId, emailAddress)`, insert-then-DO-NOTHING,
no scope, and the natural resubscribe = delete the `(workspaceId, emailAddress)` row.
## Results (sequences length 1..4 × 2 channels = 680 cells)
- Total divergence from correct: **155 / 680 = 22.8%**
- FALSE_SEND (we send to a suppressed address): **140**, all critical (B/C involved)
- FALSE_BLOCK (we drop legitimate mail): **15**
Minimal witnesses:
| events | send | current | correct | failure |
|---|---|---|---|---|
| `B R` | T or M | ALLOW | BLOCK | resubscribe deletes the bounce row → mail to a dead mailbox |
| `C R` | T or M | ALLOW | BLOCK | resubscribe deletes the complaint row → resend to a complainer (legal) |
| `U` | T | BLOCK | ALLOW | marketing unsubscribe blocks a transactional send (password reset) |
Root cause of the critical class: a single row cannot distinguish reasons, so any resubscribe keyed
by `(workspaceId, emailAddress)` removes whatever reason happens to be stored — including permanent
suppressions. The schema, not the code, forces this.
## The `reason` column is decided by a race
Closed form: with `n` distinct-precedence events arriving in uniformly random order (SNS delivers
unordered, at-least-once), insert-DO-NOTHING keeps the **first** arrival, but the correct stored
reason is the **strongest**:
```
P(stored reason wrong) = 1 - 1/n
n=2 -> 50.0% n=3 -> 66.7%
```
Empirically over all multi-event orderings: **58.3%** of cases store the wrong reason. The `reason`
column is therefore unreliable for any reason-aware logic (reporting, reason-aware resubscribe,
audit) — its value reflects SNS arrival order, not severity.
## Three mistakes that bite later
1. **`UNIQUE(workspaceId, emailAddress)` collapses multi-reason history.** One address can be
unsubscribed *and* later hard-bounce. One row + DO-NOTHING loses the second fact. Reason-aware
resubscribe becomes impossible; resubscribe un-suppresses permanent reasons (the 140 cells).
2. **No scope dimension.** Unsubscribe is marketing-only and reversible; bounce/complaint are
all-channel and permanent. Global-only over-blocks transactional mail today, and the
transactional/marketing channel split is already on the roadmap — adding scope later is a
`UNIQUE`-key change (index rebuild + data backfill), the most expensive migration kind.
3. **Hard-delete resubscribe destroys the compliance trail and races the webhook.** GDPR wants
proof-of-consent history and complaint records retained; deleting rows erases them, and a delete
racing an SNS re-delivery can resurrect or drop a row nondeterministically.
## Recommended revision (durable, still pragmatic)
Keep a single projection table but fix the key and lifecycle. No event-sourcing required.
```
EmailGroupSuppressedRecipientEntity
id uuid pk
workspaceId uuid FK CASCADE
emailAddress varchar -- normalized (see below)
scope enum GLOBAL | MARKETING -- B/C => GLOBAL, U => MARKETING
reason enum HARD_BOUNCE | COMPLAINT | UNSUBSCRIBE
status enum ACTIVE | RELEASED -- resubscribe sets RELEASED, never deletes
providerEventId varchar null -- SES feedbackId/messageId, event-grain dedupe
createdBySource FieldActorSource
createdAt timestamptz
updatedAt timestamptz
UNIQUE(workspaceId, emailAddress, scope)
```
Write rules:
- Suppress: upsert on `(workspaceId, emailAddress, scope)`. On conflict, keep the **higher-strength**
reason and `status=ACTIVE` (precedence-max, not first-wins). Permanent reasons land in `GLOBAL`.
- Resubscribe: set `status=RELEASED` on the `MARKETING` row only. `GLOBAL` (B/C) is never touched.
- Send decision: blocked iff an `ACTIVE` row exists whose scope covers the channel
(`GLOBAL` covers all; `MARKETING` covers marketing only).
- Idempotency: dedupe on `providerEventId` when present; the unique key still backstops replays.
Running the same simulation against these rules yields **0 divergences** — false-sends and
false-blocks both go to zero.
If full auditability is later required (dispute handling, consent proof at event grain), promote to
an append-only `EmailGroupSuppressionEventEntity` log with this table as a materialized projection.
The schema above is forward-compatible with that move.
## Orthogonal: address normalization
Lowercasing is necessary but not sufficient. Same inbox, different stored strings → a bounce on one
form won't suppress the other:
- Gmail dot-folding (`a.b@gmail.com``ab@gmail.com`) and `+tag` sub-addressing.
- IDN/Unicode domains; trim/whitespace; display-name leakage.
Normalize to a canonical form before storing and before the send-time lookup, or accept a known
miss rate. Decide explicitly; do not leave it implicit.
## Cost note
Item 1's migration is **not yet committed**, so adopting the revised schema now is a regenerate, not
a data migration. Done after release, fix #2 alone is a `UNIQUE`-key change over live suppression
data. Cheapest moment to get the key right is before the first commit.
@@ -0,0 +1,25 @@
import { QueryRunner } from 'typeorm';
import { RegisteredInstanceCommand } from 'src/engine/core-modules/upgrade/decorators/registered-instance-command.decorator';
import { FastInstanceCommand } from 'src/engine/core-modules/upgrade/interfaces/fast-instance-command.interface';
@RegisteredInstanceCommand('2.9.0', 1780000902115)
export class AddEmailGroupSuppressedRecipientFastInstanceCommand implements FastInstanceCommand {
public async up(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query('CREATE TYPE "core"."emailGroupSuppressedRecipient_scope_enum" AS ENUM(\'GLOBAL\', \'MARKETING\')');
await queryRunner.query('CREATE TYPE "core"."emailGroupSuppressedRecipient_reason_enum" AS ENUM(\'HARD_BOUNCE\', \'COMPLAINT\', \'UNSUBSCRIBE\')');
await queryRunner.query('CREATE TYPE "core"."emailGroupSuppressedRecipient_status_enum" AS ENUM(\'ACTIVE\', \'RELEASED\')');
await queryRunner.query('CREATE TYPE "core"."emailGroupSuppressedRecipient_createdbysource_enum" AS ENUM(\'EMAIL\', \'CALENDAR\', \'WORKFLOW\', \'AGENT\', \'API\', \'IMPORT\', \'MANUAL\', \'SYSTEM\', \'WEBHOOK\', \'APPLICATION\')');
await queryRunner.query('CREATE TABLE "core"."emailGroupSuppressedRecipient" ("workspaceId" uuid NOT NULL, "id" uuid NOT NULL DEFAULT uuid_generate_v4(), "emailAddress" character varying NOT NULL, "scope" "core"."emailGroupSuppressedRecipient_scope_enum" NOT NULL, "reason" "core"."emailGroupSuppressedRecipient_reason_enum" NOT NULL, "status" "core"."emailGroupSuppressedRecipient_status_enum" NOT NULL DEFAULT \'ACTIVE\', "providerEventId" character varying, "createdBySource" "core"."emailGroupSuppressedRecipient_createdbysource_enum" NOT NULL, "createdAt" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(), "updatedAt" TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT now(), CONSTRAINT "IDX_EMAIL_GROUP_SUPPRESSED_RECIPIENT_WORKSPACE_EMAIL_SCOPE_UNIQUE" UNIQUE ("workspaceId", "emailAddress", "scope"), CONSTRAINT "PK_55b0607e539d7941cbaedf1328a" PRIMARY KEY ("id"))');
await queryRunner.query('ALTER TABLE "core"."emailGroupSuppressedRecipient" ADD CONSTRAINT "FK_866066f1e73b748f917fd6fcd80" FOREIGN KEY ("workspaceId") REFERENCES "core"."workspace"("id") ON DELETE CASCADE ON UPDATE NO ACTION');
}
public async down(queryRunner: QueryRunner): Promise<void> {
await queryRunner.query('ALTER TABLE "core"."emailGroupSuppressedRecipient" DROP CONSTRAINT "FK_866066f1e73b748f917fd6fcd80"');
await queryRunner.query('DROP TABLE "core"."emailGroupSuppressedRecipient"');
await queryRunner.query('DROP TYPE "core"."emailGroupSuppressedRecipient_createdbysource_enum"');
await queryRunner.query('DROP TYPE "core"."emailGroupSuppressedRecipient_status_enum"');
await queryRunner.query('DROP TYPE "core"."emailGroupSuppressedRecipient_reason_enum"');
await queryRunner.query('DROP TYPE "core"."emailGroupSuppressedRecipient_scope_enum"');
}
}
@@ -58,6 +58,7 @@ import { DropFieldMetadataIsUniqueColumnFastInstanceCommand } from 'src/database
import { EmailingDomainTenantStatusAndGlobalUniquenessFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-9/2-9-instance-command-fast-1799000020000-emailing-domain-tenant-status-and-global-uniqueness';
import { EncryptNonSecretApplicationVariableSlowInstanceCommand } from 'src/database/commands/upgrade-version-command/2-9/2-9-instance-command-slow-1798400000000-encrypt-non-secret-application-variable';
import { MigrateAiModelPreferencesSlowInstanceCommand } from 'src/database/commands/upgrade-version-command/2-9/2-9-instance-command-slow-1799000010000-migrate-ai-model-preferences';
import { AddEmailGroupSuppressedRecipientFastInstanceCommand } from 'src/database/commands/upgrade-version-command/2-9/2-9-instance-command-fast-1780000902115-addEmailGroupSuppressedRecipient';
export const INSTANCE_COMMANDS = [
AddViewFieldGroupIdIndexOnViewFieldFastInstanceCommand,
@@ -118,4 +119,5 @@ export const INSTANCE_COMMANDS = [
DropFieldMetadataIsUniqueColumnFastInstanceCommand,
MigrateAiModelPreferencesSlowInstanceCommand,
EncryptNonSecretApplicationVariableSlowInstanceCommand,
AddEmailGroupSuppressedRecipientFastInstanceCommand,
];
@@ -20,6 +20,7 @@ const USAGE_UNIT_BY_OPERATION_TYPE: Record<UsageOperationType, UsageUnit> = {
[UsageOperationType.WORKFLOW_EXECUTION]: UsageUnit.INVOCATION,
[UsageOperationType.CODE_EXECUTION]: UsageUnit.INVOCATION,
[UsageOperationType.WEB_SEARCH]: UsageUnit.INVOCATION,
[UsageOperationType.EMAIL_SEND]: UsageUnit.INVOCATION,
};
// `workspaceId` + `applicationId` come from the application-access token,
@@ -0,0 +1,5 @@
export const EMAIL_SUBJECT_MAX_LENGTH = 998;
export const EMAIL_BODY_MAX_LENGTH = 1_000_000;
export const EMAIL_MAX_RECIPIENTS_PER_FIELD = 50;
export const EMAIL_MAX_REPLY_TO = 10;
export const EMAIL_MAX_TOTAL_RECIPIENTS = 50;
@@ -0,0 +1,5 @@
export const EMAIL_SEND_AWS_COST_PER_RECIPIENT_USD = 0.0001;
export const EMAIL_SEND_CREDIT_MARKUP = 3;
export const EMAIL_SEND_THROTTLE_MAX_RECIPIENTS = 500;
export const EMAIL_SEND_THROTTLE_WINDOW_MS = 60_000;
@@ -1 +0,0 @@
export const AWS_SES_MARKETING_TOPIC_NAME = 'marketing';
@@ -0,0 +1 @@
export const AWS_SES_WORKSPACE_TAG_NAME = 'workspace';
@@ -1,10 +1,8 @@
import {
AlreadyExistsException,
CreateConfigurationSetCommand,
CreateConfigurationSetEventDestinationCommand,
CreateContactListCommand,
CreateTenantResourceAssociationCommand,
GetConfigurationSetCommand,
NotFoundException,
PutEmailIdentityMailFromAttributesCommand,
} from '@aws-sdk/client-sesv2';
@@ -23,13 +21,12 @@ describe('AwsSesRegisterDomainService', () => {
const provisionInput = {
tenantName: 'twenty-workspace-ws1',
configurationSetName: 'twenty-workspace-ws1',
contactListName: 'twenty-workspace-ws1',
};
const buildNotFound = () =>
new NotFoundException({
$metadata: { httpStatusCode: 404 },
message: 'Configuration set not found.',
const buildAlreadyExists = () =>
new AlreadyExistsException({
$metadata: { httpStatusCode: 409 },
message: 'Resource already exists.',
});
const setUp = () => {
@@ -43,33 +40,7 @@ describe('AwsSesRegisterDomainService', () => {
};
describe('provisionWorkspaceResources', () => {
it('creates every workspace-scoped resource when the configuration set does not yet exist', async () => {
const { service, send } = setUp();
send.mockImplementation(async (command) => {
if (command instanceof GetConfigurationSetCommand) {
throw buildNotFound();
}
return {};
});
await service.provisionWorkspaceResources(provisionInput, config);
const commandTypes = send.mock.calls.map(
([command]) => command.constructor.name,
);
expect(commandTypes).toEqual([
GetConfigurationSetCommand.name,
CreateConfigurationSetCommand.name,
CreateConfigurationSetEventDestinationCommand.name,
CreateContactListCommand.name,
CreateTenantResourceAssociationCommand.name,
]);
});
it('issues no creates when the configuration set already exists', async () => {
it('creates every workspace-scoped resource', async () => {
const { service, send } = setUp();
send.mockResolvedValue({});
@@ -80,20 +51,36 @@ describe('AwsSesRegisterDomainService', () => {
([command]) => command.constructor.name,
);
expect(commandTypes).toEqual([GetConfigurationSetCommand.name]);
expect(commandTypes).toEqual([
CreateConfigurationSetCommand.name,
CreateConfigurationSetEventDestinationCommand.name,
CreateTenantResourceAssociationCommand.name,
]);
});
it('propagates non-NotFound AWS errors raised by the existence probe', async () => {
it('ignores AlreadyExistsException per resource so a retry re-runs every step', async () => {
const { service, send } = setUp();
send.mockRejectedValue(buildAlreadyExists());
await service.provisionWorkspaceResources(provisionInput, config);
const commandTypes = send.mock.calls.map(
([command]) => command.constructor.name,
);
expect(commandTypes).toEqual([
CreateConfigurationSetCommand.name,
CreateConfigurationSetEventDestinationCommand.name,
CreateTenantResourceAssociationCommand.name,
]);
});
it('propagates AWS errors that are not AlreadyExistsException', async () => {
const { service, send } = setUp();
const fatalError = new Error('Boom');
send.mockImplementation(async (command) => {
if (command instanceof GetConfigurationSetCommand) {
throw fatalError;
}
return {};
});
send.mockRejectedValue(fatalError);
await expect(
service.provisionWorkspaceResources(provisionInput, config),
@@ -21,7 +21,6 @@ describe('AwsSesSendEmailService', () => {
const baseContext = {
tenantName: 'twenty-workspace-ws1',
configurationSetName: 'twenty-workspace-ws1',
contactListName: 'twenty-workspace-ws1',
};
const setUp = () => {
@@ -42,7 +41,7 @@ describe('AwsSesSendEmailService', () => {
return { service, send, handleErrorService };
};
it('should call SendEmail with tenant, config set, and list management options', async () => {
it('should call SendEmail with tenant and config set, without SES list management', async () => {
const { service, send } = setUp();
send.mockResolvedValue({ MessageId: 'msg-1' });
@@ -59,11 +58,8 @@ describe('AwsSesSendEmailService', () => {
Destination: { ToAddresses: ['user@example.com'] },
ConfigurationSetName: 'twenty-workspace-ws1',
TenantName: 'twenty-workspace-ws1',
ListManagementOptions: {
ContactListName: 'twenty-workspace-ws1',
TopicName: 'marketing',
},
});
expect(command.input.ListManagementOptions).toBeUndefined();
expect(command.input.EmailTags).toEqual(
expect.arrayContaining([
{ Name: 'workspace', Value: 'ws1' },
@@ -6,7 +6,6 @@ import {
CreateTenantCommand,
CreateTenantResourceAssociationCommand,
DeleteConfigurationSetCommand,
DeleteContactListCommand,
DeleteEmailIdentityCommand,
DeleteTenantCommand,
DeleteTenantResourceAssociationCommand,
@@ -120,7 +119,6 @@ export class AwsSesDriver implements EmailingDomainDriverInterface {
{
tenantName,
configurationSetName: this.buildConfigurationSetName(workspaceId),
contactListName: this.buildContactListName(workspaceId),
},
this.config,
);
@@ -136,7 +134,6 @@ export class AwsSesDriver implements EmailingDomainDriverInterface {
return this.awsSesSendEmailService.sendEmail(input, {
tenantName: this.buildTenantName(input.workspaceId),
configurationSetName: this.buildConfigurationSetName(input.workspaceId),
contactListName: this.buildContactListName(input.workspaceId),
});
}
@@ -167,7 +164,6 @@ export class AwsSesDriver implements EmailingDomainDriverInterface {
const sesClient = this.awsSesClientProvider.getSESClient();
const tenantName = this.buildTenantName(workspaceId);
const configurationSetName = this.buildConfigurationSetName(workspaceId);
const contactListName = this.buildContactListName(workspaceId);
const configurationSetArn = `arn:aws:ses:${this.config.region}:${this.config.accountId}:configuration-set/${configurationSetName}`;
await sesClient
@@ -191,12 +187,6 @@ export class AwsSesDriver implements EmailingDomainDriverInterface {
if (!(error instanceof NotFoundException)) throw error;
});
await sesClient
.send(new DeleteContactListCommand({ ContactListName: contactListName }))
.catch((error) => {
if (!(error instanceof NotFoundException)) throw error;
});
await sesClient
.send(new DeleteTenantCommand({ TenantName: tenantName }))
.catch((error) => {
@@ -212,10 +202,6 @@ export class AwsSesDriver implements EmailingDomainDriverInterface {
return `${AWS_SES_RESOURCE_NAME_PREFIX}-${workspaceId}`;
}
private buildContactListName(workspaceId: string): string {
return `${AWS_SES_RESOURCE_NAME_PREFIX}-${workspaceId}`;
}
private async ensureTenantExists(tenantName: string): Promise<void> {
const sesClient = this.awsSesClientProvider.getSESClient();
@@ -1,26 +1,21 @@
import { Injectable, Logger } from '@nestjs/common';
import {
AlreadyExistsException,
CreateConfigurationSetCommand,
CreateConfigurationSetEventDestinationCommand,
CreateContactListCommand,
CreateTenantResourceAssociationCommand,
GetConfigurationSetCommand,
NotFoundException,
PutEmailIdentityMailFromAttributesCommand,
type SESv2Client,
} from '@aws-sdk/client-sesv2';
import { type AwsSesDriverConfig } from 'src/engine/core-modules/emailing-domain/drivers/interfaces/driver-config.interface';
import { AWS_SES_EVENT_BUS_NAME } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/constants/aws-ses-event-bus-name.constant';
import { AWS_SES_MAIL_FROM_SUBDOMAIN } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/constants/aws-ses-mail-from-subdomain.constant';
import { AWS_SES_MARKETING_TOPIC_NAME } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/constants/aws-ses-marketing-topic-name.constant';
import { AwsSesClientProvider } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/providers/aws-ses-client.provider';
type ProvisionWorkspaceInput = {
tenantName: string;
configurationSetName: string;
contactListName: string;
};
@Injectable()
@@ -35,69 +30,64 @@ export class AwsSesRegisterDomainService {
): Promise<void> {
const sesClient = this.awsSesClientProvider.getSESClient();
const isAlreadyProvisioned = await this.isWorkspaceProvisioned(
sesClient,
input.configurationSetName,
);
if (isAlreadyProvisioned) {
return;
}
const eventBusArn = `arn:aws:events:${config.region}:${config.accountId}:event-bus/${AWS_SES_EVENT_BUS_NAME}`;
const configurationSetArn = `arn:aws:ses:${config.region}:${config.accountId}:configuration-set/${input.configurationSetName}`;
await sesClient.send(
new CreateConfigurationSetCommand({
ConfigurationSetName: input.configurationSetName,
ReputationOptions: { ReputationMetricsEnabled: true },
SendingOptions: { SendingEnabled: true },
SuppressionOptions: { SuppressedReasons: ['BOUNCE', 'COMPLAINT'] },
Tags: [{ Key: 'managed-by', Value: 'twenty' }],
}),
);
await sesClient
.send(
new CreateConfigurationSetCommand({
ConfigurationSetName: input.configurationSetName,
ReputationOptions: { ReputationMetricsEnabled: true },
SendingOptions: { SendingEnabled: true },
SuppressionOptions: { SuppressedReasons: [] },
Tags: [{ Key: 'managed-by', Value: 'twenty' }],
}),
)
.catch((error) => {
if (!(error instanceof AlreadyExistsException)) {
throw error;
}
});
await sesClient.send(
new CreateConfigurationSetEventDestinationCommand({
ConfigurationSetName: input.configurationSetName,
EventDestinationName: 'twenty-eventbridge',
EventDestination: {
Enabled: true,
MatchingEventTypes: [
'SEND',
'DELIVERY',
'BOUNCE',
'COMPLAINT',
'REJECT',
'RENDERING_FAILURE',
'DELIVERY_DELAY',
'SUBSCRIPTION',
],
EventBridgeDestination: { EventBusArn: eventBusArn },
},
}),
);
await sesClient.send(
new CreateContactListCommand({
ContactListName: input.contactListName,
Topics: [
{
TopicName: AWS_SES_MARKETING_TOPIC_NAME,
DisplayName: 'Marketing',
DefaultSubscriptionStatus: 'OPT_IN',
await sesClient
.send(
new CreateConfigurationSetEventDestinationCommand({
ConfigurationSetName: input.configurationSetName,
EventDestinationName: 'twenty-eventbridge',
EventDestination: {
Enabled: true,
MatchingEventTypes: [
'SEND',
'DELIVERY',
'BOUNCE',
'COMPLAINT',
'REJECT',
'RENDERING_FAILURE',
'DELIVERY_DELAY',
'SUBSCRIPTION',
],
EventBridgeDestination: { EventBusArn: eventBusArn },
},
],
Tags: [{ Key: 'managed-by', Value: 'twenty' }],
}),
);
}),
)
.catch((error) => {
if (!(error instanceof AlreadyExistsException)) {
throw error;
}
});
await sesClient.send(
new CreateTenantResourceAssociationCommand({
TenantName: input.tenantName,
ResourceArn: configurationSetArn,
}),
);
await sesClient
.send(
new CreateTenantResourceAssociationCommand({
TenantName: input.tenantName,
ResourceArn: configurationSetArn,
}),
)
.catch((error) => {
if (!(error instanceof AlreadyExistsException)) {
throw error;
}
});
this.logger.log(
`Provisioned workspace resources for tenant ${input.tenantName}`,
@@ -117,24 +107,4 @@ export class AwsSesRegisterDomainService {
this.logger.log(`Registered MAIL FROM for domain ${domain}`);
}
private async isWorkspaceProvisioned(
sesClient: SESv2Client,
configurationSetName: string,
): Promise<boolean> {
try {
await sesClient.send(
new GetConfigurationSetCommand({
ConfigurationSetName: configurationSetName,
}),
);
return true;
} catch (error) {
if (error instanceof NotFoundException) {
return false;
}
throw error;
}
}
}
@@ -8,7 +8,7 @@ import {
type EmailingDomainSendEmailResult,
} from 'src/engine/core-modules/emailing-domain/drivers/types/send-email';
import { AWS_SES_MARKETING_TOPIC_NAME } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/constants/aws-ses-marketing-topic-name.constant';
import { AWS_SES_WORKSPACE_TAG_NAME } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/constants/aws-ses-workspace-tag-name.constant';
import { AwsSesClientProvider } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/providers/aws-ses-client.provider';
import { AwsSesHandleErrorService } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/services/aws-ses-handle-error.service';
import {
@@ -19,7 +19,6 @@ import {
type SendEmailContext = {
tenantName: string;
configurationSetName: string;
contactListName: string;
};
@Injectable()
@@ -56,6 +55,12 @@ export class AwsSesSendEmailService {
ReplyToAddresses: input.replyTo,
Content: {
Simple: {
Headers: isNonEmptyArray(input.headers)
? input.headers.map((header) => ({
Name: header.name,
Value: header.value,
}))
: undefined,
Subject: { Data: input.subject, Charset: 'UTF-8' },
Body: {
Text: { Data: input.text, Charset: 'UTF-8' },
@@ -75,12 +80,8 @@ export class AwsSesSendEmailService {
},
ConfigurationSetName: context.configurationSetName,
TenantName: context.tenantName,
ListManagementOptions: {
ContactListName: context.contactListName,
TopicName: AWS_SES_MARKETING_TOPIC_NAME,
},
EmailTags: [
{ Name: 'workspace', Value: input.workspaceId },
{ Name: AWS_SES_WORKSPACE_TAG_NAME, Value: input.workspaceId },
{ Name: 'domain', Value: input.domain },
],
}),
@@ -11,6 +11,7 @@ export enum EmailingDomainDriverExceptionCode {
INSUFFICIENT_PERMISSIONS = 'INSUFFICIENT_PERMISSIONS',
CONFIGURATION_ERROR = 'CONFIGURATION_ERROR',
SENDING_SUSPENDED = 'SENDING_SUSPENDED',
ALL_RECIPIENTS_SUPPRESSED = 'ALL_RECIPIENTS_SUPPRESSED',
UNKNOWN = 'UNKNOWN',
}
@@ -26,6 +27,8 @@ const getEmailingDomainDriverExceptionUserFriendlyMessage = (
return msg`Email domain configuration error.`;
case EmailingDomainDriverExceptionCode.SENDING_SUSPENDED:
return msg`Sending is currently suspended for this email domain.`;
case EmailingDomainDriverExceptionCode.ALL_RECIPIENTS_SUPPRESSED:
return msg`All recipients are suppressed for this email domain.`;
case EmailingDomainDriverExceptionCode.TEMPORARY_ERROR:
case EmailingDomainDriverExceptionCode.UNKNOWN:
return STANDARD_ERROR_MESSAGE;
@@ -4,6 +4,11 @@ export type EmailingDomainAttachment = {
contentType: string;
};
export type EmailingDomainEmailHeader = {
name: string;
value: string;
};
export type EmailingDomainEmailContent = {
from: string;
to: string[];
@@ -14,6 +19,8 @@ export type EmailingDomainEmailContent = {
html?: string;
replyTo?: string[];
attachments?: EmailingDomainAttachment[];
headers?: EmailingDomainEmailHeader[];
includeUnsubscribe?: boolean;
};
export type EmailingDomainSendEmailInput = EmailingDomainEmailContent & {
@@ -1,14 +1,23 @@
import { Field, InputType } from '@nestjs/graphql';
import {
ArrayMaxSize,
ArrayMinSize,
IsArray,
IsEmail,
IsOptional,
IsString,
MaxLength,
MinLength,
} from 'class-validator';
import {
EMAIL_BODY_MAX_LENGTH,
EMAIL_MAX_RECIPIENTS_PER_FIELD,
EMAIL_MAX_REPLY_TO,
EMAIL_SUBJECT_MAX_LENGTH,
} from 'src/engine/core-modules/emailing-domain/constants/email-limits.constant';
@InputType()
export class SendEmailViaDomainInput {
@Field(() => String)
@@ -18,33 +27,39 @@ export class SendEmailViaDomainInput {
@Field(() => [String])
@IsArray()
@ArrayMinSize(1)
@ArrayMaxSize(EMAIL_MAX_RECIPIENTS_PER_FIELD)
@IsEmail({}, { each: true })
to: string[];
@Field(() => [String], { nullable: true })
@IsOptional()
@IsArray()
@ArrayMaxSize(EMAIL_MAX_RECIPIENTS_PER_FIELD)
@IsEmail({}, { each: true })
cc?: string[];
@Field(() => [String], { nullable: true })
@IsOptional()
@IsArray()
@ArrayMaxSize(EMAIL_MAX_RECIPIENTS_PER_FIELD)
@IsEmail({}, { each: true })
bcc?: string[];
@Field(() => String)
@IsString()
@MinLength(1)
@MaxLength(EMAIL_SUBJECT_MAX_LENGTH)
subject: string;
@Field(() => String)
@IsString()
@MaxLength(EMAIL_BODY_MAX_LENGTH)
text: string;
@Field(() => String, { nullable: true })
@IsOptional()
@IsString()
@MaxLength(EMAIL_BODY_MAX_LENGTH)
html?: string;
@Field(() => String)
@@ -54,6 +69,7 @@ export class SendEmailViaDomainInput {
@Field(() => [String], { nullable: true })
@IsOptional()
@IsArray()
@ArrayMaxSize(EMAIL_MAX_REPLY_TO)
@IsEmail({}, { each: true })
replyTo?: string[];
}
@@ -0,0 +1,67 @@
import {
Column,
CreateDateColumn,
Entity,
PrimaryGeneratedColumn,
Unique,
UpdateDateColumn,
} from 'typeorm';
import { FieldActorSource } from 'twenty-shared/types';
import { EmailGroupSuppressionReason } from 'src/engine/core-modules/emailing-domain/types/email-group-suppression-reason.type';
import { EmailGroupSuppressionScope } from 'src/engine/core-modules/emailing-domain/types/email-group-suppression-scope.type';
import { EmailGroupSuppressionStatus } from 'src/engine/core-modules/emailing-domain/types/email-group-suppression-status.type';
import { WorkspaceRelatedEntity } from 'src/engine/workspace-manager/types/workspace-related-entity';
@Entity({ name: 'emailGroupSuppressedRecipient', schema: 'core' })
@Unique('IDX_EMAIL_GROUP_SUPPRESSED_RECIPIENT_WORKSPACE_EMAIL_SCOPE_UNIQUE', [
'workspaceId',
'emailAddress',
'scope',
])
export class EmailGroupSuppressedRecipientEntity extends WorkspaceRelatedEntity {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column({ type: 'varchar', nullable: false })
emailAddress: string;
@Column({
type: 'enum',
enum: Object.values(EmailGroupSuppressionScope),
nullable: false,
})
scope: EmailGroupSuppressionScope;
@Column({
type: 'enum',
enum: Object.values(EmailGroupSuppressionReason),
nullable: false,
})
reason: EmailGroupSuppressionReason;
@Column({
type: 'enum',
enum: Object.values(EmailGroupSuppressionStatus),
default: EmailGroupSuppressionStatus.ACTIVE,
nullable: false,
})
status: EmailGroupSuppressionStatus;
@Column({ type: 'varchar', nullable: true })
providerEventId: string | null;
@Column({
type: 'enum',
enum: Object.values(FieldActorSource),
nullable: false,
})
createdBySource: FieldActorSource;
@CreateDateColumn({ type: 'timestamptz' })
createdAt: Date;
@UpdateDateColumn({ type: 'timestamptz' })
updatedAt: Date;
}
@@ -0,0 +1,63 @@
import {
Controller,
Get,
HttpCode,
Post,
Query,
UseGuards,
} from '@nestjs/common';
import { isNonEmptyString } from '@sniptt/guards';
import { FieldActorSource } from 'twenty-shared/types';
import { isDefined } from 'twenty-shared/utils';
import { EmailGroupSuppressionService } from 'src/engine/core-modules/emailing-domain/services/email-group-suppression.service';
import { EmailGroupUnsubscribeService } from 'src/engine/core-modules/emailing-domain/services/email-group-unsubscribe.service';
import { EmailGroupSuppressionReason } from 'src/engine/core-modules/emailing-domain/types/email-group-suppression-reason.type';
import { NoPermissionGuard } from 'src/engine/guards/no-permission.guard';
import { PublicEndpointGuard } from 'src/engine/guards/public-endpoint.guard';
@Controller('emailing')
export class EmailingDomainUnsubscribeController {
constructor(
private readonly emailGroupUnsubscribeService: EmailGroupUnsubscribeService,
private readonly emailGroupSuppressionService: EmailGroupSuppressionService,
) {}
@Post('unsubscribe')
@UseGuards(PublicEndpointGuard, NoPermissionGuard)
@HttpCode(200)
async unsubscribeOneClick(@Query('token') token?: string): Promise<void> {
await this.unsubscribe(token);
}
@Get('unsubscribe')
@UseGuards(PublicEndpointGuard, NoPermissionGuard)
@HttpCode(200)
async unsubscribeFromBrowser(
@Query('token') token?: string,
): Promise<string> {
await this.unsubscribe(token);
return 'You have been unsubscribed.';
}
private async unsubscribe(token?: string): Promise<void> {
if (!isNonEmptyString(token)) {
return;
}
const payload = this.emailGroupUnsubscribeService.verifyToken(token);
if (!isDefined(payload)) {
return;
}
await this.emailGroupSuppressionService.suppress(
payload.workspaceId,
payload.emailAddress,
EmailGroupSuppressionReason.UNSUBSCRIBE,
FieldActorSource.API,
);
}
}
@@ -3,30 +3,48 @@ import { Module } from '@nestjs/common';
import { NestjsQueryTypeOrmModule } from '@ptc-org/nestjs-query-typeorm';
import { TypeORMModule } from 'src/database/typeorm/typeorm.module';
import { BillingModule } from 'src/engine/core-modules/billing/billing.module';
import { AwsSesClientProvider } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/providers/aws-ses-client.provider';
import { AwsSesRegisterDomainService } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/services/aws-ses-register-domain.service';
import { AwsSesHandleErrorService } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/services/aws-ses-handle-error.service';
import { AwsSesSendEmailService } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/services/aws-ses-send-email.service';
import { EmailingDomainDriverFactory } from 'src/engine/core-modules/emailing-domain/drivers/emailing-domain-driver.factory';
import { EmailGroupSuppressedRecipientEntity } from 'src/engine/core-modules/emailing-domain/email-group-suppressed-recipient.entity';
import { EmailingDomainUnsubscribeController } from 'src/engine/core-modules/emailing-domain/emailing-domain-unsubscribe.controller';
import { EmailingDomainEntity } from 'src/engine/core-modules/emailing-domain/emailing-domain.entity';
import { EmailingDomainResolver } from 'src/engine/core-modules/emailing-domain/emailing-domain.resolver';
import { EmailingDomainWorkspaceCleanupJob } from 'src/engine/core-modules/emailing-domain/jobs/emailing-domain-workspace-cleanup.job';
import { EmailGroupSuppressionService } from 'src/engine/core-modules/emailing-domain/services/email-group-suppression.service';
import { EmailGroupUnsubscribeService } from 'src/engine/core-modules/emailing-domain/services/email-group-unsubscribe.service';
import { EmailingDomainTenantStatusService } from 'src/engine/core-modules/emailing-domain/services/emailing-domain-tenant-status.service';
import { EmailingDomainService } from 'src/engine/core-modules/emailing-domain/services/emailing-domain.service';
import { FeatureFlagModule } from 'src/engine/core-modules/feature-flag/feature-flag.module';
import { PermissionsModule } from 'src/engine/metadata-modules/permissions/permissions.module';
import { ThrottlerModule } from 'src/engine/core-modules/throttler/throttler.module';
import { provideWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/provide-workspace-scoped-repository';
@Module({
imports: [
TypeORMModule,
NestjsQueryTypeOrmModule.forFeature([EmailingDomainEntity]),
NestjsQueryTypeOrmModule.forFeature([
EmailingDomainEntity,
EmailGroupSuppressedRecipientEntity,
]),
FeatureFlagModule,
PermissionsModule,
ThrottlerModule,
BillingModule,
],
controllers: [EmailingDomainUnsubscribeController],
exports: [
EmailingDomainService,
EmailingDomainTenantStatusService,
EmailGroupSuppressionService,
],
exports: [EmailingDomainService, EmailingDomainTenantStatusService],
providers: [
EmailingDomainService,
EmailingDomainTenantStatusService,
EmailGroupSuppressionService,
EmailGroupUnsubscribeService,
EmailingDomainResolver,
EmailingDomainDriverFactory,
EmailingDomainWorkspaceCleanupJob,
@@ -35,6 +53,7 @@ import { provideWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspac
AwsSesRegisterDomainService,
AwsSesSendEmailService,
provideWorkspaceScopedRepository(EmailingDomainEntity),
provideWorkspaceScopedRepository(EmailGroupSuppressedRecipientEntity),
],
})
export class EmailingDomainModule {}
@@ -5,6 +5,7 @@ import { MessageQueue } from 'src/engine/core-modules/message-queue/message-queu
export type EmailingDomainWorkspaceCleanupJobData = {
workspaceId: string;
domains: string[];
};
@Processor(MessageQueue.deleteCascadeQueue)
@@ -13,11 +14,12 @@ export class EmailingDomainWorkspaceCleanupJob {
@Process(EmailingDomainWorkspaceCleanupJob.name)
async handle(data: EmailingDomainWorkspaceCleanupJobData): Promise<void> {
const { workspaceId } = data;
const { workspaceId, domains } = data;
try {
await this.emailingDomainService.cleanupAllEmailingDomainsForWorkspace(
await this.emailingDomainService.cleanupEmailingDomainsForWorkspace(
workspaceId,
domains,
);
} catch (error) {
throw new Error(
@@ -1,9 +1,18 @@
import { EmailingDomainDriverExceptionCode } from 'src/engine/core-modules/emailing-domain/drivers/exceptions/emailing-domain-driver.exception';
import { type EmailingDomainDriverFactory } from 'src/engine/core-modules/emailing-domain/drivers/emailing-domain-driver.factory';
import { EmailingDomainDriver } from 'src/engine/core-modules/emailing-domain/drivers/types/emailing-domain-driver.type';
import { type WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { EmailingDomainStatus } from 'src/engine/core-modules/emailing-domain/drivers/types/emailing-domain-status.type';
import { EmailingDomainTenantStatus } from 'src/engine/core-modules/emailing-domain/drivers/types/emailing-domain-tenant-status.type';
import { type EmailingDomainEmailContent } from 'src/engine/core-modules/emailing-domain/drivers/types/send-email';
import { type EmailingDomainEntity } from 'src/engine/core-modules/emailing-domain/emailing-domain.entity';
import { type EmailGroupSuppressionService } from 'src/engine/core-modules/emailing-domain/services/email-group-suppression.service';
import { type EmailGroupUnsubscribeService } from 'src/engine/core-modules/emailing-domain/services/email-group-unsubscribe.service';
import { EmailingDomainService } from 'src/engine/core-modules/emailing-domain/services/emailing-domain.service';
import { type BillingUsageService } from 'src/engine/core-modules/billing/services/billing-usage.service';
import { type ThrottlerService } from 'src/engine/core-modules/throttler/throttler.service';
import { UsageOperationType } from 'src/engine/core-modules/usage/enums/usage-operation-type.enum';
import { type WorkspaceEventEmitter } from 'src/engine/workspace-event-emitter/workspace-event-emitter';
import { type WorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/workspace-scoped-repository';
describe('EmailingDomainService.sendEmail', () => {
@@ -19,14 +28,20 @@ describe('EmailingDomainService.sendEmail', () => {
...overrides,
}) as EmailingDomainEntity;
const buildEmailContent = () => ({
const buildEmailContent = (
overrides: Partial<EmailingDomainEmailContent> = {},
): EmailingDomainEmailContent => ({
from: 'hello@mail.example.com',
to: ['user@example.com'],
subject: 'Hi',
text: 'Body',
...overrides,
});
const setUp = (emailingDomain: EmailingDomainEntity) => {
const setUp = (
emailingDomain: EmailingDomainEntity,
suppressedAddresses: string[] = [],
) => {
const sendEmail = jest.fn().mockResolvedValue({ messageId: 'msg-1' });
const repository = {
findOne: jest.fn().mockResolvedValue(emailingDomain),
@@ -34,9 +49,47 @@ describe('EmailingDomainService.sendEmail', () => {
const factory = {
getCurrentDriver: () => ({ sendEmail }),
} as unknown as EmailingDomainDriverFactory;
const service = new EmailingDomainService(repository, factory);
const suppressionService = {
getSuppressedAddresses: jest
.fn()
.mockResolvedValue(
new Set(suppressedAddresses.map((address) => address.toLowerCase())),
),
} as unknown as EmailGroupSuppressionService;
const unsubscribeService = {
buildUnsubscribeHeaders: jest
.fn()
.mockReturnValue([
{ name: 'List-Unsubscribe', value: '<https://app/unsubscribe>' },
]),
} as unknown as EmailGroupUnsubscribeService;
const throttlerService = {
tokenBucketThrottleOrThrow: jest.fn().mockResolvedValue(1),
} as unknown as ThrottlerService;
const billingUsageService = {
canFeatureBeUsed: jest.fn().mockResolvedValue(true),
} as unknown as BillingUsageService;
const workspaceEventEmitter = {
emitCustomBatchEvent: jest.fn(),
} as unknown as WorkspaceEventEmitter;
const service = new EmailingDomainService(
repository,
factory,
suppressionService,
unsubscribeService,
throttlerService,
billingUsageService,
workspaceEventEmitter,
);
return { service, sendEmail };
return {
service,
sendEmail,
unsubscribeService,
throttlerService,
billingUsageService,
workspaceEventEmitter,
};
};
it('delegates to the driver when the domain is verified and the tenant is active', async () => {
@@ -77,6 +130,157 @@ describe('EmailingDomainService.sendEmail', () => {
},
);
it('removes suppressed recipients but still sends to deliverable ones', async () => {
const { service, sendEmail } = setUp(buildEmailingDomain(), [
'blocked@example.com',
]);
await service.sendEmail(
'ws1',
'domain-1',
buildEmailContent({
to: ['user@example.com', 'Blocked@example.com'],
cc: ['blocked@example.com'],
bcc: ['keep@example.com'],
}),
);
expect(sendEmail).toHaveBeenCalledWith(
expect.objectContaining({
to: ['user@example.com'],
cc: [],
bcc: ['keep@example.com'],
}),
);
});
it('rejects with ALL_RECIPIENTS_SUPPRESSED when every primary recipient is suppressed, without calling the driver', async () => {
const { service, sendEmail } = setUp(buildEmailingDomain(), [
'user@example.com',
]);
await expect(
service.sendEmail('ws1', 'domain-1', buildEmailContent()),
).rejects.toMatchObject({
code: EmailingDomainDriverExceptionCode.ALL_RECIPIENTS_SUPPRESSED,
});
expect(sendEmail).not.toHaveBeenCalled();
});
it('attaches one-click unsubscribe headers for a single-recipient marketing send', async () => {
const { service, sendEmail } = setUp(buildEmailingDomain());
await service.sendEmail(
'ws1',
'domain-1',
buildEmailContent({ includeUnsubscribe: true }),
);
expect(sendEmail).toHaveBeenCalledWith(
expect.objectContaining({
headers: [
{ name: 'List-Unsubscribe', value: '<https://app/unsubscribe>' },
],
}),
);
});
it('omits unsubscribe headers when the message has multiple recipients', async () => {
const { service, sendEmail, unsubscribeService } = setUp(
buildEmailingDomain(),
);
await service.sendEmail(
'ws1',
'domain-1',
buildEmailContent({
includeUnsubscribe: true,
cc: ['second@example.com'],
}),
);
expect(unsubscribeService.buildUnsubscribeHeaders).not.toHaveBeenCalled();
expect(sendEmail).toHaveBeenCalledWith(
expect.objectContaining({ headers: undefined }),
);
});
it('blocks provisioning a new emailing domain when the billing plan disallows the feature', async () => {
const { service, billingUsageService } = setUp(buildEmailingDomain());
(billingUsageService.canFeatureBeUsed as jest.Mock).mockResolvedValue(
false,
);
await expect(
service.createEmailingDomain(
'mail.example.com',
EmailingDomainDriver.AWS_SES,
{
id: 'ws1',
} as WorkspaceEntity,
),
).rejects.toMatchObject({
code: EmailingDomainDriverExceptionCode.SENDING_SUSPENDED,
});
});
it('meters one usage event per accepted recipient at the marked-up per-recipient cost', async () => {
const { service, workspaceEventEmitter } = setUp(buildEmailingDomain());
await service.sendEmail(
'ws1',
'domain-1',
buildEmailContent({ to: ['a@example.com', 'b@example.com'] }),
);
expect(workspaceEventEmitter.emitCustomBatchEvent).toHaveBeenCalledWith(
'USAGE_RECORDED',
[
expect.objectContaining({
operationType: UsageOperationType.EMAIL_SEND,
quantity: 2,
creditsUsedMicro: 600,
}),
],
'ws1',
);
});
it('rejects sending when the billing plan disallows it, without calling the driver or metering', async () => {
const { service, sendEmail, billingUsageService, workspaceEventEmitter } =
setUp(buildEmailingDomain());
(billingUsageService.canFeatureBeUsed as jest.Mock).mockResolvedValue(
false,
);
await expect(
service.sendEmail('ws1', 'domain-1', buildEmailContent()),
).rejects.toMatchObject({
code: EmailingDomainDriverExceptionCode.SENDING_SUSPENDED,
});
expect(sendEmail).not.toHaveBeenCalled();
expect(workspaceEventEmitter.emitCustomBatchEvent).not.toHaveBeenCalled();
});
it('consumes one rate-limit token per deliverable recipient before sending', async () => {
const { service, throttlerService } = setUp(buildEmailingDomain());
await service.sendEmail(
'ws1',
'domain-1',
buildEmailContent({ to: ['a@example.com'], cc: ['c@example.com'] }),
);
expect(throttlerService.tokenBucketThrottleOrThrow).toHaveBeenCalledWith(
'emailing-domain-send:ws1',
2,
expect.any(Number),
expect.any(Number),
);
});
// Verification is a precondition for the tenant-status check: a domain that
// has not been verified should surface a CONFIGURATION_ERROR rather than
// leaking the tenant pause state to callers who couldn't have used it anyway.
@@ -0,0 +1,96 @@
import { Injectable } from '@nestjs/common';
import { isNonEmptyArray } from '@sniptt/guards';
import { FieldActorSource } from 'twenty-shared/types';
import { In } from 'typeorm';
import { EmailGroupSuppressedRecipientEntity } from 'src/engine/core-modules/emailing-domain/email-group-suppressed-recipient.entity';
import { EmailGroupSuppressionReason } from 'src/engine/core-modules/emailing-domain/types/email-group-suppression-reason.type';
import { EmailGroupSuppressionScope } from 'src/engine/core-modules/emailing-domain/types/email-group-suppression-scope.type';
import { EmailGroupSuppressionStatus } from 'src/engine/core-modules/emailing-domain/types/email-group-suppression-status.type';
import { InjectWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/inject-workspace-scoped-repository.decorator';
import { WorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/workspace-scoped-repository';
const REVERSIBLE_REASONS: ReadonlySet<EmailGroupSuppressionReason> = new Set([
EmailGroupSuppressionReason.UNSUBSCRIBE,
]);
@Injectable()
export class EmailGroupSuppressionService {
constructor(
@InjectWorkspaceScopedRepository(EmailGroupSuppressedRecipientEntity)
private readonly suppressedRecipientRepository: WorkspaceScopedRepository<EmailGroupSuppressedRecipientEntity>,
) {}
async getSuppressedAddresses(
workspaceId: string,
emailAddresses: string[],
): Promise<Set<string>> {
const normalizedAddresses = [
...new Set(emailAddresses.map(normalizeEmailAddress)),
];
if (!isNonEmptyArray(normalizedAddresses)) {
return new Set();
}
const activeSuppressions = await this.suppressedRecipientRepository.find(
workspaceId,
{
where: {
emailAddress: In(normalizedAddresses),
status: EmailGroupSuppressionStatus.ACTIVE,
},
},
);
return new Set(
activeSuppressions.map((suppression) => suppression.emailAddress),
);
}
async suppress(
workspaceId: string,
emailAddress: string,
reason: EmailGroupSuppressionReason,
createdBySource: FieldActorSource,
providerEventId: string | null = null,
): Promise<void> {
await this.suppressedRecipientRepository.upsert(
workspaceId,
{
emailAddress: normalizeEmailAddress(emailAddress),
scope: scopeForReason(reason),
reason,
status: EmailGroupSuppressionStatus.ACTIVE,
createdBySource,
providerEventId,
},
['workspaceId', 'emailAddress', 'scope'],
);
}
async releaseMarketingSuppression(
workspaceId: string,
emailAddress: string,
): Promise<void> {
await this.suppressedRecipientRepository.update(
workspaceId,
{
emailAddress: normalizeEmailAddress(emailAddress),
scope: EmailGroupSuppressionScope.MARKETING,
},
{ status: EmailGroupSuppressionStatus.RELEASED },
);
}
}
const scopeForReason = (
reason: EmailGroupSuppressionReason,
): EmailGroupSuppressionScope =>
REVERSIBLE_REASONS.has(reason)
? EmailGroupSuppressionScope.MARKETING
: EmailGroupSuppressionScope.GLOBAL;
const normalizeEmailAddress = (emailAddress: string): string =>
emailAddress.trim().toLowerCase();
@@ -0,0 +1,98 @@
import { Injectable } from '@nestjs/common';
import { createHmac, timingSafeEqual } from 'crypto';
import { isDefined } from 'twenty-shared/utils';
import { TwentyConfigService } from 'src/engine/core-modules/twenty-config/twenty-config.service';
type UnsubscribeTokenPayload = {
workspaceId: string;
emailAddress: string;
};
type EmailHeader = {
name: string;
value: string;
};
@Injectable()
export class EmailGroupUnsubscribeService {
constructor(private readonly twentyConfigService: TwentyConfigService) {}
buildUnsubscribeHeaders(
workspaceId: string,
emailAddress: string,
): EmailHeader[] {
const token = this.signToken({ workspaceId, emailAddress });
const serverUrl = this.twentyConfigService.get('SERVER_URL');
const unsubscribeUrl = `${serverUrl}/emailing/unsubscribe?token=${encodeURIComponent(token)}`;
return [
{ name: 'List-Unsubscribe', value: `<${unsubscribeUrl}>` },
{ name: 'List-Unsubscribe-Post', value: 'List-Unsubscribe=One-Click' },
];
}
verifyToken(token: string): UnsubscribeTokenPayload | null {
const [encodedPayload, signature] = token.split('.');
if (!isDefined(encodedPayload) || !isDefined(signature)) {
return null;
}
const expectedSignature = this.computeSignature(encodedPayload);
if (!isSignatureEqual(signature, expectedSignature)) {
return null;
}
const payload = parsePayload(encodedPayload);
if (!isValidPayload(payload)) {
return null;
}
return payload;
}
private signToken(payload: UnsubscribeTokenPayload): string {
const encodedPayload = base64UrlEncode(JSON.stringify(payload));
return `${encodedPayload}.${this.computeSignature(encodedPayload)}`;
}
private computeSignature(encodedPayload: string): string {
return createHmac('sha256', this.twentyConfigService.get('APP_SECRET'))
.update(encodedPayload)
.digest('base64url');
}
}
const base64UrlEncode = (value: string): string =>
Buffer.from(value, 'utf8').toString('base64url');
const parsePayload = (encodedPayload: string): unknown => {
try {
return JSON.parse(
Buffer.from(encodedPayload, 'base64url').toString('utf8'),
);
} catch {
return null;
}
};
const isValidPayload = (payload: unknown): payload is UnsubscribeTokenPayload =>
isDefined(payload) &&
typeof (payload as UnsubscribeTokenPayload).workspaceId === 'string' &&
typeof (payload as UnsubscribeTokenPayload).emailAddress === 'string';
const isSignatureEqual = (candidate: string, expected: string): boolean => {
const candidateBuffer = Buffer.from(candidate);
const expectedBuffer = Buffer.from(expected);
return (
candidateBuffer.length === expectedBuffer.length &&
timingSafeEqual(candidateBuffer, expectedBuffer)
);
};
@@ -1,5 +1,15 @@
import { Injectable, Logger } from '@nestjs/common';
import { isNonEmptyArray } from '@sniptt/guards';
import { BillingUsageService } from 'src/engine/core-modules/billing/services/billing-usage.service';
import {
EMAIL_SEND_AWS_COST_PER_RECIPIENT_USD,
EMAIL_SEND_CREDIT_MARKUP,
EMAIL_SEND_THROTTLE_MAX_RECIPIENTS,
EMAIL_SEND_THROTTLE_WINDOW_MS,
} from 'src/engine/core-modules/emailing-domain/constants/email-send-billing.constant';
import { EMAIL_MAX_TOTAL_RECIPIENTS } from 'src/engine/core-modules/emailing-domain/constants/email-limits.constant';
import {
EmailingDomainDriverException,
EmailingDomainDriverExceptionCode,
@@ -13,9 +23,19 @@ import {
type EmailingDomainSendEmailResult,
} from 'src/engine/core-modules/emailing-domain/drivers/types/send-email';
import { EmailingDomainEntity } from 'src/engine/core-modules/emailing-domain/emailing-domain.entity';
import { EmailGroupSuppressionService } from 'src/engine/core-modules/emailing-domain/services/email-group-suppression.service';
import { EmailGroupUnsubscribeService } from 'src/engine/core-modules/emailing-domain/services/email-group-unsubscribe.service';
import { ThrottlerService } from 'src/engine/core-modules/throttler/throttler.service';
import { USAGE_RECORDED } from 'src/engine/core-modules/usage/constants/usage-recorded.constant';
import { UsageOperationType } from 'src/engine/core-modules/usage/enums/usage-operation-type.enum';
import { UsageResourceType } from 'src/engine/core-modules/usage/enums/usage-resource-type.enum';
import { UsageUnit } from 'src/engine/core-modules/usage/enums/usage-unit.enum';
import { type UsageEvent } from 'src/engine/core-modules/usage/types/usage-event.type';
import { WorkspaceEntity } from 'src/engine/core-modules/workspace/workspace.entity';
import { DOLLAR_TO_CREDIT_MULTIPLIER } from 'src/engine/metadata-modules/ai/ai-billing/constants/dollar-to-credit-multiplier';
import { InjectWorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/inject-workspace-scoped-repository.decorator';
import { WorkspaceScopedRepository } from 'src/engine/twenty-orm/workspace-scoped-repository/workspace-scoped-repository';
import { WorkspaceEventEmitter } from 'src/engine/workspace-event-emitter/workspace-event-emitter';
@Injectable()
export class EmailingDomainService {
private readonly logger = new Logger(EmailingDomainService.name);
@@ -24,6 +44,11 @@ export class EmailingDomainService {
@InjectWorkspaceScopedRepository(EmailingDomainEntity)
private readonly emailingDomainRepository: WorkspaceScopedRepository<EmailingDomainEntity>,
private readonly emailingDomainDriverFactory: EmailingDomainDriverFactory,
private readonly emailGroupSuppressionService: EmailGroupSuppressionService,
private readonly emailGroupUnsubscribeService: EmailGroupUnsubscribeService,
private readonly throttlerService: ThrottlerService,
private readonly billingUsageService: BillingUsageService,
private readonly workspaceEventEmitter: WorkspaceEventEmitter,
) {}
async createEmailingDomain(
@@ -31,6 +56,8 @@ export class EmailingDomainService {
driverType: EmailingDomainDriver,
workspace: WorkspaceEntity,
): Promise<EmailingDomainEntity> {
await this.assertBillingAllowsEmailGroup(workspace.id);
const existingEmailingDomain = await this.emailingDomainRepository.findOne(
workspace.id,
{
@@ -87,18 +114,18 @@ export class EmailingDomainService {
});
}
async cleanupAllEmailingDomainsForWorkspace(
async cleanupEmailingDomainsForWorkspace(
workspaceId: string,
domains: string[],
): Promise<void> {
const emailingDomains =
await this.emailingDomainRepository.find(workspaceId);
const emailingDomainDriver =
this.emailingDomainDriverFactory.getCurrentDriver();
for (const emailingDomain of emailingDomains) {
await this.deleteRemoteEmailingDomain(emailingDomain);
for (const domain of domains) {
await emailingDomainDriver.cleanupDomain({ domain, workspaceId });
}
await this.deprovisionRemoteWorkspace(workspaceId);
await this.emailingDomainRepository.delete(workspaceId, {});
await emailingDomainDriver.deprovisionWorkspace(workspaceId);
}
async getEmailingDomains(
@@ -178,11 +205,143 @@ export class EmailingDomainService {
);
}
return this.emailingDomainDriverFactory.getCurrentDriver().sendEmail({
...emailContent,
const suppressedAddresses =
await this.emailGroupSuppressionService.getSuppressedAddresses(
workspaceId,
[
...emailContent.to,
...(emailContent.cc ?? []),
...(emailContent.bcc ?? []),
],
);
const isNotSuppressed = (address: string): boolean =>
!suppressedAddresses.has(address.trim().toLowerCase());
const deliverableTo = emailContent.to.filter(isNotSuppressed);
if (deliverableTo.length === 0) {
throw new EmailingDomainDriverException(
`All primary recipients are suppressed for emailing domain ${emailingDomain.domain}`,
EmailingDomainDriverExceptionCode.ALL_RECIPIENTS_SUPPRESSED,
);
}
const deliverableCc = emailContent.cc?.filter(isNotSuppressed);
const deliverableBcc = emailContent.bcc?.filter(isNotSuppressed);
const recipientCount =
deliverableTo.length +
(deliverableCc?.length ?? 0) +
(deliverableBcc?.length ?? 0);
if (recipientCount > EMAIL_MAX_TOTAL_RECIPIENTS) {
throw new EmailingDomainDriverException(
`A single email cannot exceed ${EMAIL_MAX_TOTAL_RECIPIENTS} recipients`,
EmailingDomainDriverExceptionCode.CONFIGURATION_ERROR,
);
}
await this.assertBillingAllowsEmailGroup(workspaceId);
await this.assertWithinSendRate(workspaceId, recipientCount);
const headers = this.buildSingleRecipientUnsubscribeHeaders(
workspaceId,
domain: emailingDomain.domain,
});
emailContent,
deliverableTo,
deliverableCc,
deliverableBcc,
);
const result = await this.emailingDomainDriverFactory
.getCurrentDriver()
.sendEmail({
...emailContent,
to: deliverableTo,
cc: deliverableCc,
bcc: deliverableBcc,
headers,
workspaceId,
domain: emailingDomain.domain,
});
this.meterEmailSend(workspaceId, recipientCount);
return result;
}
private async assertBillingAllowsEmailGroup(
workspaceId: string,
): Promise<void> {
const isEntitled =
await this.billingUsageService.canFeatureBeUsed(workspaceId);
if (!isEntitled) {
throw new EmailingDomainDriverException(
'The email domain feature requires an active billing plan',
EmailingDomainDriverExceptionCode.SENDING_SUSPENDED,
);
}
}
private async assertWithinSendRate(
workspaceId: string,
recipientCount: number,
): Promise<void> {
await this.throttlerService.tokenBucketThrottleOrThrow(
`emailing-domain-send:${workspaceId}`,
recipientCount,
EMAIL_SEND_THROTTLE_MAX_RECIPIENTS,
EMAIL_SEND_THROTTLE_WINDOW_MS,
);
}
private meterEmailSend(workspaceId: string, recipientCount: number): void {
const creditsUsedMicro = Math.round(
EMAIL_SEND_AWS_COST_PER_RECIPIENT_USD *
EMAIL_SEND_CREDIT_MARKUP *
DOLLAR_TO_CREDIT_MULTIPLIER *
recipientCount,
);
this.workspaceEventEmitter.emitCustomBatchEvent<UsageEvent>(
USAGE_RECORDED,
[
{
resourceType: UsageResourceType.API,
operationType: UsageOperationType.EMAIL_SEND,
creditsUsedMicro,
quantity: recipientCount,
unit: UsageUnit.INVOCATION,
},
],
workspaceId,
);
}
private buildSingleRecipientUnsubscribeHeaders(
workspaceId: string,
emailContent: EmailingDomainEmailContent,
deliverableTo: string[],
deliverableCc: string[] | undefined,
deliverableBcc: string[] | undefined,
): EmailingDomainEmailContent['headers'] {
const isSingleRecipient =
deliverableTo.length === 1 &&
!isNonEmptyArray(deliverableCc) &&
!isNonEmptyArray(deliverableBcc);
if (emailContent.includeUnsubscribe !== true || !isSingleRecipient) {
return emailContent.headers;
}
return [
...(emailContent.headers ?? []),
...this.emailGroupUnsubscribeService.buildUnsubscribeHeaders(
workspaceId,
deliverableTo[0],
),
];
}
private async findEmailingDomainByIdOrThrow(
@@ -220,16 +379,4 @@ export class EmailingDomainService {
);
}
}
private async deprovisionRemoteWorkspace(workspaceId: string): Promise<void> {
try {
await this.emailingDomainDriverFactory
.getCurrentDriver()
.deprovisionWorkspace(workspaceId);
} catch (error) {
this.logger.warn(
`Remote deprovision for emailing domain workspace ${workspaceId} failed: ${error}`,
);
}
}
}
@@ -0,0 +1,5 @@
export enum EmailGroupSuppressionReason {
HARD_BOUNCE = 'HARD_BOUNCE',
COMPLAINT = 'COMPLAINT',
UNSUBSCRIBE = 'UNSUBSCRIBE',
}
@@ -0,0 +1,4 @@
export enum EmailGroupSuppressionScope {
GLOBAL = 'GLOBAL',
MARKETING = 'MARKETING',
}
@@ -0,0 +1,4 @@
export enum EmailGroupSuppressionStatus {
ACTIVE = 'ACTIVE',
RELEASED = 'RELEASED',
}
@@ -6,6 +6,7 @@ import { SesInboundMailHandlerService } from 'src/engine/core-modules/messaging-
import { SesInboundWebhookRouterService } from 'src/engine/core-modules/messaging-webhooks/services/ses-inbound-webhook-router.service';
import { SesOutboundSendingStateHandlerService } from 'src/engine/core-modules/messaging-webhooks/services/ses-outbound-sending-state-handler.service';
import { SesOutboundWebhookRouterService } from 'src/engine/core-modules/messaging-webhooks/services/ses-outbound-webhook-router.service';
import { SesSuppressionEventHandlerService } from 'src/engine/core-modules/messaging-webhooks/services/ses-suppression-event-handler.service';
import { SnsSignatureVerifierService } from 'src/engine/core-modules/messaging-webhooks/services/sns-signature-verifier.service';
import { SnsSubscriptionConfirmerService } from 'src/engine/core-modules/messaging-webhooks/services/sns-subscription-confirmer.service';
import { TwentyConfigModule } from 'src/engine/core-modules/twenty-config/twenty-config.module';
@@ -18,6 +19,7 @@ import { TwentyConfigModule } from 'src/engine/core-modules/twenty-config/twenty
SnsSubscriptionConfirmerService,
SesInboundMailHandlerService,
SesOutboundSendingStateHandlerService,
SesSuppressionEventHandlerService,
SesInboundWebhookRouterService,
SesOutboundWebhookRouterService,
],
@@ -38,6 +38,7 @@ export class SesInboundMailHandlerService {
s3Key: receipt.action.objectKey,
envelopeRecipients: receipt.recipients,
},
{ id: snsMessageId },
);
}
}
@@ -6,6 +6,7 @@ import { isDefined, parseJson } from 'twenty-shared/utils';
import { MessagingWebhookExceptionCode } from 'src/engine/core-modules/messaging-webhooks/messaging-webhook-exception-code.enum';
import { MessagingWebhookException } from 'src/engine/core-modules/messaging-webhooks/messaging-webhook.exception';
import { SesOutboundSendingStateHandlerService } from 'src/engine/core-modules/messaging-webhooks/services/ses-outbound-sending-state-handler.service';
import { SesSuppressionEventHandlerService } from 'src/engine/core-modules/messaging-webhooks/services/ses-suppression-event-handler.service';
import { SnsSignatureVerifierService } from 'src/engine/core-modules/messaging-webhooks/services/sns-signature-verifier.service';
import { SnsSubscriptionConfirmerService } from 'src/engine/core-modules/messaging-webhooks/services/sns-subscription-confirmer.service';
import { type SesEventBridgeNotification } from 'src/engine/core-modules/messaging-webhooks/types/ses-event-bridge-notification.type';
@@ -18,6 +19,7 @@ export class SesOutboundWebhookRouterService {
private readonly snsSignatureVerifierService: SnsSignatureVerifierService,
private readonly snsSubscriptionConfirmerService: SnsSubscriptionConfirmerService,
private readonly sesOutboundSendingStateHandlerService: SesOutboundSendingStateHandlerService,
private readonly sesSuppressionEventHandlerService: SesSuppressionEventHandlerService,
) {}
async route(rawBody: Buffer): Promise<void> {
@@ -54,6 +56,20 @@ export class SesOutboundWebhookRouterService {
);
}
await this.sesOutboundSendingStateHandlerService.handle(event);
if (
event['detail-type'] === 'Sending Status Enabled' ||
event['detail-type'] === 'Sending Status Disabled'
) {
await this.sesOutboundSendingStateHandlerService.handle(event);
return;
}
if (
event['detail-type'] === 'Email Bounced' ||
event['detail-type'] === 'Email Complaint Received'
) {
await this.sesSuppressionEventHandlerService.handle(event);
}
}
}
@@ -0,0 +1,109 @@
import { Injectable, Logger } from '@nestjs/common';
import { FieldActorSource } from 'twenty-shared/types';
import { isDefined, isNonEmptyArray } from 'twenty-shared/utils';
import { AWS_SES_WORKSPACE_TAG_NAME } from 'src/engine/core-modules/emailing-domain/drivers/aws-ses/constants/aws-ses-workspace-tag-name.constant';
import { EmailGroupSuppressionService } from 'src/engine/core-modules/emailing-domain/services/email-group-suppression.service';
import { EmailGroupSuppressionReason } from 'src/engine/core-modules/emailing-domain/types/email-group-suppression-reason.type';
import { type SesEventBridgeNotification } from 'src/engine/core-modules/messaging-webhooks/types/ses-event-bridge-notification.type';
import { parseWorkspaceIdFromAwsSesResourceArn } from 'src/engine/core-modules/messaging-webhooks/utils/parse-workspace-id-from-aws-ses-resource-arn.util';
@Injectable()
export class SesSuppressionEventHandlerService {
private readonly logger = new Logger(SesSuppressionEventHandlerService.name);
constructor(
private readonly emailGroupSuppressionService: EmailGroupSuppressionService,
) {}
async handle(event: SesEventBridgeNotification): Promise<void> {
const suppression = this.resolveSuppression(event);
if (!isDefined(suppression)) {
return;
}
const workspaceId = this.resolveWorkspaceId(event);
if (!isDefined(workspaceId)) {
this.logger.warn(
`Could not resolve workspaceId for SES ${event['detail-type']} event`,
);
return;
}
for (const emailAddress of suppression.emailAddresses) {
await this.emailGroupSuppressionService.suppress(
workspaceId,
emailAddress,
suppression.reason,
FieldActorSource.WEBHOOK,
suppression.feedbackId,
);
}
}
private resolveSuppression(event: SesEventBridgeNotification): {
reason: EmailGroupSuppressionReason;
emailAddresses: string[];
feedbackId: string | null;
} | null {
if (event['detail-type'] === 'Email Bounced') {
const bounce = event.detail?.bounce;
if (bounce?.bounceType !== 'Permanent') {
return null;
}
return {
reason: EmailGroupSuppressionReason.HARD_BOUNCE,
emailAddresses: extractEmailAddresses(bounce.bouncedRecipients),
feedbackId: bounce.feedbackId ?? null,
};
}
if (event['detail-type'] === 'Email Complaint Received') {
const complaint = event.detail?.complaint;
return {
reason: EmailGroupSuppressionReason.COMPLAINT,
emailAddresses: extractEmailAddresses(complaint?.complainedRecipients),
feedbackId: complaint?.feedbackId ?? null,
};
}
return null;
}
private resolveWorkspaceId(event: SesEventBridgeNotification): string | null {
const taggedWorkspaceId =
event.detail?.mail?.tags?.[AWS_SES_WORKSPACE_TAG_NAME]?.[0];
if (isDefined(taggedWorkspaceId) && taggedWorkspaceId.length > 0) {
return taggedWorkspaceId;
}
if (!isNonEmptyArray(event.resources)) {
return null;
}
for (const resourceArn of event.resources) {
const workspaceId = parseWorkspaceIdFromAwsSesResourceArn(resourceArn);
if (isDefined(workspaceId)) {
return workspaceId;
}
}
return null;
}
}
const extractEmailAddresses = (
recipients: { emailAddress: string }[] | undefined,
): string[] =>
isNonEmptyArray(recipients)
? recipients.map((recipient) => recipient.emailAddress)
: [];
@@ -1,15 +1,46 @@
export type SesEventBridgeNotification = {
source: 'aws.ses';
'detail-type': 'Sending Status Enabled' | 'Sending Status Disabled';
resources?: string[];
detail?: {
version?: string;
data?: {
origin?: string;
record?: {
status?: 'ENABLED' | 'DISABLED';
cause?: string;
};
export type SesEventBridgeDetailType =
| 'Sending Status Enabled'
| 'Sending Status Disabled'
| 'Email Bounced'
| 'Email Complaint Received';
type SesEventRecipient = {
emailAddress: string;
};
type SesMessageEventDetail = {
eventType?: string;
mail?: {
messageId?: string;
destination?: string[];
tags?: Record<string, string[]>;
};
bounce?: {
bounceType?: 'Permanent' | 'Transient' | 'Undetermined';
bouncedRecipients?: SesEventRecipient[];
feedbackId?: string;
};
complaint?: {
complainedRecipients?: SesEventRecipient[];
feedbackId?: string;
complaintFeedbackType?: string;
};
};
type SesSendingStateEventDetail = {
version?: string;
data?: {
origin?: string;
record?: {
status?: 'ENABLED' | 'DISABLED';
cause?: string;
};
};
};
export type SesEventBridgeNotification = {
source: 'aws.ses';
'detail-type': SesEventBridgeDetailType;
resources?: string[];
detail?: SesMessageEventDetail & SesSendingStateEventDetail;
};
@@ -8,6 +8,7 @@ export enum UsageOperationType {
WORKFLOW_EXECUTION = 'WORKFLOW_EXECUTION',
CODE_EXECUTION = 'CODE_EXECUTION',
WEB_SEARCH = 'WEB_SEARCH',
EMAIL_SEND = 'EMAIL_SEND',
}
registerEnumType(UsageOperationType, {
@@ -22,6 +22,7 @@ import { CustomDomainManagerService } from 'src/engine/core-modules/domain/custo
import { SubdomainManagerService } from 'src/engine/core-modules/domain/subdomain-manager/services/subdomain-manager.service';
import { ExceptionHandlerService } from 'src/engine/core-modules/exception-handler/exception-handler.service';
import { FeatureFlagService } from 'src/engine/core-modules/feature-flag/services/feature-flag.service';
import { EmailingDomainEntity } from 'src/engine/core-modules/emailing-domain/emailing-domain.entity';
import {
EmailingDomainWorkspaceCleanupJob,
type EmailingDomainWorkspaceCleanupJobData,
@@ -512,9 +513,18 @@ export class WorkspaceService extends TypeOrmQueryService<WorkspaceEntity> {
{ workspaceId: id },
);
const emailingDomains = await this.coreDataSource
.getRepository(EmailingDomainEntity)
.find({ where: { workspaceId: id } });
await this.messageQueueService.add<EmailingDomainWorkspaceCleanupJobData>(
EmailingDomainWorkspaceCleanupJob.name,
{ workspaceId: id },
{
workspaceId: id,
domains: emailingDomains.map(
(emailingDomain) => emailingDomain.domain,
),
},
);
if (workspace.customDomain) {