Paul RastoinandGitHub
5544b5dcfe
# Introduction The initial motivation was that in the workspace migration create action some universal foreign key aggregators weren't correctly deleted before returned due to constant missconfiguration <img width="2300" height="972" alt="image" src="https://github.com/user-attachments/assets/9401eb02-2bb2-4e69-9c5f-9a354ff61079" /> It also meant that under the hood some optimistic behavior wasn't correctly rendered for some aggregators ## Solution Refactored the `ALL_METADATA_RELATIONS` as follows: This way we can infer the FK and transpile it to a universalFK, also the aggregators are one to one instead of one versus all available Making the only manual configuration to be defined the `foreignKey` and `inverseOneToManyProperty` ``` ┌──────────────────────────────────────┐ ┌─────────────────────────────────────────────┐ │ ALL_MANY_TO_ONE_METADATA_FOREIGN_KEY│ │ ALL_ONE_TO_MANY_METADATA_RELATIONS │ │──────────────────────────────────────│ │─────────────────────────────────────────────│ │ Derived from: Entity types │ │ Derived from: Entity types │ │ │ │ │ │ Provides: │ │ Provides: │ │ • foreignKey │ │ • metadataName │ │ │ │ • flatEntityForeignKeyAggregator │ │ Standalone low-level primitive │ │ • universalFlatEntityForeignKeyAggregator │ └──────────────┬───────────────────────┘ └──────────────┬──────────────────────────────┘ │ │ │ foreignKey type + │ inverseOneToManyProperty │ universalForeignKey derivation │ keys (type constraint) │ │ ▼ ▼ ┌───────────────────────────────────────────────────────────────┐ │ ALL_MANY_TO_ONE_METADATA_RELATIONS │ │───────────────────────────────────────────────────────────────│ │ Derived from: │ │ • Entity types (metadataName, isNullable) │ │ • ALL_MANY_TO_ONE_METADATA_FOREIGN_KEY (FK → universalFK) │ │ • ALL_ONE_TO_MANY_METADATA_RELATIONS (inverse keys) │ │ │ │ Provides: │ │ • metadataName │ │ • foreignKey (replicated from FK constant) │ │ • inverseOneToManyProperty │ │ • isNullable │ │ • universalForeignKey │ └──────────────────────────┬────────────────────────────────────┘ │ ┌──────────────────┼──────────────────┐ │ │ │ ▼ ▼ ▼ ┌───────────────────┐ ┌────────────────┐ ┌──────────────────────┐ │ Type consumers │ │ Atomic utils │ │ Optimistic utils │ │───────────────────│ │────────────────│ │──────────────────────│ │ • JoinColumn │ │ • resolve-* │ │ • add/delete flat │ │ • RelatedNames │ │ • get-* │ │ entity maps │ │ • UniversalFlat │ │ │ │ • add/delete │ │ EntityFrom │ │ │ │ universal flat │ │ │ │ │ │ entity maps │ └───────────────────┘ └────────────────┘ │ │ │ (bridge via │ │ inverseOneToMany │ │ Property → │ │ ONE_TO_MANY for │ │ aggregator lookup) │ └──────────────────────┘ ``` ### Previously ``` ┌─────────────────────────────────────────────────────────────────────┐ │ ALL_METADATA_RELATIONS │ │─────────────────────────────────────────────────────────────────────│ │ Derived from: Entity types │ │ │ │ Structure: { [metadataName]: { manyToOne: {...}, oneToMany: {...},│ │ serializedRelations?: {...} } } │ │ │ │ manyToOne provides: │ │ • metadataName │ │ • foreignKey │ │ • flatEntityForeignKeyAggregator (nullable, often wrong/null) │ │ • isNullable │ │ │ │ oneToMany provides: │ │ • metadataName │ │ │ │ Monolithic single source of truth │ └──────────────────────────┬──────────────────────────────────────────┘ │ │ manyToOne entries transformed via │ ToUniversalMetadataManyToOneRelationConfiguration │ ▼ ┌─────────────────────────────────────────────────────────────────────┐ │ ALL_UNIVERSAL_METADATA_RELATIONS │ │─────────────────────────────────────────────────────────────────────│ │ Derived from: ALL_METADATA_RELATIONS (type-level transform) │ │ │ │ Structure: { [metadataName]: { manyToOne: {...}, oneToMany: {...} │ │ } } │ │ │ │ manyToOne provides: │ │ • metadataName │ │ • foreignKey │ │ • universalForeignKey (derived: FK → replace Id → UniversalId) │ │ • universalFlatEntityForeignKeyAggregator (derived from │ │ flatEntityForeignKeyAggregator → replace Ids → UniversalIds) │ │ • isNullable │ │ │ │ oneToMany: passthrough from ALL_METADATA_RELATIONS │ │ │ │ Duplicated monolith with universal key transforms │ └──────────────────────────┬──────────────────────────────────────────┘ │ ┌──────────────────┼──────────────────────┐ │ │ │ ▼ ▼ ▼ ┌───────────────┐ ┌────────────────────┐ ┌──────────────────────┐ │ Type consumers│ │ Atomic utils │ │ Optimistic utils │ │───────────────│ │────────────────────│ │──────────────────────│ │ • JoinColumn │ │ • resolve-entity- │ │ • add/delete flat │ │ • RelatedNames│ │ relation-univ-id │ │ entity maps │ │ • Universal │ │ (ALL_METADATA_ │ │ (ALL_METADATA_ │ │ FlatEntity │ │ RELATIONS │ │ RELATIONS │ │ From │ │ .manyToOne) │ │ .manyToOne) │ │ │ │ │ │ │ │ Mixed usage │ │ • resolve-univ- │ │ • add/delete univ │ │ of both │ │ relation-ids │ │ flat entity maps │ │ constants │ │ (ALL_UNIVERSAL_ │ │ (ALL_UNIVERSAL_ │ │ │ │ METADATA_REL │ │ METADATA_REL │ │ │ │ .manyToOne) │ │ .manyToOne) │ │ │ │ │ │ │ │ │ │ • resolve-univ- │ │ universalFlatEntity │ │ │ │ update-rel-ids │ │ ForeignKeyAggregator │ │ │ │ (ALL_UNIVERSAL_ │ │ read directly from │ │ │ │ METADATA_REL │ │ the constant │ │ │ │ .manyToOne) │ │ │ │ │ │ │ │ │ │ │ │ • regex hack: │ │ │ │ │ │ foreignKey │ │ │ │ │ │ .replace(/Id$/, │ │ │ │ │ │ 'UniversalId') │ │ │ └───────────────┘ └────────────────────┘ └──────────────────────┘ ```
220 lines
7.3 KiB
Plaintext
220 lines
7.3 KiB
Plaintext
---
|
|
description: Main guide for creating syncable entities in Twenty's workspace migration system
|
|
globs: ["**/metadata-modules/**", "**/workspace-migration/**"]
|
|
alwaysApply: false
|
|
---
|
|
|
|
# Creating a New Syncable Entity - Main Guide
|
|
|
|
This is the main guide for creating **syncable entities** in Twenty's workspace migration architecture.
|
|
|
|
## Documentation Structure
|
|
|
|
This main guide provides a high-level overview and navigation hub.
|
|
|
|
**⚡ Skills** (`.cursor/skills/syncable-entity-*/SKILL.md`) - Concise, action-oriented implementation guides for each step. Reference these when creating a new syncable entity.
|
|
|
|
**When to use:**
|
|
- Start here for architecture overview and workflow
|
|
- Reference specific skills (`@syncable-entity-types-and-constants`) when implementing each step
|
|
|
|
## What is a Syncable Entity?
|
|
|
|
A syncable entity is a metadata entity that:
|
|
- Has a **`universalIdentifier`**: A unique identifier used for syncing entities across workspaces/applications
|
|
- Has an **`applicationId`**: Links the entity to an application (Twenty Standard or Custom applications)
|
|
- Participates in the **workspace migration system**: Can be created, updated, and deleted through the migration pipeline
|
|
- Is **cached as a flat entity**: Denormalized representation for efficient validation and change detection
|
|
|
|
Examples: `skill`, `agent`, `view`, `viewField`, `role`, `pageLayout`, etc.
|
|
|
|
## Architecture Overview
|
|
|
|
```
|
|
Input DTO → Transform → Universal Flat Entity → Builder/Validator → Runner → Database
|
|
↓
|
|
Cache Service
|
|
```
|
|
|
|
**Key Components:**
|
|
- **TypeORM Entity**: Database model extending `SyncableEntity`
|
|
- **Flat Entity**: Denormalized type (no relations, dates as strings) - for caching
|
|
- **Universal Flat Entity**: Flat entity with foreign keys mapped to universal identifiers - for migrations
|
|
- **Transform Utils**: Convert DTOs to universal flat entities
|
|
- **Builder/Validator**: Validate and create migration actions
|
|
- **Runner**: Execute actions against the database
|
|
|
|
## Implementation Steps
|
|
|
|
Follow these skills in order:
|
|
|
|
### 1️⃣ **Foundation: Types & Constants** → `@syncable-entity-types-and-constants`
|
|
|
|
**What:** Define all types, entities, and register in central constants
|
|
|
|
**Tasks:**
|
|
- Create TypeORM entity (extends `SyncableEntity`)
|
|
- Define flat entity types
|
|
- Define action types (universal + flat)
|
|
- Register in 5 central constants
|
|
|
|
**Why first:** Everything else depends on these types
|
|
|
|
---
|
|
|
|
### 2️⃣ **Data Layer: Cache & Transform** → `@syncable-entity-cache-and-transform`
|
|
|
|
**What:** Handle conversion between different representations
|
|
|
|
**Tasks:**
|
|
- Create cache service
|
|
- Create entity-to-flat conversion
|
|
- Create input transform utils
|
|
- Handle foreign key resolution
|
|
|
|
**Dependencies:** Requires Step 1
|
|
|
|
---
|
|
|
|
### 3️⃣ **Business Logic: Builder & Validation** → `@syncable-entity-builder-and-validation`
|
|
|
|
**What:** Validate business rules and create actions
|
|
|
|
**Tasks:**
|
|
- Create validator service (never throws, never mutates)
|
|
- Create builder service
|
|
- Wire into orchestrator (⚠️ critical!)
|
|
|
|
**Dependencies:** Requires Steps 1-2
|
|
|
|
---
|
|
|
|
### 4️⃣ **Execution: Runner & Actions** → `@syncable-entity-runner-and-actions`
|
|
|
|
**What:** Execute migration actions against the database
|
|
|
|
**Tasks:**
|
|
- Create action handlers (create/update/delete)
|
|
- Implement transpilation methods
|
|
- Create universal-to-flat conversion utilities
|
|
|
|
**Dependencies:** Requires Steps 1-3
|
|
|
|
---
|
|
|
|
### 5️⃣ **Assembly: Integration** → `@syncable-entity-integration`
|
|
|
|
**What:** Wire everything together
|
|
|
|
**Tasks:**
|
|
- Register in 3 NestJS modules
|
|
- Create service and resolver layers
|
|
- Use exception interceptor
|
|
|
|
**Dependencies:** Requires Steps 1-4
|
|
|
|
---
|
|
|
|
### 6️⃣ **Testing: Integration Tests** (**MANDATORY**) → `@syncable-entity-testing`
|
|
|
|
**What:** Comprehensive test suite
|
|
|
|
**Tasks:**
|
|
- Create test utilities
|
|
- Write failing tests (all validator exceptions)
|
|
- Write successful tests (all CRUD operations)
|
|
- Use snapshot testing
|
|
|
|
**Dependencies:** Requires all previous steps
|
|
|
|
---
|
|
|
|
## Quick Reference
|
|
|
|
### Multi-Agent Workflow
|
|
|
|
For parallel development:
|
|
1. **Agent 1** (Foundation): Complete Step 1 first - unblocks everyone
|
|
2. **Agent 2** (Cache): Can start immediately after Step 1
|
|
3. **Agent 3** (Builder): Can work in parallel with Agent 4 after Step 1
|
|
4. **Agent 4** (Runner): Can work in parallel with Agent 3 after Step 1
|
|
5. **Agent 5** (Integration): Assembles everything after Steps 2-4
|
|
|
|
### Key Design Principles
|
|
|
|
| Layer | Responsibility | Can Throw? | Can Mutate? |
|
|
|-------|---------------|------------|-------------|
|
|
| Transform Utils | Data transformation | Yes (input validation) | N/A (creates new) |
|
|
| Validator | Business rule validation | **No** (returns errors) | **No** |
|
|
| Builder | Action creation | **No** (returns errors) | **No** |
|
|
| Runner | Database operations | Yes (DB errors) | Yes (via TypeORM) |
|
|
|
|
### Common Pitfalls
|
|
|
|
⚠️ **Most Commonly Forgotten:**
|
|
1. Wiring builder in orchestrator service
|
|
2. Registering in all 3 modules (builder, validators, action handlers)
|
|
3. Setting `universalIdentifier` correctly in entity-to-flat conversion
|
|
|
|
⚠️ **Common Mistakes:**
|
|
1. Using regular IDs instead of universal identifiers in transform utils
|
|
2. Throwing exceptions in validators/builders
|
|
3. Mutating entity maps in validators/builders
|
|
4. Forgetting to handle JSONB properties with `SerializedRelation`
|
|
|
|
### File Locations
|
|
|
|
```
|
|
packages/twenty-shared/src/metadata/
|
|
└── all-metadata-name.constant.ts
|
|
|
|
packages/twenty-server/src/engine/metadata-modules/
|
|
├── my-entity/ # Step 1
|
|
│ └── entities/
|
|
├── flat-my-entity/ # Steps 1-2
|
|
│ ├── types/
|
|
│ ├── constants/
|
|
│ ├── services/
|
|
│ └── utils/
|
|
└── flat-entity/constant/ # Step 1 (central registries)
|
|
├── all-entity-properties-configuration-by-metadata-name.constant.ts
|
|
├── all-one-to-many-metadata-relations.constant.ts
|
|
├── all-many-to-one-metadata-foreign-key.constant.ts
|
|
└── all-many-to-one-metadata-relations.constant.ts
|
|
|
|
packages/twenty-server/src/engine/workspace-manager/workspace-migration/
|
|
├── workspace-migration-builder/ # Step 3
|
|
│ ├── builders/my-entity/
|
|
│ └── validators/services/
|
|
└── workspace-migration-runner/ # Step 4
|
|
└── action-handlers/my-entity/
|
|
```
|
|
|
|
### Complete Checklist
|
|
|
|
Before considering complete:
|
|
- [ ] All 6 guides completed
|
|
- [ ] TypeORM entity extends `SyncableEntity`
|
|
- [ ] All constants registered (5 central registries)
|
|
- [ ] Cache service with correct decorator
|
|
- [ ] Transform utils return universal flat entities
|
|
- [ ] Validator never throws/mutates
|
|
- [ ] Builder wired in orchestrator (⚠️ critical!)
|
|
- [ ] All 3 action handlers implemented
|
|
- [ ] All 3 modules updated
|
|
- [ ] **Integration tests written (MANDATORY)**
|
|
- [ ] **All failing scenarios covered**
|
|
- [ ] **All successful use cases tested**
|
|
|
|
---
|
|
|
|
## Need Help?
|
|
|
|
Reference the appropriate skill for step-by-step guidance:
|
|
- `@syncable-entity-types-and-constants` - Types, entities, constants
|
|
- `@syncable-entity-cache-and-transform` - Cache & transform
|
|
- `@syncable-entity-builder-and-validation` - Builder & validation
|
|
- `@syncable-entity-runner-and-actions` - Runner & actions
|
|
- `@syncable-entity-integration` - Integration & wiring
|
|
- `@syncable-entity-testing` - Testing patterns
|