- move cron commands logic to a workspace service
- add commands to enqueue not started and dequeue staled workflows
- stop dispatching one job per workspace. Handle all workspaces in one
job
- run enqueue cron every 5 minutes instead of every minutes
Follow up:
- add a separated redis key to control if there are not started
workflows. It will avoid fetching workflow runs for each workspace
This Pr
- adds a `route` table to the core schema
- adds a controller to trigger route
For now, we need to add a `/s/<workspace_id>` prefix to all routes
We plan to create a custom domain table in order to let the users create
subdomains for their workspace, and so to link their routes to a
subdomain. Thank to that, we will be able to identify workspace_id from
domain name, and the prefix could be `/s`. If we create a native
dedicated subdomain for routes for each workspaces, the prefix could be
completely removed!
Here the follow up ticket to do that ->
https://github.com/twentyhq/twenty/issues/14240
This Pr continues the extensibility journey
- adds a `core.databaseEventTrigger` table
- add a oneToMany relation between `core.serverlessFunction` and
`core.databaseEventTrigger`
- add a job `CallDatabaseEventTriggerJobsJob` triggered by
`EntityEventsToDbListener` triggering `serverlessFunction` based on the
`core.databaseEventTrigger.settings.eventName`
- add a new `trigger-queue` to carry this job
- renamed `DatabaseEventTriggerListener` into
`WorkflowDatabaseEventTriggerListener`
## Context
Introducing a new SyncableEntity abstraction that can be extended by
Entities that we want to "sync" via their universal identifier (the
class will add this uuid to the entity). Starting with views, will
refactor existing syncable entities such as objects/fields later
This Pr begins the extensibility journey
- adds a `core.cronTrigger` table
- add a oneToMany relation between core.serverlessFunction and
`core.cronTrigger` (one serverlessFunction can be triggered by multiple
cronTriggers)
- add a job to trigger a serverless function
- adds a cron to trigger serverlessFunction (via the trigger job) based
on the core.cronTrigger.setting.pattern
- adds a command to register the cron
- add the command in `cron-register-all.command.ts`
## Context
To simplify the way we inject our default datasource, I've recently
removed the token injection that was confusion since we only had once
configured on the module level. Now I'm removing TypeORM service which
allows us to instantiate a new Datasource with the same parameters as
the default one, it was redundant and confusing.
## Context
View tables were missing some FK, we are also introducing delete cascade
on those tables when needed to leverage pg cascade deletion instead of
having to implement it.
Also renaming those classes with Entity suffix to follow repo
guidelines.
## Context
We are about to move view from workspace schema to core schema however
favorite table is still referencing the old table which means we can't
insert new records without breaking the constraint. This PR removes it
(and we don't have any plan to add a new one between core /
workspaceSchema, in fact, we might move the favorite table to core
schema as well in the future)
Parallel code path to read and write core views when
IS_CORE_VIEW_ENABLED.
Migrated view key to an enum.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
Improve workflow enqueue cron : instead of relying on the cache to know
how many workflows we can enqueue, query the DB. Then set the cache and
process the not started workflows.
Also adding a second cron that will look for workflows enqueued one hour
ago or more and put these back in the not started status. This will
allow the first cron to start these again.
Created:
- Services
- Resolvers
- Controllers
- Tests for services
- Integration tests for GraphQL and Rest
Updated the Rest API playground
Added new feature flag `IS_CORE_VIEW_ENABLED`
Updated `viewFilter` `operand` and `view` `type` to be enums rather than
strings and generated migration file.
Closes https://github.com/twentyhq/core-team-issues/issues/1259
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
# Introduction
- deferrable constraint on foreignKey definition
- fix relation creation input transpilation to flat field metadata
- fix validation to allow contextual validation on not already created
field metadata ( usefull for relation, and will be necessary for the
import )
This PR aims to improve performances as we are making A LOT of queries
after having added events emission to the ORM layer. While investigating
issues, I've come across multiple problems I will describe below
## Add index on workspace.activationStatus as we are querying it a lot
As per title
## Add logs on core datasource destroy
It seems that we have postgres connection pool destruction in
production. I cannot reproduce locally but I suspect the Query Timeouts
to be the root cause. I'm fixing most of the Query Timeouts cause in
this PR but I'm adding the logs so we have more information in case in
keeps happening in production.
## GraphQL query runner createMany
It was using a for loop on each record. This is as issue has we will
emit an event separately for each record => we should always try to
batch events.
Replacing by a save on all records. Note that this is not perfect as we
should avoid using save (bad performances), and use insert + updateMany
instead. As I have follow up discussions regarding permissions, I
haven't replaced it by insert and updateMany yet. Using save instead of
a for loop makes the problem less worrying.
## Introduce updateMany in ORM @Weiko @ijreilly FYI
.save(manyRecords) is bad as it's querying the data for no good reason
(and doing a select for each record...).
We already have .insert(), i'm introducing .updateMany()
I think our ORM layer should be simplified a lot but I'm not starting
the refacto yet
## Fixing ORM @Weiko @ijreilly FYI
- Fixing events emission in delete function
- Fixing events emission in insert function
- make sure everything is batched
## Events performance @Weiko @ijreilly FYI
Do not emit timelineActivity db events as this does not seem useful and
is quite heavy
## Messaging and Calendar performance @bosiraphael FYI
Rework many functions to make sure they are batched. This does not touch
the driver layer and I have heavily tested it. (multiple account, with
multiple channels, common thread, common messages, etc...)
## Workflow Trigger relation Fetch performance @martmull FYI
Improved performance by batching
- Improve migration script to make it idempotent
- Fix update on view and view filters (`displayValue` exists on
workspace view filter but not on core view filter,
`kanbanFieldMetadataId` exists on workspace view but not on core view)
Closes https://github.com/twentyhq/core-team-issues/issues/1248
- Create listeners on each CRUD operation for all view related objects
and update the core views accordingly
Some fields have to be parsed since we changed the data model a little
bit when switching to core views.
Duplicated the existing command `AddEnqueuedStatusToWorkflowRunCommand`.
Adding two steps:
- fetch the `workflowRun` object of the selected workspace
- using that object metadata id in the status field selection
We will have floating steps in our workflow with the branch design.
Currently, a step without parent is considered linked to the trigger. We
need to distinguish the 2 cases. Thus this PR:
- add `nextStepIds` to workflowVersion.trigger
- create a command to migrate existing triggers
