This PR update the redis subscription iterator wrapper with an heartbeat
system. Heartbeat interval are based on event stream TTL. Those refresh
the event stream TTL and active streams in redis.
I also cleaned a few functions that were not used anymore.
---------
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
This PR implements SSE update events across the main components of the
application : tables, boards, calendars, show pages.
There is still work to do on other event type and on making sure
everything works fine, but this first implementation should be robust
enough to start with.
Some problems encountered along the way :
- Events are returning raw Postgres output, because they are not
normalized by the GraphQL layer, so we ended up with amountMicros as
string values, which the frontend does not like, so I implemented a
small util in our `formatResult` generic pipeline to turn amountMicros
to a number if it's a string value. We could implement other formatters
for composite fields if we see problems with events.
-`action` property was missing in SSE events, which is required by the
frontend to know what kind of event it is.
# QA
https://github.com/user-attachments/assets/393b45ac-59d2-48b0-855b-2ce9c4b8ae57https://github.com/user-attachments/assets/cb214e7a-1595-4b85-bdb6-87630993d2e2
- new channel EVENT_STREAM_CHANNEL based on event stream id
- on event, perform the matching and publish only to the right streams
- store a list of active streams per workspace
- store the user id along with the queries for each stream
Bonus:
- remove onSubscriptionMatch
This PR is a follow-up of https://github.com/twentyhq/twenty/pull/16966
and implements a new mechanism to handle SSE events.
It creates only one event stream per browser tab, then use mutations to
tell the backend which query to listen to, without re-mounting the event
stream connexion.
Then each event that comes from this unique subscription is then
dispatched in a new JavaScript CustomEvent, per queryId, on which
specific hooks add an event listener.
This PR introduces the generic tooling as well as the handling of update
events on table.
- moved a few gql types to twenty shared to re-use in server
- added a new endpoint, onEventSubscription that expect a streamId to
create a connection
- two new endpoints to store queries in Redis
- updated the existing batch channel to directly use object record type
We were using a global ValidationPipe in main.ts. This is an issue as
@Controllers should return HttpExecption and @Resolvers should return
GraphqlErrors
Removing the global pipe and creating a ResolverValidationPipe able to
generate GraphqlError. We also need to handle the exception in a filter
to avoid nest to think it's unhandled and make it flow to logs
Next step:
- it would be nice to have both @UsePipes(ResolverValidationPipe) +
@UseFilters(GraphqlValidationExceptionFilter) come together. This should
be possible if we create a @GraphQLResolver annotation
related to https://github.com/twentyhq/core-team-issues/issues/601
## Done
- add a `onDbEvent` `Subscription` graphql endpoint to listen to
database_event using what we have done with webhooks:
- you can subscribe to any `action` (created, updated, ...) for any
`objectNameSingular` or a specific `recordId`. Parameters are nullable
and treated as wildcards when null.
- returns events with following shape
```typescript
@Field(() => String)
eventId: string;
@Field()
emittedAt: string;
@Field(() => DatabaseEventAction)
action: DatabaseEventAction;
@Field(() => String)
objectNameSingular: string;
@Field(() => GraphQLJSON)
record: ObjectRecord;
@Field(() => [String], { nullable: true })
updatedFields?: string[];
```
- front provide a componentEffect `<ListenRecordUpdatesEffect />` that
listen for an `objectNameSingular`, a `recordId` and a list of
`listenedFields`. It subscribes to record updates and updates its apollo
cached value for specified `listenedFields`
- subscription is protected with credentials
## Result
Here is an application with `workflowRun`
https://github.com/user-attachments/assets/c964d857-3b54-495f-bf14-587ba26c5a8c
---------
Co-authored-by: prastoin <paul@twenty.com>
