Canifa Architecture++

Before: one monolith

- ERP: integration of all apps
- Problem:
- Database: bottleneck
- Needs a very strong server
- Hard to scale horizontally
- One monolith, One team

© Trobz 2009-2016 - All rights reserved 2

After: splitting load and concerns
- One instance per functional domain
- Retail
- PLM
- CRM
- Problem: how do we sync the common data?
- Products
- Customers
- Orders
- ...

© Trobz 2009-2016 - All rights reserved 3

TOC
1) Domain objects
2) Expectations
3) Challenges
4) Potential components
5) Potential architectures

© Trobz 2009-2016 - All rights reserved 4

 1) Domain objects
- User
- res_users
- Product
- product_product
- Supplier
- res_partner
- Customer
- res_partner
- Order
- pos_order

© Trobz 2009-2016 - All rights reserved 5

2) Expectations
- Use cases
- Customer information synchronized on all systems
- Dividing load of POS in multiple sub-systems
- Accounting/Report system that aggregates all sessions/orders
- Multi-master for some entities:
- CRM: creates new Customers by importing marketing lists
- Retail: creates new Customers from the POS
- PLM: creates new Customers from the Ecommerce website

© Trobz 2009-2016 - All rights reserved 6

3) Challenges
3) Challenges
- At which level should we capture changes?
- Database:
- Pros: Captures all changes, including manual/batches fixes, queries bypassing ORM
- Cons: verbose (especially because of the ORM that produces non-optimized statements)
- Models
- Pros: Easy to plug in create/write functions
- Cons: Adds a bit of load on sender side
- At which level should we integrate changes?
- Database:
- Pros:
- Generic tools available: native logical replication/debezium + kafka connect
- Cons:
- Logical replication: sequences: needs to maintain a gap
- Debezium:
- Works at statement level, with one topic per table
- risk of losing statements orders
- Models:
- Odoo sync worker
© Trobz 2009-2016 - All rights reserved 8
3) Challenges (2)
- Change events order should be strictly respected
- Regardless of the model
- E.g.
- res.partner:create
- pos.order:create (with a reference to the partner previously created)
- we need the sync to respect that order strictly
- Tables ids
- Local sequences
- 3 solutions
- Add new global key field/or replace native odoo primary keys by global keys scheme
- Save source / source_id (which gives a sort of of global key)
- Enforce some attributes as keys (eg reference field of customer)

© Trobz 2009-2016 - All rights reserved 9

3) Challenges (3)
About data inconsistencies

- Monolith: one big database

- ACID
- Consistency enforced
- Cost

- New architecture: decoupled, asynchronous, eventually inconsistent

- For some time, some parts of the system don't have the exact same view on common records
- This is temporary
- In most business cases, it's ok

- Always remember that even with a single instance, some use cases can create data inconsistencies, e.g.:
- User A open Partner1 in edit mode
- User B opens same Partner1 in edit mode
- User A changes its name to Partner1-A, email to partner1a@trobz.com and saves
- User B changes its name to Partner1-B and saves
- Final name of Partner1 is Partner1-B, with email partner1a@trobz.com
© Trobz 2009-2016 - All rights reserved 10
4) Potential components
Sync using single master + API
- Solution
- We define a single master for each entity
- Whenever we want to create/update a common entity, we call the master through a REST API
- Problems
- Synchronous
- Strong coupling
- Strong change of Odoo
- Need to handle with errors

© Trobz 2009-2016 - All rights reserved 12

Sync using db logical replication
- Solutions
- Native logical replication
- Pglogical: more advanced
- Fine grained replication
- Problems
- Sequences issues
- Receivers limited to postgresql databases
- Manual process to define publishers/subscribers
- Needs to define good practices/helpers
- Direct coupling of the most important components (databases)
- Risk when things go wrong
- Conclusion
- Good for simple use cases: report instance

© Trobz 2009-2016 - All rights reserved 13

Sync using connector jobs
- Instance A -> B
- on create()/write(), create job
- Job connects to instance B via xmlrpc to perform action
- Pros:
- Connector based
- Cons:
- Not real-time; depends on number of jobs, depends on local system load and target system load
- Instance A (sender) needs credentials of Instance B to connect via xmlrpc
- Needs target system(s) to be up
- If you add one target system, you double the number of jobs
- Conclusion
- For 1-to-1 sync

© Trobz 2009-2016 - All rights reserved 14

Sync using connector jobs + kafka
- Instance A -> B
- on create()/write(), create job
- Job pushes change to Kafka
- Pros:
- Connector based
- If you add one target system, doesn't change anything
- Don't need target system(s) to be up
- No need for target system(s) credentials anymore
- Cons:
- Not real-time; almost real-time but there can be a lag, depending on number of jobs/local system load
- Needs to define precisely
- scope of data to sync
- behavior on conflict
- Conclusion:
- For n-to-n sync

© Trobz 2009-2016 - All rights reserved 15

Event platforms #hype
- "State of the art"
- All tech companies built their new architectures upon an event platform
- Linkedin: https://engineering.linkedin.com/blog/2016/04/kafka-ecosystem-at-linkedin
- Yelp: https://engineeringblog.yelp.com/2016/07/billions-of-messages-a-day-yelps-real-time-data-pipeline.html
- Trivago: https://fr.slideshare.net/ClemensValiente/kafka-at-trivago
- Netflix: https://quickbooks-engineering.intuit.com/lessons-learnt-from-netflix-keystone-pipeline-with-trillions-of-daily-messages-64cc91b3c8ea
- Microsoft: https://azure.microsoft.com/en-us/blog/processing-trillions-of-events-per-day-with-apache-kafka-on-azure/
- Uber...
- Proved to scale with big data volumes
- Enables real-time sync/analytics/business dashboard
- Plan for the future
- avoid the spaghetti mess
- more data!

© Trobz 2009-2016 - All rights reserved 16

Before

© Trobz 2009-2016 - All rights reserved 17

After

© Trobz 2009-2016 - All rights reserved 18

After: impact on teams
Loosely coupled teams:

© Trobz 2009-2016 - All rights reserved 19

5) Potential architectures
- No ultimate out-of-the-box solution
- Depending on the use case

© Trobz 2009-2016 - All rights reserved 20

PoC-01: data sync
PoC-01: data sync

© Trobz 2009-2016 - All rights reserved 22

Consensus rule #1: ownership
- Assumption: an object created on one given instance will probably be important to that instance
- for example a new Customer created on the POS instance will be linked to a POS order
- if another instance is only interested in Customers, it might one day submit a change on that Customer that
can't be applied on the POS instance: deletion for example, because of that link to a POS order

- So we choose to define the original producer of an object as its "owner"

- Whenever another instance modifies an object, its owner will:
- apply changes locally (validation)
- broadcast the change event if it's successful

- If changes from non-owners were immediately applied on all instances, we could face some conflicts:
- Let's imagine a simultaneous change on object A by I1 and I2
- I3 receives change from I1 first, I2 second
- I4 receives from I2 first, I2 second
- If changes are incompatible, I3 and I4 don't have the same view on the object anymore
- With the ownership rule, it's the vision of the object's owner that gets finally replicated on all nodes
- In other words, the owner is the source of truth regarding its own objects

© Trobz 2009-2016 - All rights reserved 23

Consensus rule #2: identities
- Each instance involved has a unique `SYNC_SOURCE_IDENTIFIER` (e.g. `pos` for the POS instance)

- Common objects synchronized between instances are identified by their unique `(source, source_id)`, where:
- `source` is the identifier of the owner
- `source_is` is the record id on the instance of the owner

© Trobz 2009-2016 - All rights reserved 24

Synchronization based on connector jobs
Sending changes

- We don't send the event synchronously to Kafka because:

- we don't want calls completion time to be impacted by the potential issues that we could meet (Kafka down:
need to retry)
- we don't know yet if the calls' transaction is actually going to be commited:
- if not we would need to compensate the event that has been sent
- the job will only be created if the transaction is commited
- `INSERT into queue_job` doesn't require any heavy lock so impact on db load is small

- Now, if we want to rely on these events for synchronization of other instances, we need to make sure that:
- the job is successfully processed
- no failure accepted
- if one job fails, all subsequent jobs are stuck until the situation is resolved
- jobs are processed in the same exact order that they are created

- To match these requirements, we use a dedicated channel: sequential, with a capacity of 1

© Trobz 2009-2016 - All rights reserved 25

Synchronization based on connector jobs (2)
Receiving changes

- We want model events to be processed strictly in the same order they were sent (at source level)

- We don't want events sent by a given instance to be in a position to block the processing of events from
another instance

- So we use a single dedicated process to consume model events from other instances
- But a job is created for each event
- And we have a dedicated channel per source to process them:
- sequential, with a capacity of 1

© Trobz 2009-2016 - All rights reserved 26

PoC-02: report instance
Poc-02: report instance

© Trobz 2009-2016 - All rights reserved 28