openerp 6.1 framework changes
DESCRIPTION
http://openerp.com Technical Presentation of the main changes in OpenERP 6.1 at the API and framework level: architecture changes, API evolution, new UI features, etc. See also the related presentation about the new 6.1 Web layer: http://www.slideshare.net/openobject/openerp-61-web-framework-tutorialTRANSCRIPT
@odony – 6.1 Framework Changes – 05.2012
OpenERP 6.1Technical Changes atAPI/Framework level
@odony – 6.1 Framework Changes – 05.2012
OpenERP 6.1
● Released February 2011● Normal release – non LTS = limited support● Release Notes
● http://bit.ly/openerp61RN
● Focus● Usability
● Social features
● New web client
● Cleanup
@odony – 6.1 Framework Changes – 05.2012
Agenda
● Framework & API● Architecture changes
● API evolution
● New views and UI features
● Other important bits
● Exercises, Q&A
Agenda
@odony – 6.1 Framework Changes – 05.2012
Evolution: Architecture changes
● Unified server-side deployment
● Scalability through multi-processing● Use all those cores on modern hardware
● Bypass Python GIL issues
● Reliability (cpu/memory/crashes)
● WSGI● Standard Python solution – OpenERP 6.1 is compliant
● Gunicorn - a WSGI compliant HTTP server
● Statelessness
● Distributed scheduled tasks (cron)
@odony – 6.1 Framework Changes – 05.2012
Unified server-side deployment
● No more need to deploy web client● Web client is now an OpenERP module● Only one package to install● Pre-loaded server-wide via –-load=web● All modules in same place (addons_path)● openerp-web still available stand-alone
● For testing/debugging purposes
● Needs to have copy of normal modules too!
@odony – 6.1 Framework Changes – 05.2012
Scalability through multi-processing
● Use all those cores on modern hardware● Bypass Python GIL limitations● Improve reliability (cpu/memory/crashes)
→ What does this all mean??
@odony – 6.1 Framework Changes – 05.2012
Multi-processing and GIL (1)
● CPython has a Global Interpreter Lock/GIL● Simplifies low-level memory management,
C calls, etc.● 1 Python thread = 1 OS thread● Parallel execution of threads forbidden● Cooperative multi-tasking on 1 CPU● Worse performance on multi-CPU/core!
● Battle for GIL on the different virtual CPUs!
@odony – 6.1 Framework Changes – 05.2012
Multi-processing and GIL (2)
● Cooperative multi-tasking● Release on I/O or every 100 ticks● 1 tick ≈ 1 interpreter instruction
@odony – 6.1 Framework Changes – 05.2012
Scalability through multi-processing
● multiprocessing module was an option● Using Unix tools and real processes was
more flexible and easier to scale up● Managing the processes was not
necessary, many tools exist to do that (e.g. Gunicorn)
● Just had to make OpenERP WSGI-compliant
@odony – 6.1 Framework Changes – 05.2012
WSGI (1)
● Web Server Gateway Interface for Python● WSGI applications can be embedded in
WSGI-compliant web servers● Many web frameworks support WSGI
● Django, CherryPy, …
● Many HTTP servers support WSGI● Apache(+mod), nginx(+mod), tornado, gunicorn, …
@odony – 6.1 Framework Changes – 05.2012
WSGI (2)
● Very simple API: one method
● environ = HTTP environment map● HTTP_HOST, REQUEST_METHOD, etc.
● start_response = callback(status,headers)
● Returns response as unicode string
● OpenERP's entry point: wsgi.core.application
@odony – 6.1 Framework Changes – 05.2012
Gunicorn (1)
● WGSI compliant HTTP Server● Python port of Ruby Unicorn● Pre-fork multi-process model● Flexible and easily configurable● UNIX only (fork support needed)● See also gunicorn.org
@odony – 6.1 Framework Changes – 05.2012
Gunicorn (2)
● Master “Arbiter” process● Starts and monitors workers (crash/timeout/...)
● Restarts killed workers fork→
● Increment/Decrement worker count (TTIN/TTOU signals)
● Doesn't speak HTTP naive dispatch→
● Forked worker processes● Speak HTTP Perform WSGI translation→
● Dispatch to application entry point
● Can restart after given number of requests
@odony – 6.1 Framework Changes – 05.2012
Gunicorn (3)
● OpenERP can't use normal config file or
command-line options● Config has to be provided via Gunicorn● gunicorn.conf, written in Python!
● Contains Gunicorn configuration (workers, limits, etc.)
● Contains OpenERP configuration
● Sets up requests/application hooks (pre/post request...)
● Arbiter controls max-requests limit● Workers control memory/cpu limits
@odony – 6.1 Framework Changes – 05.2012
Gunicorn (4)
$ gunicorn openerp:wsgi.core.application -c gunicorn.conf.pyRUN
CONF
@odony – 6.1 Framework Changes – 05.2012
Good to know (1)
● Gunicorn must be used explicitly→● Linear speed-up for CPU-bound tasks, less for I/O or DB
● Werkzeug built-in WSGI-compliant server→● openerp-server script
● multi-threaded mono-process
● Proxy mode specific handling (X-Forwarded..)● When deploying behind reverse proxy (e.g. with ProxyPass)
● Gunicorn: wsgi.proxied.application entry point instead
● Werkzeug: --proxy_mode
@odony – 6.1 Framework Changes – 05.2012
Good to know (2)
● No cron job with Gunicorn● Could be killed at any point!
● Start separate openerp-cron-worker job● Anywhere!
More about cron jobs in a minute...
@odony – 6.1 Framework Changes – 05.2012
OpenERP 6.0 deployment model
OpenERP Server PostgreSQLRequests OpenERP
Web Client
@odony – 6.1 Framework Changes – 05.2012
OpenERP WorkerOpenERP Worker
OpenERP 6.1 deployment model (1)
OpenERP WorkerOpenERP Worker
PostgreSQLRequests
@odony – 6.1 Framework Changes – 05.2012
OpenERP WorkerOpenERP Worker
OpenERP 6.1 deployment model (2)
OpenERP WorkerOpenERP Worker
PostgreSQLRequests
OpenERP WorkerOpenERP WorkerOpenERP WorkerOpenERP Worker
LoadBalancer
@odony – 6.1 Framework Changes – 05.2012
Statelessness (1)
● OpenERP 6.0 was still partially stateful:
1) Model registry (pool)
2) Wizards (osv_memory) kept in-memory
3) Application caches – menus, perms, … !
4) Background task processing: cron
● Not suitable for multi-processing
@odony – 6.1 Framework Changes – 05.2012
Caches and Model Registries
● Caches and Registries have state● Must be cleared on all workers when
invalidated – across multiple machines!● Signaling works with 2 DB sequences
● base_registry_signaling
● base_cache_signaling
● Before each request, both are checked● After each request, increment if changed
@odony – 6.1 Framework Changes – 05.2012
TransientModel
● osv_memory → TransientModel● Real database-backed model
● Automatically vacuum-cleaned
● Based on size limit or time limit
● Default: 2 hours
● Has special and implicit access rights● Everyone can perform CRUD
● But only access on your own records
● Can be used indifferently by multiple workers
@odony – 6.1 Framework Changes – 05.2012
Background Tasks (cron) (1)
● Problem:● Each worker possibly runs a cron processor
● Must prevent duplicate run of same task
● Must benefit from multi-processing when multiple cron processors
are running (each runs different tasks)
● Solution:● Database row-level mutex using LOCK
● Cron processor can be run on any number of servers
● Disabled on Gunicorn workers by default
● Can be run separately with openerp-cron-worker script added in 6.1
● Cron tasks can't be changed while running (locked)
@odony – 6.1 Framework Changes – 05.2012
Background Tasks (cron) (2)
● And by the way, cron tasks are executed in parallel in OpenERP 6.1
● Also if running mono-process, up to number or thread specified with –-max-cron-threads (default: 4)
● Allows multiple tasks to be executed at the same time
@odony – 6.1 Framework Changes – 05.2012
Agenda
● Framework & API● Architecture changes
● API evolution
● New views and UI features
● Other important bits
● Exercises, Q&A
Agenda
@odony – 6.1 Framework Changes – 05.2012
Date Storage and Computation (1)
● 6.0 stored timestamps in “server timezone”● Worked well for small teams with only one TZ
● Database not portable, Reports incorrect for cross-TZ teams, DST issues!
● 6.1 stores timestamps as UTC always!
● Big change, but Right Thing to Do™● Database becomes portable
● Dates monotonically incremented in DB
● Simplifies conversion for display UTC Client TZ↔
● Timestamps computation works as before on server-side
● Watch out!● “Dates” have no TZ
● Server-side dates must be initialized in client TZ
@odony – 6.1 Framework Changes – 05.2012
Date Storage and Computation (2)
● Server-side● All computation in UTC
● PDF reports use client TZ from context/user
● Database● All timestamps stored as TIMESTAMP WITHOUT TZ
● SELECT now() at time zone ‘UTC’ (should not need it)
● Client● Converted back and forth between UTC and client TZ
● Web uses browser TZ for display (JS limitation)
@odony – 6.1 Framework Changes – 05.2012
Date Storage and Computation (3)
@odony – 6.1 Framework Changes – 05.2012
Date Storage and Computation (4)
● Be very careful with date (that)!● Pure dates should be taken in user TZ● It's very tempting to use datetime.date.today()
● But may be wrong for half of the planet!● Use fields.date.context_today() instead
@odony – 6.1 Framework Changes – 05.2012
Unified Mail Subsystem
● 6.0 had 3+ independent mail stacks
● 6.1 has one unified mail stack (mail)● Per-database SMTP settings (SSL, Priority, Debug)
● Single point of extension
● Single mail queue/archive
● Common features● Scheduler
● Template (email_template)
● Composition wizard
ir.mail_server
mail.messagemail.thread
compositionscheduler
email_template
@odony – 6.1 Framework Changes – 05.2012
Many2Many auto-naming and inheritance
● What happens when you _inherit a model containing m2m field, w/ different _name?
● Problem more apparent with abstract osv_memory models
● In 6.1 many2many have auto names● rel_table, rel_from, rel_to are automatically computed
● Simpler/Partial declaration, clean inherit
● rel_table override needed for multiple relationships
@odony – 6.1 Framework Changes – 05.2012
YAML improvements (1)
● 6.1 YAML tests:● 65% coverage, 150 scenarios, 1800 test steps
● Simulating workflows works well
● But what about UI: views, on_change, … ?
● Done! YAML will now automatically use form
views and on_change methods● Better coverage
● Less code: on_change can fill in implicit values
● Can be overridden with view attribute
@odony – 6.1 Framework Changes – 05.2012
YAML improvements (2)
@odony – 6.1 Framework Changes – 05.2012
Simplified Configuration (1)
● Setup process streamlined – 1-click install● Kanban module view – ordered by sequence
● Configuration Overview Dashboard● Based on ir.action.todo and ir.actions.todo.category
● New ir.actions.todo● Reminder: group_ids, action_id, and note
● States: Todo or Done
● Types: Manual, Manual Once, Automatic
● Most are Manual or Manual Once now
● New category_id for Configuration Overview
@odony – 6.1 Framework Changes – 05.2012
Simplified Configuration (2)
@odony – 6.1 Framework Changes – 05.2012
Logging
● Python has a decent logging system● netsvc.notifyChannel() deprecated in 6.0● Convenience hierarchical loggers● One _logger per file recommended
_logger = logging.getLogger(__name__)
● New –-log-handler config parameter● --log-handler=PREFIX:LEVEL
● --log-handler=openerp.netsvc.rpc.request:DEBUG
● --log-handler=werkzeug:CRITICAL
@odony – 6.1 Framework Changes – 05.2012
Agenda
● Framework & API● Architecture changes
● API evolution
● New views and UI features
● Other important bits
● Exercises, Q&A
Agenda
@odony – 6.1 Framework Changes – 05.2012
Kanban (1)
● Kanban (カンバン ): Japanese billboard● Toyota's Lean/JIT production: “pull” or
“demand-driven” SCM● Kanban cards signal need to produce
@odony – 6.1 Framework Changes – 05.2012
Kanban (2)
● Kanban method, applied to Management● Signboard shows whole workflow at a glance
● Limit WIP (work-in-progress) for each state (Pull!)
● Monitor flow
@odony – 6.1 Framework Changes – 05.2012
Kanban (3)
● kanban views are written in Qweb● Automatic column layout with Group By● Drag & Drop reassigns grouped column● One mandatory template “kanban-box”● Can use normal OpenERP fields● Any CSS/HTML you like
@odony – 6.1 Framework Changes – 05.2012
Kanban (4)
● Typical structure
@odony – 6.1 Framework Changes – 05.2012
Kanban (5)
● Drag&Drop is nice, but group_by only shows existing data
● How about missing columns?● Solution: add a _group_by_full map● Return type must be like read(),
depending on the column type
@odony – 6.1 Framework Changes – 05.2012
Kanban (6)
● group_by_full, complex example
@odony – 6.1 Framework Changes – 05.2012
Kanban (7)
Exercise: add test kanban view for res.partner
@odony – 6.1 Framework Changes – 05.2012
Kanban (8)
● More details about Kanban API:
http://pad.openerp.com/kanban-api
(temporary location)
@odony – 6.1 Framework Changes – 05.2012
Customizable dashboards
● Dashboards now feature multiple layouts● Saved in ir.ui.view_custom, per user● Filters menu now has “Add to dashboard”● Works with kanban views too
@odony – 6.1 Framework Changes – 05.2012
New many2one widget
● Many2One fields = eternal UI problem● 6.1 widget:
● Auto-complete selection as you type
● Intuitive quick creation
● New ORM method: name_create(cr, uid, name, context)
● If fails, default to normal form creation
● Disabled with options='{"quick_create": false}'
@odony – 6.1 Framework Changes – 05.2012
New statusbar widget
● Old state selection was not user-friendly● statusbar widget shows workflow steps
● Only for selection fields● Options: list of visible values, colors
@odony – 6.1 Framework Changes – 05.2012
Agenda
● Framework & API● Architecture changes
● API evolution
● New views and UI features
● Other important bits
● Exercises, Q&A
Agenda
@odony – 6.1 Framework Changes – 05.2012
Module Manifest changes
● Web-related keys: js, css, qweb● active auto_install → (backwards-compat): 'depends' triggers!
● complexity (easy|normal|expert)
● application (bool): Application or Extra
● sequence (integer): Application install screen order
● New category: 'Hidden' – Technical, hidden by default
● Module icon for install screen/static/src/img/icon.png
● OpenERP Apps● Description in RST, one main section
● 'images': ['/path/to/screenshot.png', '/path/to/screenshot2.png']
@odony – 6.1 Framework Changes – 05.2012
EDI Subsystem (1)
● Goal: bring companies up to par with people and social network tools
● EDI module installed with Sales, Account● EDI Setup
● Company Addres & Logo
● Outgoing Email Server (SMTP)
● User emails
● Customers/Contacts emails
@odony – 6.1 Framework Changes – 05.2012
EDI Subsystem (2)
@odony – 6.1 Framework Changes – 05.2012
EDI Subsystem (3)
● Fine-tuning● Payment options
● Paypal account on company
● Bank accounts on company
● Email templates
● EDI Preview templates: Qweb
● Opt-out● Per-customer
● Global – workflow level
● Global – template
@odony – 6.1 Framework Changes – 05.2012
EDI Subsystem (4)
● EDI Document is simple JSON
● EDI engine is generic
@odony – 6.1 Framework Changes – 05.2012
Agenda
● Framework & API● Architecture changes
● API evolution
● New views and UI features
● Other important bits
● Exercises, Q&A
Agenda
@odony – 6.1 Framework Changes – 05.2012
Exercises
● Gunicorn:● Install Gunicorn and start OpenERP w/ it
● Kanban● Add a kanban view to the model of your choice
● Set a default group_by on the menu action
● Implement a working _group_by_full
● Add the Kanban view to a dashboard
● EDI● Modify an EDI Preview template to show Quotations