challenge to add openstack api validation framework · openstack community development is hard...

Challenge to add OpenStack

API Validation Framework

2013.05.31

Ken’ichi Ohmichi

NEC-Soft, Ltd.

([email protected])

NEC Corporation 2013

Agenda

▌OpenStack

▌OpenStack community development

▌OpenStack functions through RESTful API

▌Issues of RESTful APIs

▌Fail-Fast architecture

▌API validation framework

▌Development process

▌Schedule for Havana

OpenStack

▌ The Open Source Cloud Operating System▌ Controls virtual machines, storage/networking resources. ▌ Consists of multiple functional components.� Compute: Nova� VM image: Glance� Identity: Keystone� Networking: Quantum (will be renamed)� Block storage: Cinder� Object Storage: Swift� Dashboard: Horizon� Metering: Ceilometer� Orchestration: Heat

OpenStack community development

▌ The community is very active.� OpenStack community consists of many developers, users and operators.� They are cooperating each other on mailing-list, IRC and development tools such as Gerrit.� The community releases new version of OpenStack every six months, and the functions increases by each version.

OpenStack community development

▌ The development routine

1. Submit a bug-report / blueprint to OpenStack Launchpad.2. “git clone” necessary source code from OpenStack github.3. Modify the source code as you like.4. “git commit” with the id of bug-report / blueprint.5. “git review” for posting a patch into Gerrit system.6. (Reviewers) review the patch and submit their feedbacks with score on

Gerrit.Score +2: ApproveScore +1: Looks good do me, but someone else must approveScore -1: I would prefer that you didn’t merge this.Score -2: Do not merge

7. Apply reviewer’s feedback to your patch, and retry “git review” for reposting it.

8. (Reviewers) re-review the patch. When core reviewer marks it as +2, the patch is merged to OpenStack github. Congratulations☺

OpenStack functions through RESTful API

▌ Control functions through RESTful API directly.� EX) We can create a virtual machine by sending API request with following parameters to http://<openstack>:8774/<tenant-id>/servers with POST method. The parameters can be specified with JSON/XML format.

{"server": {

"flavorRef": "1","imageRef": "70a599e0-31e7-49b7-b260-868f441e862b","name": "new-server-01"

}

▌ Control functions with dashboard or command line.

▌ These tools uses RESTful API in the background. Create vm by

pushing here.

Dashboard


Issue/Solution of RESTful APIs

▌Issue� Not all API parameters are completely validated, so some API

operations execute without parameter validations.

▌Solution� By validating every API parameters just after receiving API request, it

makes better OpenStack quality.� Because the basic design of OpenStack is “Fail-Fast” architecture.

Fail-Fast Architecture

▌ Fail-Fast system does not retry/cleanup if something wrong happens.� Ex) If something wrong happens after adding a DB record during one API operation, it does not remove the added record. As the result, incomplete record remains in DB.

▌ The antonym is Fail-Safe.▌ The implementation cost of Fail-Safe tends to higher:� “What should be cleaned up when error handling?”� “What should it do when double errors (something wrong in error handling)?”

▌ On Fail-Fast system is easy-debugging because developers can know what happens easily.

Op-01

Op-02

Op-03

Record-01

add

Fail-Fast

remains

Op-01

Op-02

Op-03

Record-01

add

Fail-Safe

removeddel


API validation framework: Goals

▌I have proposed API validation framework to OpenStack community for comprehensive validation.

▌This framework goals are:� To validate every API parameter.� To return an error response before API operation, if invalid API

parameter. � To unify the error message format of the response, if the same

cause.ex) “.. is not string.", ".. is not integer."

▌By comprehensive validation just after receiving API request, itmakes better OpenStack quality because it can reduce incomplete situations like incomplete record remaining.

Merit


API validation framework: Validation Point (discussing now)

▌The API validations are executed before each API method.

▌The proposed framework is implemented with jsonschema library.▌This framework can validate requests of both XML and JSON, because jsonschema handles “dict” data after JSON/XML deserialization.

� I heard often “Can this framework handle XML request?” due to its library name.

� The answer is “Yes, it can.”

Nova

Client RouterJSON/XMLDeserialization

request

(2) Execute the selected validation

(1) Select a validation and a method

corresponding to API

(3) Execute the selected method

API ValidationAPI ValidationAPI Validation

API methodAPI methodAPI method


API validation framework: API Schema (discussing now)

▌“API schema” = API parameter data format.� Ex) The API schema of the “Change Password” API.

{'type' : 'object','properties' : {

'changePassword': {'properties' : {

'adminPass': {'type': 'string', 'required': True},}

}}

}

▌Need many API schema definitions because OpenStack has many APIs (hundreds).


API validation framework: Merits

▌In addition to the goals, there are two merits of the framework.

▌Clarification of API Parameter Definitions� The type and acceptable range and length of each parameter are clarified in the API

schema definitions.� By defining the schemas of every API, developers will be able to create applications

more easily using Nova APIs.

▌Cleaned-up Code� The amount of Nova code can be reduced by merging validations and error response

methods.� Ex) the “min_count” parameter of “create a virtual machine instance” API

try:min_count = int(str(min_count))

except ValueError:msg = _('min_count must be an integer value')raise exc.HTTPBadRequest(explanation=msg)

if min_count < 1:msg = _('min_count must be > 0')raise exc.HTTPBadRequest(explanation=msg)

'min_count': {'type': 'integer', 'minimum': 1},

Development process

▌ March, 2013� Created some prototype for Nova.� Proposed the prototype to OpenStack community.� Could listen positive responses.

▌ April, 2013� Had a session of OpenStack Design Summit for this framework in Portland, Ore.� Could make consensus for its purpose.� Could know it is good that the first target of this framework is Nova v3 API.� Nova v3 API will be implemented for Havana, and compatibility issues do not happen even if applying strict validation.


Schedule for Havana

▌Now we are discussing what is best validation mechanism.� We need more time for consensus.

▌We have 4 months until Havana-3 meaning deadline for new code.� Make Framework Core Patches.� Make API schema Definition Patches.

• This is very hard work, because Nova has many APIs.• Merge by middle of August.

we

MayApr JulyJune SepAug Oct

AD 2013

h3h1 h2

rc1rc2

rc3 Havana

Framework core

Deadline for new code

API schema definition

▌ OpenStack community development is hard work, but contribution to the community is very interesting because the community consists of skillful developers and they give good questions, advices, and answers.

▌ I am glad if many people are into OpenStackcommunity.

Thanks


Related URLs

▌Blueprint “Nova API Validation Framework”� https://blueprints.launchpad.net/nova/+spec/nova-api-validation-fw

▌Framework core patch for Oslo� https://review.openstack.org/#/c/25884/

▌Framework core for Nova� https://review.openstack.org/#/c/25358/

▌Nova v3 API� https://blueprints.launchpad.net/nova/+spec/nova-v3-api

▌How to contribute to OpenStack� https://wiki.openstack.org/wiki/How_To_Contribute

challenge to add openstack api validation framework · openstack community development is hard...

Documents