qt webkit going mobile

54
Qt WebKit goes Mobile Kenneth Rohde Christiansen FOSDEM 2011

Upload: kenneth-rohde-christiansen

Post on 11-Nov-2014

4.746 views

Category:

Technology


5 download

DESCRIPTION

WebKit is everywhere, on desktops, mobiles, - and as an integrated part of the Qt platform, it is on Linux-based MeeGo as well.n this session we will quickly introduce the WebKit project, and look at why it has become so popular on the mobile platform. We will look at some of the mobile adaptions normally done by the different WebKit ports and at the work we have been doing in Qt WebKit to make it more fit for mobile usage.Some of the technologies we will touch, includes: frame flattening, tiling (including challenges), viewport meta information, touch events and fuzzy hit testing.We hope the session will make people aware of the importance of the Qt WebKit project and its mobile efforts; create discussions and use-cases that we can use for integrating even better with, in particular, open source projects targeting mobile usage and the open source community in general.

TRANSCRIPT

Page 1: Qt WebKit going Mobile

Qt WebKit goes Mobile

Kenneth Rohde Christiansen FOSDEM 2011

Page 2: Qt WebKit going Mobile

Introduction

2

Welcome everyone!

Page 3: Qt WebKit going Mobile

Introduction

• Name is Kenneth Rohde Christiansen • Works in the Nokia, Application and Service Frameworks subentry (basically Qt Development Frameworks + browser/web teams)

• Among other, part of the Qt team developing and adapting WebKit for Nokia

3

Page 4: Qt WebKit going Mobile

Relevance to FOSDEM

• Qt and Qt WebKit are fully open source projects, and we actively encourage contributions.

• Long time Linux support, now also targeting the open source MeeGo project

4

Page 5: Qt WebKit going Mobile

Today

5

Today we will identify what is needed to make a great mobile browser

Page 6: Qt WebKit going Mobile

Today

6

… and how to accomplish that

Page 7: Qt WebKit going Mobile

Rendering Engine

7

Page 8: Qt WebKit going Mobile

First we need a great, flexible Web Engine

is our recommendation WebKit

8

Page 9: Qt WebKit going Mobile

Rendering Engine

• WebKit is already a part of Qt (and MeeGo) • It is highly modular given it is not a browser, but an API for a.o. making one

• That is what I know about ;-)

9

Why ?

Page 10: Qt WebKit going Mobile

Rendering Engine

But the most important reason: WebKit is the engine powering most of the mobile web browser in use That means: It is proven technology – it can be done Less bootstrapping – a lot of things already in place

10

Page 11: Qt WebKit going Mobile

Targeting a small form factor and different input

11

Page 12: Qt WebKit going Mobile

Adaptions

• Nokia’s track record not that great, or is it? • First to bring WebKit to mobile phones and did quite well at the point

Lot has changed since then • Availability of 3G and Wi-fi • Reduced data cost / plans • More powerful devices – with larger screens • Totally new interaction model; touch!

12

Page 13: Qt WebKit going Mobile

Adaptions

This means that, people today actually make use of their data plans The better connection and faster devices means that we can load and process normal pages and are not limited to WAP pages as before.

ie. brings the real internet to devices

13

Page 14: Qt WebKit going Mobile

Adaptions

We have lots of challenges:

•  Small form factor – things doesn’t fit, becomes too small •  Load and layout/paint is slow compared with desktop •  Touch interaction requires a very responsive user experience •  Fingers are huge – content is small

• Web apps… let’s talk about that later

14

Page 15: Qt WebKit going Mobile

Adaptions: small form factor

• We need good support for zooming

Some great ideas exists • Double tap to zoomable area (vertically centered where you clicked); zoom fully out

if trying to zoom to a similar target. •  Pinch zoom, preferable with moving – pinch and move simoustainly.

15

Page 16: Qt WebKit going Mobile

Double tap to zoomable area

16

Page 17: Qt WebKit going Mobile

Pinch zoom

17

Page 18: Qt WebKit going Mobile

Adaptions: small form factor

Did that solve our issues?

Still too small text, or text being cut when zoomed in too much.

Solutions?

Android: text reflowing iOS: enforce min. font sizes depending on the width of the zoomable area (eg. <div>).

18

Page 19: Qt WebKit going Mobile

Text reflowing

19

Not my personal favorite • Hard to get right – lots of bugs related to that

on Android • Relayouting is expensive • “interrupted” animation

Text reflowing on Android

Page 20: Qt WebKit going Mobile

Adaptions: big, imprecise fingers

• Problem: We need to interact with the content • We have a very little screen size, so we miss overview and a way to scroll to the

important area, fast.

• Solution? •  For news sites with columns: Double tap to area / back out, shows overview and

makes you go to an area fast. •  For other pages we need another solution:

•  Panning; pan around with your finger •  Added kinetics; when you lift you finger it will continue your motion with kinetics

20

Page 21: Qt WebKit going Mobile

Adaptions: big, imprecise fingers

Give visual feedback: • Scroll indicators (fade out when not in use; focus is contents) • A bounce back or similar way to show the user reached the boundaries Deal with imprecise fingers: • Ignore kinetics for minor moves – We do not want a finger ‘click’ to start kinetic

panning • Lock panning in the horizontal or vertical direction • More complicated: Improve hit testing to look for area close by.

21

Page 22: Qt WebKit going Mobile

Hit testing

Fingers click an area, not a single point as a mouse

• We need heuristics to guess where the user tries to click: This depends on finger geometry.

• We need the hit testing to look for clickable things close to the actual click.

22

Page 23: Qt WebKit going Mobile

Frame flattening

• Panning depends on the whole web content being one • Thus, we need to eliminate sub-scrollable areas

• Expand iframes to the size of their contents (done of iOS, etc) • Expand framesets to become one flattened area

Examples coming up ;-)

23

Page 24: Qt WebKit going Mobile

Frame flattening

24

No one wants to interact with a page like this

Page 25: Qt WebKit going Mobile

Frame flattening

25

Now what about this?

This is easily pannable

Page 26: Qt WebKit going Mobile

Conclusion

Touch is a very nice way to interact with web contents. It feels quite natural and easy. But there is a one big but…

You feel if the page blocks

26

Page 27: Qt WebKit going Mobile

Now why is that?

Web browsers actually stall a lot • During loading • While laying out the content • While painting • While running JavaScript

You can feel this on the desktop, but desktops are fast, and more important, you don’t touch the contents Big screen, less need for scrolling, not pinch zoom, etc

27

Page 28: Qt WebKit going Mobile

Touch requires a non-blocking UI

28

Page 29: Qt WebKit going Mobile

Back to the drawing board

• Do we need to redesign the engine? • Or is there a simpler idea

Redesigning the engine will be very complex and there is not much we can do with regard to loading.

29

Page 30: Qt WebKit going Mobile

Solution

Separate UI and web engine in different threads or processes

30

Page 31: Qt WebKit going Mobile

Two ideas

1.  Store painting sequences and replay them on the UI. 2.  Share painting area (shared memory, or memory mapped IO)

Quite complicated. We want a non blocking UI, so everything has to have an asynchronous API.

31

Page 32: Qt WebKit going Mobile

Process/thread separation

In a way we need the process separation just like e.g. Chrome, but not for security reasons (that is an extra bonus) but for the sake of a non-blocking user experience. Example: Fixed elements cannot be handled/painted by the web process as it would be invoked for each scrolling step, meaning it would block: We need to decouple the dependency on the web process for all kinds of interaction (panning, pinching, rotation, etc)

32

Page 33: Qt WebKit going Mobile

Asynchronous Tiling

Basic idea: Imagine that the contents was just one big image. Then we could easily ensure 60 fps panning, pinch zoom, etc. That is the basic idea!

33

Page 34: Qt WebKit going Mobile

Basics

34

Viewport

Full size tiles

Edge tiles

Page 35: Qt WebKit going Mobile

Basics

35

Viewport

Full size tiles

Edge tiles

Tiling means that we device our contents into tiles and we handle scrolling etc. on the UI side. Basically you can think of the contents being one big tiled image on the UI side. In practice an intelligent image which doesn’t need to paint tiles not partially visible.

Page 36: Qt WebKit going Mobile

Basics

When a tile becomes visible:

•  If painted, blit it. •  If unavailable, show some kind of indicator as background image, ie. checker

boards (known from graphics apps).

When web page area change:

•  If visible, update the changed area of the tiles (all tiles need to be in sync) – you can mark dirty on non-visible tiles.

•  The dirty regions can eventually be merged smartly.

36

Page 37: Qt WebKit going Mobile

The magic number

We want to ensure 60fps for panning, pinch zoom and rotation. Repaints, loading and JavaScript makes it hard to keep that promise. So we need to:

•  Suspend JavaScript execution while panning, pinch etc. • Defer loads •  Ignore repaints from the web process.

37

Page 38: Qt WebKit going Mobile

Pinch zooming

When zooming painting is involved (we change the scale) We cannot control how long it will take to paint – painting web contents is basically painting rectangles, lines etc… very vector graphics like. Solution:

•  Freeze the backing store (do not repaint tiles; the page is suspended) •  Scale the tiles (basically scaling an image; zooming in will show pixelated contents) • Unfreeze on end, repainting all tiles at the right scale

38

Page 39: Qt WebKit going Mobile

Rotation

Rotation is a bit tricky. For normal pages we basically multiply/divide with the device width/height factor, so that you look at the same contents.

39

Page 40: Qt WebKit going Mobile

Rotation, layouting

The iPhone gave page authors some ways to control how the page is laid out For instance, the page author can decide that the contents has a max scale: Example: Google.com has a min and max scale of 1.0, so instead of applying the difference in the width in landscape and portrait, we just rotate!

40

Page 41: Qt WebKit going Mobile

Rotation, layouting continued

41

Now what happens, this doesn’t look good!

Page 42: Qt WebKit going Mobile

Relayout is the solution

42

We need to relayout with a different width

Page 43: Qt WebKit going Mobile

Rotation

Recall that layouts are expensive, so if we layout during the rotation we cannot ensure 60 fps. Thus, only layout after rotation has finished; with the above as intermediate step (show checkerboards) 43

Page 44: Qt WebKit going Mobile

Rotation

Summing up: Rotation starts: freeze backing store / suspend JS, paint, loads During rotation: Keep old contents, show checkerboards where no contents is After rotation: Relayout (if needed) and unfreeze / resume.

44

Page 45: Qt WebKit going Mobile

Web Apps!

Let’s not get into why web apps makes sense, or when then do. That would be another presentation J If interested, check out: http://tinyurl.com/4hx36lt which is my presentation: “Connecting technology for great experiences” which touches QML and Web for application developers.

Web Apps makes sense in many cases, so how can we support them? 45

Page 46: Qt WebKit going Mobile

Web Apps

How should web apps behave? • Basically like real apps • A one to one scale – one css pixel is one device pixel – no scaling • No pinch zoom of the whole apps, it is an app not a page! • It has to be able to deal with input

46

Page 47: Qt WebKit going Mobile

Touch events

Mobile browsers support touch events in addition to mouse events This allows apps to take advantage of multi touch etc. We need to support that. This means that each event has to go to the web process and be treated before we can use it on the UI for panning etc. A bit slower. Trick: Only send events to the web process if the page is actually using touch events (have listeners registered).

47

Page 48: Qt WebKit going Mobile

Viewport meta tag

Introduced by the iPhone. • Layouts with a width of 980 by default. • Works with most sites, but some break. ie. www.apple.com which needs a

width > 1000 or so. • Added a meta tag to give control to the web authors

48

Page 49: Qt WebKit going Mobile

Viewport meta tag

Features: • Define layout width (and height, though seldom used in practice) • Define initial, minimum and maximum scale • Define non-user-scalable (basically remove pinch zoom)

Nota bene: It is defines given portrait mode.

<meta contents=“viewport” value=“width=320, maximum-scale=1”> Only comma allowed as separators, though some Android versions allowed semicolon.

49

Page 50: Qt WebKit going Mobile

Avoid hardcoded values

There was a need to avoid hardcoding values such as the the iPhone width of 320. Introduced device-width, device-height <meta contents=“viewport” value=“width=device-width, maximum-scale=1”>

50

Page 51: Qt WebKit going Mobile

DPI, a pixel is not a pixel

Facts: • Most contemporal mobile pages were designed with the iPhone in mind • Buttons, hit areas are adapted to the sizes and DPI of the iPhone which is 160.

First Android phones has the same DPI, but some low end models had lower and almost all current models have a DPI of 240. For pages designed for the iPhone, they need to be scaled up by 1.5 for text, hit areas to not be too small. (240 / 160 = 1.5) Android defines a pixel at 160 as a DIP = density independent pixel.

51

Page 52: Qt WebKit going Mobile

DPI and the meta tag

Android added the target-densitydpi extension <meta contents=“viewport” value=“width=device-width”>

on a 240 DPI device uses 320 as device-width. On the other hand <meta contents=“viewport” value=“width=device-width, target-densitydpi=device-dpi”>

uses 420.

52

Page 53: Qt WebKit going Mobile

Summing up

Making a browser for touch devices has different requirements than that of a desktop browser. Focus is touch interaction, requires touch events, engine/application separation, and many tricks to make things work smoothly. Web apps requires touch events and the ability to control whether pinch zoom etc is allowed and how the contents should be laid out.

53

Page 54: Qt WebKit going Mobile

Questions? I hope you enjoyed the show and have some good questions!

54

Feel free to send questions to [email protected] and you are welcome to join #qtwebkit at irc.freenode.net