android display stack high level overview
DESCRIPTION
DisplayTRANSCRIPT
1
Android Display Stack –High level overview ............................................................................ 1
Android UI .............................................................................................................................. 2
Android applications .............................................................................................................. 3 Android Framework Widgets ............................................................................................................................................... 3 In android Terminology… Views and ViewGroups ....................................................................................................... 3
Other android UI packages: ................................................................................................................................................... 4 Custom Graphics: ....................................................................................................................................................................... 4 More Control: ............................................................................................................................................................................... 4 Summary: ...................................................................................................................................................................................... 5
View to Surface ...................................................................................................................... 6 Summary ....................................................................................................................................................................................... 6 Surface Flinger ........................................................................................................................ 8 Producer-‐Consumer design ................................................................................................................................................... 8 Image Producers – Image Consumers (SurfaceTexture/ SurfaceTextureClient) ........................................... 8
SurfaceFlinger ............................................................................................................................................................................. 9 SurfaceFlinger Composition .................................................................................................................................................. 9 Hardware Composer ............................................................................................................. 11
Appendix 1: Miscellaneous ................................................................................................... 12 WindowManager ..................................................................................................................................................................... 12 Gralloc .......................................................................................................................................................................................... 12 FrameBuffer .............................................................................................................................................................................. 12 Back buffering .......................................................................................................................................................................... 12 Appendix 2: Android widget usage example ......................................................................... 13 Static layout (Defined in res/layout/some_layout_name.xml) ........................................................................... 13 a. Layout defined via xml file .............................................................................................................................................. 13 b. This defined view will be added on Activity’s create method. ......................................................................... 14 c. This activity will be invoked by Intent/Default as defined in Manifest ....................................................... 14
Dynamic layout (Dynamic components added/removed in code) ................................................................... 14 a. UI widgets are added in code ......................................................................................................................................... 14 b. This activity is invoked dynamically from another activity ............................................................................. 15
Android Display Stack –High level overview High level overview of different components in UI stack.
2
Android UI Typical android UI components from System Services and Android Applications: Ref: https://developer.android.com/design/get-‐started/ui-‐overview.html
1. System Services: a. Status Bar b. Navigation Bar
2. Notification Services https://developer.android.com/design/patterns/notifications.html
3. App UI a. Action Bar: https://developer.android.com/design/patterns/actionbar.html b. Navigation Drawer:
https://developer.android.com/design/patterns/navigation-‐drawer.html c. Content Area
3
Android applications
Android Framework Widgets Android applications typically use predefined android’s widgets like TextView, RadioButton etc. These are defined in android.widget package. (Link to complete list of widgets)
Figure 1: Sample android UI with basic widgets like TextView, Button
The widgets are typically laid out using predefined android layouts like LinearLayout, FrameLayout etc., Layouts are defined in android.view.ViewGroup. package. (Link to common layouts) Android allows XML based layout design. Sample code example is in appendix 1.
In android Terminology… Views and ViewGroups Widgets referred above implement android.view.View and layouts are derived from android.view.ViewGroup.
Ref: http://developer.android.com/guide/topics/ui/overview.html
4
Other android UI packages: There are several other pre-‐defined UI components in android framework that applications can use. Example. android.webkit. *, android.view. * etc.,
Custom Graphics: Obviously, pre-‐defined widgets are not the only way. Android applications can do a lot more depending on their use case:
1. Drawable: Ref: http://developer.android.com/guide/topics/graphics/2d-‐graphics.html#drawables Android’s graphics library under android.graphics.drawable package provides interfaces to draw shapes and images in a view.
2. CustomVIew: Existing android views can be customized to draw dynamically in non-‐performance intensive use cases. These app-‐defined classes will extend android’s View class, override onDraw(Canvas) to provide their own draw implementation. Convenience here is that pre-‐defined Canvas is already provided. Draws should simply be placed on this.
3. SurfaceView: This extends the android View class and is used to perform operations on a secondary non-‐UI thread to maintain usablility.
4. Canvas: Drawing custom graphics on a user defined Canvas is useful in performance intensive apps like video games. A new canvas can be created from a bitmap.
More Control:
1. OpenGL: If android applications want to render complex graphics like 3D animation and are short of framework APIs, they can directly use OpenGL using the wrapper APIs: http://developer.android.com/training/graphics/opengl/index.html Package reference: http://developer.android.com/reference/android/opengl/package-‐summary.html
2. Renderscript: For graphical computation intensive applications that perform image processing etc., RenderScript framework is provided. These are built on top of Hardware composers. http://developer.android.com/guide/topics/renderscript/compute.html
5
Summary: In a nutshell, there is wide variety of android applications. (Basic, performance intensive, containing video, image processing etc.,) Depending on their use case, these use different UI alternatives permissible in android UI stack.
1. Use predefined widgets and components 2. Draw custom graphics on a new canvas or define custom widgets (Drawables) and
custom views from existing Views and pre-‐defined Canvas. 3. Display graphics using openGL directly against the android framework APIs. 4. Use RenderScript framework.
Figure 2: Apps creating surfaces in different possible ways
6
View to Surface View is converted to a Surface/Texture/Buffer Data by the JNI libraries. Each type of UI component, has its own JNI implementation libraries, ex., GLSurfaceView is implemented on OpenGL; TextView on Canvas -‐> Skia/OpenGL, RenderScript on OpenGL etc., Following diagrams visually represents this architecture.
Drawing happens in two steps. I) Measuring, where each of the child components in the tree rooted at the View are validated in-‐order on the layout. II) Layout, each of the child components are positioned in the layout based on the measurements in step I. Ref: https://developer.android.com/guide/topics/ui/how-‐android-‐draws.html
Summary 1. Activity receives focus -‐> Draws its layout. 2. Starts from the root node in the layout. 3. Traverses in-‐order across the layout hierarchy tree from ViewRoot. 4. Every node in the tree can be another Layout(ViewGroup) or a View(a Drawable)
a. ViewGroup => Draw each of my children in this order. b. View => Draw me in this manner.
5. Happens in two passes. a. Measure to compute size. b. Layout to position based on the size in step a.
6. Draw equals converting it to buffer data/off-‐screen surface, which is consumed by SurfaceFlinger
7
8
Surface Flinger
Producer-‐Consumer design SurfaceFlinger implements Android’s ImageConsumer in the Image Producer/Consumer design.
Image Producers – Image Consumers (SurfaceTexture/ SurfaceTextureClient) Image Stream Producers (SurfaceTextureClient) include any process that produces a graphical buffer. Image Stream Consumers (SurfaceTexture) refer to any process that are capable of consuming the graphical buffers from producers. Binder binds these producers (SurfaceTextureClient) and their consumers (SurfaceTexture). Java doc: http://developer.android.com/reference/android/graphics/SurfaceTexture.html Ref: https://source.android.com/devices/graphics.html Examples:
1. View-‐Finder/ Camera preview produces images. Camera app consumes these graphical buffers to save the pictures.
2. All UI applications that draw something are image producers; SurfaceFlinger, being a system service is a consumer with special privilege to consume and modify the graphical buffers from all these image producers.
9
SurfaceFlinger 1. SurfaceFlinger is yet another android openGL native program in its implementation. 2. It can do something more, being a system service running all the time. 3. This system service is not accessible from applications. (Some services are.) 4. It is an ‘Image Consumer’ of all images (Offscreen surfaces) drawn to display driver. 5. ‘Image Producers’ include application surfaces (typical application content area)
and system services (Status bar etc.,) 6. SurfaceFlinger is also responsible for allocating these surfaces. 7. It is the only Image Consumer that has special privileges to modify the image stream
before it goes to the display driver. 8. SurfaceFlinger is responsible ONLY FOR COMPOSITING the surfaces and NOT FOR
RENDERING Views to Surfaces. 9. It works with ‘Window Manager’ another system service to get metadata on the
window like z-‐ordering etc., 10. It does composition using either Hardware Composer or OpenGL. Hw Composer
exploits GPU, improves performance. OpenGL handles complex composition requirements like 3D.
11. The modified/composited surface is put in system buffer = seen on screen. 12. It runs indefinitely waiting to pick a screen update and is a contributor to battery
drain. Ref: http://pierrchen.blogspot.in/2014/02/what-‐is-‐surfaceflinger-‐in-‐android.html
SurfaceFlinger Composition
10
11
Hardware Composer Why SurfaceFlinger should offload some of its composition to hardware?
1. Hardware Composition will happen in parallel on a dedicated hardware, while it works on next layers on its openGL implementation using CPU cycles.
2. Power saving Ref: http://pierrchen.blogspot.jp/2014/02/android-‐graphic-‐composition-‐hardware.html Implementation of hardware composers is certainly hardware dependent, but Google has guidelines: https://source.android.com/devices/graphics.html
12
Appendix 1: Miscellaneous
WindowManager
Gralloc
FrameBuffer This the actual abstraction of the memory mapped region in hardware framebuffer (/dev/fb0)
Back buffering Explanation ref: http://www.kandroid.org/online-‐pdk/guide/display_drivers.html
13
Appendix 2: Android widget usage example Sample UI on a basic Android application: Reference: http://developer.android.com/training/basics/fragments/index.html https://developer.android.com/training/basics/firstapp/building-‐ui.html Output: Snippets below is to display the below widget and display the text entered on clicking Send.
Figure 3Output of basic UI app with TextBox
User applications add UI components in two ways:
Static layout (Defined in res/layout/some_layout_name.xml)
a. Layout defined via xml file HelloWorld/res/layout/activity_main.xml
<?xml version="1.0" encoding="utf-8"?> <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent" android:orientation="horizontal" > <EditText android:id="@+id/edit_message" android:layout_width="0dp" android:layout_height="wrap_content" android:layout_weight="1" android:hint="@string/edit_message" /> <Button android:layout_width="wrap_content" android:layout_height="wrap_content" android:onClick="sendMessage"
14
android:text="@string/button_send" /> </LinearLayout>
b. This defined view will be added on Activity’s create method.
/HelloWorld/res/layout/activity_main.xml public class MainActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.activity_main); } … }
c. This activity will be invoked by Intent/Default as defined in Manifest HelloWorld/AndroidManifest.xml
<activity android:name="com.example.helloworld.MainActivity" android:label="@string/app_name" > <intent-filter> <action android:name="android.intent.action.MAIN" /> <category android:name="android.intent.category.LAUNCHER" /> </intent-filter>
</activity>
Dynamic layout (Dynamic components added/removed in code)
a. UI widgets are added in code HelloWorld/src/com/example/helloworld/DisplayMessageActivity.java public class DisplayMessageActivity extends Activity { @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); // Get the message from the intent
15
Intent intent = getIntent(); String message = intent.getStringExtra(MainActivity.EXTRA_MESSAGE); // Create the text view TextView textView = new TextView(this); textView.setTextSize(40); textView.setText(message); // Set the text view as the activity layout setContentView(textView); } }
b. This activity is invoked dynamically from another activity /HelloWorld/res/layout/activity_main.xml <Button android:layout_width="wrap_content" android:layout_height="wrap_content" android:onClick="sendMessage" android:text="@string/button_send" /> HelloWorld/src/com/example/helloworld/MainActivity.java /** Called when the user clicks the Send button */ public void sendMessage(View view) { Intent intent = new Intent(this, DisplayMessageActivity.class); EditText editText = (EditText) findViewById(R.id.edit_message); String message = editText.getText().toString(); intent.putExtra(EXTRA_MESSAGE, message); startActivity(intent); }