using multitouch and sensors in java

Using multi-touch and sensors in Java

Felipe Pedroso

2

Agenda

•The future of PCs• JavaFX and Touch Support•Using JNI to work with sensors

Mobile Computing Lines are Blurring

Ultrabook™ Convertible & Detachable

Ultrabook™TabletSmartphone

Consumption Usages Creation/Productivity

Notebook

4

Reshaping the PC Experience

Ambient Light

Sensor

GPSCompass

Near Field Communicati

on

Gyroscope

Ultrabook™

Accelerometer

Multi-Touch

Context Aware Sensors

OK, they have all those features…

… but how can we implement them using Java?

6

To allow users to TOUCH my Java App...

• Should I use...• AWT?• Swing?• SWT?

• Actually, you can use them but...• They aren’t made for multi-touch (OK, there’s a way to do it)• You must optimize your UI controls to be more ‘touch friendly’

• So, how to do it in a simple and easy way?

7

8

Actions supported

• Touch events: Down, Move and Up• Tap (Down and Up) / Double tap• Drag and Drop (Down, Move and Up)• Hold

• Gestures• Swipe: Up, Down, Left and Right• Zoom: Pinch / Spread• Rotate

9

What I need to do to manipulate components?

• Your components must extend the Node class or any of its subclasses (StackPane, ImageView, etc)

• Set the proper EventHandler to handle the action.

10

What I need to do to manipulate components?

• Apply a transform to the component (Translation, Rotation and Scale):

• Let’s dive into some code!

What about the sensors?

JNI, I choose you!

12

Available sensors on Ultrabooks

• Accelerometer• Proper acceleration in three axis (x, y, z)

• Gyrometer• Device orientation

• Magnetometer• Strenght and direction of magnetic fields

• GPS (Global Positioning System)• Location and Time information

• NFC• Near Field Communication

• Ambient Light Sensor• Ambient Light Level

13

API Windows – Sensor Fusion

14

API Windows – Namespaces• Windows.Sensors.* Common sensors

• Accelerometer• Gyrometer• Inclinometer• OrientationSensor• SimpleOrientationSensor• Compass• LightSensor

• Windows.Devices.Geolocation GPS• Geolocator

• Windows.Networking.Proximity NFC• ProximityDevice

15

Windows API – How to

• Get the default object of your sensor using the GetDefault method

• You can call the GetCurrentReading() to get the current value of the sensors or...

• ... work with the ReadingChanged event• Set the attribute ReportInterval (please, respect the MinimumReportInterval to avoid problems)• Delegate a method to handle the event (something like setting a method as a listener)• Handle the event!

• This procedure is valid only for sensors from Windows.Sensors namespace

16

How can I access that?

17

Using JNI to access the sensors from Windows.Sensors

1. [Java] Create a native method to register the object that will handle the events that come from the sensor

2. Generate the header file using javah

3. [C++] Create a DLL Project in Visual Studio with the DLLs and namespaces of the Windows 8 APIS

4. [C++] Use a variable to keep a reference to the object and the ID (GetMethodID) of the method that will handle the event

18

Using JNI to access the sensors from Windows.Sensors

5. [C++] Initialize the sensor and delegate a C++ method to handle the event

6. [C++] Redirect the delegate method call to the Java method using the function CallVoidMethod with the following parameters: the Java Object, the method ID and it’s parameters

7. [Java] Handle the event!

A lot of text? Please, show me the code!

19

What about Linux?

• Touch works fine, but there aren’t clear APIs to read sensors. If you know how to do it, let’s work together!

20

Wants to know more about Intel Software?http://software.intel.com/pt-br/

21

Windows Community

http://software.intel.com/en-us/windows

22

www.javafxcommunity.com

23

JFXtras.org

Take aways

• The PC experience is being reshaped

• JavaFX + Touch = AWESOME

• Using JNI to access the sensors can be LEGENDARY!

Any doubts?(if yes, the answer is 42)

Thanks a lot!

Felipe Pedrosofacebook.com/felipe.a.pedroso

@felipeapedroso