thesis report - strobed ir illumination for image quality …1252323/fulltext01.pdf ·...
TRANSCRIPT
Strobed IR Illumination for Image Quality Improvement in Surveillance Cameras
STEVE DARMADI
KTH ROYAL INSTITUTE OF TECHNOLOGY
I N F O R M A T I O N A N D C O M M U N I C A T I O N T E C H N O L O G Y
DEGREE PROJECT IN EMBEDDED SYSTEMS
SECOND LEVEL, 30 CREDITS
STROBED IR ILLUMINATION FOR IMAGE QUALITY IMPROVEMENT IN
SURVEILLANCE CAMERAS
THESIS REPORT
by
STEVE DARMADI
SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE
KTH ROYAL INSTITUTE OF TECHNOLOGY
-ii-
ABSTRACT
Infrared (IR) illumination is commonly found in a surveillance camera to improve night-time recording
quality. However, the limited available power from Power over Ethernet (PoE) connection in network-
enabled cameras restricts the possibilities of increasing image quality by allocating more power to the
illumination system.
The thesis explored an alternative way to improve the image quality by using strobed IR illumination.
Different strobing methods will be discussed in relation to the rolling shutter timing commonly used in
CMOS sensors. The method that benefits the evaluation scenario the most was implemented in a prototype
which is based on a commercialized fixed-box camera from Axis. The prototype demonstrated how the
synchronization of the sensor and the strobing illumination system can be achieved.
License plate recognition (LPR) in a dark highway was chosen as the evaluation scenario and an analysis on
the car movements was done in a pursue of creating an indoor test. The indoor test provided a controlled
environment while the outdoor test exposed the prototype to real-life conditions. The test results show that
with strobed IR, the output image experienced brightness improvement and reduction in rolling shutter
artifact, compared to constant IR. The theoretical calculation also proved that these improvement does not
compromise the average power consumption and eye-safety level of the illumination system.
Keyword: surveillance camera, infrared, strobing, rolling shutter, license plate recognition
-iii-
SAMMANFATTNING
Infraröd (IR) belysning påträffas ofta i övervakningskameror för att förbättra bildkvalitén vid
videoinspelning på natten. Den begränsade tillgängliga effekten från Power over Ethernet-anslutningen
(PoE) i nätverksaktiverade kameror sätter dock en övre gräns för hur mycket effekt som kameran tillåts
använda till belysningssystemet, och därmed hur pass mycket bildkvalitén kan ökas.
I detta examensarbete undersöktes ett alternativt sätt att förbättra bildkvalitén genom att använda blixtrande
(eng: ”strobed”) IR-belysning. Olika strobe-metoder undersöktes i relation till rullande slutare, vilket är den
slutar-metod som vanligtvis används i CMOS-sensorer. Den metod som gav mest fördelaktiga resultat vid
utvärdering implementerades i en prototyp baserad på en kommersiell nätverkskamera av Fixed box-typ
från Axis Communications. Denna prototyp visade framgångsrikt ett koncept för hur synkronisering av
bildsensorn och belysningssystemet kan uppnås.
Registreringsskyltigenkänning (LPR) på en mörk motorväg valdes som utvärderingsscenario och en analys
av bilens rörelser gjordes för att skapa en motsvarande testuppställning inomhus. Inomhustesterna gav en
kontrollerad miljö medan testerna utomhus utsatte prototypen för verkliga förhållanden. Testresultaten visar
att med strobed IR blev bilden från kameran både ljusare och uppvisade mindre artefakter till följd av
rullande slutare, jämfört med konstant IR-belysning. Teoretiska beräkningar visade också att dessa
förbättringar inte påverkar varken kamerans genomsnittliga effektförbrukning eller ögonsäkerheten för
belysningssystemet negativt.
Nyckelord
övervakningskamera, infraröd, strobing, rullande slutare , registreringsskyltigenkänning
-iv-
ACKNOWLEDGMENTS
This thesis project was carried out in Axis Communication AB, so my thanks go to these gurus:
• Jenny Karlsson, for LPR advice and the outdoor test company,
• Andreas Karlsson, who helped us on PLD problems,
• Anders Svensson, who made sure that everything ran smoothly,
• Christian Adielsson, Johan Kjörnsberg, and Ola Synnergren, the triplet which guided every of our
baby steps.
On the KTH side, my gratitude goes to Mark T. Smith who reassured us about the administration and
academic point-of-view. I would also thank all my life support in Lund: Sofia Collberg, Fei Shenyang, Martin
Nilsson, Karolis Poskus, Zhou Zhuo Hang, and the Indonesian communities in Skåne and Stockholm.
Now it comes the most important part. Carlos Tormo is the man whom I successfully persuaded to be part
of the project. So, countless thanks go to him for his 24/7 passion in making the project both enjoyable and
memorable.
-v-
ABBREVIATIONS
ADC Analog to Digital Converter
CCD Charge-coupled Device
CMOS Complementary Metal–Oxide–Semiconductor
EIT Extend Integration Time
FPS Frame per Second
IP Internet Protocol
IR Infrared
I/O Input/output
LPR License Plate Recognition
LVDS Low-voltage Differential Signaling
MCU Micro-controller Unit
PD Powered Device
PLD Programmable Logic Device
PoE Power over Ethernet
PWM Pulse-width Modulation
RPM Rotation per Minute
STS Shared-time Strobing
WDR Wide Dynamic Range
WFS Whole-frame Strobing
-vi-
TABLE OF CONTENT
Abstract ........................................................................................................................................................................... ii
Sammanfattning ............................................................................................................................................................ iii
Acknowledgments ........................................................................................................................................................ iv
Abbreviations ................................................................................................................................................................. v
Table of Content .......................................................................................................................................................... vi
1 Introduction .......................................................................................................................................................... 1
1.1 Problem Statement ................................................................................................................................... 2
1.2 Goal............................................................................................................................................................. 2
1.3 Benefits, Ethics, and Sustainability ........................................................................................................ 2
1.4 Methodology .............................................................................................................................................. 2
1.5 Scope and Limitation ............................................................................................................................... 3
2 Strobing Methods ................................................................................................................................................. 4
2.1 Whole-frame Strobing .............................................................................................................................. 5
2.2 Shared-time Strobing ................................................................................................................................ 7
2.3 Method comparison ................................................................................................................................. 8
3 License Plate Recognition .................................................................................................................................10
4 Evaluation Method ............................................................................................................................................17
4.1 Chosen Strobing Method ......................................................................................................................17
4.2 Indoor Test ..............................................................................................................................................20
4.3 Outdoor Test ...........................................................................................................................................21
5 System Implementation .....................................................................................................................................24
5.1 LED Driver .............................................................................................................................................24
5.2 Strobe Signal Generation .......................................................................................................................24
5.2.1 Sync codes .........................................................................................................................................25
5.2.2 PLD and ARTPEC Modification ..................................................................................................27
5.3 Eye-safety .................................................................................................................................................30
6 Results and Analysis ...........................................................................................................................................31
6.1 Indoor Test ..............................................................................................................................................31
6.2 Outdoor Test ...........................................................................................................................................32
6.3 Implementation Feasibilities .................................................................................................................36
7 Conclusion and Future Work ...........................................................................................................................37
7.1 Conclusion ...............................................................................................................................................37
7.2 Future Work ............................................................................................................................................37
Bibliography .................................................................................................................................................................38
Appendix .......................................................................................................................................................................39
-1-
1 INTRODUCTION
Many surveillance cameras are equipped with an adjustable IR-cut filter to improve usability in low-light
condition. During daytime, this filter is enabled to avoid color misinterpretation due to sensor sensitivity
outside the visible light range. When the scene gets darker than a certain level, the filter is disabled (usually
by moving the filter glass off the sensor) to allow the sensor to collect more light from any wavelength
within the sensor sensitivity. The infrared light, however, will distort the overall perceived image. Hence,
the output image is converted to grayscale when the filter is removed.
However, in some darker conditions, this method is not good enough to produce an acceptable image.
Longer shutter times and higher sensor gains are usually set to get more details. In other cases where there
is virtually no light source (or its reflection) in the scene, a near-infrared (IR-A) light emitter is added either
built-in or as an external accessory.
Being a pioneer in IP cameras, Axis Communications AB (Axis) is a company which focuses on network-
enabled products. The ability to have data and power connections through the same cable (i.e., PoE) is one
of the main selling points for their current main business (i.e., video surveillance). The convenience of
installation and cabling system, however, has an impact on the power that the camera can use. The most
common PoE standard (class 3, type 1, 802.3af) can supply at most 12.95 W to the powered device,
restricting the power consumption of different sub-systems in the camera, like the IR illumination.
As an example, one of Axis network cameras has a built-in IR emitter which draws 2A at 1.9V [1]. The
power is delivered to the LED through two power stages (PoE 48V to 3.3V and LED driver) which create
10% losses in each stage according to an internal testing data. Therefore, using equation (1), it is logical to
assume that the IR illumination costs almost 5W.
𝑃𝑎𝑐𝑡𝑢𝑎𝑙 = 𝑃𝑖𝑑𝑒𝑎𝑙 × 𝜇1 × 𝜇2 (1)
Meanwhile, other component groups also consume large portions of available power, as listed in Table 1.
Axis overcomes this problem by carefully scheduling group operations to avoid unwanted bootup due to
power failure. Given the strict power condition, increasing the IR emitter power (to get better image quality)
without using additional strategy is unfavorable.
Table 1. Components Power Consumption in One of Axis’ Built-in IR Network Camera
Group Power (W) ratio to PD PoE (%)
PoE PD (class 3, type 1) (available power) 12.95 100
Heater 4.2 32
Iris adjustment 1.8 14
Lens zoom and focus 1 8
IR illumination 5 39
other functions (e.g., MCUs) 7.1 55
Due to the high power-demand, building a camera with built-in IR illumination would be challenging,
especially when recording fast moving objects in a low-light environment. Hence, Axis is looking for ways
to improve the image quality through strobing the IR light. Strobing is an illumination technique which
consists of emitting flash of light in a cyclic manner. This method is expected to improve the level of
luminous energy delivered to the object by only emitting IR when the sensor is acquiring the image
(integrating). The non-integrating time will save energy and thus, enable designers to choose a more
powerful IR emitter.
The reduction of exposure time in strobing can potentially reduce the hazard introduced to the human eye.
According to IEC-62471 (Photobiological Safety of Lamps and Lamp Systems), particularly its application on
-2-
repetitively pulsed devices [2], the system can be evaluated by averaging the pulsed emission. Therefore,
image quality improvements through IR power increment can be less harmful using strobed IR.
1.1 Problem Statement
The main question brought from the background is:
Can the image quality of surveillance cameras be improved by utilizing strobed IR?
This high-level question can be broken down into several sub-questions as follows:
1. How to synchronize frame capture (sensor) with strobed IR light?
2. How does strobed IR performs compared to constant IR regarding image quality?
1.2 Goal
The research aims to build a prototype which can answer the questions mentioned in the problem statement.
Furthermore, it also aims to:
• Prove that IR illumination can be well synchronized with sensor timing.
• Present strobing method/s that can provide significant improvements in image quality.
• Evaluate image quality difference between constant light and proposed strobing methods in a
related use case.
1.3 Benefits, Ethics, and Sustainability
The thesis presents an early verdict of strobed IR implementation for surveillance camera industries,
especially from image quality perspective. Possibilities of reducing motion blur and improving object
brightness in very challenging situations will bring more-detailed footages to the user. This means that the
camera would provide more information and be more reliable in surveillance activities. On the other hand,
the strobing function would not create new ethical questions, as it does not alter how the camera works as
a surveillance device. Concerns about eye safety for people who are exposed to the camera’s IR radiation
will be addressed in Section 5.3.
1.4 Methodology
The methodology used resembles a conceptual research method with five research stages, as shown in
Figure 1. The idea of strobing IR light was built on top of an existing surveillance camera system. Hence,
the first stage of the research is understanding how the system works through a literature study. The second
stage is closely related with the first one. It consists of shutter timing analysis to produced new strobing
methods that suit the rolling shutter. This stage also includes analysis on object movement in License Plate
Recognition (LPR) which produces the speed transformation formulas that supports the development of
the evaluation method.
The third stage is the evaluation method itself. It consists of two different tests: an indoor test, which
simulates the license plate characters’ movement to check the prototype performance before the real LPR
outdoor test. The fourth stage consists of prototyping activities, where a commercial product will be
modified to produce the intended strobing function for both tests. In this stage, an eye-hazard calculation
will also be done with the implemented specification. The prototype would be evaluated with the tests, and
later the test results will be analyzed qualitatively and presented.
-3-
Figure 1. Research Methodology
1.5 Scope and Limitation
Strobing light itself can be beneficial for many applications including the well-known flashbulb in still-image
photography, and time-of-flight based distance measurement[3]. In video surveillance application, light
strobing opens possibilities for image quality and power efficiency improvement.
However, these two benefits could act as opposite perspectives. This thesis focuses on discovering image
quality improvements while keeping the power consumption below or at least similar with constant IR.
Meanwhile, work on power efficiency perspective was conducted by another writer [4] during the same
timeframe. Although the two works are closely related since the prototype implementation was based on
the same platform, many differences can be found. One example is the usage of Whole-frame Strobing
(WFS) technique in this work, is limited to the theoretical overview. Comparison between strobing
techniques (Section 2.3) concluded that it gives no meaningful benefit on image quality. However, the use
of WFS can be found in the other writer’s work as it is beneficial for the power efficiency improvement.
Other research limitations are explained in the following points:
• The prototype is built on a commercialized Axis fixed-box camera which will be addressed as the
development platform for the rest of report. This model is using Programmable Logic Device
(PLD) to bridge the sensor and main microcontroller. Hence, it will ease the strobing signal
generation (access to modification, reliability, and complexity compared to implementing it in
microcontroller). For test purposes, the development platform will be equipped with a zoom
telephoto lens (f=12.5-50mm).
• Any development related to sensor timing will be based on one capture mode (1920x1080
@25FPS, readout rate = 60FPS). More details are discussed in Section 4.1.
• The scenario chosen to evaluate the prototype is License Plate Recognition (LPR). A Spanish EU
license plate is used as a reference (due to its availability).
• The research will focus on image quality improvement, which will be evaluated qualitatively.
Therefore, analysing the LPR algorithm is outside the scope of the project.
Literature Study
Development platform, rolling shutter, LED driver
Strobing Method and LPR Analysis
Two strobing methods and speed tranformation formulas
Evaluation Method
Indoor and Outdoor test
Implementation (prototyping)
Strobing signal generation and eye-hazard calculation
Test and result analysis
Image quality comparison between strobed and constant IR
-4-
2 STROBING METHODS
This part of the report will discuss possible strobing methods suitable for rolling shutter sensors. It should
be noted that the analysis provided here is based on the simplified assumption that the recording scene is
totally dark (no other source of light except the IR emitter).
Rolling shutter is a sensor timing commonly found in Complementary Metal Oxide Semiconductors
(CMOS) sensors. Many modern imaging devices, including Axis cameras, utilize this sensor type due to its
faster readout, lower heat, and reduced power consumption compared to Charge Coupled Devices (CCD)
sensors [5]. However, CMOS sensors generally adopt rolling shutter, as opposed to global shutter, which is
found in CCD. The use of this mechanism introduces the so-called rolling shutter artifact, shown in Figure 2.
The artifact is introduced as the pixels are integrated (collect or exposed to light) in a per-row basis.
Figure 2. Images with (left) and without (right) Rolling Shutter Artifact
note: both images are captured by the same camera, but the right picture is produced by using shared-time
strobing to simulate a global shutter.
This behavior is unique to CMOS sensors, as shown in Figure 3. The type of sensor has several Analog to
Digital Converters (ADC)s, each serving a column of pixels. The ADC array will digitalize one column at
the same time and make the digital output accessible, before proceeding to the next line. This digitalization
and output-providing process is called the readout (represented in blue boxes in Figure 4). The delay between
readouts of different lines forces the lines to integrate light (collect photons) at a slightly different time
(approx. 14.82uS at 60FPS-full HD, more details in Section 4.1) which eventually will create the rolling
shutter artifact. This effect will be more pronounced if the captured objects are faster.
While rolling shutter creates a distinct artifact, it also introduces a limitation on how the IR light can be
strobed in a synchronized manner. For a given integration time (𝑇𝑖), every line in the sensor will accumulate
light for the same duration (e.g., 8 time-units in Figure 4). The time unit will be further called as H-unit,
which also represents the smallest time difference that the sensor can work with, depending on the selected
drive mode (discussed in Section 4.1).
Figure 3. CCD and CMOS Sensor Basic Schematic
-5-
Figure 4. Rolling Shutter Timing
In constant IR, the emitter will be always active, including the entire integration time. This means that, every
line will receive the same amount of light from the emitter, regardless some artifacts which are caused by
moving objects. Meanwhile, in strobed IR, the emitter will be turned on and off periodically. Thus, the next
question is when the emitter should switch state.
If there was a disparity between the length of the strobe experienced by different lines, two different
unwanted effects would happen. The first effect is intensity gradient, as explained in [6]. As shown in Figure
5, the upper lines receive longer strobe compared to the lower ones. As a result, the upper part of the picture
will appear brighter. The lowest line will instead be completely dark because of the light absence. The second
effect is motion blur variation. If a long strobe length is used to capture a fast moving object, part of the
object that is exposed longer by the light will form a longer trail (blurrier).
Figure 5. Unsynchronized Strobing Light Timing
To avoid these effects, the strobe length (𝑇𝑂𝑁) should equally overlap with each line’s integration time. The
next section will discuss two different methods to fulfill this requirement: whole-frame strobing (WFS), and
shared-time strobing (STS).
2.1 Whole-frame Strobing
The idea of WFS is to turn off the emitter at times when none of the lines are integrating. Since during the
integration time the emitter works the same way as constant IR, theoretically this method will not alter the
image output. Energy saving will happen during the off cycle, and hence the average power consumption
can be reduced. In Figure 6, each frame has 9H frame period (𝑇𝐹𝑃). The H unit is defined as the smallest
time step which the sensor can differentiate (more about this are explained in section 4.1). However, the
integration only happens at 4 ≤ 𝑡 ≤ 8, so afterwards, the IR light can be turned off until the integration of
next frame occurs (𝑡 = 13).
The minimum strobe length for the given scenario in Figure 6 is:
𝑇𝑂𝑁_𝑚𝑖𝑛 = 𝑇𝑖 + (𝑁𝑣𝑎𝑙𝑖𝑑 − 1) (2)
= 2 + (4 − 1)
4 Integration time
3 Shutter timing
2 Readout
1
1 2 3 4 5 6 7 8 9 10 11 12 13
shared time (Ts)
integration time
line number
t[H]
IR state
4 Integration time
3 Shutter timing
2 Readout
1
1 2 3 4 5 6 7 8 9
FRAME PERIOD
IR OFF IR ON IR OFF
line number
t[H]
-6-
= 5
Moreover, the average power consumption for WFS in this case is:
𝑃𝐴𝑉𝐺 =𝑇𝑂𝑁
𝑇𝐹𝑃× 𝑃𝐿𝐸𝐷 (3)
=5
9 𝑃𝐿𝐸𝐷
With WFS activated, 𝑃𝐴𝑉𝐺 is almost half of the original consumption, 𝑃𝐿𝐸𝐷. However, this number will vary
much as 𝑇𝑂𝑁 depends heavily on the frame period, the integration time, and the number of valid lines
(𝑁𝑣𝑎𝑙𝑖𝑑). Valid lines are lines that contains image data, as opposed to blanking lines which exist just to justify
the sensor timing. Discussion about the sensor data sequence will be covered in Section 4.1.
In another case, as shown in Figure 7, a higher 𝑁𝑣𝑎𝑙𝑖𝑑 will force the integration time to extend further
towards the previous frame, leaving no time for off state and thus, the system becomes a constant IR again.
Similar condition happens when using a longer integration time. To keep the readout position consistent,
the extension of integration time happens backwards (towards the previous frame). Therefore, the free time
between adjacent frames is decreasing, so less off time would be available as seen in Figure 8 .
Figure 6. Whole-Frame Strobing
Figure 7. Whole-frame Strobing with More Lines
IR state
4
3
2
1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Integration time Shutter timing Readout
FRAME 1 FRAME 2 FRAME 3
IR ON IR OFF IR ON IR OFF IR ONIR OFF
line number
t[H]
IR state IR OFF
8
7
6 Integration time
5 Shutter timing
4 Readout
3
2
1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
FRAME 1 FRAME 2
IR ON
line number
t[H]
-7-
Figure 8. Whole-frame Strobing with Longer Integration Time
2.2 Shared-time Strobing
Another method that can be used to fit the strobing pulse is the Shared-time Strobing (STS). This method
relies on the fact that, in some conditions (e.g., Figure 9), the integration time is long enough to provide a
common integration time for all lines (shared-time). Similar ideas also appear in earlier research such as [6]
and [7], but with different motivations.
At the positive side, STS will be significantly more efficient compared to WFS. The method will make sure
that every line is utilizing the emitted light whenever the IR is on. Moreover, the strobed IR will create a
virtual global shutter (removing the rolling shutter artifacts) as every line is considered being exposed only
when the emitter is on. Nevertheless, the minimal requirement for integration time could be huge compared
to the strobe length itself.
As an example, consider having an initial scene with constant IR as seen in Figure 10. Each frame is having
2H of integration time. To keep the brightness level, extension of integration time (EIT) must be done to
provide equivalent shared time. The result will look similar to Figure 9. The minimum required integration
time can be calculated using equation (4).
𝑇𝑖_𝑚𝑖𝑛 = 𝑇𝑂𝑁 + (𝑁𝑣𝑎𝑙𝑖𝑑 − 1) (4)
= 2 + (4 − 1)
= 5
The formula also shows that the 𝑇𝑖_𝑚𝑖𝑛 is very dependent to the 𝑁𝑣𝑎𝑙𝑖𝑑. If a full-HD sensor with 1080 lines
is used, the resulting 𝑇𝑖_𝑚𝑖𝑛 is 1082. For a large 𝑇𝑂𝑁, this method would be unfavorable, as 𝑇𝑖_𝑚𝑖𝑛 would
be very large and tend to grow beyond the available frame period. Moreover, a too large 𝑇𝑂𝑁 would make
the system act like the constant IR. No significant improvement can be expected from this technique, unless
the conditions is restricted to short 𝑇𝑂𝑁.
IR state IR OFF IR OFF
4
3
2
1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Integration time Shutter timing Readout
FRAME 1 FRAME 2 FRAME 3
IR ON IR ON IR ON
line number
t[H]
-8-
Figure 9. Shared-time Strobing
Figure 10. Example Case Requiring EIT for Shared-time Strobing
2.3 Method comparison
Notice that the formula in STS (4) is very similar to (2) in WFS, with the position of 𝑇𝑖_𝑚𝑖𝑛 and 𝑇𝑂𝑁 being
swapped. The complete formulas after taking the upper limit (𝑇𝐹𝑃) into account can be found in (5) and
(6) for WFS and STS respectively. The upper limit in (6) protects the system from dropping the frame
rate, while in (5), the upper limit becomes a boundary between WFS and constant IR.
𝑇𝑖 + (𝑁𝑣𝑎𝑙𝑖𝑑 − 1) ≤ 𝑇𝑂𝑁 ≤ 𝑇𝐹𝑃 (5)
𝑇𝑂𝑁 + (𝑁𝑣𝑎𝑙𝑖𝑑 − 1) ≤ 𝑇𝑖 ≤ 𝑇𝐹𝑃 (6)
Having these equations sorted out, the energy consumption ratio (compared to constant IR) of each
strobing method can be analyzed. The general energy consumption ratio is:
Γ =𝑇𝑂𝑁
𝑇𝐹𝑃
(7)
For WFS, the largest energy reduction will happen when the shortest 𝑇𝑖 is used for a given 𝑁𝑣𝑎𝑙𝑖𝑑.
Meanwhile, there is no reduction at all when WFS is used for tight timing scenario as in Figure 7
(𝑇𝑂𝑁 = 𝑇𝐹𝑃). The energy reduction ratio range for WFS can be written as follows:
𝑇𝑖 + (𝑁𝑣𝑎𝑙𝑖𝑑 − 1)
𝑇𝐹𝑃≤ Γ𝑊𝐹𝑆 ≤ 1 (8)
While for STS, the situation is more dynamic since 𝑇𝑂𝑁 is the first thing to set.
0 ≤ Γ𝑆𝑇𝑆 ≤𝑇𝑂𝑁
𝑇𝐹𝑃
(9)
IR state
4
3
2
1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
Integration time Shutter timing Readout
FRAME 1 FRAME 2 FRAME 3
IR ON IR OFF IR ON IR OFF IR ON IR OFFIR OFF
line number
t[H]
IR state
4
3
2
1
1 2 3 4 5 6 7 8 9
FRAME PERIOD
IR ON
line
t[H]
-9-
However, the limitation of STS is not on energy reduction but instead on the maximum achievable 𝑇𝑂𝑁.
Condition (10) should be fulfilled to prevent frame rate drop, and substituting this into (4) will yield
𝑇𝑂𝑁_𝑚𝑎𝑥(11). With a long 𝑇𝑂𝑁, the requirement of 𝑇𝑖_𝑚𝑖𝑛 (equation (4)) will be enormous. The tendency
of requiring large 𝑇𝑖 will force the sensor to collect more light and hence, making STS sensitive to motion
blur if there is any ambient light source.
𝑇𝑖 ≤ 𝑇𝐹𝑃 (10)
𝑇𝑂𝑁_𝑚𝑎𝑥 = 𝑇𝐹𝑃 − (𝑁𝑣𝑎𝑙𝑖𝑑 − 1) (11)
Choosing between one of the previous methods is merely prioritizing between small 𝑇𝑖 (short integration
time) or small 𝑇𝑂𝑁 (short strobing length). However, the priority will vary on the shooting condition.
Table 2 gives some examples of requirements in three different conditions.
Table 2. Suitable Method for Different Shooting Conditions
Condition Requirements Suitable method
High-speed
object;
low ambient
light
𝑇𝑖 does not matter (if smaller than
𝑇𝐹𝑃) as ambient light is not significant
compared to IR emitter brightness.
However, 𝑇𝑂𝑁 should be small.
STS.
As a small 𝑇𝑂𝑁 is mandatory, the LED average
power can be greatly reduced. Alternatively, if
the LED average power is kept the same, the
peak power can be increased to improve scene
brightness.
High-speed
object;
medium
ambient
light
𝑇𝑖 should be small since ambient light
is significant. 𝑇𝑂𝑁 should be long
enough to cover the integration time.
WFS.
The IR average power reduction depends
heavily on 𝑁𝑣𝑎𝑙𝑖𝑑 and 1H period.
High-speed
object;
high
ambient
light
𝑇𝑖 should be small since ambient light
is significant.
WFS or no IR at all.
If the shooting condition is very bright, there is
a chance to get good image quality without any
IR light.
-10-
3 LICENSE PLATE RECOGNITION
LPR is one of the video surveillance scenarios where the recorded objects (vehicles) are very dynamic. The
system works by feeding an image or a sequence of images (e.g., video) to an algorithm, in which a series of
image processing stages and optical character recognition are done. The scenario of LPR itself can vary
depending on the surveillance objectives, such as toll or parking payment, traffic surveillance, and road-rule
enforcement.
The quality of the input image plays an essential role in the success of LPR [8]. Some applications (i.e., traffic
surveillance) require the system to work real-time in different conditions (outdoor, day to night). This means
that the camera used should be designed to yield blur-free and reasonably bright images, regardless of the
availability of ambient light.
This requirement fits perfectly on possible improvements that strobing can achieve. As discussed in Chapter
2, strobing works best when the required integration time is short. On the other hand, the efficiency boost
that strobing provides might be utilized to improve the brightness of the license plate, and/or to reduce the
motion blur. LPR with strobing becomes even more relevant when designing a built-in IR camera which
has a tight power limit.
After choosing LPR as one of the evaluation methods, we need to specify a scenario. Night-time recording
of cars, which travels in a dark highway, is one of the hardest scenarios in LPR (also in general photography).
It requires a fast shutter speed, typically between 500 to 2000uS, according to the common practice in Axis.
This is done to freeze the object while keeping acceptable brightness and avoiding excessive noise. By using
this extreme case, we could maximize the image quality improvement that strobed IR can offer, exactly
where image brightness improvement is restricted in constant IR (due to PoE power limitation).
However, evaluating the performance regarding motion blur for a camera is not straightforward. To extract
valid conclusions, a controlled environment is needed, in which the quality of images can be analyzed for
different camera configurations in the same condition.
An outdoor test with real vehicles is an option that can be useful, but there are a few external factors that
can pose problems and cause difficulties when analyzing specific aspects of the camera performance. For
example, some cars’ headlights will be stronger than others, and some plates will be more reflective than
others, which will yield different license plate image quality for the same camera settings. Thus, a comparison
of two different camera configurations, in these two different situations, might not be fair.
We need to consider that, while it is possible to have different cameras shooting at the same scene with
constant IR, it is not when using strobed IR. This is because the cameras must be synchronized with the
light emitter, which at this point is only possible with one camera.
Meanwhile, a controlled scenario in a laboratory allows testing different camera configurations in the same
conditions. Unfortunately, LPR testing in a laboratory means moving a license plate (or similar object) at
high speeds towards the camera, which is a hard task considering the limited dimensions of a laboratory.
For example, to capture the object in 100km/h in a 30m room, the object should accelerate from 0 to
100km/h in 2.16s. Therefore, we need to find a way to simulate the object movement, without moving the
object towards the camera.
Since the video is recorded in the sensor plane (2-dimensional), movements in the optical axis (3rd
dimension) of the camera will be translated into movements in the sensor plane. The direction of both
movements is better described in Figure 11. If we mathematically translate movement in the optical axis
into an equivalent movement in the sensor plane, the test setup can be greatly simplified.
In Figure 12, the proposed setup is shown. This setup consists of a reflective spinner with similar letters
found in EU license plates. The spinner will be spin using a stepper motor at a certain speed. In a particular
angular speed, each character will move in various linear speeds, depending on its position relative to the
shaft. The positions of the characters are as listed in Table 3.
-11-
Figure 11. Direction of Object Movement (left) and Car Velocity Components (right)
The next step is to correlate the car speed in a real case with the characters’ speed in the spinner. At first,
we need to understand what causes the movement of the projected image (at the sensor). The object
movements can be broken down into two different types of movement, shown in Figure 11. The first type
is the movement in sensor plane (X and Y direction), while the second type is in the optical axis (Z direction).
In the LPR situation shown in Figure 11, the car movement consists of both movement types. The spinner,
on the other hand, is considered to create movements only in the sensor plane (assuming the spinner is
aligned with the sensor plane). Since we can only do the comparison in the sensor plane domain, the optical
axis movement of the car (𝑉𝑐𝑎𝑟_𝑧) must be transformed first.
Figure 12. Spinner Bar
Table 3. Character Position and Speed in The Spinner
Characters from center
distance from center [cm]
First 5
Second 15
Third 25
Fourth 35
Fifth 45
Sixth 55
Both types of movements are affected by the optical properties, such as the relation between magnification,
distance, and height. The relationship is described in (12) and can be seen in Figure 13.
𝑀 =
𝑑𝑖
𝑑𝑜=
ℎ𝑖
ℎ𝑜
(12)
The spinner moves in both X and Y directions (∆ho), thus creates a shift in the projected image (∆hi) as
shown in Figure 14. Meanwhile, movement due to 𝑉𝑐𝑎𝑟_𝑧 can be perceived as a change in magnification.
With a constant focal length (di = distance between sensor plane and point of convergence), a change in
magnification would modify the projected image height as seen in Figure 15..
-12-
Figure 13. The Optical Properties
Figure 14. Object Movement in X or Y Direction (sensor plane)
Figure 15. Object Movement in Z Direction (optical axis)
As explained earlier, we would like to relate the speed of the spinner (XY) with the car speed (XYZ). To
generate the similar level of motion blur, both car and spinner should generate an equivalent movement in
their projected image. First, we will calculate the equivalent spinner movement (XY) of the car Z component
in (13).
∆ℎ𝑖_𝑧 = ∆ℎ𝑖_𝑥𝑦 (13)
ℎ𝑖2_𝑧 − ℎ𝑖1_𝑧 = ℎ𝑖2_𝑥𝑦 − ℎ𝑖1_𝑥𝑦 (14)
We will derive the equation one by one for each side to improve readability (without writing the extra
subscript to differentiate direction). We start with the Z movement (left-hand side in (14)). Using (12), all
ℎ𝑖 in (14) can be substituted with its ℎ𝑜counterparts to yield (15) and (16).
-13-
(𝑀2 ∙ ℎ𝑜2) − (𝑀1 ∙ ℎ𝑜1) (15)
𝑑𝑖2
𝑑𝑜2∙ ℎ𝑜2 −
𝑑𝑖1
𝑑𝑜1∙ ℎ𝑜1
(16)
Furthermore, ℎ𝑜2 and ℎ𝑜1 can be reduced to ℎ𝑜 since we will consider car speed in Y direction later. Also,
𝑑𝑖2 and 𝑑𝑖1are 𝑑𝑖 as the focal length is constant, so finally (16) can be reduced to (17).
(
1
𝑑𝑜2−
1
𝑑𝑜1) ∙ ℎ𝑜𝑑𝑖
(17)
Merging the fractions will yield (18) and (19).
(
𝑑𝑜1 − 𝑑𝑜2
𝑑𝑜1𝑑𝑜2) ∙ ℎ𝑜𝑑𝑖
(18)
(
∆𝑍
𝑑𝑜1𝑑𝑜2) ∙ ℎ𝑜𝑑𝑖
(19)
At last, we substitute ∆𝑍 with the measurable variables in (20). 𝑉𝑐𝑎𝑟_𝑧 is the car speed in Z direction, while
𝑇𝑂𝑁 is the strobe length. 𝑇𝑂𝑁 can also be the shutter speed if constant IR is used. The final equation for Z
direction is shown in 21.
∆𝑍 = 𝑉𝑧∙ 𝑇𝑂𝑁 (20)
(
𝑉𝑐𝑎𝑟_𝑧 ∙ 𝑇𝑂𝑁
𝑑𝑜1𝑑𝑜2) ∙ ℎ𝑜𝑑𝑖
(21)
Now we are finished with the Z direction (car movement). For the XY direction (spinner movement), the
derivation is much more straightforward as the magnification does not change due to a constant object
distance (𝑑𝑜1 = 𝑑𝑜2). The final equation for the spinner movement is shown in (22).
(
𝑑𝑖
𝑑𝑜) ∙ (𝑉𝑋𝑌. 𝑇𝑂𝑁)
(22)
Before we put back (21) and (22) side by side as in (14), the directional subscripts are added to avoid
confusion between different directions. Assuming focal length for both directions are the same (𝑑𝑖_𝑍 =
𝑑𝑖_𝑋𝑌), (23) can be reduced to (24).
(
𝑉𝑐𝑎𝑟_𝑧 ∙ 𝑇𝑂𝑁
𝑑𝑜1_𝑍. 𝑑𝑜2_𝑍) ∙ ℎ𝑜𝑍
∙ 𝑑𝑖_𝑍 = (𝑑𝑖_𝑋𝑌
𝑑𝑜_𝑋𝑌) ∙ (𝑉𝑋𝑌. 𝑇𝑂𝑁)
(23)
(
𝑉𝑐𝑎𝑟_𝑧 ∙ 𝑇𝑂𝑁
𝑑𝑜1_𝑍. 𝑑𝑜2_𝑍) ∙ ℎ𝑜𝑍
= (1
𝑑𝑜_𝑋𝑌) ∙ (𝑉𝑋𝑌. 𝑇𝑂𝑁)
(24)
Knowing that ∆𝑍 is simply 𝑑𝑜1_𝑍 − 𝑑𝑜2_𝑍, we can finally derive (24) into two useful equations, (25) and
(26), which correlate car movement in the Z direction (𝑉𝑧) with spinner movement in XY direction (𝑉𝑋𝑌).
Furthermore, 𝑉𝑋𝑌 can represent 𝑉𝑍_𝑥 or 𝑉𝑍_𝑦 depending on the height of the object ℎ𝑜_𝑍 given to the
equation (whether it is the width or the height of the license plate).
-14-
𝑉𝑋𝑌 =
ℎ𝑜_𝑍 ∙ 𝑑𝑜_𝑋𝑌
𝑑𝑜1_𝑍(𝑉𝑐𝑎𝑟_𝑧 . 𝑇𝑂𝑁 − 𝑑𝑜1_𝑍)∙ 𝑉𝑐𝑎𝑟_𝑧
(25)
𝑉𝑐𝑎𝑟_𝑧 =
𝑑𝑜1_𝑍2 . 𝑉𝑋𝑌
𝑑𝑜_𝑋𝑌 . ℎ𝑜_𝑍 + 𝑇𝑂𝑁 . 𝑑𝑜1_𝑍. 𝑉𝑋𝑌
(26)
Notice that the analysis is not finished yet since car movement in Y direction should also be considered.
Figure 16 explains where should the 𝑉𝑐𝑎𝑟_𝑦 be included to the conversion.
Figure 16. Vehicle to Equivalent Spinner Speed Conversion
To review the significance of object movement in each direction to motion blur creation, we will do an
example case for the conversion theory. Like Figure 11, the camera is mounted on a bridge, looking down
to a highway. The car is assumed to move straight, so there is no X component involved in the movement.
The license plate dimensions are taken from EU standards.
Table 4. Parameters of an Example Case
Variable Abbreviation Value
Car speed 𝑉𝑐𝑎𝑟 33.33 m/s (120 km/h) License plate height ℎ𝑜_𝑍𝑦 0.114 m
License plate width ℎ𝑜_𝑍𝑥 0.52 m
Camera mounting height ℎ𝑐𝑎𝑚 5 m
Camera to license plate
distance 𝑑𝑜1_𝑍
14.62 m
Camera to Spinner distance 𝑑𝑜_𝑋= 𝑑𝑜_𝑦 20 m
Viewing angle α 70 deg Strobe length 𝑇𝑂𝑁 10-3 s
For the given values in Table 4, the calculations could be done as follows:
1. Calculate the z and y component of the car speed.
𝑉𝑐𝑎𝑟_𝑧 = 𝑉𝑐𝑎𝑟 ∙ 𝑠𝑖𝑛𝛼 = 31.32 𝑚/𝑠
𝑉𝑐𝑎𝑟_𝑦 = 𝑉𝑐𝑎𝑟 ∙ 𝑐𝑜𝑠𝛼 = 11.39 𝑚/𝑠
2. Calculate 𝑉𝑋 from Z movement.
𝑉𝑋 =ℎ𝑜_𝑍𝑥
∙ 𝑑𝑜_𝑋
𝑑𝑜1_𝑍(𝑑𝑜1_𝑍
− 𝑉𝑐𝑎𝑟𝑧∙𝑇𝑂𝑁)∙ 𝑉𝑐𝑎𝑟𝑧
-15-
= 1.52 m/s
3. Calculate 𝑉𝑌 from Z movement.
𝑉𝑌 =ℎ𝑜_𝑍𝑦
∙ 𝑑𝑜_𝑌
𝑑𝑜1_𝑍(𝑑𝑜1_𝑍
− 𝑉𝑐𝑎𝑟𝑧∙𝑇𝑂𝑁)∙ 𝑉𝑐𝑎𝑟𝑧
= 0.334 m/s
4. Calculate total 𝑉𝑋 and 𝑉𝑌 .
𝑇𝑜𝑡𝑎𝑙𝑉𝑌= 𝑉𝑌 + 𝑉𝑐𝑎𝑟_𝑦 = 11.72
𝑚
𝑠= 42.2 𝑘𝑝ℎ
𝑇𝑜𝑡𝑎𝑙𝑉𝑋= 𝑉𝑋 = 1.52
𝑚
𝑠= 5.47 𝑘𝑝ℎ
From this example, it can be concluded that ‘the movement of the car in the optical axis’ (𝑉𝑐𝑎𝑟_𝑧) does not
contribute significantly to the movement projected in the sensor plane (𝑉𝑋 and 𝑉𝑌). Figure 17 shows that
even with a bigger viewing angle (which increases 𝑉𝑐𝑎𝑟_𝑧), 𝑉𝑌 ratio to 𝑉𝑐𝑎𝑟_𝑦 is still very small (lower than
0.1). In other words, the blur generated by the optical axis movement of the car, is negligible when compared
to the blur generated by the sensor plane movement.
Figure 17. Contribution of Optical Axis Movement in Producing Motion Blur
On the other hand, the speed conversion shows that the dominant movement comes from the car
movement in Y axis (sensor plane). The viewing angle affects 𝑉𝑐𝑎𝑟_𝑦 heavily, and so does the 𝑇𝑜𝑡𝑎𝑙𝑉𝑌, as
shown in Figure 18. Thus, using a high viewing angle in LPR will help reducing the blur caused by the
movements in Y axis.
Figure 18. 𝑇𝑜𝑡𝑎𝑙𝑉𝑌 relation to Viewing Angle
-16-
However, it is important to note that this speed conversion utilizes over-simplification of the optical system
and geometry. These are the assumptions that have been made:
• The lens is an ideal thin lens with specific properties, e.g., has no barrel distortion and known
point of convergence.
• The angled view of the real case also affects how the LPR algorithm performs to some degree,
which cannot be reproduced by the spinner (entirely perpendicular to the camera).
• The equations assume that the optical axis of the camera intersects the license plate at its center,
yielding minimum motion blur.
Also, we need to point out that only motion blur is being evaluated, while resolution and LED power are
not considered yet. From the previous equations, we could wrongly conclude that higher viewing angles
would yield better image quality. While this is true in terms of motion blur, the resolution of the camera will
limit the camera’s angle (since a greater angle means longer distance to the license plate). If the resolution is
not the limiting factor (e.g., a proper zoom lens is used), then the LED power is the only remaining variable
to consider (even more critical when zoom lens are used).
Accuracy test of the equations is somewhat challenging to do and out of the project bound. Therefore, the
proposed method serves as a rough approximation which might ease LPR related product development. It
also helps to understand what factors are contributing to the motion blur.
-17-
4 EVALUATION METHOD
As the flow of the research has been briefly explained in Section 1.4, this chapter will focus on the evaluation
method. In general, there are two kinds of evaluation that will be done to answer the research question:
• An indoor test with the proposed method discussed in the LPR chapter will be done in order to
estimate if the prototype is good enough for an outdoor test, which is more difficult to do and
more resource consuming.
• An outdoor test which resembles a real-life situation of the LPR application.
The section of each test will discuss in detail the motivation of the test as well as the test setup and test
cases. However, it is crucial to specify which strobing method is appropriate for image quality improvements
in LPR before determining the test cases for each test.
4.1 Chosen Strobing Method
In this section, we will analyze the timing properties of the sensor used in the development platform. The
sensor output in this platform is sent through a low-voltage differential signaling (LVDS) communication
protocol and the output data sequence follows the pixel arrangement shown in Figure 19.
The sensor has recording pixels (gradient color) in the center, and extra surrounding pixels for color
processing and optical blanking for calibration (white and blue respectively). Meanwhile, the rest of the
diagram represents the additional data which is also output to the data sequence (does not represent physical
pixels). The device which receives the sensor output (i.e., the PLD in the camera) will go through every line
and use the same timing (by detecting sync codes) to move between lines.
The white and colorful lines are categorized as valid lines, while the grey ones as blanking lines. The number
of valid lines (𝑁𝑣𝑎𝑙𝑖𝑑) is fixed. The number of blanking lines (𝑁𝑏𝑙𝑎𝑛𝑘𝑖𝑛𝑔), however, can be adjusted to fit a
certain timing requirement.
Figure 19. Sensor Pixel Arrangement
In Table 5, several timing combinations that can be achieved by the sensor is listed. Notice that there are
two different terms for rate. The first one, readout rate, is the speed of data transfer. The higher the rate,
the faster the sensor can move between different lines, and thus it determines the smallest applicable time
step (1H period).
-18-
Table 5. Sensor Drive Mode Combinations at 1080p, LVDS 12-bit Output
note: some combinations (3 and 4) are generated by further calculation (not provided in the datasheet).
Recording Pixels Readout
rate
[FPS]
1H
period
[uS]
𝑵𝒕𝒐𝒕𝒂𝒍
Resulting
frame rate
[FPS]
𝑻𝑶𝑵𝐦𝐢𝐧(𝑾𝑭𝑺) or
𝑻𝒔_𝒎𝒂𝒙(𝑺𝑻𝑺)
[uS]
H
[pixels] V
[lines]
1
1920 1080
25 35.6 1125 25 497.8
2 60 14.8 1125 60 207.4
3 60 14.8 2700 25 23540.7
4 60 14.8 2250 30 16874.1
Meanwhile, the resulting frame rate is the actual rate of the output video. This rate is affected by the total
number of lines (𝑁𝑡𝑜𝑡𝑎𝑙), which value is made adjustable due to the flexibility of 𝑁𝑏𝑙𝑎𝑛𝑘𝑖𝑛𝑔. For example,
we can provide more integration time for a given frame period by increasing 𝑁𝑡𝑜𝑡𝑎𝑙 from the baseline (1125
lines). Longer integration times become available at the cost of frame rate reduction. In Table 5, this is
shown when moving from case 2 into case 3 or 4.
If seen from a different perspective, increasing the 𝑁𝑡𝑜𝑡𝑎𝑙 can also benefit the strobing application. Consider
moving from case 1 to case 3. The maximum shared time (𝑇𝑠_𝑚𝑎𝑥) for STS is increased, while the frame
rate is kept constant. For a given frame period, 𝑇𝑠_𝑚𝑎𝑥 can be calculated as follows:
𝑇𝑠_𝑚𝑎𝑥[𝐻] = (𝑁𝑡𝑜𝑡𝑎𝑙 − (𝑁𝑣𝑎𝑙𝑖𝑑 + 1)) (27)
𝑇𝑠_𝑚𝑎𝑥[𝑢𝑆] = (𝑁𝑡𝑜𝑡𝑎𝑙 − (𝑁𝑣𝑎𝑙𝑖𝑑 + 1)) × 1𝐻𝑝𝑒𝑟𝑖𝑜𝑑[𝐻/𝑢𝑆] (28)
Notice that the 𝑇𝑠_𝑚𝑎𝑥 [uS] can increase, not because of its value in H unit alters, but due to the amplification
of the unit conversion (1𝐻𝑝𝑒𝑟𝑖𝑜𝑑). Figure 20 shows that, higher readout rates make the rolling shutter look
less angled and create more shared time to do STS.
Figure 20. Shutter Timing Diagram of Different Readout Rates for STS
note: strobe lengths are shown in the brighter shades
WFS application is also benefiting from higher 𝑁𝑡𝑜𝑡𝑎𝑙 as the minimum strobe length (𝑇𝑂𝑁_𝑚𝑖𝑛) drops. The
timing of higher readout speed in Figure 21 provides longer off time for better power saving with WFS.
Equation (2) can be converted to yield 𝑇𝑂𝑁_𝑚𝑖𝑛[𝑢𝑆] by simply multiplying 1H period:
𝑇𝑂𝑁_𝑚𝑖𝑛[𝑢𝑆] = (𝑁𝑡𝑜𝑡𝑎𝑙 − (𝑁𝑣𝑎𝑙𝑖𝑑 + 1)) × 1𝐻𝑝𝑒𝑟𝑖𝑜𝑑[𝐻/𝑢𝑆] (29)
frame rate = 25fps;
readout rate = 25fps (1H period = 35.53uS)
frame rate = 25fps;
readout rate = 60fps (1H period = 14.82uS)
-19-
Figure 21. Shutter Timing Diagram of Different Readout Rates for WFS
note: strobe lengths are shown in the darker shades
Initially, the readout rate is designed to match the desired frame rate (e.g., the first combination in Table 5).
However, it has been shown that other combinations can be generated by modifying the total number of
vertical lines. This mechanism is already implemented in the development platform, which has a fixed 60FPS
readout rate for all capture modes in full-HD resolution. For this work, the default capture mode used is
described in Table 6. The wide dynamic range (WDR) mode is disabled to avoid unwanted artifacts such as
motion blur and ghosting. Details on this can be found in [9].
Table 6. Default Capture Mode Parameters Used in The Development Platform
Parameter Value
capture mode name 1080p 1920x1080 (16:9) @25/30FPS
wide dynamic range OFF
recording pixels, H[pixels] x V[lines] 1920x1080
total number of pixels (output format) 2200x2700 (LVDS 12-bit)
valid lines 1110
frame rate – capture frequency 25FPS – 50hz
readout rate – 1H period 60FPS – 14.82uS
At this point, the specific parameter needed to compare the available strobing methods, such as readout rate
and frame rate, has been determined. Using a typical shutter time used for LPR (𝑇𝑖 = 2000𝑢𝑆), we could
check if the proposed suitable method presented in Table 2 is accurate.
Now we will evaluate the average power reduction of WFS first. Using equation (2), the 𝑇𝑂𝑁_𝑚𝑖𝑛 for the
given shutter time and number of valid lines is:
𝑇𝑂𝑁_𝑚𝑖𝑛 = 𝑇𝑖 + (𝑁𝑣𝑎𝑙𝑖𝑑 − 1) .
= 2000𝑢𝑆 + (1110𝐻 × 14.8148𝑢𝑆/𝐻 − 1)
= 18444𝑢𝑆
This number is almost half of the original 𝑇𝑂𝑁 for constant IR (always on throughout the frame period).
Hence, with the same average power consumption, the usage of WFS is only able to multiply the allowable
peak power for less than three time the original. Meanwhile, STS with the same configuration only need
𝑇𝑂𝑁 = 2000𝑢𝑆 which is much smaller than what WFS need. Hence, for a high-speed and low ambient light
condition, STS would be the appropriate strobing method. Along with the significant improvement in
allowable peak power, STS will also help reducing the rolling shutter artifacts to improve the overall image
quality. Meanwhile, the use of WFS will not be demonstrated in this thesis since it is more suitable for
improvements which are oriented to power efficiency [4].
frame rate = 25fps;
readout rate = 60fps (1H period = 14.82uS)
frame rate = 25fps;
readout rate = 25fps (1H period = 35.53uS)
-20-
4.2 Indoor Test
In accordance with the proposed method discussed in Section 2.3, we will evaluate the prototype with the
spinner placed in the dark room. The spinner is driven by a stepper motor at its maximum constant velocity,
and the equivalent car speed of the characters can be found in Table 7.
Table 7. Equivalent Car Speed of Characters on The Spinner
Spinner RPM
Characters from center
Equivalent car speed [km/h]
α=60° α=70° α=80°
120
First 4.5 6.6 13.0
Second 13.6 19.8 39.1
Third 22.6 33.1 65.1
Fourth 31.7 46.3 91.2
Fifth 40.7 59.5 117.2
Sixth 49.8 72.7 143.3
All listed speeds can be simulated in a single recording as all the characters (the whole bar) are fit into the
shooting frame. Therefore, the next step is to determine the test setup which can compare the image quality
of strobed IR and constant IR. The strobed IR will have more emitter (four LEDs), while constant IR will
use one LED. By having more emitter, strobed IR is expected to deliver brighter image without consuming
more power in average.
Meanwhile, the LED module which will be used for all cases should be identical to yield a fair comparison.
The LED module type (described in Table 8) is taken from a reference model which uses the module as a
built-in IR. Moreover, this camera model also serves as one of the test models for LPR development in
Axis. The development platform, on the other hand, is not equipped with built-in IR and instead chosen
because it has a PLD.
Table 8. LED Module Specifications for Indoor Testing
parameter value
View angle 60°
Typical current, 𝐼 700mA
Forward voltage, 𝑉 1.85V
Radiometric power 700mW @1A
Radiant Intensity, 𝐼𝑒 n/a
Peak power consumption, 𝑃𝐿𝐸𝐷 1.295W
Moving to the test cases, the strobed IR will have one case, while constant IR will have three cases. The first
case of constant IR will have a shutter speed similar to the strobe length of the strobed IR case. The second
and the third configuration are cases where higher gain or longer shutter speed is used respectively. This
will be done to show the quality deterioration that constant IR illumination experiences when matching the
brightness level to the strobed IR one (since it uses more LEDs).
The configurations (gain and iris value) will be tuned to yield acceptable exposure for strobed IR. The first
and the second case of constant IR will use similar gain and iris value while the third case will have 6 dB
more gain. All configurations will be evaluated at 𝑇𝑂𝑁 = 2000𝑢𝑆 for strobing and 𝑇𝑖 = 2000𝑢𝑆 for
constant IR which is a typical shutter speed used in LPR. The full list of test cases can be seen in Table 9.
-21-
Table 9. Indoor Test Cases
Iris Gain (dB) 𝑻𝒊 (uS) 𝑻𝑶𝑵 (uS) 𝑵𝑳𝑬𝑫
1
F2.26
0 40000 2000 4
2 0 2000 constant 1
3 0 8000 constant 1
4 6 2000 constant 1
With this strobe length, the LEDs in strobed IR (four LEDs) are expected to consume less average power
compared to constant IR, even though they have a higher peak power demand. Using the power
consumption value given in Table 8, we can theoretically calculate the average power consumption for
strobed and constant case, especially with the chosen shutter time, using the following equation:
𝑃𝑎𝑣𝑔 =𝑇𝑂𝑁
𝑇𝐹𝑃× 𝑃𝐿𝐸𝐷 × 𝑁𝐿𝐸𝐷 (30)
The results in Table 10 clearly show that with this setup, the number of LEDs (𝑁𝐿𝐸𝐷 ) in strobed IR can
still be added before it surpasses the constant IR consumption for higher brightness. Nevertheless, this
addition will be done in the outdoor test where the distance between the camera and the license plate is
much bigger. For this test, 4 LEDs are assumed to be enough. The result of this test will be discussed in
Section 6.1.
Table 10. Average Power Consumption of Strobed and Constant IR in Indoor Test
Constant IR (1 LEDs) Strobed IR (4 LEDs)
1.295W 0.259W, for: 𝑇𝑂𝑁 = 2000𝑢𝑆; 𝑇𝐹𝑃 = 40000𝑢𝑆
4.3 Outdoor Test
While the indoor test has a very controlled environment (total darkness) and simulated speed, the outdoor
test will evaluate the system with a real LPR situation. The outdoor test will be done on a highway where
cars usually travel between 80 to 120km/h. The lighting situation is also critical, as we want the recording
place to be as dark as possible. Some of the city roads in Lund (where the thesis is conducted) may have
cars traveling at the desired speed. However, they are mostly well-lit by street lights or having lights coming
from nearby buildings. Therefore, we limit the location candidate to the most accessible highway, the
European route E22 which passes the eastern part of Lund.
This highway does not have any light sources aside from the lights of the cars, which makes it suitable for
the test. Furthermore, it also separates the city center with the eastern part of Lund, thus making it is easy
to find pedestrian bridges which can be used to place the test setup. The coordinate of the chosen location
is 55.703645° latitude, 13.219716° longitude.
After settling with the location, some earlier tests were carried out using any available car that passed the
road as the object. Nevertheless, the results were hard to analyze since different cars have various speed,
license plate mounting positions, and conditions (e.g., cleanliness). Some older cars also have non-reflective
plates which push the brightness of the captured image to an unacceptable level. At last, it was decided to
use the same car which will travel at 100km/h when it reaches the bridge. The sample car is of Spanish
origin due to the its accessibility, so a Spanish license plate with EU badge becomes the object.
Since we already know that the viewing angle will affect the motion blur of the car, this test will also compare
the image quality between three viewing angles shot with strobed IR. The complete test cases for the outdoor
test is shown in Table 11.
-22-
Table 11. Test Cases for Outdoor Test
Iris α (°) 𝑻𝑶𝑵 (uS) Gain (dB) 𝑻𝒊 (uS) 𝑰𝑳𝑬𝑫 (mA)
1
F1.63
80
2000 9.4 18444 700
2 constant 9.4 2000 350
3 constant 21.3 2000 350
4 constant 9.4 8000 350
5 70
2000 9.4 18444 700
6 constant 9.4 2000 350
7 60 2000 9.4 18444 700
Test cases number 1 to 4 resemble the indoor test cases, except for the use of shorter 𝑇𝑖 (from equation
(4)) which yield just enough shared time for the strobing to happen (instead of having it extended
throughout one frame period). By doing this, we can reduce the blur caused by the ambient light (that might
come from unwanted light sources such as the car light). Another difference in the outdoor test is the use
of more powerful LEDs. Some pre-test done before the final test shows that even with 𝑁𝐿𝐸𝐷 = 4
configuration used in the indoor test, the license plate is simply not bright enough. It is important to note
that the distance between the bridge (where the camera is) and the car is approximately 50 meters when
using viewing angle (α) of 80°.
Moreover, Table 12 shows that the LED configuration is actually producing the same amount of radiometric
power compared to the one used in the indoor test. The difference is located in its narrow view angle, and
hence the radiometric power will be more concentrated. Unfortunately, the radiant intensity of the LED
module used in the indoor test is unknown, so we cannot compare them in a fairer way.
Table 12. LED Module Specifications for Outdoor Test [10]
parameter value
Brand, model Osram, SFH 4786S
View angle 30°
Radiometric power 700mW@1A
Radiant Intensity, 𝐼𝑒 1800mW/sr @1A
To further anticipate the under-exposed condition, the new LED module will be organized in a string
consisting of 8 LEDs. It will also be driven in two different current levels depending on the IR light (strobed
or constant). In general, LEDs can be driven in a higher than typical current if it is driven in a pulsed-way.
For strobed IR, the current is set at 700mA while it is set at 350mA for constant current. These current
levels are safe for the LEDs as it is under the maximum current allowed stated in [10]. The complete LED
driving conditions are listed in Table 13.
Table 13. LED Driving Condition for Outdoor Test
parameter Strobed IR Constant IR
𝑇𝑂𝑁 2000uS 40000uS
𝐼𝐿𝐸𝐷 700mA 350mA
Forward voltage, 𝑉 3.44V 3.24V
Peak power consumption per module, 𝑃𝐿𝐸𝐷 2.4W 1.13W
Number of LEDs, 𝑁𝐿𝐸𝐷 8 1
Total peak power consumption 19.2W 1.13W
Average power consumption, 𝑃𝐴𝑉𝐺 0.96W 1.13W
-23-
The average power consumption for this LED module in constant IR is comparable (slightly lower) than
its counterpart used in the indoor test. However, the most notable part is the LED array used for strobed
IR is still consuming less average power than the one for constant IR although it has significantly higher
peak power consumption.
-24-
5 SYSTEM IMPLEMENTATION
After theoretical studies and evaluation method has been defined, it is time to implement the idea in the
prototype. This chapter is divided into three different sections, which will cover the choice for LED driver,
how the strobing signal is generated, and the impact of the implemented system to the human-eye.
5.1 LED Driver
The chosen LED driver for the prototype is the AN-1750, which is the evaluation board of TI LM3406.
This driver works as a buck regulator controlled current source designed to deliver current up to 1.5A to
the LED. It also supports driving a LED string with a total forward voltage of 33V (fsw=500kHz, toff-min=
230ns)[11].
The unique feature of this driver is the two different dimming techniques (DIM1 and DIM2), which is part
of the research focus in [4]. DIM1 is an input for logic level pulse width modulation (PWM) signal for
controlling the dimming of the LED. This method provides a better efficiency as it disables the driver
output when DIM1 is set low. Meanwhile, DIM2 input is controlling another dimming method which uses
an additional MOSFET connected in parallel with the LED string to turn off (dim) the LEDs. This method
would provide shorter fall time while consuming more energy in the off state (due to non-zero shunt
current) [11]. Using the DIM2 input requires an inversed logic PWM as the voltage is transferred to the gate
of the NPN shunt transistor. In this thesis, the DIM2 is chosen for the sake of strobing responsiveness.
The tests done in [4] showed that both rise and fall time of the driver using DIM2 (900nS and 100nS
respectively) are less than smallest step in the sensor timing (14.82uS) which will be discussed in Section 4.1.
Figure 22. Schematic of TI AN-1750 (LM3406 Evaluation Board) [11]
5.2 Strobe Signal Generation
This section will explain how the sensor and the IR light can be synchronized according to the STS method.
In general, there are four subsystems that are part of the synchronization path as shown in Figure 23. In the
end, an LED driver required a logic-level pulse to control the LED state. Meanwhile, in the beginning, the
sensor output is only accessible by the PLD through low-voltage differential signaling (LVDS). Typical
video synchronization signal such as horizontal and vertical sync is actually available at the sensor output
pins. However, these pins are not connected to any path in the printed circuit board (PCB). Therefore, the
only way to access the sensor timing is through analyzing the LVDS data sequence, especially the sync code.
In the development platform, the PLD is responsible for transforming the raw sensor data to ready-to-
process data for the Axis’ proprietary CPU (ARTPEC). There is an already implemented frame-grabber
module in the PLD code which detects the sync code to separate valid lines (useful information) and
blanking lines. Hence, we just need to export the extracted sync codes from the frame-grabber and use it as
-25-
an input for the IR Strobing module that will be implemented in the PLD. This module will generate the
required strobing signal and the logic-leveled sync codes for debugging purposes. The access to these signals
will be physically available through unused I/O pins in the PLD. Details on the strobe signal generation will
be discussed in Section 5.2
Figure 23. System Architecture
After the strobing signal generation is done, the next step is to make the strobing parameter dynamic
(changeable on the fly). This parameter can be stored in a register inside the PLD, and the value can be
updated from ARTPEC through I2C communication. Hence, the user can quickly change the parameter
from the ARTPEC side. The modification of the ARTPEC code will be discussed in Section 5.2.2.
At last, the access for other sensor configurations (i.e., sensor gain, iris, etc.) is available in the camera’s web
interface, so modification for this purpose is unnecessary.
5.2.1 Sync codes
The sensor LVDS output is designed so that the image data from every line (including blanking lines) is
surrounded by two sync codes as seen in Figure 24. Each code can be one of four types listed in Table 14.
Figure 24. Sync Codes Timing (right)
Table 14. Definition of Sync Codes
Sync code Definition
SOL Start of a valid line
EOL End of a valid line
SOF Start of a blanking line
EOF End of a blanking line
-26-
By being able to detect different sync codes, we can determine the switching point of the LED (high/low
logic level of the strobing signal). The shared-time strobing (STS) needs to set the strobing signal from high
to low before the first valid line is readout. To be precise, the LED should be switch off at the 13th EOL
(best explained in Figure 25. LED Switching Point for Shared-time Strobing
note: the number of lines drawn does not represent the real sensorA). However, to simplify the system and
to give some margin for the LED to be completely off, the strobing signal will be set low earlier when the
first SOL is detected. The difference will not be significant as the sensor receives light just 10% less (𝑇𝑖 =
2000𝑢𝑆).
The next task is to determine when to switch the LED back on. This is trickier than the previous transition,
as the switch on point does not lie precisely at the first sync code of any kind. The LED must be turned on
after every line has started integrating. In other words, the last line will start to integrate at the nth EOF as
drawn in Figure 25. The final switching points are presented in Table 15.
The period between the last valid line readout and this nth is called 𝐻𝑂𝐿𝐷𝑆𝑇𝐵. It can be calculated using
(31).
𝐻𝑂𝐿𝐷𝑆𝑇𝐵 = 𝑇𝐹𝑃 − (𝑇𝑖 + 1) (31)
Table 15. Shared-time Strobing Switching Points
State transition Switching point
on to off First SOL
off to on nth EOF, n = 𝐻𝑂𝐿𝐷𝑆𝑇𝐵
Figure 25. LED Switching Point for Shared-time Strobing
note: the number of lines drawn does not represent the real sensor
Idle / unused
Integration time
Shutter timing
Ti Ti Readout (active lines)
Blanking line
Readout (blanking lines)
LED state
sensor output
IR ONLED OFF LED OFF
nth
blanking linen-1th
blanking linelast blanking line first valid line
V.BLK EOF SOF V.BLK EOF
IR OFF IR ON
SOL EOLDATA SOFSOF V.BLK EOF
BLANKING VALID LINES BLANKING VALID LINES
TsTs
HOLDSTB
TFPline
t[H]
-27-
5.2.2 PLD and ARTPEC Modification
As the sync codes from the sensor are already extracted in the PLD, the strobing signal generation will also
be implemented there. The PLD code is written on SystemVerilog [12], and a new module which includes
two parts will be added to the code.
The first part is a counter which will be triggered by either a rising edge in EOF (count up) or SOL (reset).
The second part is a combinatorial logic which sets the strobing signal (IR_STB) high or low depending on
the counter position. A logic high will be output when the counter value is greater than or equal to
𝐻𝑂𝐿𝐷𝑆𝑇𝐵, otherwise, IR_STB is set to logic low. The approximated register-transfer logic (RTL)
implementation of this module can be seen in Figure 26.
Figure 26. RTL Implementation of Strobing Signal Generation Module
Whenever the sensor is within the valid lines, the counter will always be reset, and hence the LED will be
kept off. Meanwhile, outside the valid lines (during the blanking lines), the counter will work freely with the
count-up being triggered by the rising edge of EOF every time a blanking line has passed. The LED will be
switched on after 𝐻𝑂𝐿𝐷𝑆𝑇𝐵 blanking lines occur.
The needed EOF and SOL (in the form of logic signal) are taken from the existing frame grabber module,
and the IR_STB output signal is connected to an I/O pin which later will be connected to the LED driver
input. Along with IR_STB, the sync codes (EOF and SOL) are also made accessible for measurement
through the neighboring pins. A hardware rework was done to connect the PLD pins to an add-on PCB
which is mounted on the back side of the camera (shown in Figure 27).
Figure 27. Development Platform Rework for External Connections (connected to probes)
-28-
The variable 𝐻𝑂𝐿𝐷𝑆𝑇𝐵 is implemented as a register in the PLD. The PLD acts as a slave device in I2C bus,
while the ARTPEC as the master. Meanwhile, additional codes are implemented in the ARTPEC to update
the 𝐻𝑂𝐿𝐷𝑆𝑇𝐵 value every time the camera boots up, or when the sensor backend software is restarted. The
variable can be modified by the user through an external text file which is accessible through SSH connection
with the camera. Using this technique, the PLD does not have to be flashed whenever a new value is desired.
We also found that this is the easiest way to change the variable, as making changes in the web user interface
would require too much effort.
The implementation worked successfully when the signals were examined with an oscilloscope. For initial
testing shown in Figure 28 and Figure 29, the combinatorial logic was set to HIGH when the LED needs
to be on. However, in later implementation, the logic was changed to suit the inverted logic of DIM2 input
of the LED driver.
Figure 28. IR activated when 𝑐𝑜𝑢𝑛𝑡 ≥ 𝐻𝑂𝐿𝐷𝑆𝑇𝐵, 𝐻𝑂𝐿𝐷𝑆𝑇𝐵 = 21
Figure 29. IR Deactivated when 1st SOL is Detected (Counter is Reset)
In the last iteration, however, the system was modified to simplify the test procedure. The 𝐻𝑂𝐿𝐷𝑆𝑇𝐵 is no
longer input manually. Instead, three pre-determined modes which suite the test cases were employed, each
holding different 𝐻𝑂𝐿𝐷𝑆𝑇𝐵 values. The procedure to switch modes are using the same 𝐻𝑂𝐿𝐷𝑆𝑇𝐵 update
procedure.
Legend
IR_STB
EOF
SOL
Legend
IR_STB
EOF
SOL
-29-
Table 16. Strobing Modes Implemented in The Prototype
Mode number 𝑯𝑶𝑳𝑫𝑺𝑻𝑩 (H) 1H=14.82uS
Suitable 𝑻𝒊 (uS)
1 1454 2000
2 1522 1000
3 1553 500
After having the strobing signal generation working correctly, we need to prove that the sensor timing is
synchronized with the strobed IR light. To do this, we need to create an unsynchronized test instead. The
test goal was to produce an intensity gradient (discussed in Chapter 2) where the midpoint of the gradient
is located in a determined area (in this case, the center of the frame). Figure 30 explains how the parameters
can be derived to produce an intensity gradient at the center of the frame. Short integration time is used to
make the strobing light stop shining exactly where the center line (𝑁𝑉𝐴𝐿𝐼𝐷
2) starts integrating. As the light
will automatically turn off when the first line is readout, the value of 𝑇𝑖 shall be the same as the number of
valid lines (in H unit) which we would like to expose (which is also 𝑁𝑉𝐴𝐿𝐼𝐷
2 for the center line). Meanwhile,
𝐻𝑂𝐿𝐷𝑆𝑇𝐵 can be set to any value that is less than 𝐻𝑂𝐿𝐷𝑆𝑇𝐵, which can be calculated in (32):
𝐻𝑂𝐿𝐷𝑆𝑇𝐵_𝑚𝑎𝑥 = 𝑇𝐹𝑃 − 𝑇𝑖 − 𝑁𝑉𝐴𝐿𝐼𝐷 − 1 (32)
For the chosen capture mode, 𝑁𝑉𝐴𝐿𝐼𝐷 = 1110𝐻, hence we can set 𝑇𝑖 = 550𝐻 or 8255uS. However, the
closest number 1/125s was used to match the available camera preset. The acquired image (Figure 31) shows
that the intensity gradient occurs. The upper half of the object is also (spinner) invisible because of not
exposed by the IR light. Therefore, we can confirm that the sensor timing is synchronized with the IR light.
Figure 30. Synchronization Test Timing Diagram
Figure 31. Intensity Gradient in Synchronization Test
-30-
5.3 Eye-safety
Eye-safety calculation for the implemented system will be done according to the guide provided on [13],
which is based on IEC-62471 [14]. The calculation guideline [13] is created mainly for designing proximity
sensing device with pulsing IR (similar to how we use the IR light in this project). Figure 32 shows a
simplified flow of the evaluation procedure.
Figure 32. Eye-safety Evaluation Procedure
For all type of hazards shown in Figure 32, the exposure can be calculated with the radiometric data of the
emitter found in the datasheet. The data from LED configuration found in the outdoor test is chosen for
exposure calculation, as it is the strongest light source configuration that is used in the project. The next
step is to calculate the exposure for each hazard type. Afterward, the limit exposure values for each hazard
type are generated with the given time that one particular risk group specifies. The requirement matching
always starts from the lowest risk group (exempt) to the higher risk groups if the previous risk group
requirement is not met. As the risk group goes higher, the exposure limit becomes less stringent. Meanwhile,
this also means that the device will have more precautions as not to danger the user. The result of this eye-
safety evaluation for the implemented system is presented in Table 17. In short, the LED configuration
used in strobed IR for the outdoor test met all requirements for the lowest risk group (exempt group).
Table 17. Eye-safety Evaluation Results
module: OSRAM SFH 4786S; 𝑁𝐿𝐸𝐷=8; 𝐼𝐿𝐸𝐷=700mA; 𝑇𝑂𝑁=2000uS; 𝑇𝐹𝑃=40000uS
Exposure
Exempt Group
Limit
Requirement
status Safety Factor
Corneal Hazard 17.5W/m2 100W/m2 Met 5.78
Retina Hazard 53,781.63W/m2/sr 1,772,151.9W/m2/sr Met 32.95
Retina Hazard –
weak stimulus
53,781.63W/m2/sr 379,746.8W/m2/sr Met 7.06
note: complete calculation of this evaluation procedure can be found in the Appendix.
-31-
6 RESULTS AND ANALYSIS
The test result of the prototype will be discussed chronologically in this chapter. Indoor and outdoor test
sections will compare the difference between results using strobed and constant IR in various test cases.
6.1 Indoor Test
The results shown in Figure 33 compare the snapshot of each recording from all four indoor test cases listed
in Table 9. This group of snapshots was taken when the spinner position is close to horizontal to avoid the
rolling shutter artifact, that we will discuss later.
Generally, all images show that the setup (spinner) managed to reflect the IR light strong enough to provide
a decent contrast with the letters and the background (the wall of the dark room). As the default
configuration (iris, gain) was tuned for the strobed IR, Image 1 is correctly exposed. Meanwhile, Image2,
which was taken with constant IR, is underexposed, although the time of each frame receiving light from
the IR for both cases are the same (𝑇𝑂𝑁_1 = 𝑇𝑖_2). Therefore, it is clear that the additional three LEDs
brings a significant exposure improvement in strobed IR.
The attempt to match the exposure by increasing the shutter time four times (to 8000uS) in constant IR
(Image 3) improves the exposure with a tradeoff of prominent blur starting from the third letter. If we look
back to the equivalent car speed for each character in Table 7, this shutter time will start to produce blur
when the car travels at more than 61km/h with 80° viewing angle. The equivalent speed would be even
lower for steeper angle (e.g., 60°). Meanwhile for strobed IR, if we consider the sixth character as acceptable,
then the system is expected to perform well up to 143km/h for 80° viewing angle. The same motion blur
level applies to constant IR with the same shutter time with strobed IR (Image 2) regardless of being
underexposed.
However, in this indoor test, a higher gain managed to improve strobed IR with acceptable results. Any
intrusive noise is not seen even though the gain is doubled from the original configuration. However, the
exposure level does not seem to be a match yet with Image 1.
IMAGE 1 - gain = 0.0 dB, Ti = 40000uS, TON = 2000uS, NLED = 4
IMAGE 2- gain = 0.0 dB, Ti = 2000uS, TON = constant, NLED = 1
IMAGE 3 - gain = 0.0 dB, Ti = 8000uS, TON = constant, NLED = 1
IMAGE 4 - gain = 6.2 dB, Ti = 2000uS, TON = constant, NLED = 1
Figure 33. Indoor Test Results 1 (Brightness and Motion Blur)
-32-
Moving to the rolling shutter artifacts comparison, we need to take the snapshot of the recording when the
spinner is close to fully vertical so that the object will occupy more lines of the sensor. The results are shown
in Figure 34 with the spinner bar in constant IR (Image 6) appears bent. The upper edge of the spinner
experienced the most shift because it is the latest line to be integrated. The spinner rotates clockwise, hence
it makes sense that the later integration results in bending in a clockwise manner for the half upper/right
part of the spinner. Meanwhile, the image taken with strobed IR (Image 5) looks normal. Therefore, it is
proofed that the strobed IR with STS can create a virtual global shutter when the strobing light is the only
light source.
IMAGE 5 - gain = 0.0 dB, Ti = 40000uS,
TON = 2000uS, NLED = 4
IMAGE 6 - gain = 6.2 dB, Ti = 2000uS,
TON = constant, NLED = 1
Figure 34. Indoor Test Results 2 (Rolling Shutter Artifacts)
6.2 Outdoor Test
The result of the outdoor test is presented in two different crops, as seen in Figure 35. A full crop gives a
view of the overall situation while a tight crop which isolates the car’s front side gives a better view of the
license plate. From the full crop we can tell that Image 1 (in which strobing is used) has a brighter overall
impression compared to the all other images shot with constant IR (Image 2 to 4). One reason is the use of
longer shutter time that STS requires in order to provide enough shared time for the 2000uS strobe length
(𝑇𝑂𝑁). Another possible explanation for this is the existence of another car behind the object car which lit
up the road on the upper part of the frame.
In Figure 35, the vast brightness difference between strobed and constant IR can be seen. Image 1 was taken
with strobed IR which was equipped with a powerful LED array (NLED = 8, ILED = 700mA). Even with an
almost straight viewing angle (80°), the IR illumination is able to balance the brightness of the license plate
with the bright lights coming from the car’s headlights. Meanwhile, the illumination in constant IR (Image
2) did not provide enough light to expose the license plate well. Neither the image which was taken with a
longer shutter time (Image 3) nor a higher gain (Image 4) produced comparable plate brightness to Image
1. This time (in the outdoor test), the gain is increased by approximately 12dB from the original settings
which make the noise apparent especially in the area around the headlights. On the other hand, the case
with longer shutter time (Image 4) has comparable brightness with Image 3, but it suffers from motion blur.
While generally the images taken with constant IR (Image 2 to 4) is darker than the one with strobing, the
license plate is better isolated with constant IR. This isolation might help the LPR algorithm to identify the
plate easier. However, for some image processing cases, a more balanced image (brighter surroundings) is
better because the darker part of the frame (e.g., road, tree) does not need much shadow lifting (a processing
algorithm which is found in some generations of Axis cameras to improve the detail of the image).
-33-
IMAGE 1 - α=80°, gain = 9.4dB, Ti = 18444uS, TON = 2000uS, ILED = 700mA, NLED = 8
IMAGE 2 - α=80°, gain = 9.4dB, Ti = 2000uS, TON = constant, ILED = 350mA, NLED = 1
IMAGE 3 - α=80°, gain = 21.3dB, Ti = 2000uS, TON = constant, ILED = 350mA, NLED = 1
IMAGE 4 - α=80°, gain = 9.4dB, Ti = 8000uS, TON = constant, ILED = 350mA, NLED = 1
Figure 35. Outdoor Test Result 1 (Strobing versus Constant IR)
note: each picture on the right is cropped to the same number of pixels.
-34-
The drawback of this processing is that it creates more noise, particularly in the lifted areas (dark area) and
hence, the noise reduction algorithm will work harder to smoothen the image, finally reducing the detail of
the license plate. Determining which is the best for the LPR algorithm (better isolation or more balanced
exposure) will be hard as it depends heavily on the algorithms involved (LPR vs. shadow lifting + noise
reduction algorithm) and outside the scope of the project. Hence, it is hard at this point to tell whether the
longer shutter time characteristic of strobed IR is desired over the better isolation provided by constant IR
regardless of the plate brightness.
The next group of images compares the image quality between different viewing angles, each taken with
strobed IR. From Chapter 3 we know that a higher angle will produce less motion blur as the car movement
is translated into a lower equivalent sensor plane movement compared to shooting at a lower angle. A similar
relation is found in the test results shown in Figure 36. Out of the three angles, Image 1 (same image with
Image 1 in Figure 35) with α=80° is the only image without noticeable motion blur.
IMAGE 1 - α=80°, gain = 9.4dB, Ti = 18444uS, TON = 2000uS, ILED = 700mA, NLED = 8
IMAGE 5 - α=70°, gain = 9.4dB, Ti = 18444uS, TON = 2000uS, ILED = 700mA, NLED = 8
IMAGE 7 - α=60°, gain = 9.4dB, Ti = 18444uS, TON = 2000uS, ILED = 700mA, NLED = 8
Figure 36. Outdoor Test Results 2 (Strobed IR - Different Viewing Angle)
However, the lower angle made the distance between the camera (and its IR LEDs) and the car lower, thus
making the license plate looks noticeably brighter as shown in Image 5 and Image 7. Image 5 has a well-
exposed image while Image 7 has an overexposed license plate and it becomes difficult to determine whether
-35-
the loss of detail is due to motion blur or highlight clipping. At this point, we can confirm that the brightness
improvement from lowering the angle can lead to more motion blur.
After discussing possible brightness improvement with lower angles, now it becomes interesting to see how
much constant IR can be improved by using a lower angle. Figure 37 shows that the image taken with α=70°
has a significant improvement over the shot with α=80°. However, the same motion blur problem occurs
as previously discussed.
IMAGE 2 - α=80°, gain = 9.4dB, Ti = 2000uS, TON = constant, ILED = 350mA, NLED = 1
IMAGE 6 - α=70°, gain = 9.4dB, Ti = 2000uS, TON = constant, ILED = 350mA, NLED = 1
Figure 37. Outdoor Test Results 3 (Constant IR - Different Viewing Angle)
Beside from the cases planned in Section 4.3, it is also interesting to check how STS will perform if the
condition is not so dark. In one occasion of the outdoor test, the setup was set earlier when the sun was not
fully set. The same outdoor test was redone with the same strobing method, same strobing length, but
different iris and gain (F1.99, Gain = 0.0dB) to compensate the brighter skylight. The resulting image (Image
8 in Figure 38) shows two different license plates visible at the same time with different level of sharpness
and shape. The sharper one positioned approximately in the middle of the blurry plate image, which
occupies more space in the frame. This blurry image is most likely a result of light integration of ambient
light, which extends much longer than the strobe itself. As the ambient light was less intense than the IR,
the image appears less bright compared to the sharp one. With this being said, strobing with STS can actually
produce acceptable results as long as the light of the strobed IR is the primary reflected light from the license
plate. Moreover, if we consider using a faster development platform in the future, we might have a higher
readout rate, which can lower the minimum integration time for a given strobe length and thus filter out the
ambient light even more. As an example, with readout rate of 120FPS, we could reduce the integration time
from 18444uS (60FPS readout rate) to 10222uS for a 2000uS-long strobe.
-36-
IMAGE 8 - α=70°, gain = 9.4dB, Ti = 2000uS, TON = constant, ILED = 350mA, NLED = 1
Figure 38. STS Result in a Not-so-dark Environment
6.3 Implementation Feasibilities
Both indoor and outdoor tests have explained the possible image quality improvements that strobed IR can
offer to the constant IR. In the outdoor test, the development platform is equipped with 8 LEDs, which
were driven with higher than specified current. This configuration might raise questions such as whether
the real average power consumption is close to the theoretical calculation and whether the negative impact
of the high peak power consumption is manageable. The answer to these power-related question can be
found in [4] as that research was done in parallel with this thesis and with a similar strobing method.
On the other hand, the indoor test has helped ease the development process significantly. An outdoor test
might be the closest resemblance of LPR activity, but to do a proper outdoor test one particular car has to
run in a loop as much as the number of test cases. It is a very resource consumptive process, and there are
also other variables such as traffic and weather condition that might prevent the test to happen. Indoor test
adversely could provide a stable environment at any given time.
The thesis has also emphasized the improvement for LPR scenario, especially in the extreme condition (dark
highway). However, this does not limit improvements in more typical applications. Surveillance activities
that are focused on human behavior, for example, can also benefit from this technique as night time
recording is usually done with slower shutter speed. The scene brightness improvement for this scenario
might not be as significant, but the ability to use a shorter strobe length may help to fight the motion blur.
The last concern about strobing comes from the individualistic nature of the implemented system. The
thesis only presents a way to synchronize one image sensor with one IR system. If several cameras exist in
the same environment, then a more advanced synchronization is needed to avoid the flickering effect that
occurs for the unsynchronized cameras. The PoE cable (the only cable that is connected to a network
camera) is probably not a suitable media to transfer the strobing signal as low latency is a must. A different
way to address this problem is to use narrower LED modules to prevent the area of interest being lit up by
multiple IR systems.
-37-
7 CONCLUSION AND FUTURE WORK
7.1 Conclusion
To begin with, we revisit the first research question on how to synchronize frame capture (sensor) with
strobed IR. The rolling shutter timing analysis yielded two possible strobing methods which are the whole-
frame strobing and the shared-time strobing. The latter was chosen due to a more beneficial outcome that
can be carried out for the evaluation scenario (license plate recognition). The strobing light is actuated
through a LED driver that supports responsive pulse-width modulation dimming. The strobing signal for
this driver is generated in the PLD by utilizing the sync codes which is provided in the sensor output data
stream. The variable needed to control the strobe length can be adjustable by modifying the text file which
is stored in the ARTPEC (main CPU).
The second research question is how does strobed IR performs compared to constant IR. To evaluate both
systems we first determined the evaluation scenario (LPR). A speed transformation technique was derived
to understand the car movement better. The use of a spinner with license plate characters was proposed to
mimic the car movement in real life. The indoor test shows that strobed IR improved object brightness and
reduced the rolling shutter artifacts when compared to constant IR illumination. In the outdoor test,
constant IR required a lower viewing angle (shorter camera-to-car distance) to correctly expose the license
plate, while strobed IR managed to yield acceptable image even in a high angle (which is more susceptible
to glare from car’s headlights). The ability to shot in high angle (80°) also makes it easier to produce blur-
free images as the car speed translates to a lower sensor plane speed. A general advantage of the strobed IR
is that the system has more headroom to compensate other limitations such as a noisy sensor and lenses
with small iris.
Now we need to address the main research question. The image quality of surveillance camera can be
improved by utilizing strobed IR. The research has demonstrated the possible improvements, especially in
license plate recognition application without increasing the average power consumption (even lower than
constant IR as seen on Figure 39) and compromising eye-safety.
Figure 39. Theoretical IR LED Power Consumption
note: strobe length = 2000uS; frame period = 40000uS; duty cycle = 0.05
7.2 Future Work
Possibilities of using strobed IR to get improvements in other surveillance activities should be discovered,
as this thesis only cover a small part of it. Even in LPR, the recording condition can change throughout the
day and hence a system that is able to switch between different strobing techniques or constant light is
worth to be explored. Moreover, it is also interesting to see if synchronized strobing of multiple network
cameras, which record the same scene, is feasible.
0.30W
1.30W
0.96W
1.13W
5.18W
1.30W19.20W
1.13W
0 2 4 6 8 10 12 14 16 18 20
Indoor Test - Strobing (4 LEDs)
Indoor Test - Constant (1 LED)
Outdoor Test - Strobing (8 LEDs)
Outdoor Test - Constant (1 LED)
Peak power Average power
-38-
BIBLIOGRAPHY
[1] “OSRAM SFH 4715A Datasheet.” OSRAM, 27-Sep-2016. [2] A. M. Earman, “Application Note: Eye Safety for Proximity Sensing Using Infrared Light-emitting
Diodes.” Renesas, 28-Apr-2016. [3] P. L. Bender et al., “The Lunar Laser Ranging Experiment: Accurate ranges have given a large
improvement in the lunar orbit and new selenophysical information,” Science, vol. 182, no. 4109, pp. 229–238, Oct. 1973.
[4] C. Tormo, “Energy efficiency and power consumption improvement of IR illumination for surveillance cameras,” KTH Royal Institute of Technology, Stockholm, 2018.
[5] “Technical white paper: CCD and CMOS Technology.” Axis Communications, 2010. [6] “Technical note: Rolling Shutter vs. Global Shutter.” QImaging, 2014. [7] J. C. Lee, “Electronic Device with Modulated Light Flash Operation for Rolling Shutter Image Sensor,”
US20150002734A1, 01-Jan-2015. [8] S. Du, M. Ibrahim, M. Shehata, and W. Badawy, “Automatic License Plate Recognition (ALPR): A
State-of-the-Art Review,” IEEE Trans. Circuits Syst. Video Technol., vol. 23, no. 2, pp. 311–325, Feb. 2013.
[9] “White paper: WDR solutions for forensic value.” Axis Communications, Oct-2017. [10] “Datasheet: IR OSLUX (810nm) - 30° / 8° tilted, SFH 4786S, version 1.0.” OSRAM, 23-Oct-2017. [11] “User Guide: AN-1750 LM3406 Evaluation Board.” Texas Instruments, May-2013. [12] “IEEE Standard for SystemVerilog–Unified Hardware Design, Specification, and Verification
Language,” IEEE Std 1800-2017 Revis. IEEE Std 1800-2012, pp. 1–1315, Feb. 2018. [13] Renesas Electronics Corporation, “Eye Safety for Proximity Sensing Using Infrared Light-emitting
Diodes,” Application Note AN1737, Apr. 2016. [14] “IEC 62471: Photobiological safety of lamps and lamp systems.” IEC, 2006.
-39-
APPENDIX
Eye Safety Calculation according to [1]. Some modifications were done to calculate the average radiated
power properly. Emission Limits are set to pass Exempt Risk Group.
LED (SOURCE) PROPERTIES
1 Nominal wavelength 810 nm
2 Emission half-angle 15 deg
max. radiant intensity, Ie_MAX 2500 mW/sr
measured at (IMEAS) 1000 mA
3 lens diameter 3.160 mm
4 extended source area, A 7.843 mm2
DRIVING CONDITIONS
5 NLED 8
6 drive current, ILED 700 mA
7 strobe length, TON 2 ms
8 frame period, TFP 40 ms
9 average emitted power, Ie 0.700 W/sr
VIEWER CONDITIONS
10 distance from eye to light source, d 0.200 m
11 fully dilated pupil diameter 7.000 mm
12 mean source extension, Z=[(l+w)/2] 0.003 m
13 Angular subtense of source, α 0.016 rad
CORNEAL EXPOSURE HAZARD
14 Exposure time 1000 s
15 Cornea Exposure (Ee) 17.500 W/m2
16 Exposure Limit (EIR) 101.221 W/m2
safety factor 5.784
RETINAL THERMAL HAZARD
17 Exposure time 10 s
18 Limit of angular subtense on retina 0.0110
19 Angular subtense on retina, αeff 0.016
20 Burn hazard weight function, Rλ 0.603
21 Burn hazard weighted radiance (LR) 53781.63407 W/m2/sr
22 Burn hazard weighted radiance limit 1772151.899 W/m2/sr
safety factor 32.95087495
RETINAL THERMAL HAZARD, WEAK STIMULUS
23 Exposure time 1000 s
24 Limit of angular subtense on retina 0.0110
25 Angular subtense on retina, αeff 0.016
26 Burn hazard weighted radiance for weak stimulus (LIR) 53781.63407 W/m2/sr
27 Retinal hazard thermal limit 379746.8354 W/m2/sr
safety factor 7.060901775
-40-
Table 18. Exposure Time for Exposure Limit Calculation
RISK EXPOSURE TIME
EXEMPT LOW RISK MOD RISK
Retina Thermal LR 10 10 0.25
Retina Thermal, weak stimulus LIR 1000 100 10
Corneal 1000 100 10
Table 19. Limit of Angular Subtense for Different Time Range
TIME RANGE LIMIT OF ANGULAR SUBTENSE (RADIANS)
t < 0.25s 0.0017
0.25s < t < 10s 0.011√(𝑡/10)
t > 10s 0.011
Table 20. Emission Limits for Each Risk Group
RISK EMISSION LIMITS
UNITS EXEMPT LOW RISK MOD RISK
Retina Thermal LR 28000/α 28000/α 71000/α W/m2/sr
Retina Thermal, weak stimulus LIR 6000/α 6000/α 6000/α W/m2/sr
Calculation note:
1,2,3 Taken from LED module datasheet [10].
4 Area of lens, where the worst-case surface area is used (assuming that the whole radiometric
power is emitted from 1 LED).
5 Number of identical LED modules
6,7,8 Depends on strobing and shooting parameters
9 𝐼𝑒 = 𝐼𝑒_𝑀𝐴𝑋∙𝑁𝐿𝐸𝐷𝐼𝐿𝐸𝐷
𝐼𝑀𝐸𝐴𝑆∙
𝑇𝑂𝑁
𝑇𝐹𝑃
10 200mm is taken to adhere with IEC 62471 standard
11 Worst-case pupil size (biggest opening)
12 𝑍 =𝑙+𝑤
2; l = w = light source lens diameter
13 α =𝑒𝑦𝑒−𝑡𝑜−𝑙𝑖𝑔ℎ𝑡 𝑠𝑜𝑢𝑟𝑐𝑒 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒
𝑙𝑒𝑛𝑠 𝑑𝑖𝑎𝑚𝑒𝑡𝑒𝑟
14, 17, 18 Depends on the risk group, see Table 18.
15 𝐸𝑒 =𝐼𝑒
𝑑2 [𝑤
𝑚2]
-41-
16 18000 ∙ 𝑡−0.75 [𝑤
𝑚2] (𝑡 ≤ 1000𝑠)
100 [𝑤
𝑚2] (𝑡 > 1000𝑠)
18 Limit of angular subtense from given exposure time (17), see Table 19.
19 Equals to (13), unless (18) is bigger then use (18)
20 𝑅𝜆 = 10700−𝜆
500
21,26 𝐿𝑅 ≈𝐼𝑒∙𝑅𝜆
𝐴
22, 27 See Table 20.
TRITA-EECS-EX-2018:450
www.kth.se