stride : scene text recognition in-device
TRANSCRIPT
STRIDE : Scene Text Recognition In-DeviceRachit S Munjal
On-Device AISamsung R&D Institute
Bangalore, [email protected]
Arun D PrabhuOn-Device AI
Samsung R&D InstituteBangalore, India
Nikhil AroraOn-Device AI
Samsung R&D InstituteBangalore, India
Sukumar MoharanaOn-Device AI
Samsung R&D InstituteBangalore, India
Gopi RamenaOn-Device AI
Samsung R&D InstituteBangalore, India
Abstract—Optical Character Recognition (OCR) systems havebeen widely used in various applications for extracting semanticinformation from images. To give the user more control over theirprivacy, an on-device solution is needed. The current state of theart models are too heavy and complex to be deployed on-device.We develop an efficient lightweight scene text recognition (STR)system, which has only 0.88M parameters and performs real-timetext recognition. Attention modules tend to boost the accuracyof STR networks but are generally slow and not optimized fordevice inference. So, we propose the use of convolution attentionmodules to the text recognition networks, which aims to providechannel and spatial attention information to the LSTM moduleby adding very minimal computational cost. It boosts our wordaccuracy on ICDAR 13 dataset by almost 2%. We also introducea novel orientation classifier module, to support the simultaneousrecognition of both horizontal and vertical text.
The proposed model surpasses on-device metrics of inferencetime and memory footprint and achieves comparable accuracywhen compared to the leading commercial and other open-sourceOCR engines. We deploy the system on-device with an inferencespeed of 2.44 ms per word on the Exynos 990 chipset device andachieve an accuracy of 88.4% on ICDAR-13 dataset.
I. INTRODUCTION
Mobile phones are ubiquitous today. Billions of imagesare captured every day for personal and social use. This hasdriven the need for image understanding tasks such as objectdetection and optical character recognition (OCR). OCR is oneof the most renowned and foremost discussed Computer Visiontask, which is used to convert the text in images to electronicform in order to analyze digitized data. The recognition of texthas a wide range of applications such as image retrieval, sceneunderstanding, keyword-based image hashing, document anal-ysis, and machine translation. Though OCR pipelines workwell to retrieve text from scanned documents, these traditionalmethods fail to work on images occurring in natural scenes.We focus on analyzing and extracting text from real-worldscene images, commonly known as Scene Text Recognition(STR).
STR poses a great challenge due to the large variance oftext patterns and fonts, complex foreground-background vari-ations, imperfect imaging conditions, and highly complicated
backgrounds. This makes it a much more complex task thanconventional OCR. With the advent of deep learning, deepneural networks have been used to solve the task of scene textrecognition. These networks are computationally expensive,aiming to achieve the best possible recognition results onpopular benchmarks. Some of them also require the use oflexicon dictionaries or language-model-based corrections toimprove the prediction accuracy. This increases the memoryconsumption of the network further. But, due to the natureof downstream applications, it becomes necessary to processimages in real-time, using the limited computation powerof smartphones. We aim to build an efficient and compactgeneral-purpose text recognition system, supporting diversesymbols and characters and consequently, multiple languages.
Other than dealing with perturbations such as blurriness,contrast mismatch, altered brightness, and so on, a general-purpose OCR system must also be able to deal with vari-ations in orientation. [1], [2] have proposed transformationmodules to normalize the text image into a straight line.But vertical text, in which horizontal characters are stackedvertically, is not handled by the current OCR systems. Thisorientation is different from orthogonally rotated top-to-bottom
Fig. 1. 1 and 2 are orthogonally rotated images, whereas 3 is a type of verticalimage which is not handled by conventional OCR.
arX
iv:2
105.
0779
5v1
[cs
.CV
] 1
7 M
ay 2
021
or bottom-to-top text which can be handled by clockwise oranti-clockwise rotation of the text. Here on, for the ease ofunderstanding, we will denote the vertically stacked horizontaltext as Vertical Text and the orthogonally rotated verticaltext as simply Rotated Text. The distinction between thesetwo types of images is shown in Fig. 1. Vertical texts arepredominantly found in East-Asian scripts including Chinese,Korean, Vietnamese and Japanese. Some methods recognizevertical text by detecting character boundaries and recognizingeach character in the word. But, due to improper characterboundaries in scene text scenarios, high latency, and less word-level context in the case of character-level recognition, thisapproach is not very efficient and accurate.
In this paper, we propose STRIDE: Scene Text RecognitionIn-Device, a light-weight CNN-LSTM based network, whichperforms real-time text recognition of multi-oriented, multi-scale scene text images on-device. We develop 4 modelsto support 4 different scripts and cumulatively support 34languages. Each network has around 0.88M parameters, whichis at least 10x smaller than existing models with comparableprecision. These optimizations are covered in more detail infurther sections. The main contributions of this paper are asfollows:
1 Simultaneous recognition of horizontal and vertical text2 Addition of convolution attention blocks and analyzing
its impact on OCR systems3 Developing a device ready text recognition solution, with
state of the art inference speed and comparable accuracyThe rest of the paper is organized in the following way.
Section II talks about related works. We elucidate the workingof our pipeline in section III. Section IV concentrates on theexperiments we conducted and the corresponding results weachieved. The final section V takes into consideration thefuture improvements which can be further incorporated.
II. RELATED WORKS
Though the OCR problem has received attention for manydecades, the main focus was document images [3]. AlthoughOCR on document images is now well-developed, these meth-ods fail utterly when applied to natural scene images due to alarge number of variations in scene images.
In the last decade, the advent of deep learning has ledto substantial advancements in STR. Earlier methods weresegmentation-based methods [4] that aimed to locate thecharacters, use character classifier to recognize the characters,and then group characters into text lines. PhotoOCR [5] wasone such segmentation-based method that used a deep neuralnetwork trained on the extracted histogram of oriented gradient(HOG) features for character classification. The performanceof segmentation-based methods are constrained by their depen-dence on accurate detection of individual characters, which isa very challenging problem. Also, most of the methods [6], [7]rely on post-OCR correction, such as lexicon set or languagemodels to capture context beyond a character. This leads toan increase in their time and memory consumption.
Segmentation-free methods focus on mapping the entiretext image to a target string. Jaderberg [8] treated the wordrecognition problem as a multi-class classification problem,where each class represents a different word. But, such anapproach is not scalable.
Recent methods have construed the problem of scene textrecognition as an image-based sequence recognition. CRNN[9] was the first combined application of CNN and RNNfor text recognition. It consisted of a fully-convolutional partof VGG [10], followed by 2 Bi-LSTM layers. For lexicon-free prediction, the network was trained using connectionisttemporal classification (CTC) [11] to find the label sequencewith the highest probability.
Multiple variants have hence been proposed to improve theperformance of CRNN. Image processing techniques such asbackground removal, super-resolution [12], and image recti-fication [1] have been employed to reduce the load on thedownstream stages for learning complex features. Better con-volutional feature extractors such as Resnet [13] and RCNN[14] have been used to better extract the text features fromcomplex images. But these feature extractors are very deepand have high latency.
Along with deep feature extractors, attention mechanism[1], [15], [16] is often combined with RNN for character-sequence decoding. This can help in extracting text fromirregular scene text crops and achieve better performancein regular word crops too. [17] conduct a fair comparisonto identify the impact of variations in different modules.Although the newer variants have achieved better performanceon various benchmarks, they have come at a cost to memoryand computation.
Convolution based Attention modules such as convolutionalblock attention module [18] and global squeeze-excite blocks[19] have shown to increase the performance of convolutionalnetworks on benchmarks such as Imagenet for detection andsegmentation tasks. The effect of these attention modules hasnot been explored in the domain of text recognition till now.
The Global Squeeze-Excite blocks provide channel atten-tion to the network and can be easily integrated with anyrecognition feature extraction. It recalibrates channel-wisefeature responses by explicitly modeling inter-dependenciesbetween channels. Global Squeeze-Excite blocks have beenshown to increase the search space with very little effecton time and memory [20]. Alternatively, Convolutional BlockAttention Module (CBAM) proposed modification to the SEblock, which helps to provide spatial attention in addition tochannel attention. CBAM contains two sequential sub-modulescalled the Channel Attention Module (CAM) and the SpatialAttention Module (SAM), which are applied in that particularorder.
Most of the current STR models assume the input imageas horizontal. Even the irregular scene text recognizers fail onvertical text images due to the structure of the network [21].Some models use vertical information [22]–[24]. But, thesemodels use attention mechanism, which makes them infeasibleto be used on-device due to their computation requirements.
Fig. 2. STRIDE Network Pipeline: The word boxes detected from the text localization network are passed to the feature extractor, after applying selectiverotation. The orientation of each word is classified separately and passed to the sequence model with the temporal word features extracted.
The demand for on-device OCR has lead to prominentfirms offering OCR such as Google’s Mlkit1 and Apple’s Textrecognition in vision framework 2. But these products supporta minimal number of languages on-device. PP-OCR [25] is anopen-source on-device OCR supporting multiple scripts. But,we outperform the network in terms of both accuracy andinference speed.
III. NETWORK ARCHITECTURE
In this section, we describe our network, which takes animage and bounding box of the word as input and convertsit into a sequence of labels. The network is based on theConvolutional Recurrent Neural Network (CRNN) architecture[9]. We incorporate certain modifications to the network, toperform simultaneous recognition of horizontal and verticaltext. We also focus on building a compact network, suitablefor on-device inferencing without compromising on accuracy.The network architecture is shown in Fig. 2. The proposedmodel consists of four components: selective rotation, featureextraction, sequence modeling, and prediction. Each compo-nent of the network is carefully designed and optimized tominimize the overall network size.
A. Selective Rotation
Text in scene images comes in diverse shapes and varieties.The natural text present on signboards, restaurant boards,banners, etc. is generally skewed or rotated. The recogni-tion model is made to handle perspective information, textskewness, and rotation up to a certain level. But, as the skewincreases it starts impacting the recognition results. Rectifyingand pre-processing each word crop adds to the computationcost of the network. To handle this trade-off between accuracyand inference time, we extract rotation angle and skewnessinformation from the localization module [26] and pass onlythe highly rotated words through a computer vision-based
1developers.google.com/ml-kit/vision/text-recognition2developer.apple.com/documentation/vision/recognizing text in images
perspective transformation. The model is made invariant tosmall amounts of distortions in the text.
B. Feature ExtractionThe feature extractor block consists of CNN and max-
pooling layers which extracts attributes related to the text andlearns feature representation for both horizontal and verticaltext. It has 4 convolution layers with kernel size of 3x3.The first two convolutions operate on full word crop width,which is followed by an average pool layer across the widthdimension. So, the features fed to the following convolutionlayers and the sequence model are represented by half theoriginal word width.
Low-level features play a huge role to distinguish subtlechanges in characters due to diacritics in Latin script and tocorrectly identify minute differences in Chinese and Koreancompound characters. So, to preserve the important low-leveldetails, we pass a residual skip-connection from the secondconvolution layer and concatenate it with the output of thefourth layer. All crops are scaled to a fixed height of 16while preserving the aspect ratio. Thus, all input word-cropsare of size 16 ∗ width ∗ 3, where the width of the input isdetermined from the aspect ratio of the original word crop.Each convolution layer is followed by a max-pool layer acrossthe height dimension. So, the final output from the featureextractor is of size 1 ∗ (width/2) ∗ 164, 164 representing thenumber of channels fed to the sequence modeler.
Attention mechanism provides localized information andgenerally boosts the recognition accuracy when used in theencoder or decoder blocks of an OCR network. But, thiscomes with an additional time and network complexity. [17]shows the time gain when the attention module is used inconjunction with the base network. This additional overheadof conventional attention methods and transformer modulesmakes it unsuitable to be directly deployed for on-deviceinference.
To overcome this drawback and take advantage of the im-proved feature discriminability, we propose the use of convolu-
Fig. 3. Feature Extractor and Orientation Classifier Module. CBAM is used to get channel and character region attention information. The detected orientationis concatenated to the extracted features and fed to the LSTM. Input Image is of size: 16*width*3, where height =16 and 3 denotes the RGB channels.
tion attention blocks in the encoder networks for recognition. Itprovides attention information to the sequence modeling blockwith very minimal computational overhead. These blocks canbe integrated with any recognition network.To the extent ofour knowledge, the effect of these attention modules has notbeen explored in the domain of text recognition.
We experiment with Global Squeeze-Excite (GSE) Blocksand Convolution Block Attention modules, both of whichprovide channel or region-specific localized information asdescribed in Section 2. The GSE Blocks squeeze globalspatial information into a channel descriptor and this channelattention information is mapped to the input feature. WhereasConvolution Block Attention Module (CBAM) provides bothspatial and channel attention, through its two sequential sub-modules called the Channel Attention Module (CAM) and the
Fig. 4. Feature maps extracted after the third convolution layer. CBAM blocksare able to clearly separate the characters from its background.
Spatial Attention Module (SAM).Fig. 3 shows the components of the CBAM block. In a semi-
supervised way, SAM tries to learn the region of interest foreach character in the word, which helps in providing a betterbackground-foreground separation. Whereas, the CAM mod-ule focuses on learning the discriminative features betweencharacters, that leads to excitation of desired channels.
Fig. 4 shows the feature maps from the third convolutionlayer of the network with and without the CBAM block. Thefeature maps of the network using CBAM provide better atten-tion and character separation, which when fed to the sequencemodeling block leads to better recognition results. Ablationstudy of SE and CBAM blocks for the text recognition modelis performed in section IV. Empirically, CBAM provides thebest overall result, with minimal computational overhead.
C. Orientation Classifier
Due to the frequent use of vertical texts in East-Asian scriptsincluding Chinese, Korean, Vietnamese, and Japanese, thereis a need to recognize vertical text along with the horizontaltext. To solve this problem, we aim to recognize horizontaland vertical text simultaneously with a single model, withoutany additional overhead.
Predicting orientation at character-level is difficult due tothe orthogonal nature of character pairs like Z and N or Hand I. Hence, the orientation of the word requires a globalword-level context, rather than a character-level context. Toget this word-level information, we apply Global AveragePooling (GAP) across the width dimension. This information
is then fed to a fully connected layer with sigmoid activation topredict a single orientation per word, instead of the per-pixelvalue of orientation across the width. Fig 3 shows the workingof our orientation classifier. The prediction of the orientationclassifier can be represented as:
ypi = σ ( Dense ( GAPaxis=2(CB1))) (1)
where CB1 is the output from the first CBAM block andypi is the prediction of the orientation classifier.
During training, we jointly optimize the recognition networkand the orientation classifier. The orientation classifier istrained using binary cross-entropy loss and it is defined as:
Lo = − 1
N
N∑i=1
yi log(ypi ) + (1 − yi) log(1 − ypi ) (2)
where yi is the actual orientation, N is the batch size andLo defines the orientation loss.
D. Sequence modeling
Sequence modeling captures contextual information withina sequence of characters, for the next stage to predict eachcharacter. Our sequence modeling stage consists of 1 Bi-LSTMlayer. LSTM has shown strong capability for learning mean-ingful structure from an ordered sequence. Another importantproperty of the LSTM is that the rate of changes of the internalstate can be finely modulated by the recurrent weights, whichcontributes to its robustness against localized distortions of theinput data.
The LSTM receives input from both the orientation classifierand feature extractor. The orientation information is a pieceof non-temporal information that is fed to the LSTM with thetemporal word features extracted from the convolution layers,the width of the image being the time dimension. There area few ways to feed non-temporal data to the bi-directionalLSTM. It can be fed as the first and last timestamp of thetemporal sequence. The disadvantage of this approach is that ifthe input sequence is long enough, the orientation informationmay not be properly propagated to all the timestamps, and itmight forget the conditioning data. The approach we followis to append the orientation information to each timestamp,effectively adding the feature across the width dimension ofthe convolution features. We are able to retain the accuracyof horizontal text on the benchmark dataset, in addition tosupporting vertical word recognition, as shown in Table II.
For faster inference, we use LSTM with recurrent projectionlayer [27] to decrease the computational complexity. Therecurrent projection layer projects hidden states at every timestep to a lower dimension. The projected hidden states inboth directions are then concatenated and fed to the predictionstage.
E. Prediction
The prediction stage outputs a sequence of characters fromthe identified features of the word crop. The per-frame pre-dictions made from the LSTM are fed to a fully connected
Fig. 5. Synthetic dataset created by rendering font on varied backgroundsand using data augmentation techniques.
layer to obtain per-frame probability distribution over thelabels. Finally, Connectionist Temporal Classification (CTC)[11] is used to transform frame-wise classification scores tolabel sequence. If xt is the per-frame probability distributionover the set L′, where L′ represents the set of all labels inaddition to blank symbol and the ground truth label sequenceis represented by y∗, then CTC defines the objective functionto be minimized as follows:
Lc = −∑
log p (y∗ | x) (3)
Combined with orientation loss defined in Eq. 2, the com-bined loss equation is:
L = Lc + λLo (4)
where λ is a hyper-parameter controlling the trade-offbetween the two losses. λ is set to 1 in our experiments.
For fast inference, we use the greedy decoder, which as-sumes the most probable path to be the most probable labelingto estimate the character sequence.
IV. EXPERIMENTS
A. Datasets
The diversity of training datasets plays an important role increating a robust model with high performance. But, the sizeof the existing real-world datasets is small to train a highlyaccurate scene text recognizer. The real-world vertical imagesare also limited in number and a few were collected from thedatasets mentioned below. Labeling scene text images is costly.Thus, we rely on synthetic datasets for training our network.HVSynth: We create our own synthetic dataset for horizon-tal and vertical text, by generating text images with basicdata augmentation techniques such as rotation, perspectivedistortion, blurring, etc. We also apply text blending, to addbackground noise to the text. We create a total of 5L samplesfor horizontal text and 1L for vertical text. Fig. 5 and Fig. 6shows some samples of synthetic horizontal and vertical text.
Along with our synthetic datasets, we use standard syntheticdatasets such as:MJSynth (MJ) [8] is a synthetic dataset designed for SceneText Recognition. It contains 8.9M word box images generatedusing 90k alpha-numeric words. We use this dataset to trainour Latin model.SynthText (ST) [28] is another synthetically generated dataset.Even though SynthText was an arrangement for detection, it
Fig. 6. Vertical Natural and Synthetic Text Samples
has been also used for Scene Text Recognition by croppingoff word boxes. SynthText has around 6M training data oncewe render the text samples as word boxes. We also use thesame method to generate word crops for all the other scripts.
We use the following real-world datasets for training andevaluation:ICDAR 2013 (IC13) [29] was created for the ICDAR 2013Robust Reading competition for reading camera capturedscene texts. The dataset has 848 images for training and 1095images for testing.ICDAR 2015 Incidental Text (IC15) [30] consists of a lot ofirregular text and highly blurred images. It has 4468 trainingimages and 2077 testing images.IIIT5k-Words (IIIT5k) [31] consists of images crawled fromgoogle search result. The dataset consists of 5000 images thatare split into a training set of 2000 images and an evaluationset of 3000 images.Street View Text (SVT) [32] consists of 247 images for train-ing and 647 images for evaluation. The images are outdoorstreet images collected using Google Street view. Some imagesare severely corrupted by noise, blur, and low resolution.ICDAR 2019 multi-lingual scene text (MLT) [33] consistsof around 90k word crops belonging to 7 scripts. Though thedataset was created for script identification, the word cropshave been annotated and can be used for training STR.
B. Training details
We implement the network using Tensorflow 2.3 [34] frame-work. All experiments are carried out on GeForce GTX 1080tiGPU, and 16GB RAM. The training batch size is 64. We useAdam optimizer [35] to train the models with initial learningrate 1e− 3. The learning rate was halved when the validationloss did not decrease for more than 2 epochs, and the trainingwas stopped once the learning rate reached 1e− 5.
Training CTC is hard and the models take a lot of timeto converge [36]. For improving stability during training, weexplore curriculum learning strategies. Curriculum learningwas proposed to address the challenges of training deepneural networks under non-convex training criteria. For afew epochs, we train our network on a subset of trainingimages consisting of shorter words and clear text. After everyepoch, we increased the maximum length of the words alongwith the degree of perturbations. Adopting curriculum trainingstrategies significantly fastened the training of the network.
TABLE IExperimental Results
Model Params IIIT SVT IC13 IC15CRNN 8.3M 82.9 81.6 89.2 64.2RARE 10.8M 86.2 85.8 91.1 68.9
STAR-Net 48.7M 87.0 86.9 91.5 70.3Rosetta 44.3M 84.3 84.7 89.0 66.0
Our Model 0.88M 82.5 81.3 88.4 68.7
Both synthetic and natural scene text contains few wronglyannotated images and highly complex background images.Also, many images of the MjSynth, SynthText, and IIIT5kdatasets have case-insensitive labeling, which makes it difficultfor the model to learn case information. To tackle these data-related issues, we weakly supervise our model, by monitoringper image loss. We reduce the training data instances that havea very high loss on a trained model, which could possibly bedue to wrong annotations or hard examples. We also correctthe wrong case annotations of the dataset, by taking input fromthe model, in case of incorrect case predictions. The droppedimages are fed back to the data pipeline after a few epochs,to make up for the dropped hard examples.
C. Results on benchmark datasets
Though there are a few benchmark recognition datasets forLatin, there are no such datasets for other scripts. There areend-to-end benchmarks for Chinese, but none for recognition.The recognition word accuracies on the public real-worddatasets mentioned in section IV-A, obtained by some state-of-the-art techniques and our proposed model STRIDE are givenin Table I. The results of the other models are taken from [17]which compares different STR models.
Though our network is a minuscule version of CRNN hav-ing around a tenth of its parameters, the recognition accuraciesof our model is on-par with CRNN. We perform better in IC15,since it consists of a few vertical and orthogonally rotatedimages which our model can handle. By using computationallyintensive modules such as attention for decoding (in RARE)and Resnet for feature extraction (in Star-Net), the perfor-mance gap widens. But due to the on-device constraints, wecannot integrate these modules in our model.
D. Result on HVSynth dataset
The performance of our model on the HVSynth datasetis shown in Table II. Our model is able to predict well onboth horizontal and vertical word crops with a negligibleloss on horizontal word crops performance, compared to our
TABLE IIVertical Text Classification and Recognition Result on Latin synthetic dataset
Network HorizontalWord
Accuracy
VerticalWord
Accuracy
OrientationClassifierAccuracy
Base 94.77 - -Base + Orientation
Classifier94.25 96.12 97.34
Fig. 7. Complex Pass Cases
base model. The orientation classifier is able to learn thedistinction between horizontal and vertical text and predictsthe orientation with high accuracy.
E. Ablation study on attention modules
Table III shows the impact of attention modules on therecognition accuracies of our Latin model on IC13. As we cansee, convolutional attention blocks can boost the performanceof the network with little effect on latency and size. Thus, us-ing CBAM, which provides both spatial and channel attention,helps in better extraction of the features of the characters.
F. Results Analysis
Our model performs substantially well while handling com-plex images like those shown in Fig. 7. We also investigatethe failure cases of our model and some of them are shownin Fig. 8. A few common reasons for failure are elucidatedbelow:• Text Merging with Background: Model fails to handle
cases where the text merges with the background. Bet-ter feature extractors may improve performance in thisdomain.
• Shadow and Blur: Selective failure cases are observeddue to reflections in low resolution textured images.
TABLE IIIAblation Study of GSE and CBAM modules on IC13 data
Module WordAccuracy
CharAccuracy
Parameters Time
Base Network 86.5 92.2 850k 2.2 ms1 GSE Block 87.4 92.8 859k 2.28 ms2 GSE Block 87.7 93.1 868K 2.35 ms
2 CBAM Block 88.4 93.4 886K 2.44 ms
Fig. 8. Fail Cases
TABLE IVOn-Device Statistics
Script No. ofcharacterssupported
No. ofparameters (in
mil)
Time taken (inms)
Latin 236 0.88 2.44Korean 1330 0.99 2.81
Japanese 2647 1.11 3.21Chinese 5949 1.43 3.93
• Smudged Text: Existing models do not explicitly handlesmudged text in low resolution cases; super-resolutionmodules or image pyramids might improve performance.
• Occluded Text: Current research methods do not sub-stantially exploit contextual information to overcome oc-clusion. Future researches may utilize superior languagemodels to maximally make use of context.
G. On-Device Statistics
The model details for various scripts are shown in TableIV. The number of parameters of the model varies acrossscripts due to the difference in the number of characterssupported, which affects the number of parameters in the lastfully connected layer. For speed comparisons, we find out thetime taken to process a word crop of size 16∗64 on S20 whichhas Exynos 990 chipset and 8 GB RAM.
V. FUTURE WORK
Though the model performs well on regular datasets, themodel does not perform well on irregular datasets containingcurved text and distorted text due to model limitations. [17]has clearly shown that such images can be deciphered by usingcomputationally expensive modules such as 2-D attention[21]. Other solutions to read irregular word crops involvedeveloping efficient pre-processing modules that can transformand normalize the image [1]. We plan to explore both avenuesto develop a robust on-device OCR that can handle both typesof word crops.
Another interesting scope of future work will be to targetchallenging scripts like Arabic which are written right to left.The model also faces difficulty in extracting text written incalligraphic fonts and handwritten text. The model can be fine-tuned with such datasets and it remains to be seen how thissimilar architecture will perform in such cases.
VI. CONCLUSION
In this paper, we have presented STRIDE, a lightweightOCR solution. OCR is one of the most important and popularareas of research especially for scene text given the pop-ularity of high-resolution cameras in recent times, and thebillions of images captured daily. In this paper, we presentedSTRIDE, our novel, lightweight, lexicon-free, and on-devicesolution with real-time results. We have demonstrated thatthe architecture of our proposed system is universal, andscales to document and scene text, and to other scripts andlanguages as well. The system can be used to recognize bothhorizontally and vertically aligned text. We also introduce the
use of convolution attention blocks in STR networks and showits impact on accuracy through ablation studies. Additionally,we have compared our methods to previous works. We haveshown how we improve upon previous approaches and handlethe complex scenarios and challenges of scene text, at thesame time optimizing the network for real-time performanceon-device. The number of LSTM layers is decided to achieve atrade-off between model performance and model size. Stack-ing more LSTM layers didn’t give any significant boost inmodel accuracy. We are able to achieve comparable resultswith the current SOTA recognition models with only 0.88Mparameters and an On-Device inference time of 2.44ms. Wehope this will enable the use of OCR on camera images andscene text as a starting point for various downstream Visionand NLP tasks, entirely on-device giving real-time results andprotecting the user’s privacy at the same time.
REFERENCES
[1] B. Shi, X. Wang, P. Lyu, C. Yao, and X. Bai, “Robust scene textrecognition with automatic rectification,” in Proceedings of the IEEEconference on computer vision and pattern recognition, 2016, pp. 4168–4176.
[2] B. Shi, M. Yang, X. Wang, P. Lyu, C. Yao, and X. Bai, “Aster:An attentional scene text recognizer with flexible rectification,” IEEEtransactions on pattern analysis and machine intelligence, vol. 41, no. 9,pp. 2035–2048, 2018.
[3] G. Nagy, “Twenty years of document image analysis in pami.”[4] X. Chen, L. Jin, Y. Zhu, C. Luo, and T. Wang, “Text recognition in the
wild: A survey,” arXiv preprint arXiv:2005.03492, 2020.[5] A. Bissacco, M. Cummins, Y. Netzer, and H. Neven, “Photoocr: Reading
text in uncontrolled conditions,” in Proceedings of the IEEE Interna-tional Conference on Computer Vision, 2013, pp. 785–792.
[6] T. Wang, D. J. Wu, A. Coates, and A. Y. Ng, “End-to-end text recog-nition with convolutional neural networks,” in Proceedings of the 21stinternational conference on pattern recognition (ICPR2012). IEEE,2012, pp. 3304–3308.
[7] X. Liu, T. Kawanishi, X. Wu, and K. Kashino, “Scene text recognitionwith cnn classifier and wfst-based word labeling,” in 2016 23rd Inter-national Conference on Pattern Recognition (ICPR). IEEE, 2016, pp.3999–4004.
[8] M. Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman, “Syntheticdata and artificial neural networks for natural scene text recognition,”arXiv preprint arXiv:1406.2227, 2014.
[9] B. Shi, X. Bai, and C. Yao, “An end-to-end trainable neural networkfor image-based sequence recognition and its application to scenetext recognition,” IEEE transactions on pattern analysis and machineintelligence, vol. 39, no. 11, pp. 2298–2304, 2016.
[10] K. Simonyan and A. Zisserman, “Very deep convolutional networks forlarge-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[11] A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber, “Connection-ist temporal classification: labelling unsegmented sequence data withrecurrent neural networks,” in Proceedings of the 23rd internationalconference on Machine learning, 2006, pp. 369–376.
[12] D. Jain, A. D. Prabhu, G. Ramena, M. Goyal, D. P. Mohanty, S. Mo-harana, and N. Purre, “On-device text image super resolution,” arXivpreprint arXiv:2011.10251, 2020.
[13] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for imagerecognition,” in Proceedings of the IEEE conference on computer visionand pattern recognition, 2016, pp. 770–778.
[14] C.-Y. Lee and S. Osindero, “Recursive recurrent nets with attentionmodeling for ocr in the wild,” in Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition, 2016, pp. 2231–2239.
[15] Z. Cheng, F. Bai, Y. Xu, G. Zheng, S. Pu, and S. Zhou, “Focusingattention: Towards accurate text recognition in natural images,” inProceedings of the IEEE international conference on computer vision,2017, pp. 5076–5084.
[16] W. Liu, C. Chen, K.-Y. K. Wong, Z. Su, and J. Han, “Star-net: a spatialattention residue network for scene text recognition.”
[17] J. Baek, G. Kim, J. Lee, S. Park, D. Han, S. Yun, S. J. Oh, andH. Lee, “What is wrong with scene text recognition model comparisons?dataset and model analysis,” in Proceedings of the IEEE InternationalConference on Computer Vision, 2019, pp. 4715–4723.
[18] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutionalblock attention module,” in Proceedings of the European conference oncomputer vision (ECCV), 2018, pp. 3–19.
[19] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” inProceedings of the IEEE conference on computer vision and patternrecognition, 2018, pp. 7132–7141.
[20] M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard,and Q. V. Le, “Mnasnet: Platform-aware neural architecture search formobile,” in Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition, 2019, pp. 2820–2828.
[21] H. Li, P. Wang, C. Shen, and G. Zhang, “Show, attend and read: A simpleand strong baseline for irregular text recognition,” in Proceedings of theAAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp.8610–8617.
[22] X. Yang, D. He, Z. Zhou, D. Kifer, and C. L. Giles, “Learningto read irregular text with attention mechanisms,” in Proceedingsof the Twenty-Sixth International Joint Conference on ArtificialIntelligence, IJCAI-17, 2017, pp. 3280–3286. [Online]. Available:https://doi.org/10.24963/ijcai.2017/458
[23] Z. Cheng, Y. Xu, F. Bai, Y. Niu, S. Pu, and S. Zhou, “Aon: Towardsarbitrarily-oriented text recognition,” in Proceedings of the IEEE Con-ference on Computer Vision and Pattern Recognition, 2018, pp. 5571–5579.
[24] C. Choi, Y. Yoon, J. Lee, and J. Kim, “Simultaneous recognition ofhorizontal and vertical text in natural images,” in Asian Conference onComputer Vision. Springer, 2018, pp. 202–212.
[25] Y. Du, C. Li, R. Guo, X. Yin, W. Liu, J. Zhou, Y. Bai, Z. Yu, Y. Yang,Q. Dang et al., “Pp-ocr: A practical ultra lightweight ocr system,” arXivpreprint arXiv:2009.09941, 2020.
[26] R. S. Munjal, M. Goyal, R. Moharir, and S. Moharana, “Telcos:Ondevice text localization with clustering of script,” in press, 2021.
[27] H. Sak, A. W. Senior, and F. Beaufays, “Long short-term memoryrecurrent neural network architectures for large scale acoustic modeling,”2014.
[28] A. Gupta, A. Vedaldi, and A. Zisserman, “Synthetic data for textlocalisation in natural images,” in Proceedings of the IEEE conferenceon computer vision and pattern recognition, 2016, pp. 2315–2324.
[29] D. Karatzas, F. Shafait, S. Uchida, M. Iwamura, L. G. i Bigorda, S. R.Mestre, J. Mas, D. F. Mota, J. A. Almazan, and L. P. De Las Heras,“Icdar 2013 robust reading competition,” in 2013 12th InternationalConference on Document Analysis and Recognition. IEEE, 2013, pp.1484–1493.
[30] D. Karatzas, L. Gomez-Bigorda, A. Nicolaou, S. Ghosh, A. Bagdanov,M. Iwamura, J. Matas, L. Neumann, V. R. Chandrasekhar, S. Lu et al.,“Icdar 2015 competition on robust reading,” in 2015 13th InternationalConference on Document Analysis and Recognition (ICDAR). IEEE,2015, pp. 1156–1160.
[31] A. Mishra, K. Alahari, and C. Jawahar, “Scene text recognition usinghigher order language priors,” in BMVC-British Machine Vision Confer-ence. BMVA, 2012.
[32] K. Wang, B. Babenko, and S. Belongie, “End-to-end scene text recog-nition,” in 2011 International Conference on Computer Vision. IEEE,2011, pp. 1457–1464.
[33] N. Nayef, F. Yin, I. Bizid, H. Choi, Y. Feng, D. Karatzas, Z. Luo, U. Pal,C. Rigaud, J. Chazalon et al., “Icdar2017 robust reading challenge onmulti-lingual scene text detection and script identification-rrc-mlt,” in2017 14th IAPR International Conference on Document Analysis andRecognition (ICDAR), vol. 1. IEEE, 2017, pp. 1454–1459.
[34] M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin,S. Ghemawat, G. Irving, M. Isard et al., “Tensorflow: A system for large-scale machine learning,” in 12th {USENIX} symposium on operatingsystems design and implementation ({OSDI} 16), 2016, pp. 265–283.
[35] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014.
[36] F. Borisyuk, A. Gordo, and V. Sivakumar, “Rosetta: Large scale systemfor text detection and recognition in images,” in Proceedings of the24th ACM SIGKDD International Conference on Knowledge Discovery& Data Mining, 2018, pp. 71–79.