realtime webcam sudoku solver - codeproject®

8,617,442 members and growing! (64,158 online) 438 Sign out

Home Articles Quick Answers Discussions Learning Zones Features Help! The Lounge Search site

doicanhden

Article Browse Code Stats Revisions Alternatives

Prize winner in Competition "Best C++/MFC article of August 2011"

Prize winner in Competition "Best overall article of August 2011"

110 » Platforms, Frameworks & Libraries » Game Development » Games

Licence LGPL3First Posted 8 Aug 2011Views 74,058Downloads 8,568Bookmarked 253 times

Realtime Webcam Sudoku SolverBy Bojan Banko | 8 Aug 2011

C++ Dev Intermediate

Sudoku solver via a webcam: A nice computer vision application.

See AlsoMore like thisMore by this author

4.97 (233 votes)

Add your ownalternative versionDownload demo project - 170 KB

Download source - 307 KB

Introduction

This application might not have any practical value, but it is great from a learning perspective. I wantedto learn about computer vision. Computer vision is one of the most exciting areas of modern computing.It is also a difficult area. What is simple and obvious for the human brain is very difficult for a computer.Many things are still impossible with the current level of IT progress.

This application is implemented using low level C++ because I wanted to learn how things work underthe hood. If you’d like to start with a computer vision application, you should take an existing library(like OpenCV) and start from there. You can find several tutorials here on CodeProject. The webcamimage gathering source code is from from Vadim Gorbatenko (AviCap CodeProject).

The webcam acquires one image at a time (a frame). The flow of frames gives the impression of motion.The next image explains what this application does within a webcam frame.

Related Articles

http://www.codeproject.com/Articles/238114/Realtime-Webcam-Sudoku-Solver

4/1/2012 10:14 PM 1 trong 14

The numbers below the image are delays in milliseconds measured on my 2.8GHz PC with the webcamset to 640x480 pixels. The milliseconds show some interesting results. For example, the slowest step isthe webcam image gathering. 100 ms means that you will get 10 frames per second. This is poor.Webcams are typically slow. You can make them faster on account of quality. A lower resolution makesthe camera faster, but the image quality may become useless. Another surprise is why the conversion toblack and white is so slow (12ms). On the other hand, what is expected to be slow is OCR and thesolver. But those are surprisingly fast – only 7 ms. I will explain each step in detail and show how thingscould be improved. The source code function that invokes the above process isDoSomeImageProcessing().

How conversion to black and white works

Thresholding

Every computer vision application starts with the conversion from color (or grayscale) to monochromeimage. In the future, there will probably be some color-aware algorithms that will have some use ofcolors, but today computer vision applications are all monochrome based (daltonists).

The simplest method to convert color to monochrome is “global thresholding”. Suppose you have a pixelwith red intensity 200, green 200, and blue 200. Since the intensity goes from 0 to 255, this pixel isvery intensive in light. The threshold between black and white is the middle: 256/2=128. The meanintensity of our pixel is (200+200+200)/3 = 200, which is above the threshold value 128, so it will beconverted to white. But this simple global thresholding is not useful in real world applications. The nextimage shows that.

For a good conversion to monochrome, we will use adaptive thresholding. Adaptive thresholding doesn’tuse the fixed thresholding value 128. Instead, it calculates the threshold value separately for eachpixel. It reads 11x11 surrounding pixels and sums their intensity. Then the average mean value of sumsbecomes the threshold value for that pixel. The formula for the pixel is threshold = sum/121, where121=11x11, and sum is the surrounding intensity summed. If the central pixel intensity is greater thanthe calculated mean threshold, it will become white, otherwise black. On the next image, we arethresholding the pixel marked red. The same calculation is required for all other pixels. This is thereason why this is so slow. The program needs width*height*121 pixel reads.

Sudoku Solver and GeneratorSimple LINQ Sudoku SolverAnother Sudoku Solver and GeneratorSudoku Solver with LINQ (C# 3.0)Sudoku SolverSudoku (Suduku) SolverOO, Patterns and Sudoku Solver:Part 2Logical Sudoku SolverVisual Basic Sudoku Solver andGeneratorSuDoku Solver and GeneratorSudoku SolverOO, Patterns and Sudoku Solver:Part 1Building a Sudoku Solver in F#SudokuXSudokuWnd - a control thatimplements the Sudoku puzzleSolving SudokuSudoku as a CSPInformed Search Algorithms to SolveSudoku SamuraiA Sudoku Teacher Using BoostLibraries


4/1/2012 10:14 PM 2 trong 14

Integral image

This can be optimized by using the “integral image”. Integral image is an array of integers int[imagewidth*image height]. The concept is simple and powerful. The following image is an example.Suppose we have an image 12x12 pixels with all intensity values set to 1 for simplicity (in a real worldapplication, the sample image won’t be that simple). The integral image is the sum of all the pixels fromthe top and from the left to that pixel.

The next image is an example of how the integral image can be useful. The goal is to calculate the sumof all the pixels in the gray rectangle.

The formula is: sum=D-C-B+A. In our case: 110-44-40+16 = 42 (try to count manually).

Instead of reading all the pixels from the gray rectangle (which can be much bigger than this example),we only need four memory reads. This is a significant optimization. But even with this optimization, theconversion to black and white is still one of the heaviest steps.


4/1/2012 10:14 PM 3 trong 14

How to detect rotation

A webcam is not a scanner. A webcam picture is never perfectly aligned. We must expect the image tobe skewed and rotated. To detect the angle of rotation, we will use the fact that the Sudoku picturealways has some horizontal and vertical lines. We will detect the most expressive (strongest) linesomewhere near the center of the picture. The most expressive line is not affected by the noise. Thealgorithm to detect lines in a monochrome image is called Hough transform.

How it works: You must recall the school math formula of the line: y = (x * cos(theta) + rho) /sin(theta).

Where theta is the angle of the line and rho is the distance of the line from the center coordinate (0,0).

The important thing to note is that every line can be expressed with only two variables: theta and rho.

The program traverses through every pixel in the monochrome image. It skips white pixels. When onblack pixel, it tries to draw all the possible lines passing through that pixel (virtual lines). It works witha resolution of 1 degree. This means that every pixel will have 180 virtual lines passing through it. Whynot 360? Because the angles 180-360 are the copy of angles 0-180.

There is an array of virtual lines called an accumulator. The accumulator is a two-dimensional array withdimensions called theta and rho. The same theta and rho as in the line formula. Every virtual linerepresents one value in the accumulator. By the way, that is why this method is called transformation -it transforms the lines from [x, y] to [theta, rho] array. Every virtual line will increment a value in theaccumulator, raising the probability that this virtual line is a real line. This is a kind of voting. The reallines will have majority of votes.

After all pixels and their 180 virtual lines have voted, we must find the biggest values in theaccumulator. (By the way, why the name “accumulator”? Because it accumulates votes.) The winner isthe most expressive (strongest) line. Its dimensions theta and rho can be used with the above line


4/1/2012 10:14 PM 4 trong 14

formula to draw it.

The next image is a simple example. On the left, we have a three pixel line. Your eye can see a diagonalline that goes from the upper-left to the lower-right corner, but this is not that obvious for the computer.

How the line detection works: Look at the above image. The Hough transform algorithm skips the whitepixels. Every black pixel draws four green virtual lines (actually, it's 180 lines, but we take only fourhere for simplicity) passing through that pixel. The first pixel votes the lines A, B, C, and D. The secondpixel votes E, B, G, H. The third pixel will vote the lines I, B, K, L. Note that line B is common for allthree black pixels. It will get 3 votes. All other lines will get only one vote in the accumulator.Therefore, the virtual line B must be the winner - the real line.

The next image is a more complex example. On the left is the image of Sudoku grid lines. On the rightis the accumulator array after the Hough transformation algorithm execution. The bright areas in theaccumulator means there are many votes. Blackness means there is no chance to find a line there.Concentrate only on the brightest points. Each line (marked with letters A-U) has a bright point in theaccumulator. You can see that all the lines are slightly rotated (approx. 6 degrees). Line A is less rotatedthan line K. Because the image is not only rotated but also skewed. Also, if you look deeper, you can seehow lines A and B are closer than B and C. You can see that both on the left and the right image.

Hough transformation is important to understand if you want to learn visual pattern recognition ingeneral. The concept of virtual lines that can yield real lines by voting can be extended to any othergeometrical shape. Like circles or rectangles. The line is the simplest geometrical shape, so it should bethe simplest to understand.

If you need to locate the circle(s), you will need a three-dimensional accumulator with the dimensions:x, y, and r, where x, y are coordinates and r is the radius of the virtual circles. Again, the highest(lightest) votes in such an accumulator are the real circles on the image.

Hough transformation execution can be optimized by limiting the area and the angles of the originalimage. We don’t need all the lines from the source image. To detect the rotation angle, we only need a


4/1/2012 10:14 PM 5 trong 14

single line somewhere near the center of the image. From that line angle, we assume all other lines arerotated the same.

How to detect grid lines

In order to extract the numbers from the grid, we need to precisely locate where a Sudoku grid startsand where it ends. This part is the simplest part for the human brain, but unexpectedly, this part is themost difficult for the computer. Why? Why not use the lines detected by the Hough transformation asdescribed in the previous section? The answer is because there is a lot of noise. In most cases, a Sudokugrid printed in a magazine or newspaper is never alone. There are other grids and lines around it thatmakes the noise. For example, look at this one:

It is difficult to tell for the computer which lines are Sudoku lines and which are surrounding noise.Where is the end of one grid and start of another.

To solve that problem, we won’t detect black lines. Instead, we will detect the white areas around theSudoku grid. On the next image, you can see how. The green line 1 is never interrupted with the blackpixels from the Sudoku grid, while line 2 is interrupted at least 10 times. This means that line 1 hasmore probability to be outside the grid. By counting how many times any horizontal and vertical line isinterrupted with the black pixels, we can conclude that the green lines on the next image are probablythe boundaries of the Sudoku grid. Simple enough - just count how many transitions from the white toblack pixels are under the line. You don’t need to run the lines at the resolution of 1 pixel. It’s goodenough to skip every 3 pixels for speed.


4/1/2012 10:14 PM 6 trong 14

After we detect the boundaries, we will run Hough transformation inside those boundaries to preciselydetect the grid lines. So far we didn’t care about the images skewing and other image imperfections.Only about coarse image rotation. This step will improve that. By running the Hough transformation ona limited area of the grid, we will get the precise position of all the grid lines. This will help to locate thedigits in the cells.

TODO: This step could be improved to be more insensitive to noise. My plan for the next version is tocombine the above method with “tilted Haar-like features” to detect the corners of the grid. I hope thiscould improve quality. The problem may be that Haar-like features are good with solid areas, but wedeal with lines. Lines occupy smaller areas, so the difference between a light and dark rectangle is notso big.

I wonder what other options are to detect the 10x10 grid…

How OCR works

After we locate the blobs inside the grid cells, we need to recognize them. We have a relatively easytask. Only numbers from 1 to 9. Not the entire alphabet.

Theory

Every recognition algorithm has these steps:

Determine featuresTrain (learning step)Classify (runtime recognition)

Determine features is a part of the application design. Features are for example: The number 1 is talland thin. This is what it makes it different from others. The number 8 has two circles, one above theother, etc.

The feature definition can be a very hard and unintuitive job, depending on the things to recognize. Forexample, what features would you use to recognize somebody’s face? Not any face. The specific face.

Zone features

In this application, we will use zone density features. The next step (that is already done in advance) isto train the application by providing training pictures of digits 1-9. You can see these pictures in the.\res folder. The pictures are resized to 5x5 pixels, normalized, and stored in the static arrayOCRDigit::m_zon[10][5][5], which looks like:

The resizing to 5x5 is called zoning. The above array is called density features.

Normalizing means that those 5x5 pictures have density values in the range 0 to 1024. Withoutnormalizing, the zones would be incomparable.


4/1/2012 10:14 PM 7 trong 14

What happens at run-time: When a blob is isolated from a webcam’s image, it’s resized to 5x5 pixels.Then these 25 pixels are compared, one by one, with the nine trained density feature arrays. The goalis to find the minimal difference in pixel intensity. Less difference means more similarity.

This method is insensitive to blob size scale, since we always use 5x5 zones. It is sensitive to rotation,but we already know what the rotation is and can adjust it. The problem is that it is sensitive to positionoffset and also it does not work with negative images (white on black) and with really rotated images(like upside down).

Width/height ratio feature

Number 1 is a special case. Since it’s similar to 4 and 7, the above method is not reliable. The specificfeature of number 1 is: If the blob’s width is less than 40% of the blob’s height, it must be number 1. Noother digit is so thin. In addition to the above 25 zone features, this is the 26th feature we are checking.

For the classification step, we use the k-nearest neighbor with k=1, which means that we are detectingonly one closest neighbor.

TODO

For the next version, the OCR quality can be improved by introducing other features. For example, thedigits 5, 6, and 9 are very similar if used with zone features. To differentiate them, we could use profilefeatures. The profile feature is the number of pixels (distance) between the bounding box of the bloband the edge of the blob.

In the next image, you can see that the right profiles are similar for 5 and 6, but are different for 9. Theleft profiles for 5 and 9 are similar, but are different for 6.


4/1/2012 10:14 PM 8 trong 14

There are other possible improvements. Professional OCR engines use many different features. Some ofthem can be very exotic.

Fixing the OCR result

After OCR is done, the results are logically corrected based on Sudoku rules. The same digit cannot befound in the same row, column, or 3x3 block. If this rule is broken, the OCR result must be wrong andneeds a correction. We will replace the wrong result with the next probable result. For example, in Fig.12 above, the result is 5 because the diff=5210, which is the smallest. The next probable result is 6.Because it has next diff=5508. So, we will replace result 5 with 6. To decide which of two conflicteddigits needs correction, we take the one with the smallest difference between the first and the seconddiff. The source code is in SudSolver::Fixit().

How the Sudoku solver works

There are more different methods to solve the Sudoku puzzle. Here we will use three simple methodsworking together and helping each other: naked singles, hidden single, and brute force.

Method 1. Brute force

Also called backtracking. This is the most common method used by programmers and which always givesa solution, no matter how difficult it is. But brute force could be very slow, depending on the number ofrecursive iterations needed. You can never know in advance how much iteration is needed. Brute forceis a “trial and error” method. It tries all the combinations of possible values 1-9 on all empty cells untilall the cells are filled with consistent values. There could be more than one solution, but we are happyto find the first one only.

The first step is to prepare the table of candidates – possible values for each empty cell. The imagebelow explains what the candidates are. They are in blue color. By Sudoku rules, the first cell cancontain only 1, 4, or 8. For example, 3 cannot be there, because it is already present two cells below.


4/1/2012 10:14 PM 9 trong 14

The brute force will try to combine all the little blue numbers until it finds a solution. See the first cell.The algorithm will start with the value 1, and then on the fifth cell, it will take number 3 and so on. Ifany of the selected numbers is not consistent with the other values, the algorithm will try with adifferent one. For example, the sixth cell from the left has also number 3 as a candidate, but since thisis not consistent with the fifth cell, the algorithm will try with the next candidate, which is 4, etc.

Brute force can be very slow if the solution requires many iterations. For example, the next image is a“near worst case” puzzle for the brute force algorithm (source: Wiki). Because it tries with all thepossible values and the correct one is the last candidate in the sequence of candidates. Fortunately,there is a solution to that problem - you should not start from the top left cell. Any other initial cell willget the solution faster. We will use this trick to speed up the brute force method.

There are other possible optimizations to make the brute force faster, like sorting the recursionsequence from the cells with the smallest number of candidates to the cells with the maximal number ofcandidates. But we don’t use such optimizations because they are only partial optimizations. They allhave a “worst case” where they are not fast enough for real-time applications. Instead, we will use atime-boxed, three retries, random sequence optimization. The trick is to abort backtracking if it takestoo long. Then re-sort randomly the sequence for recursion and try again from scratch.

Method 2. Naked single candidates

The image below explains this method. If a cell has a single candidate, we are 100% sure this is a validvalue for that cell. After we set that value, the next step is to rebuild the list of candidates. The list ofcandidates gradually reduces until all the candidates are singles. This is an obvious and simple methodfor computers. Not that obvious for humans, though. Human players cannot keep a list of candidates intheir head.


4/1/2012 10:14 PM 10 trong 14

Method 3. Hidden single

The image below explains this method. Look at the number 7. If you are a Sudoku player, you willprobably instantly see that the number 7 must be there.

Even if this cell has four candidates: 4, 7, 8, 9 (see Fig. 15), the trick is that we search for a uniqueinstance of candidates inside the 3x3 block, column, or row. This method probably cannot solve theentire puzzle, but works nicely together with method 2. When method 2 runs out of single candidates,method 3 can help.


4/1/2012 10:14 PM 11 trong 14

Putting all methods together

For the webcam solver, the speed is very important. Brute force is not fast enough for our application.Therefore, we will use a combination of all three methods. The methods 2 and 3 are very fast, but cansolve only simple puzzles. Since we get the input from a noisy webcam, we often get very hard or evenunsolvable puzzles (because of OCR unreliability). We must assume the puzzles will often be very hardto solve. Even if the original puzzle is meant to be a simple one.

How to read the next diagram: On the left side, there are fast methods 2 and 3. Only if they areunsuccessful, we will jump to the right side, which is slow brute-force. Even if methods 2 and 3 areunsuccessful to solve, they do a great job of solving at least some cells, reducing the job for brute-force.

Only if methods 2 and 3 cannot solve it, the program falls down to the brute force method. And eventhen, the brute force is limited to 600000 iterations to keep the algorithm time-boxed. There will bethree re-tries after which the program gives up. Between each retry, the recursion sequence isrearranged randomly with the hope that the new sequence will lead to a fast solution. If brute-force failsafter three re-tries, it’s not a complete failure. We may have more luck with the next camera frame. Theabove diagram is implemented in SudSolver::SolveMe().

Buffering the solution

When the solution is found, we keep track of the most recent solutions in an array of typeSudResultVoter. Buffering is needed because the OCR is not 100% reliable and we get wrongsolutions from time to time. To avoid a fluctuating solution, we will always show the strongest solutionthat has been found recently (among the last 12 solutions, to be precise). From time to time, the arrayis reset, forgetting the old solution and giving a chance to a new Sudoku that is currently in focus.

Video


4/1/2012 10:14 PM 12 trong 14

Article Top Rate this: Poor Excellent Vote

TODO

Some ideas for the future:

Rewrite the program for Android. I wonder how that would work on SmartPhones.Parallelize some functions to use multicore processors. Today, almost all PCs have at leastdual-core processor. Normally, parallel tasking is used to improve performance. But with thewebcam solver, we don’t need more speed, we need more quality. The idea is that parallel tasksshould perform the same operation on the same image frame but with different settings. After alltasks are joined, we will take the task with the best result and discharge others.

History

08 August 2011 - First release.

License

This article, along with any associated source code and files, is licensed under The GNU Lesser GeneralPublic License (LGPLv3)

About the Author

Bojan Banko

Croatia

Member

Programmer since 1994.

Comments and Discussions


4/1/2012 10:14 PM 13 trong 14

Add a Comment or Question Search this forum Go

Permalink | Advertise | Privacy | MobileWeb03 | 2.5.120330.1 | Last Updated 8 Aug 2011

Article Copyright 2011 by Bojan BankoEverything else Copyright © CodeProject, 1999-2012

Terms of Use

Profile popups Noise Medium Layout Normal Per page 25 U pdate

Refresh First PrevNext

Mazen el Senih 16:18 15 Mar '12

Shahin Khorshidnia 10:22 12 Mar '12

kartalyildirim 3:53 2 Mar '12

Paulo Matias 14:02 25 Jan '12

Mario Majcica 16:08 23 Jan '12

thatraja 9:06 10 Jan '12

Rajesh Anuhya 4:49 10 Jan '12

sisira 10:09 7 Jan '12

sisira 10:07 7 Jan '12

Nigam Patel 0:28 5 Jan '12

Member 2614807 5:58 3 Jan '12

pablosfor 6:57 20 Dec '11

Smithers-Jones 4:24 1 Dec '11

Filip D'haene 18:36 17 Nov '11

Mahdi Nejadsahebi 15:33 25 Oct '11

RKnGl 7:10 30 Sep '11

Irwan Hassan 11:54 28 Sep '11

_groo_ 9:18 28 Sep '11

Bojan Banko 9:34 28 Sep '11

aeastham 4:45 28 Sep '11

josip cagalj 3:45 27 Sep '11

Andrei Azzopardi a.k.a.C_Coder

3:39 27 Sep '11

Alexander Voronin 2:49 27 Sep '11

jerviz_76 2:43 27 Sep '11

Lucky Vdb 1:19 27 Sep '11

Last Visit: 10:06 28 Mar '12 Last Update: 10:57 1 Apr '12 12 3 4 5 Next »

General News Suggestion Question Bug Answer Joke Rant Admin

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Layout: fixed | fluid

Great article , idea ,solution

My vote of 5

Good work.. [modified]

My vote of 5

My vote of 5 (I'll do even more if possible)

Congratulations......

My vote of 5

16-bit Video Format support?

My vote of 5

My vote of 5

My vote of 5

My vote of 5

My vote of 5

My vote of 5

My vote of 5

My vote of 5

My vote of 5

Vote of 5, and a question

Re: Vote of 5, and a question

My vote of 5

Odlično

My vote of 5

My vote of 5

My vote of 5

My vote of 5


4/1/2012 10:14 PM 14 trong 14

realtime webcam sudoku solver - codeproject®

Documents

generator

brute force

hough transformation

source code

brute force

sudoku puzzle

human brain

gray rectangle