Augmented Reality Sudoku Solver: Building the Graphical User Interface
This article is a continuation of the Augmented Reality Sudoku Solver, we are going to build a GUI-based AR Sudoku Solver. In this article, we are going to build the GUI components of the project using PyGame. There are several sub-components of the project each implemented in separate files to ensure code-reusability.
We aim to solve any sudoku of the N dimension, where N is a non-prime. The project will be implemented in two forms -
- An option to load an image saved on the system or use the webcam to feed the image to the program and then play the game on the system.
- An option to use Augmented Reality and solve the sudoku by showing the puzzle to the webcam.
The project uses Dancing Links in the form of Algorithm X to find the solution of the Sudoku puzzle. Sudoku is a well-known NP-Complete problem and Algorithm X is a means of implementing a form of greedy depth-first search to find the appropriate solution. The project will be blogged in 4 parts -
- Part I — Understanding the Sudoku Solver i.e. the Algorithm that goes into solving the Sudoku.
- Part II — Processing Image from Camera to be able to extract the grid of the Sudoku.
- Part III — Processing the Image and the respective model to detect the numerical values in each cell.
- Part IV — Building the GUI using PyGame.
Part IV: Building the GUI using the PyGame -
The Complete GUI for the game is built using 3 components -
- Button: We build a mouse clickable rounded rectangular button. These buttons are used to help us select choices for various operations.
- Camera Window: This is a window integrated with the camera feed used to Click an Image or run the Augmented Reality mode.
- Game Window: This is the main GUI of the entire project and is used to run all the other parts along with the main game.
Button Class -
Here, we first initialize the class with the following variables -
- X: Central X-Coordinate of the Button Location.
- Y: Central Y-Coordinate of the Button Location.
- Button Width: This is the pixel width of the Button.
- Button Height: This is the pixel height of the Button.
- Color: This is the color of the button in RGB.
- Hover Color: This is the color of the button when the mouse hovers over the button, it is set to 75% of the original color of the button.
- Button Curvature Radius: This is the radius of curvature for the rounded rectangular shape of the button.
- Text: The text used to display on the button.
Draw Button -
This method is used to draw the button over a “pygame.surface” object. The “pygame.surface” object can be thought of as a 2-dimensional Image over which other images/objects can be displayed. We first define the rectangular shape of the button. We then define the color of the button based on whether it is located under the mouse or not. Next, we define four circular outlines which we display over the corners of the rectangular outline.
Then we fill the image with the defined color, fill the outlines and the text with black color. Finally, we display the button on the surface.
Clicked -
This method returns True if the mouse button is clicked while the mouse was over the button, else returns Fasle.
Check if the Button is Under the Mouse -
This method gets the corrdinates of the mouse and checks if the coordinates of the button lie in the region of the button space. If the x-coordinate lies withing the width of the button and similarly if the y-coordinate lies within the height it returns True, else returns False.
Camera Window Class -
Here, we first initialize the class with the following variables -
- Camera Source: Index of the camera source, which can be a camera or a feed from a video.
- Capture Image: Boolean variable indicating whether to include a capture button or not.
Context Manager: Enter -
This method is a special dunder method which enables the use of the object with the “with statement contexts”. Here, we first define our camera object and read a frame from it. Then we check if the window is meant to capture image or not. If so, we include a button with the camera icon which can be used to click an image. If the window is meant for Augmented Reality viewing we load a Home Button on the top left which when clicked takes us back to the main game menu.
Draw Camera Window -
We first fill the color with white color and then extract a frame from the camera which is converted from BGR scheme to RGB. We then create a “pygame.surface” object with the image as horizontally flipped to ensure that the digits do not appear mirrored. We then display the capture button or the home button depending on the purpose of the window. Next, we iterat through the events to check for any clicks.
Context Manager: Exit -
This is the counterpart of the “__enter__” dunder method and is called when we move out of the “with-context”.
Game Window Class -
This is the main GUI class that encapsulates all the other components of the project. Here, we first initialize the class with the following variables -
- Box Rows: Number of rows in a sub box of the puzzle.
- Box Columns: Number of columns in a sub box of the puzzle.
- Num of Rows: Total Number of Rows and Columns in the puzzle.
- Play Area Width: Total Pixel Width of the Game Area.
- Play Area Height: Total Pixel Height of the Game Area.
- Top Left Area Coordinates: Top left coordinates of where the puzzle is placed in the window.
- Puzzle Matrix: This is sudoku puzzle matrix.
- Initial Copy of Puzzle Matrix: Since the Game solution takes happens in place it modifies the original puzzle matrix. This is a backup copyof the initial matrix.
- Solution List: List of solutions for the current puzzle.
- Selected Solution: Solution that is selected to be displayed on clicking the solve button.
- Game Window: This is game window screen object.
- Selected Box: This is the presently highlighted box where we input the numerical value.
- Locked Boxes: List of non-zero positions of the puzzle which cannot be modified.
- Home Icon: Icon for the home button to navigate to the main menu.
- Home Button: Button to navigate to the main menu.
- Load Image Button: Button to load an image from a file.
- Load from Camera Button: Button to click an image from the camera.
- Solve Puzzle Button: Button used to reveal the puzzle solution.
- Play Game Button: Button in the main menu used to play the game.
- Augmented Reality Button: Button in the main menu used to load the Augmented Reality.
Get Locked Box Positions -
This method returns the non-zero positions from the puzzle matrix.
Draw Game Window -
We first fill the window with white color and display the Sudoku Title. Next , we iterate through the number of rows and columns to create the horizontal and vertical lines. Finally we iterate through the puzzle matrix, where if the element is a non-zero value we display the number in the respective cell. We check if the position is that of a locked cell, then we display the number in black, else we check if its a possible number for that cell or not and accordingly display the text in blue or red respectively.
Note: The “solved” boolean is used after the solve button is clicked to put the non locked position values in green.
Mouse Click Handler -
We first check if the “Load Image from File” button is clicked. If so then we use the “askopenfilename” method from the Tkinter library to open the image.
Then we use the “SudokuImageProccessing” object to load the image and get the matrix. If the image is valid, the puzzle is loaded on the screen and the matrix and its dimensions are saved, else it prompts error dialog box saying “Unable to load image”.
Next, we check for the “Capture Image” button. If so the we open the Camera Window object with “capture_image” as True and read data from the window object indefinitely until an image is returned. Similar to the previous check we load the image and to the “SudokuImageProccessing” object and try extracting the matrix. If the image is loaded successfully we load the puzzle on the screen and the matrix and its dimensions are saved, else it prompts error dialog box saying “Image not clear”.
Next, we check if the “Solve” button is clicked. We first reset the matrix to the initial puzzle matrix. Then we iterate through the matrix finding the zero positions in puzzle and for each position we fill the correct solution value in green. After this we hold until a keystroke or a mouse button click and then switch to the congratulations screen.
Next, we perform the “Home” button check which simply returns a False indicating that we need to go back to the main menu.
Finally we check if the mouse was clicked in the puzzle area and if so, we select the closest box as the highlighted box.
Game Play -
This method is responsible for handling the input part of the game and checking if the solution is correct. We first iterate through the events to check if the event occurred was a mouse click, close action, numerical value input, deleted key, or an arrow key input. In case it is a numerical value input, we fill the value in the currently selected box and display the appropriate color blue or red depending on whether the input is possible or not. If it was a delete key we remove the current entry and set it back to zero if the box is not locked. If the key is an arrow key, we move the box in the direction of the key.
Augmented Reality -
This is the Augmented Reality implementation code. Here we first open the Camera Window object and intilialize the solution and frame count as None and 0 respectively. Then using the “suppress(Exception)” context we disable the occurrence of any exceptions. Then we process the image for every 10th frame to get the matrix , its dimensions and get the solution. Then we plot this solution over the image for 10 frames. This means that every 10 frames we check the image for a new matrix, this is done to ensure that the frames are smooth and to avoid any lag in the GUI due to processing time.
Main Menu -
This method is the menu GUI for the project and includes two buttons: one for the Game Play and the other for Augmented Reality.
Graceful Exit -
This method is used to ensure that the last played matrix and its dimensions are saved before exiting the GUI.
Main Program -
This is the main driver program for the project. It loads the menu and runs the Game Play and the Augmented Reality Components in an infinite loop.
Do check out other parts of this article series -
