Intro

Hi, I recently came up with an idea to build a mobile app that let’s you take a photo of a handwritten formula and send it to server.

I usually build mobile apps with Java, but this time I didn’t want to limit myself to Android platform only. Moreover, I found it hard to manipulate camera preview with Java. So I started searching for a solution that would compromise both multi-platform support and easy camera preview manipulation. Shortly after I came across Apache Cordova framework.

In this post, I’ll guide you through building app with Cordova and running it on Android. On top of that, I’ll teach you how to program all necessary functionality with powerful AngularJS framework.

Setup

# Install nodejs
# Nodejs is a package manager, distributing javascript packages
sudo apt-get install nodejs

# Install cordova using nodejs
npm install -g cordova

# Create a new cordova project
cordova create path/to/myApp

# Navigate to project folder
cd path/to/myApp

# Add platforms you want to build your app for
# List available platforms: cordova platform
cordova platform add <platform_name>

# Add camera preview plugin
cordova plugin add cordova-plugin-camera-preview

Code

Now, www is the only directory that we’ll make changes in. The rest of the directories are related to Cordova itself.

We start off with creating an AngularJS application. Note that we also need to include ui-router submodule.

// js/app.js

var app = angular.module('app', ['ui.router']);

Now it’s time to define an AngularJS service. But, what is a service? A service is a function, or an object that is available for, and limited to, your AngularJS application.

Why one should use services?

  • Services allow you to persist and share data across your application.
  • You don’t need to repeat yourself. You may refer to the single piece of functionality from within your application.

Plug in your server’s IP address into images_endpoint variable.

// js/services.js

app.service('photoService', ['$http', '$q', function($http, $q) {
    // Address where images are to be sent
    this.images_endpoint = 'http://IP_address:8000/api/photos/';

    // This method crops an image
    this.crop = function(base64_img, rect_width, rect_height, x_coord, y_coord) {
        var deferred = $q.defer();

        // image variable will contain ORIGINAL image
        var image = new Image();

        // canvas variable will contain CROPPED image
        var canvas = document.createElement('canvas');
        var ctx = canvas.getContext('2d');

        // Load original image onto image object
        image.src = 'data:image/png;base64,' + base64_img;
        image.onload = function(){

            // Map rectangle onto image taken
            var x_axis_scale = image.width / window.screen.width
            var y_axis_scale = image.height / window.screen.height
            // INTERPOLATE
            var x_coord_int = x_coord * x_axis_scale;
            var y_coord_int = y_coord * y_axis_scale;
            var rect_width_int = rect_width * x_axis_scale;
            var rect_height_int = rect_height * y_axis_scale

            // Set canvas size equivalent to cropped image size
            canvas.width = rect_width_int;
            canvas.height = rect_height_int;

            ctx.drawImage(image,
                x_coord_int, y_coord_int,           // Start CROPPING from x_coord(interpolated) and y_coord(interpolated)
                rect_width_int, rect_height_int,    // Crop interpolated rectangle
                0, 0,                               // Place the result at 0, 0 in the canvas,
                rect_width_int, rect_height_int);   // Crop interpolated rectangle

            // Get base64 representation of cropped image
            var cropped_img_base64 = canvas.toDataURL();

            // Cropping has been finished
            deferred.resolve(cropped_img_base64);
        };

        return deferred.promise;
    };

    // This method sends an image in base64 format to the server
    this.send = function(cropped_img_base64) {
        // Ending slash in URL is necessary
        return $http.post(this.images_endpoint,
            {
                // Data sent along with a request
                'image': cropped_img_base64
            }
        );
    };
}])

We’ll make use of photoService from within an AngluarJS controller. But, what is a controller? A controller is a regular JavaScript Object, created by a standard JavaScript object constructor which controls the flow of data in the application.

// js/controllers.js

app.controller('cameraPreviewCtrl', ['$scope', '$document', 'photoService', function($scope, $document, photoService) {
    // Initialize Camera Preview
    var options = {
        x: 0,
        y: 0,
        width: window.screen.width,
        height: window.screen.height,
        camera: CameraPreview.CAMERA_DIRECTION.BACK,  // Front/back camera
        toBack: true,   // Set to true if you want your html in front of your preview
        tapPhoto: false,  // Tap to take photo
        tapFocus: true,   // Tap to focus
        previewDrag: false
    };
    // For more options
    // Take a look at docs: https://github.com/cordova-plugin-camera-preview/cordova-plugin-camera-preview#methods
    CameraPreview.startCamera(options);

    // Initialize spinner state & flash mode
    $scope.show_spinner = false;
    $scope.flash_mode = CameraPreview.FLASH_MODE.OFF;

    // Absolute paths to icons
    $scope.flash_on_icon = 'img/icons/flash_on.svg';
    $scope.flash_off_icon = 'img/icons/flash_off.svg';
    $scope.take_pic_icon = 'img/icons/btn_icon6.png';

    // Cropped image placeholder
    $scope.cropped_photo = null;

    // Rectangle element reference
    var rect = $document[0].getElementsByClassName('rectangle')[0];

    $scope.takePicture = function() {
        // SHOW loading spinner
        $scope.show_spinner = true;
        // Get rectangle size
        var rect_width = rect.offsetWidth, rect_height = rect.offsetHeight;

        // Get rectangle coordinates
        var rect_coords = rect.getBoundingClientRect();
        var x_coord = rect_coords.left, y_coord = rect_coords.top;

        CameraPreview.takePicture(function(base64_img) {

            photoService.crop(base64_img, rect_width, rect_height, x_coord, y_coord).then(
                // Photo was successfully cropped
                function successCallback(cropped_base64_img) {

                    $scope.cropped_photo = cropped_base64_img;
                    // Photo was successfully sent to server
                    photoService.send(cropped_base64_img).then(
                        function successCallback(response) {
                            // Hide spinner
                            $scope.show_spinner = false;
                            // Reset photo placeholder
                            $scope.cropped_photo = null;
                        },
                        function errorCallback(error) {
                            // Show this message when an error occurs
                            var error_message = 'Oops, something went wrong!';
                            // Hide spinner
                            $scope.show_spinner = false;
                            // Reset photo placeholder
                            $scope.cropped_photo = null;
                            // Show error popup
                            alert(error_message);
                        }
                    );
                },
                function errorCallback(error) {
                    // Show this message when an error occurs
                    var error_message = 'Could not perform cropping action!';
                    // Hide spinner
                    $scope.show_spinner = false;
                    // Reset photo placeholder
                    $scope.cropped_photo = null;
                    // Show error popup
                    alert(error_message);
                }
            );
        });
    };

    $scope.changeFlashMode = function() {
        // Trigger flash mode
        if ($scope.flash_mode === CameraPreview.FLASH_MODE.OFF) {
            $scope.flash_mode = CameraPreview.FLASH_MODE.ON;
        } else {
            $scope.flash_mode = CameraPreview.FLASH_MODE.OFF;
        }
        // Set flash mode
        CameraPreview.setFlashMode($scope.flash_mode);
    };
}])

Let’s style our camera preview.

/* css/index.css */

body {
    width: 100vw;
    height: 100vh;
    margin: 0px;
}

.cameraPreview {
    display: flex;
    width: 100%;
    height: 100%;
}
img {
    position: absolute;
    /* Prevent border showing up after button is clicked */
    border: none;
}
.rectangle {
    width: 80%;
    height: 20%;
    /* Center vertically & horizontally - only works with display: flex; */
    margin: auto;
    /* Make inner part of rectangle transparent */
    background-color: rgba(255, 255, 255, 0);
    /* SHADOW EFFECT darkens everything outside rectangle */
    box-shadow: 0 0 500px 5000px rgba(0, 0, 0, 0.5);

    /* COOL BORDER */
    border-width: 20px;
    border-style: solid;
    /* You need to place border.png into img folder */
    border-image: url('../img/borders/border.png') 50 round;

    background-size: calc(100% + 40px) calc(100% + 40px);
    background-position: center;

    /* This one is neccessary for interaction.js resize to work */
    touch-action: none;

    box-sizing: border-box;
    -moz-box-sizing: border-box;
    -webkit-box-sizing: border-box;
}
.flash_btn {
    background-size: 50px 50px;
    width: 33px;
    height: 33px;
    top: 15px;
    right: 15px;
}
.snap_btn {
    background-size: 60px 60px;
    width: 55px;
    height: 55px;
    bottom: 20px;
    left: 0px;  right: 0px;
    margin: auto;
}

@-webkit-keyframes spin {
    0% { -webkit-transform: rotate(0deg); }
    100% { -webkit-transform: rotate(360deg); }
}
@keyframes spin {
    0% { transform: rotate(0deg); }
    100% { transform: rotate(360deg); }
}
.spinner {
    position: absolute;
    width: 50px;
    height: 50px;
    bottom: 20px;
    left: 0px;  right: 0px;
    margin: auto;

    border: 2px solid transparent;
    border-radius: 50%;
    border-top: 2px solid #fff;
    -webkit-animation: spin 1s linear infinite;
    animation: spin 1s linear infinite;
}

We can now add few buttons and camera frame to body tag. Don’t forget to import necessary JavaScript and CSS files.

<!-- index.html -->

<!DOCTYPE html>
<html>
    <head>
        <meta http-equiv="Content-Security-Policy" content="default-src * gap: ws: https://ssl.gstatic.com;style-src * 'unsafe-inline' 'self' data: blob:;script-src * 'unsafe-inline' 'unsafe-eval' data: blob:;img-src * data: 'unsafe-inline' 'self' content:;">

        <meta name="format-detection" content="telephone=no">
        <meta name="msapplication-tap-highlight" content="no">
        <meta name="viewport" content="user-scalable=no, initial-scale=1, maximum-scale=1, minimum-scale=1, width=device-width">

        <!-- Styles -->
        <link rel="stylesheet" type="text/css" href="css/index.css">

        <!-- AngularJS module -->
        <script src="https://cdnjs.cloudflare.com/ajax/libs/angular.js/1.6.5/angular.min.js"></script>
        <!-- AngularJS Ui-router module -->
        <script src="https://cdnjs.cloudflare.com/ajax/libs/angular-ui-router/1.0.3/angular-ui-router.min.js"></script>

        <!-- AngularJS app code -->
        <script type="text/javascript" src="js/app.js"></script>
        <script type="text/javascript" src="js/services.js"></script>
        <script type="text/javascript" src="js/controllers.js"></script>

        <title>Basic Math OCR</title>
    </head>
    <body ng-controller="cameraPreviewCtrl">
        <div class="cameraPreview">

            <!-- Rectangle centered vertically & horizontally -->
            <div class="rectangle"
                ng-style="{'background-image': 'url(' + cropped_photo + ')'}"></div>

            <!-- Placed in top-right corner -->
            <img class="flash_btn"
                ng-click="changeFlashMode()"
                ng-src="{{ flash_mode === 'on' ? flash_on_icon : flash_off_icon }}">

            <!-- Placed at the bottom -->
            <img class="snap_btn"
                ng-click="takePicture()"
                ng-src="{{ take_pic_icon }}"
                ng-hide="show_spinner">

            <!-- Loading spinner -->
            <span class="spinner" ng-show="show_spinner"></span>
        </div>

        <script type="text/javascript" src="cordova.js"></script>
        <script type="text/javascript" src="js/index.js"></script>
    </body>
</html>

Last but not least, we need to bootstrap angular application once device is ready, that is when cordova plugins are loaded.

document.addEventListener('deviceready', function() {

    var body = document.body;
    angular.bootstrap(body, ['app']);
}, false);

This is how your www directory should look like after you got to this point.

  • hooks
  • node_modules
  • platforms
  • plugins
  • res
  • www
    • css
      • index.css
    • img
      • borders
        • border.png
      • icons
        • flash_off.svg
        • flash_on.svg
    • js
      • app.js
      • controller.js
      • index.js
      • services.js
    • index.html

Results

client-results

How to run?

# cordova platforms list - to list all available platforms
cordova run <platform_name>

Extras

We can also make camera frame interactive, so that you can resize or even drag it. In order to do so, we’ll use interact.js module.

<!-- index.html -->

<head>
    ...
    <!-- Import interact.js module -->
    <script src="https://cdnjs.cloudflare.com/ajax/libs/interact.js/1.2.9/interact.min.js"></script>

    ...
</head>
// js/index.js

...
interact('.rectangle').resizable({
    // Resize from all edges and corners
    edges: {left: true, right: true, bottom: true, top: true},

    // Keep the edges inside the parent
    restrictEdges: {
        outer: 'parent',
        endOnly: true,
    },

    // Minimum size
    restrictSize: {
        min: { width: 100, height: 50 },
    },
})
.on('resizemove', function(event) {
    var target = event.target,
        x = (parseFloat(target.getAttribute('data-x')) || 0),
        y = (parseFloat(target.getAttribute('data-y')) || 0);

    // update the element's style
    target.style.width  = event.rect.width + 'px';
    target.style.height = event.rect.height + 'px';

    target.setAttribute('data-x', x);
    target.setAttribute('data-y', y);
})

Code

This is where you’ll find the complete code for this post as well as all necessary icons: https://github.com/ThomasLech/Basic-handwritten-math-OCR-Client-application