GlusterFS - Replicate a volume over two nodes

When you are using a load balancer with two or more backend nodes(web servers) you will probably need some data to be mirrored between the two nodes. A high availability solution is offered by GlusterFS.

Within this article, I am going to show how you can set volume replication between two CentOS 7 servers.

Let's assume this:

  • node1.domain.com - 172.31.0.201
  • node2.domain.com - 172.31.0.202

First, we edit /etc/hosts of each of the servers and append this:

172.31.0.201     node1.domain.com     node1
172.31.0.202     node2.domain.com     node2

We should now be able to ping between the nodes.

PING node2.domain.com (172.31.0.202) 56(84) bytes of data.
64 bytes from node2.domain.com (172.31.0.202): icmp_seq=1 ttl=64 time=0.482 ms
64 bytes from node2.domain.com (172.31.0.202): icmp_seq=2 ttl=64 time=0.261 ms
64 bytes from node2.domain.com (172.31.0.202): icmp_seq=3 ttl=64 time=0.395 ms

--- node2.domain.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.261/0.379/0.482/0.092 ms

Installation:

Run these on both nodes:

yum -y install epel-release yum-priorities

Add priority=10 to the [epel]section in  /etc/yum.repos.d/epel.repo

[epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
priority=10
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7

Update packages and install:

yum -y update
yum -y install centos-release-gluster
yum -y install glusterfs-server

Start glusterd service, also enable it to start at boot:

service glusterd start
systemctl enable glusterd

You can use service glusterd status and glusterfsd --version to check all is working properly.

Remember, all the installation steps should be executed on both servers!

Setup:

On node1 server run:

[root@node1 ~]# gluster peer probe node2
peer probe: success.
[root@node1 ~]# gluster peer status
Number of Peers: 1

Hostname: node2
Uuid: 42ee3ddb-e3e3-4f3d-a3b6-5c809e589b76
State: Peer in Cluster (Connected)

On node2 server run:

[root@node2 ~]# gluster peer probe node1
peer probe: success.
[root@node2 ~]# gluster peer status
Number of Peers: 1

Hostname: node1.domain.com
Uuid: 68209420-3f9f-4c1a-8ce6-811070616dd4
State: Peer in Cluster (Connected)
Other names:
node1
[root@node2 ~]# gluster peer status
Number of Peers: 1

Hostname: node1.domain.com
Uuid: 68209420-3f9f-4c1a-8ce6-811070616dd4
State: Peer in Cluster (Connected)
Other names:
node1

We need to create now the shared volume, and this can be done from any of the two servers.

[root@node1 ~]# gluster volume create shareddata replica 2 transport tcp node1:/shared-folder node2:/shared-folder force
volume create: shareddata: success: please start the volume to access data
[root@node1 ~]# gluster volume start shareddata
volume start: shareddata: success
[root@node1 ~]# gluster volume info

Volume Name: shareddata
Type: Replicate
Volume ID: 30a97b23-3f8d-44d6-88db-09c61f00cd90
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: node1:/shared-folder
Brick2: node2:/shared-folder
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

This creates a shared volume named shareddata, with two replicas on node1 and node2 servers, under /shared-folder path. It will also silently create the shared-folder directory if it doesn't exist.If there are more servers in the cluster, do adjust the replica number in the above command. The "force" parameter was needed, because we replicated in the root partition. It is not needed when creating under another partition.

Mount:

In order for the replication to work, mounting the volume is needed.  Create a mount point:

mkdir /mnt/glusterfs

On node1 run:

[root@node1 ~]# echo "node1:/shareddata    /mnt/glusterfs/  glusterfs       defaults,_netdev        0 0" >> /etc/fstab
[root@node1 ~]# mount -a
[root@node1 ~]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00   38G  1.1G   37G   3% /
devtmpfs                         236M     0  236M   0% /dev
tmpfs                            245M     0  245M   0% /dev/shm
tmpfs                            245M  4.4M  240M   2% /run
tmpfs                            245M     0  245M   0% /sys/fs/cgroup
/dev/sda2                       1014M   88M  927M   9% /boot
tmpfs                             49M     0   49M   0% /run/user/1000
node1:/shareddata                 38G  1.1G   37G   3% /mnt/glusterfs

On node2 run:

[root@node2 ~]# echo "node2:/shareddata    /mnt/glusterfs/  glusterfs       defaults,_netdev        0 0" >> /etc/fstab
[root@node2 ~]# mount -a
[root@node2 ~]# df -h
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00   38G  1.1G   37G   3% /
devtmpfs                         236M     0  236M   0% /dev
tmpfs                            245M     0  245M   0% /dev/shm
tmpfs                            245M  4.4M  240M   2% /run
tmpfs                            245M     0  245M   0% /sys/fs/cgroup
/dev/sda2                       1014M   88M  927M   9% /boot
tmpfs                             49M     0   49M   0% /run/user/1000
node2:/shareddata                 38G  1.1G   37G   3% /mnt/glusterfs

Testing:

On node1:

touch /mnt/glusterfs/file01
touch /mnt/glusterfs/file02

on node2:

[root@node2 ~]# ls /mnt/glusterfs/ -l
total 0
-rw-r--r--. 1 root root 0 Sep 24 19:35 file01
-rw-r--r--. 1 root root 0 Sep 24 19:35 file02

This is how you mirror one folder between two servers. Just keep in mind, you will need to use the mount point /mnt/glusterfs in your  projects, for the replication to work.


How To Install A .pxf Windows SSL Certificate On Your Linux Web Server

Windows uses .pfx for a PKCS #12 file. PFX stands for Personal eXhange Format. This is like a bag containing multiple cryptographic information. It can store private keys, certificate chains, certificates and root authority certificates. It is password protected to preserve the integrity of the contained data.
In order to install it on our apache/nginx web server we need to convert it PEM.

Upload first the .pfx to your linux server. You will need OpenSSL installed.

On Centos run:

yum install openssl

On Ubuntu run:

sudo apt-get update
sudo apt-get install openssl

To decript the .pfx use:

openssl pkcs12 -in cert.pfx -out cert.pem

You will be prompted for the password that was used to encrypt the certificate. After providing it, you will need to enter a new password that will encrypt the private key.
The .pem file resulted will contain the encrypted public key, the certificate and some other information we will not use.Copy the key from inside and paste it to a new .key file.
Also copy the certificate from the .pem and put it in a new .cert file.

Remember to copy the whole blocks, including the dashed lines.
The private key file is still encrypted, so we have to decrypt it with:

openssl rsa -in cert.key -out cert.key

You will now be prompted for the password you set to encrypt the key. This will decrypt the private key file to itself.

To install the certificate to Nginx, you will need to import your .key and .cert in Nginx configuration file like this:

ssl_certificate /path/to/your/cert.pem;
ssl_certificate_key /path/to/your/cert.key;

For Apache use:

SSLCertificateFile /path/to/your/cert.pem;
SSLCertificateKeyFile /path/to/your/cert.key;

Android vs. iOS App Development: A Comparison & Decision Guide

Android vs. iOS App Development: A Comparison & Decision Guide [Infographic]

There’s a revolution in mobile app development happening right now with the number of apps available for download in major app stores being at an all-time high. App developers are burying themselves in work, creating the latest software applications for mobile devices and hoping that their creations make enough money for them at Apple’s App Store and the Google Play Store. Apparently, the numbers are more than encouraging.

As of March 2017, the App Store had slightly more apps available at 2.2 billion compared with Play Store’s 2.8 million apps. Also in Q1 2017, combined iOS and Android app downloads were up 15% year-over-year, reaching a total of about 25 billion new downloads globally.

With re-installs, these numbers could go much higher. In terms of in-app purchases, revenue likewise increased 45% year-over-year for a staggering $15 billion across the iOS and Android systems running worldwide.

It’s definitely a good time for mobile app companies to continue hiring app developers to work on a wide array of mobile app products—from mobile music apps to instant messaging and mobile game apps. These apps are forecast to generate close to $200 billion in in-app and in-advertising revenues in less than three years’ time. But the question is, should you develop Android or iOS apps?

The following infographic gives you a comprehensive comparison between Android and iOS app development services in terms of demographics, the range of devices, time frame of development, hardware, publishing and restrictions, cost, and revenue. These are the factors you will need to consider as you decide whether to develop on the Android or iOS platform.

Android vs. iOS App Development A Comparison & Decision Guide Infographic

Share this Image On Your Site

The Android and iOS platforms both provide a venue for your mobile app development, but they have very distinct functionalities on their own. Sometimes, you get a clear-cut distinction between the two. However, there are also areas where they go neck and neck so you’ll have to rely on your judgment and preference when choosing the appropriate platform to develop your ware.

Demographics

Market share-wise, Android has a greater reach than its competitor, although iOS users’ income levels and spending capabilities may balance things out in the overall picture.

Devices

Screen and Resolution

The operating systems of Android and iOS differ from one another to a great extent. In designing your app, you need to factor in how your app will run given a particular device’s screen and resolution, the hardware compatible with the platform, and the individual features and capabilities of various phone models.

It might be a little more challenging to develop apps on the Android platform since you’ll be designing for different screen sizes and resolution. Meanwhile, iOS has a more uniform size range so that may free you up from having too many considerations to worry about.

Manufacturers

With Android, you’re developing not only for one Android-based device but thousands of other devices produced by different vendors. Since Android has an open nature system, you can leverage it to run your apps not just on mobile phones but also on smart TVs, wearables, or media streaming devices.

Since different Android devices have different capabilities that are unique from one another, your apps should be designed to allow compatibility with all Android devices. You have to consider the whole Android family—from low-range to mid- and high-range iterations when developing and designing your apps. Otherwise, you might be leaving out a large portion of the Android market from running your apps.

On the other hand, developing your apps for iOS means you’ll have fewer configurations to make, as you only have a limited lineup of gadgets at your disposal. This also means you’ll have fewer fragmentation issues developing your apps within the iOS system.

Development Time

As a developer, you’re naturally concerned about how much time it will take you to develop your app and put it on the App Store or the Play Store. Spending more time on app development may cause a dent on your resources and your chance to generate revenue for the company.

Testing your app across Android’s wide range of devices will naturally require more time as you look for and fix issues on one device after another. If the app works well on one Android device but not on another, you have to make the necessary modifications to the app.

Development Tools

Integrated Development Environment (IDE):

Android Studio replaced Eclipse, the previous iteration of Android’s IDE. Although Android Studio has been padded with improvements, iOS’s Xcode is deemed more mature. Specifically, in terms of debugging capability, the mobile simulator, which you use to build a prototype of your app for mobile, of iOS is deemed better than that of the Android simulator. However, Android Studio makes up for this by having a better autocomplete functionality for your coding work.

Programming Language

Java and Swift have a fairly similar structure. What differentiates them is that the former may be a bit complex if you’re just beginning to code, while the latter is based on Objective-C and touted to be generally error-free.

Publishing and Restrictions

From membership application all the way to submitting and publishing an app, it would seem like you’ll have to do a look-for-a-needle-in-the-haystack sort of thing with iOS apps, with no certainty whether your app will get approved on the App Store. In contrast, you can readily publish your app on the Play Store just by signing up and uploading your APK.

The strict quality standards of iOS may benefit users on the one hand but may scare freelance developers off on the other. That said, you’re more likely to have a better experience with Android’s mobile development policies, especially if you’re a startup developer and you’re not willing to spend on annual fees.

Cost and Revenue

Both the Play Store and the App Store put developers on equal footing when it comes to revenue generation, but the interplay of cost and revenue seems to favor iOS more. This may be attributed to iOS users’ willingness to spend on app purchases. Thus, iOS would seem to be the better choice over Android cost- and revenue-wise since it’s faster and less complicated to develop on the iOS platform.

Apple devices are known to be of premium quality and work well with a wide range of peripherals. These factors make the iOS platform effective for mobile app development as you need a shorter time to design, test, and release your app.

And the Winner Is?

After pitting Android and iOS against each other, both fared fairly the same.

Android may be your go-to platform in terms of compatibility with any hardware, more lenient rules for membership and approval of apps, and affordability of publishing fees. On the other hand, iOS shows better mileage in terms of uniformity in the screen size and resolution of devices for which the apps are intended, a more streamlined interface, shorter time frame of development, and cost and revenue model.

It’s a draw between Android and iOS when you consider market demographics, IDE features, and the nature of programming language used.

With all things considered, no one can claim to be better than the other. What one lacks, the other compensates for in another area of app development. It’s really your own choice to make, depending on your business goals and availability of resources.


How to Setup Docker for Your Symfony Project

As you probably know, I am a big Symfony fan :) In the last few years I used Vagrant to set up my Symfony development environment more or less as described here. But we now have Docker and it’s time to move forward with a new setup. Docker containers are generally more lightweight than Vagrant virtual machines, so starting and stopping them is extremely fast. They also take a lot less disk space.

To setup a Docker container you need a Dockerfile file and to combine several containers we use the docker-compose.yml file. For our environment we will need two containers, one with the latest Ubuntu LTS to host the web server and our project files and one for MySQL.

The first container is defined in the Dockerfile file as below:

FROM ubuntu:16.04
ADD . /app
RUN apt-get update
RUN apt-get install -y php apache2 libapache2-mod-php7.0 php-mysql php-intl git git-core curl php-curl php-xml composer zip unzip php-zip
# Configure Apache
RUN rm -rf /var/www/* \
    && a2enmod rewrite \
    && echo "ServerName localhost" >> /etc/apache2/apache2.conf
ADD vhost.conf /etc/apache2/sites-available/000-default.conf
# Install Symfony
RUN mkdir -p /usr/local/bin
RUN curl -LsS https://symfony.com/installer -o /usr/local/bin/symfony
RUN chmod a+x /usr/local/bin/symfony
# Add main start script for when image launches
ADD run.sh /run.sh
RUN chmod 0755 /run.sh
WORKDIR /app
EXPOSE 80
CMD ["/run.sh"]

Add this file in a new folder. We also need the vhost.conf and run.sh files used in the code from above.

The vhost.conf file is used to configure the apache virtual host:

<VirtualHost *:80>
    ServerName localhost
    DocumentRoot /app/my_project/web
    <Directory /app/my_project/web>
        Options -Indexes
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>

The run.sh file runs when the container starts and just makes sure that the cache and logs folders are set up before launching the apache web server:

#!/bin/bash
mkdir -p my_project/app/cache my_project/app/logs
touch my_project/app/logs/prod.log
touch my_project/app/logs/dev.log
chgrp -R www-data .
chmod -R g+w my_project/app/cache my_project/app/logs
source /etc/apache2/envvars
tail -F /var/log/apache2/* my_project/app/logs/prod.log my_project/app/logs/dev.log &
exec apache2 -D FOREGROUND

That’s all for our main Ubuntu container, we now need to create the docker-compose.yml file:

version: "2"
services:
    mysql:
        image: mysql:5.7
        container_name: mysqldb
        ports:
            - "4000:3306"
        expose:
            - "3306"
        environment:
            MYSQL_ROOT_PASSWORD: yourrootpassword
    site:
        build: .
        container_name: myproject
        ports:
            - "8080:80"
        expose:
            - "80"
        depends_on:
            - mysql
        volumes:
          - .:/app

This will tell Docker to first start an MySQL 5.7 container, then our Ubuntu container that will be able to access the MySQL container using the mysql host name.

Start everything with the docker-compose up command.

When it’s done, open a new terminal (let the latest docker compose command run, do not stop it) and use the docker ps command to see the running containers:

Now, you can take the id of the Ubuntu container and ssh into it:

docker exec -t -i [ID] bash

Here you will start a new Symfony project as always (you will have to delete the my_project folder created by Docker first):

rm -rf my_project
symfony new my_project 2.8

If you name you project something else (yes, you will probably do that) just replace the all my_folder occurrences in the files and commands above with the actual name of your project.

After you created the new Symfony project, it is better to rebuild the Docker containers so the setup of the cache and logs folders will be as intended. In the terminal window where you launched Docker, press Ctrl+C to stop it then run docker-compose up again.

That’s it! You can now access your new Symfony project using the following URL: http://localhost:8080/. To connect to MySQL you will use the mysql host name:

parameters:
    database_host: mysql
    database_port: null
    database_name: symfony
    database_user: root
    database_password: yourrootpassword

 

Thank you! Please let me know in the comments if you have any suggestions to improve this setup or problems running it.


How to Get Nearby Locations from MySQL Database

If you have a MySQL table with locations (latitude and longitude columns) you can get a list of locations near a specific point using the following query:

SELECT *, ((ACOS(SIN(<latitude> * PI() / 180) * SIN(latitude * PI() / 180) + COS(<latitude> * PI() / 180) * COS(latitude * PI() / 180) * COS((<longitude> - longitude) * PI() / 180)) * 180 / PI()) * 60 * 1.1515 * 1.609344) as distance FROM locations HAVING distance <= <distance> ORDER BY distance ASC

In this query we have noted the latitude of the reference point with <latitude>, its longitude with <longitude> and the maximum distance to search for with <distance> (in kilometers).

To make this calculate the distance in miles, just remove the * 1.609344 from the end of the calculation:

SELECT *, ((ACOS(SIN(<latitude> * PI() / 180) * SIN(latitude * PI() / 180) + COS(<latitude> * PI() / 180) * COS(latitude * PI() / 180) * COS((<longitude> - longitude) * PI() / 180)) * 180 / PI()) * 60 * 1.1515) as distance FROM locations HAVING distance <= <distance> ORDER BY distance ASC

You can test the results using Google Maps. To measure distances in Google Maps you need to right-click and choose “Measure distance” from the popup menu.

How to Get Nearby Locations in Symfony with Doctrine

If you’re like me and use Symfony a lot, here’s how you can add a repository method for your entity to return objects ordered by distance from a specific location:

<?php
namespace AppBundle\Entity;
class LocationRepository extends \Doctrine\ORM\EntityRepository
{
    public function findClosest($lat, $lng) {
        $qb = $this->createQueryBuilder('l');
        $qb->addSelect('((ACOS(SIN(:lat * PI() / 180) * SIN(l.latitude * PI() / 180) + COS(:lat * PI() / 180) * COS(l.latitude * PI() / 180) * COS((:lng - l.longitude) * PI() / 180)) * 180 / PI()) * 60 * 1.1515) as HIDDEN distance');
        $qb->orderBy('distance');
        $qb->setParameter('lat', $lat);
        $qb->setParameter('lng', $lng);
        
        return $qb->getQuery()->getResult();
    }
}

To make this work we need to add the ACOS, SIN, COS and PI functions to Doctrine’s DQL as it does not have them by default.

First create 4 new classes in a new AppBundle/DQL folder, one for each function we need to define.

AppBundle/DQL/Acos.php

<?php
namespace ApiBundle\DQL;
use Doctrine\ORM\Query\AST\Functions\FunctionNode,
    Doctrine\ORM\Query\Lexer;
class Acos extends FunctionNode
{
    public $arithmeticExpression;
    public function getSql(\Doctrine\ORM\Query\SqlWalker $sqlWalker)
    {
        return 'ACOS(' . $sqlWalker->walkSimpleArithmeticExpression(
                $this->arithmeticExpression
        ) . ')';
    }
    public function parse(\Doctrine\ORM\Query\Parser $parser)
    {
        $parser->match(Lexer::T_IDENTIFIER);
        $parser->match(Lexer::T_OPEN_PARENTHESIS);
        $this->arithmeticExpression = $parser->SimpleArithmeticExpression();
        $parser->match(Lexer::T_CLOSE_PARENTHESIS);
    }
}

AppBundle/DQL/Sin.php

<?php
namespace ApiBundle\DQL;
use Doctrine\ORM\Query\AST\Functions\FunctionNode,
    Doctrine\ORM\Query\Lexer;
class Sin extends FunctionNode
{
    public $arithmeticExpression;
    public function getSql(\Doctrine\ORM\Query\SqlWalker $sqlWalker)
    {
        return 'SIN(' . $sqlWalker->walkSimpleArithmeticExpression(
                $this->arithmeticExpression
        ) . ')';
    }
    public function parse(\Doctrine\ORM\Query\Parser $parser)
    {
        $parser->match(Lexer::T_IDENTIFIER);
        $parser->match(Lexer::T_OPEN_PARENTHESIS);
        $this->arithmeticExpression = $parser->SimpleArithmeticExpression();
        $parser->match(Lexer::T_CLOSE_PARENTHESIS);
    }
}

AppBundle/DQL/Cos.php

<?php
namespace ApiBundle\DQL;
use Doctrine\ORM\Query\AST\Functions\FunctionNode,
    Doctrine\ORM\Query\Lexer;
class Cos extends FunctionNode
{
    public $arithmeticExpression;
    public function getSql(\Doctrine\ORM\Query\SqlWalker $sqlWalker)
    {
        return 'COS(' . $sqlWalker->walkSimpleArithmeticExpression(
                $this->arithmeticExpression
        ) . ')';
    }
    public function parse(\Doctrine\ORM\Query\Parser $parser)
    {
        $parser->match(Lexer::T_IDENTIFIER);
        $parser->match(Lexer::T_OPEN_PARENTHESIS);
        $this->arithmeticExpression = $parser->SimpleArithmeticExpression();
        $parser->match(Lexer::T_CLOSE_PARENTHESIS);
    }
}

AppBundle/DQL/Pi.php

<?php
namespace ApiBundle\DQL;
use Doctrine\ORM\Query\AST\Functions\FunctionNode,
    Doctrine\ORM\Query\Lexer;
class Pi extends FunctionNode
{
    public $arithmeticExpression;
    public function getSql(\Doctrine\ORM\Query\SqlWalker $sqlWalker)
    {
        return 'PI()';
    }
    public function parse(\Doctrine\ORM\Query\Parser $parser)
    {
        $parser->match(Lexer::T_IDENTIFIER);
        $parser->match(Lexer::T_OPEN_PARENTHESIS);
        $parser->match(Lexer::T_CLOSE_PARENTHESIS);
    }
}

The next and the last step is to add the above definitions to your app’s config.yml file:

doctrine:
    dbal:
        # ...
    orm:
        auto_generate_proxy_classes: '%kernel.debug%'
        naming_strategy: doctrine.orm.naming_strategy.underscore
        auto_mapping: true
        dql:
            numeric_functions:
                acos: ApiBundle\DQL\Acos
                sin: ApiBundle\DQL\Sin
                cos: ApiBundle\DQL\Cos
                pi: ApiBundle\DQL\Pi

That’s it. Now it will all work as expected. If you need to add more specific functions to DQL, just look at the above classes and make your own.


golang, middleware, negroni

Using Negroni middleware in Golang for specific routes with httprouter

In the last days I’ve played a little bit with negroni middleware and httprouter router. I found that the documentation about how can we use negroni middlewares for specific routes with httprouter is pretty poor. So I decided to write this article to share with you what I’ve found.

Negroni is an idiomatic approach to web middleware in Golang which will help you build and stack middleware very easily. It comes with some default middlewares like:

  • negroni.Recovery - Panic Recovery Middleware.
  • negroni.Logger - Request/Response Logger Middleware.
  • negroni.Static - Static File serving under the "public" directory.

But it also letting you create your own middlewares very easily:

func MyMiddleware(rw http.ResponseWriter, r *http.Request, next http.HandlerFunc) {
    // do some stuff before
    next(rw, r)
    // do some stuff after
}

…
n := negroni.New()
n.Use(negroni.HandlerFunc(MyMiddleware))

Negroni is BYOR (Bring your own Router) so we can use it with httprouter.

Httprouter is a Golang lightweight and high performance HTTP request router which is fast and has low memory consumption. TLDR: it is one of the fastest routers.

Here you have an example on how a simple route can be added when using httprouter:

router := httprouter.New()
router.POST("/login", loginHandler)

Where loginHandler will look like:

func loginHandler(w http.ResponseWriter, r *http.Request, params httprouter.Params) {
    // login controller logic
}

Now lets suppose you want to access GET profile endpoint but you need to be authenticated.
We will start by creating an authentication middleware:

// auth middleware
func auth(rw http.ResponseWriter, r *http.Request, next http.HandlerFunc) {
    // do some stuff before
    log.Println("auth middleware -> before executing controller")

    // call endpoint handler
    next(rw, r)

    // do some stuff after
    log.Println("auth middleware -> after the controller was executed")
}

Then we will create a handler which will return profile information after the authentication will be done. This handler will be sent to the auth middleware as a callback using "next" parameter:

func profileHandler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "This is the content of the profile controller\n")
    log.Println("executing profile controller")
}

Now let’s send everything to the router:

nProfile := negroni.New()
// set the middleware
nProfile.Use(negroni.HandlerFunc(auth))
// set the handler
nProfile.UseHandlerFunc(profileHandler)
// attach negroni middleware and handler to our route
router.Handler("GET", "/profile", nProfile)

That was simple, but now if we have a route which contains parameters inside it (like /hello/:name) we cannot access it from our handler function. To solve that, I’ve written the following functions which will help us call the handler with the parameters:

// callwithParams function is helping us to call controller from middleware having access to URL params
func callWithParams(router *httprouter.Router, handler func(w http.ResponseWriter, r *http.Request, ps httprouter.Params)) func(w http.ResponseWriter, r *http.Request) {
    return func(w http.ResponseWriter, r *http.Request) {
        params := getUrlParams(router, r)
        handler(w, r, params)
    }
}

// getUrlParams function is extracting URL parameters
func getUrlParams(router *httprouter.Router, req *http.Request) httprouter.Params {
    _, params, _ := router.Lookup(req.Method, req.URL.Path)

    return params
}

And will be used like this:

router := httprouter.New()
nHello := negroni.New()
// add auth middleware
nHello.Use(negroni.HandlerFunc(auth))
// add handler using callWithParams function so we can access the URL parameters in handler
nHello.UseHandlerFunc(callWithParams(router, helloHandler))
router.Handler("GET", "/hello/:name", nHello)

n.UseHandler(router)

log.Fatal(http.ListenAndServe(":8080", n))

 

The full working example can be found here.


Symfony OAuth Authentication for Your Mobile Application

Let’s say you built an API using Symfony and you need to access it from a mobile application using authenticated requests on behalf of your users.

Here’s how to make this work using Symfony 2.8 and Doctrine.

Install FOSOAuthServerBundle

We will use the FOSOAuthServerBundle to implement this feature. Install it using the following command:

composer require friendsofsymfony/oauth-server-bundle

Next, enable the bundle in the AppKernel.php file:

public function registerBundles()
{
    $bundles = array(
        // ...
        new FOS\OAuthServerBundle\FOSOAuthServerBundle(),
    );
}

Create OAuth model classes

To create the OAuth model classes just add the following files to your project. Here we already have FOSUserBundle installed and set up to use the ApiBundle\Entity\User class.

src/ApiBundle/Entity/Client.php

<?php
namespace ApiBundle\Entity;

use FOS\OAuthServerBundle\Entity\Client as BaseClient;
use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 */
class Client extends BaseClient
{
    /**
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    protected $id;

    public function __construct()
    {
        parent::__construct();
        // your own logic
    }
}

src/ApiBundle/Entity/AccessToken.php

<?php
namespace ApiBundle\Entity;

use FOS\OAuthServerBundle\Entity\AccessToken as BaseAccessToken;
use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 */
class AccessToken extends BaseAccessToken
{
    /**
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    protected $id;

    /**
     * @ORM\ManyToOne(targetEntity="Client")
     * @ORM\JoinColumn(nullable=false)
     */
    protected $client;

    /**
     * @ORM\ManyToOne(targetEntity="User")
     */
    protected $user;
}

src/ApiBundle/Entity/RefreshToken.php

<?php
namespace ApiBundle\Entity;

use FOS\OAuthServerBundle\Entity\RefreshToken as BaseRefreshToken;
use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 */
class RefreshToken extends BaseRefreshToken
{
    /**
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    protected $id;

    /**
     * @ORM\ManyToOne(targetEntity="Client")
     * @ORM\JoinColumn(nullable=false)
     */
    protected $client;

    /**
     * @ORM\ManyToOne(targetEntity="User")
     */
    protected $user;
}

src/ApiBundle/Entity/AuthCode.php

<?php
namespace ApiBundle\Entity;

use FOS\OAuthServerBundle\Entity\AuthCode as BaseAuthCode;
use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 */
class AuthCode extends BaseAuthCode
{
    /**
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    protected $id;

    /**
     * @ORM\ManyToOne(targetEntity="Client")
     * @ORM\JoinColumn(nullable=false)
     */
    protected $client;

    /**
     * @ORM\ManyToOne(targetEntity="User")
     */
    protected $user;
}

Configure FOSOAuthServerBundle

Import the routing configuration in your app/config/routing.yml file:

fos_oauth_server_token:
    resource: "@FOSOAuthServerBundle/Resources/config/routing/token.xml"
fos_oauth_server_authorize:
    resource: "@FOSOAuthServerBundle/Resources/config/routing/authorize.xml"

Add FOSOAuthServerBundle settings in app/config/config.yml:

fos_oauth_server:
    db_driver: orm       # Drivers available: orm, mongodb, or propel
    client_class:        ApiBundle\Entity\Client
    access_token_class:  ApiBundle\Entity\AccessToken
    refresh_token_class: ApiBundle\Entity\RefreshToken
    auth_code_class:     ApiBundle\Entity\AuthCode
    service:
        user_provider: fos_user.user_provider.username

Back to the models

Generate a migration and migrate the database:

php app/console doctrine:migrations:diff
php app/console doctrine:migrations:migrate

…or, if you’re not using migrations, just update the database schema:

php app/console doctrine:schema:update --force

Configure your application’s security

Edit your app/config/security.yml file to add FOSOAuthServerBundle specific configuration:

# ...

    firewalls:
        oauth_token: # Everyone can access the access token URL.
            pattern: ^/oauth/v2/token
            security: false
            
        api:
            pattern:    ^/api
            fos_oauth:  true
            stateless:  true
            anonymous:  true # can be omitted as its default value
    
    # ...

    access_control:
        - { path: ^/api, role: IS_AUTHENTICATED_FULLY }

Create a client

Before you can generate tokens, you need to create a Client using the ClientManager. For this, create a new Symfony command:

<?php
namespace ApiBundle\Command;

use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;

class OAuthAddClientCommand extends ContainerAwareCommand
{
    protected function configure()
    {
        $this
            ->setName('oauth:add-client')
            ->setDescription("Ads a new client for OAuth")
        ;
    }

    protected function execute(InputInterface $input, OutputInterface $output)
    {
        $redirectUri = $this->getContainer()->getParameter('router.request_context.scheme') . "://" . $this->getContainer()->getParameter('router.request_context.host');
        $clientManager = $this->getContainer()->get('fos_oauth_server.client_manager.default');
        $client = $clientManager->createClient();
        $client->setRedirectUris(array($redirectUri));
        $client->setAllowedGrantTypes(array('refresh_token', 'password'));
        $clientManager->updateClient($client);
    }
}

Now run the above command to generate your first OAuth client:

php app/console oauth:add-client

This client will be able to generate tokens and refresh tokens using the user’s username and password. You can find it’s data in the database client table. The token endpoint is at /oauth/v2/token by default.

Document using NelmioApiDocBundle

If you use the NelmioApiDocBundle to document your API, you can add these OAuth methods too. Create a new YAML file in src/ApiBundle/Resources/apidoc/oauth.yml:

grant_type_password:
    requirements: []
    views: []
    filters: []
    parameters:
        grant_type:
            dataType: string
            required: true
            name: grant_type
            description: Grant Type (password)
            readonly: false
        client_id:
            dataType: string
            required: true
            name: client_id
            description: Client Id
            readonly: false
        client_secret:
            dataType: string
            required: true
            name: client_secret
            description: client Secret
            readonly: false
        username:
            dataType: string
            required: true
            name: username
            description: Username
            readonly: false
        password:
            dataType: string
            required: true
            name: password
            description: Password
            readonly: false
    input: null
    output: null
    link: null
    description: "Get OAuth token for user using username and password"
    section: "OAuth"
    documentation: null
    resource: null
    method: "POST"
    host: ""
    uri: "/oauth/v2/token"
    response:
        token:
            dataType: string
            required: true
            description: OAuth token
            readonly: true
    route:
        path: /oauth/v2/token
        defaults:
            _controller: FOS\UserBundle\Controller\SecurityController::checkAction
        requirements: []
        options:
            compiler_class: Symfony\Component\Routing\RouteCompiler
        host: ''
        schemes: []
        methods: [ 'POST' ]
        condition: ''
    https: false
    authentication: false
    authenticationRoles: []
    cache: null
    deprecated: false
    statusCodes: []
    resourceDescription: null
    responseMap: []
    parsedResponseMap: []
    tags: []
    
grant_type_refresh_token:
    requirements: []
    views: []
    filters: []
    parameters:
        grant_type:
            dataType: string
            required: true
            name: grant_type
            description: Grant Type (refresh_token)
            readonly: false
        client_id:
            dataType: string
            required: true
            name: client_id
            description: Client Id
            readonly: false
        client_secret:
            dataType: string
            required: true
            name: client_secret
            description: client Secret
            readonly: false
        refresh_token:
            dataType: string
            required: true
            name: refresh_token
            description: Refresh token
            readonly: false
    input: null
    output: null
    link: null
    description: "Get new OAuth token using refresh token"
    section: "OAuth"
    documentation: null
    resource: null
    method: "POST"
    host: ""
    uri: "/oauth/v2/token"
    response:
        token:
            dataType: string
            required: true
            description: OAuth token
            readonly: true
    route:
        path: /oauth/v2/token
        defaults:
            _controller: FOS\UserBundle\Controller\SecurityController::checkAction
        requirements: []
        options:
            compiler_class: Symfony\Component\Routing\RouteCompiler
        host: ''
        schemes: []
        methods: [ 'POST' ]
        condition: ''
    https: false
    authentication: false
    authenticationRoles: []
    cache: null
    deprecated: false
    statusCodes: []
    resourceDescription: null
    responseMap: []
    parsedResponseMap: []
    tags: []

Add a new NelmioApiYmlProvider.php file in src/ApiBundle/Service folder:

<?php

namespace ApiBundle\Service;

use Nelmio\ApiDocBundle\Annotation\ApiDoc;
use Nelmio\ApiDocBundle\Extractor\AnnotationsProviderInterface;
use Symfony\Component\Finder\Finder;
use Symfony\Component\Routing\Route;
use Symfony\Component\Yaml\Yaml;

/**
 * Generate annotations for vendor routes to be displayed in Nelmio ApiDoc.
 */
class NelmioApiYmlProvider implements AnnotationsProviderInterface
{
    private $vendorFolder;

    public function __construct($vendorFolder)
    {
        $this->vendorFolder = $vendorFolder;
    }
    /**
     * {@inheritdoc}
     */
    public function getAnnotations()
    {
        $annotations = [];
        $configDirectories = array($this->vendorFolder);

        $finder = new Finder();

        $finder->files()->in($configDirectories);

        if (count($finder) == 0) {
            return $annotations;
        }

        foreach ($finder as $file_) {
            $data = Yaml::parse(file_get_contents($file_));

            $vendors = array_keys($data);
            foreach ($vendors as $vendor) {
                $apiDoc = new ApiDoc($data[$vendor]);
                $route = new Route(
                    $data[$vendor]['route']['path'],
                    $data[$vendor]['route']['defaults'],
                    $data[$vendor]['route']['requirements'],
                    $data[$vendor]['route']['options'],
                    $data[$vendor]['route']['host'],
                    $data[$vendor]['route']['schemes'],
                    $data[$vendor]['route']['methods'],
                    $data[$vendor]['route']['condition']
                );

                $apiDoc->setRoute($route);
                $apiDoc->setResponse($data[$vendor]['response']);
                $annotations[] = $apiDoc;
            }
        }

        return $annotations;
    }
}

Add a new service in src/ApiBundle/Resources/config/services.yml file:

services:
    nelmio_api_doc.yml_provider.api_yml_provider:
        class: ApiBundle\Service\NelmioApiYmlProvider
        arguments:
            folder: %kernel.root_dir%/../src/ApiBundle/Resources/apidoc
        tags:
            - { name: nelmio_api_doc.extractor.annotations_provider }

You’ll find now two /oauth/v2/token methods with different parameters listed in the api/doc section of your project.

That’s all! You can now use the generated client to authenticate your users in your mobile app using OAuth.

How to use the FOSOAuthServerBundle

First you will need to get an access token by making a POST request to the /oauth/v2/token endpoint with the following parameters:

grant_type=password
client_id=[client's id from the database followed by '_' then the corresponding random id]
client_secret=[client's secret]
username=[user's username]
password=[users's password]

You should get back something like this:

{
  "access_token": "ZDgxZDlkOWI2N2IyZWU2ZjlhY2VlNWQxNzM0ZDhlOWY2ZTIwOTBkNGUzZDUyOGYxOTg1ZTRjZGExOTY2YjNmNw",
  "expires_in": 3600,
  "token_type": "bearer",
  "scope": null,
  "refresh_token": "MDQ3MGIwZTk5MDkwOGM5NjhkMzk5NTUyZDJjZmYwM2YzZWViZDFhZjk0NTIyZmNjNzkyMDM0YjM4ODQ2N2VhNg"
}

Use the access token for authenticated requests by placing it in the request header:

Authorization: Bearer ZDgxZDlkOWI2N2IyZWU2ZjlhY2VlNWQxNzM0ZDhlOWY2ZTIwOTBkNGUzZDUyOGYxOTg1ZTRjZGExOTY2YjNmNw

When the access token expires, you can get a new one using the refresh_token grant type at the same /oauth/v2/token endpoint:

grant_type=refresh_token
client_id=[client's id from the database followed by '_' then the corresponding random id]
client_secret=[client's secret]
refresh_token=[refresh token received earlier]

The response should be similar to:

{
  "access_token": "MjE1NjRjNDc0ZmU4NmU3NjgzOTIyZDZlNDBiMTg5OGNhMTc0MjM5OWU3MjAxN2ZjNzAwOTk4NGQxMjE5ODVhZA",
  "expires_in": 3600,
  "token_type": "bearer",
  "scope": null,
  "refresh_token": "YzM2ZWNiMGQ5MDBmOGExNjhmNDI1YjExZTkyN2U0Mzk5ZmM4NzcwNDdhNjAzZDliMjY3YzE0ZTg5NDFlZjg3MQ"
}

Mysqldump Through a HTTP Request with Golang

So, in a previous post I explained how one can backup all databases on a server, each in its own dump file. Let's take it to the next level and make a Golang program that will let us run the dump process with a HTTP request.

Assuming you already have Go installed on the backup server, create first a project directory in your home folder for example. Copy the mysql dump script from here and save it as dump.sh in your project folder. Modify ROOTDIR="/backup/mysql/" inside dump.sh to reflect current project directory.

We will create a Golang script with two functions. One will launch the backup script when a specific HTTP request is done. The other one will put the HTTP call behind a authentication, so only people with credentials will be able to make the backup request.

package main

import (
	"encoding/base64"
	"fmt"
	"log"
	"net/http"
	"os"
	"os/exec"
	"strings"
)

var username = os.Getenv("DB_BACKUP_USER")
var password = os.Getenv("DB_BACKUP_PASSWORD")

func BasicAuth(w http.ResponseWriter, r *http.Request, user, pass string) bool {
	s := strings.SplitN(r.Header.Get("Authorization"), " ", 2)
	if len(s) != 2 {
		return false
	}
	b, err := base64.StdEncoding.DecodeString(s[1])
	if err != nil {
		return false
	}
	pair := strings.SplitN(string(b), ":", 2)
	if len(pair) != 2 {
		return false
	}
	return pair[0] == string(user) && pair[1] == string(pass)
}
func handler(w http.ResponseWriter, r *http.Request) {
	if BasicAuth(w, r, username, password) {
		cmd := exec.Command("bash", "dump.sh")
		stdout, err := cmd.Output()
		if err != nil {
			log.Fatal(err)
		}
		fmt.Fprintf(w, string(stdout))
		return
	}
	w.Header().Set("WWW-Authenticate", `Basic realm="Protected Page!!! "`)
	w.WriteHeader(401)
	w.Write([]byte("401 Unauthorized\n"))
}

func main() {
	http.HandleFunc("/backup", handler)
	http.ListenAndServe(":8080", nil)
}

This uses DB_BACKUP_USER and DB_BACKUP_PASSWORD that you will have to set as environment variables. Just append this to your ~/.bashrc file

export DB_BACKUP_USER="hello"
export DB_BACKUP_PASSWORD="password"

Now run source ~/.bashrc to load them.

Build the executable with go build http-db-backup.go where http-db-backup.go is the name of your Go file. Now you need to run the executable with sudo, but while preserving the environment: sudo -E ./http-db-backup

Now if you open your browser and open http://111.222.333.444:8080/backup (where 111.222.333.444 is your backup machine IP) the backup process will start, and you will get the output of the dump.sh in your browser when backup finishes.

We can furthermore add another function to list the directory in browser, so you can download the needed backup or backups.

func lister(w http.ResponseWriter, r *http.Request) {
	if BasicAuth(w, r, username, password) {
		http.FileServer(http.Dir(".")).ServeHTTP(w, r)
		return
	}
	w.Header().Set("WWW-Authenticate", `Basic realm="Protected Page!!! "`)
	w.WriteHeader(401)
	w.Write([]byte("401 Unauthorized\n"))
}

All you need to do is to add http.HandleFunc("/", lister) to your main() and navigate to http://111.222.333.444:8080/ . You will be able to navigate the backup directory to download the dump files.


Best Solutions to Top Mobile App Development Challenges

Best Solutions to Top Mobile App Development Challenges

More than half of the world’s population now uses a smartphone. As the price of data plans continue to decrease all over the world, web traffic coming from mobile devices increases year by year. In fact, more than half of today’s global internet traffic comes from mobile devices.

People are spending more time on their mobile devices, thereby increasing the demand for mobile apps. Recent statistics show that 52% of the time individuals spend on digital media is on mobile apps. In 2016, global mobile app revenues amounted to 88.3 billion U.S. dollars, and it’s forecasted that by 2020, global mobile app revenues from app stores and in-app advertising will reach 188.9 billion U.S. dollars.

Indeed, mobile app development is a lucrative business, given the mobile and digital revolution.

But, for every venture comes a challenge. It’s important to know the potential challenges you may face and how to address them before hiring an app developer.

running

Tight Market Competition

The app market is overcrowded, and it’s becoming a challenge for a mobile application to get noticed. Apple’s App Store has over 2.2 billion apps and Google’s Play Store has over 2.8 million apps as of March 2017. That doesn’t even include the apps in Windows Phone Store and BlackBerry App World.

Solution! Define your target market and the main goal of the app. These two details are fundamental to the overall design, UI, and UX of your mobile app.

Make sure to do your market research to study the apps that already offer similar functionality. Determine your unique selling point, and make sure that your app features a beautiful design and an easy-to-use interface.

Funding

So, you’ve already come up with a solid app idea, and you’re sure that it’s going to be a hit. It’s time to get the funding to get it built.

Mobile app development can be costly, depending on the design, function, and interface of the app. Prepare the funding before you hire an app developer to avoid delays in the launch. You should also take into consideration the cost of marketing your new app.

Solution! If you can’t personally fund your mobile app idea, there are many ways to get funding. You can try crowdfunding websites like GoFundMe, Kickstarter, AppBackr, and the like. You can also ask your relatives or friends to partner up with you. If you want your mobile app to be a solo venture, then you can try for a bank loan or personal loan from people in your social circle.

Device Compatibility

Screen Size and Device Compatibility

Users have a wide variety of devices that run on different screen sizes, operating systems and platforms. Make sure that your mobile app developer understands the importance of a responsive design. Your mobile app should work for devices of all screen sizes, from the smallest smartphone to the largest tablet computer.

Solution! OS is another aspect to consider. iOS, Android and Windows has its own UI and algorithms. Be prepared to encounter bugs and errors along the way. For this reason, it’s highly recommended to release a beta version.

Interactive UI and User-Friendliness

Poor user experience is one of the top reasons why a lot of mobile apps don’t succeed simply because it turns off users. Your mobile app must be easy to use and understand. It doesn’t need to have a wide plethora of unnecessary features.

Solution! Include a tutorial that users can go back to within the app itself. Also, mobile devices nowadays include high-tech sensors and hardware that offers a range of opportunities for creating an interactive app. Explore the possibilities, but stay true to the goal and function of your mobile app.

coding

Development Approach

There are numerous app development approaches that you can utilize, such as hybrid app development, cross-platform, native app development, or platform specific native app development. Also, you don’t want to waste time and additional resources in developing an app that wouldn’t work for multiple devices and platforms.

Solution! Strategize and decide on the tools and frameworks to use according to the UI and UX needs of your app. Platforms like OpenSourceSDK offers the ability to develop apps using C/C++ or HTML5. It also allows developers to incorporate single API across multiple platforms. Then, there are cross-platform developer tools that you can leverage for your mobile app venture, such as RhoMobile, PhoneGap, Appcelerator, Mosync, and WidgetPad.

Managing Mobile Content

Content is the life of a mobile app. Today’s mobile users demand rich content, image, and animation. However, you need to consider internet speed and bandwidth. Oftentimes, wireless network connection contributes to the failed delivery of mobile app content. As a result, users find it too bothersome to use mobile apps that don’t load quickly.

Solution! Choose a mobile content management system that simplifies content and optimizes bandwidth, such as AirWatch, Alfresco, and MobileIron. These tools can help you deliver rich content reliably.

Battery Life

Balancing Performance and Battery Life

It’s a big challenge to design a well-performing and bug-free app that contains rich media yet doesn’t consume too much battery life. Pokemon Go, for instance, is a great game with rich AR graphics and features. Users were hooked the moment they downloaded the app. But, most users are complaining that the game drains their smartphone quickly. Graphics, animation, content, and function should not compromise battery consumption.

Solution! Run tests to monitor the performance of your app.

Marketing and Promotion

Your mobile app is now among the millions of apps listed in the Apple’s App Store or Google’s Play Store. How do users find it? Branding, marketing and promotions play a huge role in the success of your mobile app.

Solution! Platforms like AppTamin, AppScend , MobileDevHQ, and Smore can help you market your mobile apps. You can also take to social media advertisements to get the word out there. Create a data-driven strategy and consult with marketing professionals about how you can get your target market to download the new app.

It’s true that mobile apps are popping up like mushrooms in app stores. But, mobile app development isn’t as easy as 1, 2, 3, launch! There are challenges that beset creators and developers. Fortunately, these challenges can be overcome with careful planning.