Moving Big data in and around Azure using Azure Data Factory.

There are numerous data storage options available on Azure, each one designed and developed for different modern data storage scenarios. These storage options could be in the form of database, data warehouse, data caches and data lakes. Usage of these depends on the application and the scale that they serve. Within databases, some applications might need relational database, some might need NOSQL, or a key-value storage, or in-memory database (for caching), or blob storage (for media and large files). Another criteria to keep in mind when selecting a database for your application is the required read-write throughput and latency. Azure has a wide array of fully-managed database services which frees up the development teams valuable time in managing, scaling and configuring these databases.

Whatever database you choose, you should also keep in mind how easy or difficult it is to move the data in and out of that database. You might have a situation in future where you need to move to a new database solution because of reasons like change in application architecture, scale, performance, or even cost. Microsoft Azure has a very powerful ETL tool called Azure Data Factory to easily move data in and around Azure at scale. It has over 80 native connectors which can serve both as source and sink.  In this blog, I would like to highlight a few features and concepts of Azure Data Factory which will serve as a quick start guide for anyone looking to do data movement and transformation on Azure.

by Siddharth Bhola Jul 24, 2019

Design patterns are a lifesaver for Extensible Applications.

All of the major programming languages have evolved tremendously over the past decade especially to make the developers job easy. C# by Microsoft has been at the top of the game of adding plethora of easy to use and extremely powerful constructs ranging from async programming to functional programming. The well-developed programming languages have made it extremely easy for beginners to pick up coding and start developing applications quickly.

However, when developing extensible applications for the future, just using the language to write down logic for applications does not do the job well. We have to think about the overall design of the application for it to be able to scale and extend well. This means utilizing appropriate design patterns wherever applicable.

by Siddharth Bhola Jul 11, 2019

Introduction to MQTT protocol for IOT applications.

In my previous blog post, I explained why the widely used application-layer protocol HTTP is not suitable for the IOT applications. There are a number of application-layer protocols developed to serve the purpose of IOT like MQTT, AMQP, CoAP, XMPP, STOMP etc. In this blog post, I will provide an introduction to MQTT protocol, its design, architecture and use-case examples.

by Siddharth Bhola Jun 26, 2019

Why HTTP is not suitable for IOT applications.

When developing modern web-applications, the most commonly used application layer protocol is HTTP. It is suitable for serving hypermedia resources in a request-response fashion over the internet. However, when it comes to the Internet of things(IOT), it tends to perform poorly mainly because it was designed to be a reliable but heavy-weight protocol for the world-wide-web. In this blog, I want to highlight the issues with HTTP with respect to IOT applications.

by Siddharth Bhola Jun 24, 2019