Actual data extracted from one of our client's database

Developing applications for Big Data

Engineering software platforms for High Volume Transaction Processing (HVTP) require an array of skills and tools in order for every piece of the application to work harmoniously and efficiently.

Start with the infrastructure and think about scalability first. Your servers must be ready to grow if needed – possibly automatically and before reaching critical points. If you think your application is demanding today, check it again tomorrow. With social media playing such an important role in marketing nowadays, growth can far exceed what you expect. Be ready to double, and then double again, the number of users you have today.

Think of each part of the software individually in order to achieve the maximum result by giving each of them equal importance. Every single part of the system must be ready for volume; thus, the physical machine, the database, and the code must be equally high-performance. On the code side, event-driven languages are excellent for dealing with memory. You have a few options here: Node.js, Erlang, Elixir, Scala, and Ruby.

Prefer a NoSQL DB (OrientDB is a great option) if your data requires a large amount of table joins. In that case, put a NoSQL hat on and start thinking about the data as part of a large net, where each record is directly connected to the subsequent records around it. If your data is indeed relational, try to denormalise your tables and create indexes where appropriate.

From the ground up, focus on code optimisation. Use design patterns like the Single Responsibility Principle. Keep things organised by applying patterns like the Model View Controller Pattern, which is valid for both front- and back-end areas.

Test! It is possible to perform simulated concurrent API connections with a third-party application. Here at Techifide, we created a dedicated application just for that. We start by opening 500K concurrent connections and increasing them up to 100M. At each stage, we check CPU and RAM usage on the API’s side, spot bottlenecks, and improve the application as soon as possible.

Talk to us!

At Techifide, we provide Software Design and Architecture for Big Data Applications, Use Case Analysis, MVP and Software Development for High Volume Transactions. Talk to us and find out how we can help you create your SaaS.