Deep Tech

Building is about getting around the obstacles that are presented to you.

Jeremy Renner

While working for a previous employer, I was tasked to build a highly performant and negligible latency system. The system was suppose to process some realtime stock market data and make decisions based on that data. It incorporated machine learning and other complex algorithms. The system was suppose to be highly scalable and fault tolerant.

The idea was to build the Dream System.

Building software can be an exhilarating experience, especially when it’s for a dream project. There’s a sense of ownership that comes with building something that you’re passionate about, and that can keep you motivated during the long hours of work and problem-solving. However, building a dream software also comes with its fair share of challenges.

One of the biggest challenges was to manage the expectations of stakeholders. Since everyone has a different idea of what the end product should look like, keeping everyone aligned and on the same page was difficult. So It was important to establish clear goals and milestones early on in the project so that we avoided any miscommunication or confusion down the line.

Another challenge and probably the most important one was to get the right tools for the job. There were many different technologies out there to choose from and as the team was a mix of experienced and new developers, the experienced developers had their own preferences and the new developers were not sure what to choose. The team was divided on the choice of technology and it soon escalated to the management and guess what happened: they decided for us (go with the most popular technology stack that can get the job done Faster)

But the question was is faster always better?

While this war went on, the project was delayed and the management was not happy. But it was essential for the team to have a common ground and move forward.
So after a lot of discussions and debates, we finally decided to go with the most popular technology stack that can get the job done Faster 🤦

Well, engineers are humans too and they can make mistakes. We made a mistake and we paid for it later.

The first week of development was smooth, we were able to get the basic system up and running. But as we started adding more features, we started to see the cracks. The system was not able to handle the load and it was crashing. We tried to optimize the code and database queries but it was not enough. We tried to scale the system, but it was not enough. We tried to add more servers, but it was not enough. We tried to add more memory, but it was not enough. We tried to add more CPU, but it was not enough.

Well it surpassed the initial estimated cost by 10x and the system was still not able to handle the load. Can you believe it? 10x the initial estimated cost and the system was still not able to handle the load. Have you been there? Have you been in a situation where you have to make a decision between the cost and the performance? Well, we were there and we had to make a decision.

This is when we as a team decided to go back to the drawing board and rethink our approach. We decided to go back to the basics and start from scratch. We started questioning on all the decisions that we made in the process of building the Dream Software and we started to realize that we made a lot of mistakes. We started to realize that:

• We made a lot of assumptions and we didn’t have enough data to back those assumptions.
• We didn’t have a clear understanding of the problem that we were trying to solve.
• Our decision of the technology stack was not apt for the problem that we were trying to solve.

After having this realization, The question was What do we do now?

Well, lets work our asses off and build the Dream Software is what we decided to do.

The team dealt with setbacks and failures. But It’s common to encounter obstacles and bugs during the development process, and it can be frustrating when progress is slow. However, it’s important to keep a positive attitude and remain persistent, knowing that each failure brings you closer to finding the solution.

Despite the challenges, the reward of seeing a dream software come to life was incredibly fulfilling. Bringing an idea to fruition and creating something that can make a difference in people’s lives is a truly satisfying experience. Overall, building a dream software requires hard work, patience, mistakes and creativity, but the end result is well worth it.