For the past year, I’ve been working on a project where I’ve been building a web product for one of the largest retailers in the world. The website itself is a private tool used internally by the company, but with over 100’000+ employees even an internal tool could receive a significant amount of user requests.
So how do you make sure that a web application is ready for a smooth release without performance issues when you finally open the flood...
Read more
A little more than 1 year ago I decided to apply to become a freelancer with the exclusive freelance platform TopTal. They claim to only recruit the Top 3% of talent and it’s definitely the crème de la crème within its category and the platform that many freelancers aspire to join.
After preparing for the application process in the same way as I would prepare for an interview with any large technology company, I was accepted to the...
Read more
This week I ran into something interesting on the current project that I’m working on. In the project, we have a PostgreSQL datamart where we store a ton of data generated from a machine learning model.
Data is added to the database every time a run finishes and each run contain hundreds of thousands of entries, on top of that we run around ~200 runs per day so that equals to at least 20M rows per day, ouch.
Sometimes some of our favorite tools make things so simple for us by abstracting away so much of the technical implementations, that we lose track of what is actually going on behind the scenes, which cause us to accidentally implement things incorrectly and have unintended bugs within our code.
I ran into one of these bugs a few days ago with Celery. I had a Celery task that accepted a deserialized yaml file as a dictionary, it then modified some...
Read more
Stability of your asynchronous background tasks is crucial for your system design. When you move the work of processing from the main application and instead leverage something like Celery, to execute the work in the background, it’s important that you can feel confident that those tasks get executed correctly without you having to babysit it and wait for the results.
There are generally two things that can go wrong as you send a task to a Celery worker to process...
Read more
These days, many of the large tech companies that people are aspiring to work for, are following a very similar recruitment process that is split up into multiple steps that slowly filter down the applicants to a few final ones.
This process usually looks something like this:
Phone Interview
Coding Challenge (Either live or using online tools)
Engineering Interview(s)
Partner/Manager/Team Leader Interview
The steps can vary slightly, but in almost all cases they will always start off by...
Read more
Reacting on calls to Celery tasks is one of the first things that you will want to dig deeper in as soon as you start scratching the surface of Celery. How do you react on when a task finishes or fails, and then trigger some other code when these events occur?
For example, I was working on a project recently where we leveraged Celery to do asynchronous communication between services in a distributed system. We had multiple applications that were...
Read more
Parsing file paths, web addresses or file names is something that we have to come back to over and over again in different situations. It could be that you want to validate the file extension of a file name, or perhaps that you want to get a hostname of a full URL.
There are plenty of different methods to break down a string of a path into smaller components, to allow you to get the information you’d like. A lot...
Read more
Personally, I feel that code reviews have been one of the key tools in my toolbox that have helped me develop the most within the past few years.
Not only has it helped me and the team that I’m working with to create a high quality, consistent and easy-to-use code base, but it has also given me great perspectives and insight into what I can do better, and how I can take the next steps into becoming a better software...
Read more
Scaling Django in production to be able to serve your application to thousands of visitors is one of the most popular topics that people wonder and ask about within the Django community. I can’t even count the times I’ve seen people ask questions related to this within Facebook groups, StackOverflow and other discussion forums.
Yes, Django scales incredibly well and we can see that from use cases such as BitBucket, Quora, Instagram, Disqus and other services that use Django to...
Read more