CSV as a file format is very versatile, almost any programs can parse it. The only issue is you can't use SQL against CSV files directly.
This is a major pain point, since using SQL is so much faster than firing up a jupyter notebook and wrangle the data in python, or use Ex...
Platform engineering is all the rage these days. Often, you'll often hear this term with the keyword DevX. How are they related?
Imagine you are working on a microservice backend. You are just starting out, so you don't have much features to work on yet. But as a PoC, you on...
Imagine having multiple services running, each has its own logs. Most people don't read them, and they shouldn't, because services emit a lot of logs! But we need them, because it's the only way to diagnose and troubleshoot system errors. But you might say "my service is not...
I think everyone who has worked with data, in any role or function, used pandas 🐼 at certain point. I first used pandas in 2017, so it's 6 years already. Things have come a long way, and so is data size I'm working with!
Pandas has its own issues, namely no native support fo...
Correction 2023-07-02: fix homelab specs and corresponding AWS EC2 instance class (it's actually 32GB RAM, not 64GB)
Congratulations, you managed to successfully deployed a few services on kubernetes! But this is not the end 👀. Unfortunately money doesn't grow on trees, and ...
Kubernetes is awesome, I think this is obvious if you have more than a handful of services to manage. If you use cloud, either VM or container-based runtime, it would provide you a dashboard to see the metrics. But what about kubernetes? Since you would have multiple service...
Background
Ever wonder how websites (and everything in between) work? Chances are you can create a project running on your local machine. It works as you expected, but to let other people access it, you have to "deploy" it. For many years, to support a lot of request volume...
Background
You want to track who access your site, Google Analytics can do that. To see the data, you can use Google Analytics dashboard, the default settings is good enough for most use cases. But what if you have a lot of tracking data, and you want to streamline a way to...
Raise your hands if you normally have to send .env files to your team members so they can start a project in dev environment.
While there is nothing wrong with this approach, it could introduce a lot of security risks, namely sharing secrets via plaintext protocol. Sure, you...
For most people, using SQL to transform data is a no-brainer, seeing it's a very versatile language, and doesn't have quite a steep learning curve compared to python. There are some cases where SQL is more suitable for a task, but the reverse can also happen as well.
For ins...
I love not having to manage databases. Hosting it on your compute is guaranteed to be cheaper, but I don't want to be constantly worrying about backups and database upgrade / maintenance.
AWS offers managed databases, known as RDS. Mostly I use postgres, which works well for...
Organizational health lies in being able to work together.
As a manager, our job is to try as much as we can to drive balance and clarity.
A happy, driven team sees the wider purpose of their work and also feels empowered to execute tasks individually.
“You can’t call yourse...
Opportunity cost
Staff engineers’ skills are usually in high demand. If you’re assigned to a single team, you may not be top of mind for solving a problem elsewhere in the org, or your manager may be unwilling to let you go.
Losing empathy
It’s easy to overfocus and forget...
Scaling, the dreaded word among developers, because this means more complexity. But why do we need scaling?
Imagine a super busy corner store. During early mornings, there might not be a lot of customers, so one cashier might be enough to handle all customers. But during aft...
For many years, installing a package on linux means either:
Compiling a binary from source, then install it. -> I think we know why this didn't catch on for the mass.
Downloading a compiled binary for your system's architecture and platform. -> This requires you to al...
Recently I work with GitHub CLI a lot, and having to constantly fire up Bitwarden app to retrieve GITHUB_TOKEN gets old real fast...
I was thinking of storing it in a gist in a password manager, luckily someone had the same idea and implemented it. The only issue is that I u...
I self hosted a lot of services, sometimes I try out a few apps that would get deleted within the same day. All this requires setting up CNAME for reverse-proxy (because I want to make sure there's no funny reverse-proxy shenanigans going on, for future reference).
I can alw...
You are building a website, it's a simple frontend that needs to call the database for [total lead drops this week]. Your website is still at an infancy stage, with only a few features. At this point, you contemplate whether you need a proper backend or not. But to deploy a ...
Back in the day, there was no cloud. If you want a lot of computing power, you need to build your own data center, and this is very expensive. Then cloud happened, and suddenly you can work with a lot of flexibility like you couldn't before. Want to try out a small deploymen...
Updated 2023-02-20: update terraform code
For frontends, if no server-side rendering is required, we can deploy it as a static site. If you already use GitHub, you might be familiar with GitHub Pages. One common use case is to deploy your personal landing page / blog via Git...
Previously I wrote about setting up ECS task on fargate backend. But we can also use EC2 as backend too, in some cases where the workload is consistent, ie scaling is not required, since EC2 would be cheaper than fargate backend, even more so if you have reserved instance on...
Imagine you have a few data pipelines to schedule. Simplest solution would be cronjob. Time goes by and next thing you know, you have around 50 pipelines to manage. The fun starts when you have to hunt down which pipeline doesn't run normally. And by then it would be super h...
For IaC, no doubt that Terraform is the leader. But there are other alternatives too, one of them is Pulumi. Currently Pulumi provides fun challenges to get started with their services. Best of all, they give you swags too! We are going to create a simple Pulumi project for ...
I have been working in the data industry since almost half a decade ago. Over time I have noticed so-called archetypes within various data engineering roles. Below are main skills and combinations I have seen over the years. This is by no means an exhaustive list, rather wha...
To deploy a web application, there are many ways to go about it. I could spin up a bare VM and set up the environment manually. To make things easier, I could have package the app into docker image. But this still means I have to "update" the app manually if I add changes to...
I often hear people ask "if you can do data engineering with SQL, then what's the point of learning spark or python?"
Data ingestion
Let's circle back at bit.
I think we all can agree that: there's a point in time where there's no data in the data warehouse (which DE-who-us...
Imagine working in a company, and they have a super cool internal module! The module works great, except that it is a private module, which means you need to install it by cloning the source repo and install it from source.
That shouldn't be an issue if you work on your loca...
For data that doesn't fit into memory, spark is often a recommended solution, since it can utilize map-reduce to work with data in a distributed manner. However, setting up local spark development from scratch involves multiple steps, and definitely not for a faint of heart....
Creating scripts are easy. But creating a small docker image is not 😅.
Not all Linux flavors are created equal, some are bigger than others, etc. But this difference is very crucial when it comes to reducing docker image size.
A simple bash script docker image
Given a Docke...
Correction 2023-07-06: I only recently realized SSM and Secrets Manager are not the same.
At my organization we use sops to check in encrypted secrets into git repos. This solves plaintext credentials in version control. However, say, you have 5 repos using the same database...
Subscribe
