For a machine learning project, usually it is divided into two main categories: research and production. For research ML project, the model would be created and used locally on a researcher's machine. For a production ML project, a deployment would be involved. Usual pattern...
When we face with engineering problems, it's too easy to fall into the trap thinking it should be solved with a technical solution. Seasoned engineers think differently, because they realize that most of the time, it's "people" or "workflow" problems. Let me provide a few ex...
Updated 2024-02-21 - update iceberg catalog ref url
Every business generates data, some very little, some do generate ginormous amount of data. If you are familiar with the basic web application architecture, there are data, application and web tier. But it doesn't end there...
Background
Think of websites you visit each day. Most likely they are hosted on a cloud provider such as AWS, GCP, Azure. The good news is it's very easy to create a simple deployment with a virtual machine, but for scalable and high-availability workloads, usual recommenda...
Background
For data processing tasks, there are different ways you can go about it:
using SQL to leverage a database engine to perform data transformation
dataframe-based frameworks such as pandas, ray, dask, polars
big data processing frameworks such as spark
Check out t...
Note: if you've seen the list elsewhere, it was probably me. I first posted this list on Data Engineering Discord and Data Engineer Cafe.
Books
Data fundamentals (good entrypoint)
Fundamentals of Data Engineering - Joe Reis & Matt Housley
Seven Databases in Seven Week...
Imagine going to your client's site to deploy a software. During the deployment process, you notice that the speed is atrociously slow. You have a suspicion that your client's network bandwidth is the issue.
To test this theory, you go to a speedtest website and run a test. ...
Some of you might be familiar with geoserver for serving spatial data as consumable WMS/WFS layers. The issue is that as far as I know, you have to manually manage assets upload and specifying styles manually. Also the tool is a bit dated.
One modern alternative is QGIS serv...
Sometimes you'll have to grant other people (or entities) access to your Kubernetes cluster. Easiest is you can give them your admin credentials, but this is similar to giving your house key to a friend, when they only need access to your living room.
You can give them diffe...
In certain cases, you can't do a full postgres replication to another instance, or you prefer a fine-grained control on what to replicate, pglogical is one way to achieve partial replication, albeit this requires more manual setup.
Below are steps I used to do a pglogical re...
People will do almost anything to not feel pain, including causing pain and abusing power.
Very few people can handle being held accountable without rationalizing, blaming, or shutting down.
Without understanding how our feelings, thoughts, and behaviors work together, it’s almost impossible to find our way back to ourselves and each other. When we don’t understand how our emotions shape our thoughts and decisions, we become disembodied from our own experiences and disconnected from each other.
Asshole assholes: They suck at work and everything else. These are the mean, jealous, insecure jerks who you’d avoid at a party, but who inevitably sit immediately next to you at the office. They cannot deliver, are deeply unproductive, so they do everything possible to defl...
Successfully deploying services are not the end, maintenance is coming to town! When you want to see how your system works, usually people rely on SaaS like Datadog or New Relic to do the heavy lifting. Also a lot of $$$ is required.
Also with SaaS like these, usually you ha...
It isn’t the changes themselves that the people in these cases resist. It’s the losses and endings that they have experienced and the transition that they are resisting.
A large university reassigned one of its vice presidents to a far less important area than the one he had...
CSV as a file format is very versatile, almost any programs can parse it. The only issue is you can't use SQL against CSV files directly.
This is a major pain point, since using SQL is so much faster than firing up a jupyter notebook and wrangle the data in python, or use Ex...
Platform engineering is all the rage these days. Often, you'll often hear this term with the keyword DevX. How are they related?
Imagine you are working on a microservice backend. You are just starting out, so you don't have much features to work on yet. But as a PoC, you on...
Imagine having multiple services running, each has its own logs. Most people don't read them, and they shouldn't, because services emit a lot of logs! But we need them, because it's the only way to diagnose and troubleshoot system errors. But you might say "my service is not...
I think everyone who has worked with data, in any role or function, used pandas 🐼 at certain point. I first used pandas in 2017, so it's 6 years already. Things have come a long way, and so is data size I'm working with!
Pandas has its own issues, namely no native support fo...
Correction 2023-07-02: fix homelab specs and corresponding AWS EC2 instance class (it's actually 32GB RAM, not 64GB)
Congratulations, you managed to successfully deployed a few services on kubernetes! But this is not the end 👀. Unfortunately money doesn't grow on trees, and ...
Kubernetes is awesome, I think this is obvious if you have more than a handful of services to manage. If you use cloud, either VM or container-based runtime, it would provide you a dashboard to see the metrics. But what about kubernetes? Since you would have multiple service...
Background
Ever wonder how websites (and everything in between) work? Chances are you can create a project running on your local machine. It works as you expected, but to let other people access it, you have to "deploy" it. For many years, to support a lot of request volume...
Background
You want to track who access your site, Google Analytics can do that. To see the data, you can use Google Analytics dashboard, the default settings is good enough for most use cases. But what if you have a lot of tracking data, and you want to streamline a way to...
Raise your hands if you normally have to send .env files to your team members so they can start a project in dev environment.
While there is nothing wrong with this approach, it could introduce a lot of security risks, namely sharing secrets via plaintext protocol. Sure, you...
For most people, using SQL to transform data is a no-brainer, seeing it's a very versatile language, and doesn't have quite a steep learning curve compared to python. There are some cases where SQL is more suitable for a task, but the reverse can also happen as well.
For ins...
I love not having to manage databases. Hosting it on your compute is guaranteed to be cheaper, but I don't want to be constantly worrying about backups and database upgrade / maintenance.
AWS offers managed databases, known as RDS. Mostly I use postgres, which works well for...
Organizational health lies in being able to work together.
As a manager, our job is to try as much as we can to drive balance and clarity.
A happy, driven team sees the wider purpose of their work and also feels empowered to execute tasks individually.
“You can’t call yourse...
Opportunity cost
Staff engineers’ skills are usually in high demand. If you’re assigned to a single team, you may not be top of mind for solving a problem elsewhere in the org, or your manager may be unwilling to let you go.
Losing empathy
It’s easy to overfocus and forget...
Scaling, the dreaded word among developers, because this means more complexity. But why do we need scaling?
Imagine a super busy corner store. During early mornings, there might not be a lot of customers, so one cashier might be enough to handle all customers. But during aft...
For many years, installing a package on linux means either:
Compiling a binary from source, then install it. -> I think we know why this didn't catch on for the mass.
Downloading a compiled binary for your system's architecture and platform. -> This requires you to al...
Recently I work with GitHub CLI a lot, and having to constantly fire up Bitwarden app to retrieve GITHUB_TOKEN gets old real fast...
I was thinking of storing it in a gist in a password manager, luckily someone had the same idea and implemented it. The only issue is that I u...
Subscribe
