If we are talking about IaC, Terraform would be on the list. It made IaC popular and help a lot of companies maintain infrastructure at scale. Especially when you have multiple sets of infrastructures to maintain, Terraform can help you reduce the setup time tremendously via...
Correction 2024-09-21: after using multi-stage build, Node image size dropped from 1.1GB to 130MB.
The first programming language I achieved proficiency was Python, so for the longest time I've been using it to do most
stuff. Last year I picked up Go, and I had a blast with...
These days there are multiple ways to deploy a workload, be it cloud-based or bare-metal. For cloud, depending on whether you are using PaaS or IaaS, backup options can vary.
Why do we need to backup? Because your workloads can contain a state, this can be stored as local fi...
Streamlit is well-loved by many people, especially among data folks due to the fact that it does not require prior web programming knowledge to get started.
Popular use cases for streamlit can be anything from a quick machine learning application poc or internal dashboards. ...
When it comes to IaC (infrastructure as code), most people might have heard of HashiCorp's Terraform (it uses HCL as DSL. Interestingly enough, Terraform also has its own CDK to translate programming languages into HCL), Pulumi or AWS CDK. The latter two support programming ...
If you look online, many sources would tell you that you should use service account to authenticate for GCP services. While this is true, it's not for all the cases.
For local development, you should use Application Default Credentials
Imagine working in a team, and you hav...
For a summarization task, there should be an input, in which it's reduced to a handful of paragraphs. This input is in text format.
You don't necessarily start from a text format though, since the source content can be audio or video files. But this means at the end, the sou...
For big data processing, spark is still king. Over the years, many improvements have been made to improve spark performance. Databricks themselves created photon, a spark engine that can accelerate spark queries, but this is proprietary to Databricks.
Other alternatives do e...
Python package management is not straightforward, seeing default package manager (pip) does not behave like node's npm, in a sense that it doesn't track dependencies versions.
This is why you should use poetry to manage python packages, since it creates a lock file, so you c...
Previously, I talked about dataframe performance, but this doesn't include writing data to destination part.
At a large scale, big data means you need to use spark for data processing (unless you prefer SQL, in which this post is irrelevant). But not many orgs need big data,...
A minimal application architecture would compose of a database, and an application backend. Serverless database is still in its infancy, but thankfully container-based runtime is very much alive and doing well. On GCP, a serverless container-based runtime do exist, known as ...
Back in 2017-2018, everyone wanted to be a data scientist. Then reality hits, that they need a data engineer for a successful machine learning project. Things didn't end there, since they also need a machine learning engineer to create production-ready code.
Some people thin...
Many data science teams are struggling with implementing end-to-end machine learning projects. It's a very common phenomenon, so if you are experiencing this, you are not alone. Having worked in every stage of data science project lifecycle, in addition to normal web service...
Over the years I've heard stories of data teams struggling with deploying machine learning models to production. Clearly
there is a pattern here. This article is my reflection on the matter.
So what's the problem?
Data scientists, by definition, create mathematical models f...
There are multiple reasons why someone wants to set up a postgres locally. Either for learning SQL or as an application's backend. Over the years I see people struggle with using postgres locally, so here are common use cases and possible issues, with solutions for each.
For...
For our production workload, we deploy the workloads on Kubernetes, in which an ingress resource is created per each deployment. Resources in ingress are GCP Load Balancer and SSL Certificate. As for DNS, we use Cloudflare since it enables CDN without extra configurations on our part.
A few months after the deployment went live initially, we were informed that the website couldn't be accessed. Turns out GCP couldn't renew the SSL Certificate (error FAILED_NOT_VISIBLE.) Looking at GCP docs, turns out if the DNS couldn't be resolved to the Load Balancer IP, it couldn't provision/renew a certificate.
The fix? Disable proxy on Cloudflare records and wait until GCP successfully renews the certs.
For a machine learning project, usually it is divided into two main categories: research and production. For research ML project, the model would be created and used locally on a researcher's machine. For a production ML project, a deployment would be involved. Usual pattern...
When we face with engineering problems, it's too easy to fall into the trap thinking it should be solved with a technical solution. Seasoned engineers think differently, because they realize that most of the time, it's "people" or "workflow" problems. Let me provide a few ex...
Updated 2024-02-21 - update iceberg catalog ref url
Every business generates data, some very little, some do generate ginormous amount of data. If you are familiar with the basic web application architecture, there are data, application and web tier. But it doesn't end there...
Background
Think of websites you visit each day. Most likely they are hosted on a cloud provider such as AWS, GCP, Azure. The good news is it's very easy to create a simple deployment with a virtual machine, but for scalable and high-availability workloads, usual recommenda...
Background
For data processing tasks, there are different ways you can go about it:
using SQL to leverage a database engine to perform data transformation
dataframe-based frameworks such as pandas, ray, dask, polars
big data processing frameworks such as spark
Check out t...
Note: if you've seen the list elsewhere, it was probably me. I first posted this list on Data Engineering Discord and Data Engineer Cafe.
Books
Data fundamentals (good entrypoint)
Fundamentals of Data Engineering - Joe Reis & Matt Housley
Seven Databases in Seven Week...
Imagine going to your client's site to deploy a software. During the deployment process, you notice that the speed is atrociously slow. You have a suspicion that your client's network bandwidth is the issue.
To test this theory, you go to a speedtest website and run a test. ...
Some of you might be familiar with geoserver for serving spatial data as consumable WMS/WFS layers. The issue is that as far as I know, you have to manually manage assets upload and specifying styles manually. Also the tool is a bit dated.
One modern alternative is QGIS serv...
Sometimes you'll have to grant other people (or entities) access to your Kubernetes cluster. Easiest is you can give them your admin credentials, but this is similar to giving your house key to a friend, when they only need access to your living room.
You can give them diffe...
In certain cases, you can't do a full postgres replication to another instance, or you prefer a fine-grained control on what to replicate, pglogical is one way to achieve partial replication, albeit this requires more manual setup.
Below are steps I used to do a pglogical re...
People will do almost anything to not feel pain, including causing pain and abusing power.
Very few people can handle being held accountable without rationalizing, blaming, or shutting down.
Without understanding how our feelings, thoughts, and behaviors work together, it’s almost impossible to find our way back to ourselves and each other. When we don’t understand how our emotions shape our thoughts and decisions, we become disembodied from our own experiences and disconnected from each other.
Asshole assholes: They suck at work and everything else. These are the mean, jealous, insecure jerks who you’d avoid at a party, but who inevitably sit immediately next to you at the office. They cannot deliver, are deeply unproductive, so they do everything possible to defl...
Successfully deploying services are not the end, maintenance is coming to town! When you want to see how your system works, usually people rely on SaaS like Datadog or New Relic to do the heavy lifting. Also a lot of $$$ is required.
Also with SaaS like these, usually you ha...
It isn’t the changes themselves that the people in these cases resist. It’s the losses and endings that they have experienced and the transition that they are resisting.
A large university reassigned one of its vice presidents to a far less important area than the one he had...
Subscribe
