Sign up

Karn Wong

Not verified No WebSub updates No webmention support Invalid

Homepage and blog by Karn Wong

Generator
Zola
Public lists
I ♥ RSS
Fetched

Karn Wong
Text • Unknown

Load balancer 101, thou shalt not be alone

Scaling, the dreaded word among developers, because this means more complexity. But why do we need scaling? Imagine a super busy corner store. During early mornings, there might not be a lot of customers, so one cashier might be enough to handle all customers. But during aft...

Karn Wong
Text • Unknown

Cross-platform package (+env) management with Nix

For many years, installing a package on linux means either: Compiling a binary from source, then install it. -> I think we know why this didn't catch on for the mass. Downloading a compiled binary for your system's architecture and platform. -> This requires you to al...

Karn Wong
Text • Unknown

Load credentials into your shell via Bitwarden CLI - Fish edition

Recently I work with GitHub CLI a lot, and having to constantly fire up Bitwarden app to retrieve GITHUB_TOKEN gets old real fast... I was thinking of storing it in a gist in a password manager, luckily someone had the same idea and implemented it. The only issue is that I u...

Karn Wong
Text • Unknown

Cloudflare DNS management with Terraform

I self hosted a lot of services, sometimes I try out a few apps that would get deleted within the same day. All this requires setting up CNAME for reverse-proxy (because I want to make sure there's no funny reverse-proxy shenanigans going on, for future reference). I can alw...

Karn Wong
Text • Unknown

Deploy more efficiently with templating

You are building a website, it's a simple frontend that needs to call the database for [total lead drops this week]. Your website is still at an infancy stage, with only a few features. At this point, you contemplate whether you need a proper backend or not. But to deploy a ...

Karn Wong
Text • Unknown

IaC is cool, until someone specified the wrong machine size 💸

Back in the day, there was no cloud. If you want a lot of computing power, you need to build your own data center, and this is very expensive. Then cloud happened, and suddenly you can work with a lot of flexibility like you couldn't before. Want to try out a small deploymen...

Karn Wong
Text • Unknown

Deploy static site with branch preview via Cloudflare Pages

Updated 2023-02-20: update terraform code For frontends, if no server-side rendering is required, we can deploy it as a static site. If you already use GitHub, you might be familiar with GitHub Pages. One common use case is to deploy your personal landing page / blog via Git...

Karn Wong
Text • Unknown

Terraform with ECS task on EC2 backend

Previously I wrote about setting up ECS task on fargate backend. But we can also use EC2 as backend too, in some cases where the workload is consistent, ie scaling is not required, since EC2 would be cheaper than fargate backend, even more so if you have reserved instance on...

Karn Wong
Text • Unknown

Intro to Dagster Cloud

Imagine you have a few data pipelines to schedule. Simplest solution would be cronjob. Time goes by and next thing you know, you have around 50 pipelines to manage. The fun starts when you have to hunt down which pipeline doesn't run normally. And by then it would be super h...

Karn Wong
Text • Unknown

Intro to Pulumi

For IaC, no doubt that Terraform is the leader. But there are other alternatives too, one of them is Pulumi. Currently Pulumi provides fun challenges to get started with their services. Best of all, they give you swags too! We are going to create a simple Pulumi project for ...

Karn Wong
Text • Unknown

Data engineer archetypes

I have been working in the data industry since almost half a decade ago. Over time I have noticed so-called archetypes within various data engineering roles. Below are main skills and combinations I have seen over the years. This is by no means an exhaustive list, rather wha...

Karn Wong
Text • Unknown

Minimal ECS task with Fargate backend

To deploy a web application, there are many ways to go about it. I could spin up a bare VM and set up the environment manually. To make things easier, I could have package the app into docker image. But this still means I have to "update" the app manually if I add changes to...

Karn Wong
Text • Unknown

What SQL can't do for data engineering

I often hear people ask "if you can do data engineering with SQL, then what's the point of learning spark or python?" Data ingestion Let's circle back at bit. I think we all can agree that: there's a point in time where there's no data in the data warehouse (which DE-who-us...

Karn Wong
Text • Unknown

Use SSH key during Docker Build without embedding the key via ssh-agent

Imagine working in a company, and they have a super cool internal module! The module works great, except that it is a private module, which means you need to install it by cloning the source repo and install it from source. That shouldn't be an issue if you work on your loca...

Karn Wong
Text • Unknown

Use Pyspark locally with Docker

For data that doesn't fit into memory, spark is often a recommended solution, since it can utilize map-reduce to work with data in a distributed manner. However, setting up local spark development from scratch involves multiple steps, and definitely not for a faint of heart....

Karn Wong
Text • Unknown

Reduce Docker image size with Alpine

Creating scripts are easy. But creating a small docker image is not 😅. Not all Linux flavors are created equal, some are bigger than others, etc. But this difference is very crucial when it comes to reducing docker image size. A simple bash script docker image Given a Docke...

Karn Wong
Text • Unknown

Secrets management with SOPS, AWS Secrets Manager and Terraform

Correction 2023-07-06: I only recently realized SSM and Secrets Manager are not the same. At my organization we use sops to check in encrypted secrets into git repos. This solves plaintext credentials in version control. However, say, you have 5 repos using the same database...

Karn Wong
Text • Unknown

Run GitHub Actions faster with cache for Pipenv and Docker build

Update 2021-11-29 Recently we create more PRs, notice that there are a lot of redundant steps (env setup before triggering checks, etc). Found out you can cache steps in GitHub Actions, so I did some research. Got it working and turns out I reduce at least 60% actions time f...

Karn Wong
Text • Unknown

ecs-cli snippets 

ecs-cli configure profile \
  --access-key $KEY \
  --secret-key $SECRET \
  --profile-name $PROFILE

### launch mode: fargate
ecs-cli configure \
  --cluster $CLUSTER \
  --default-launch-type FARGATE \
  --config-name $NAME \
  --region ap-southeast-1

ecs-cli up \
  --cluster-config $NAME \
  --vpc $VPCID\
  --subnets $SUBNETID1, $SUBNETID2

### launch mode: ec2
ecs-cli configure \
  --cluster $CLUSTER \
  --region ap-southeast-1 \
  --default-launch-type EC2 \
  --config-name $NAME

ecs-cli up --keypair $KEYPAIR \
  --extra-user-data userData.sh \
  --capability-iam --size 1 \
  --instance-type t2.large \
  --cluster-config $NAME \
  --verbose \
  --force \
  --aws-profile $PROFILE

ecs-cli compose \
  --cluster-config $NAME \
  --file docker-compose.yml up \
  --create-log-groups

Karn Wong
Text • Unknown

Self-hosting primer

Updated: 2025-01-26 Self-hosting is a practice for running and managing websites / services using your own server. Some people do this because they are concerned about their privacy, or some services are free if they host it themselves. Below are instructions for how to do s...

Karn Wong
Text • Unknown

Python venv management

Update 2023-09-09: I now revert back to plain requirements.txt, since pipenv is very poor at resolving large dependencies list. Poetry still remains funky so that's off the table as well. Update 2024-01-20: I've been using poetry in production for a few months now and so far...

Karn Wong
Text • Unknown

Don't write large table to Postgres with Pandas

We have a few tables where the data size is > 3GB (in parquet, so around 10 GB uncompressed). Loading it into postgres takes an hour. (Most of our tables are pretty small, hence the reason why we don't use columnar database). I want to explore whether there's a faster way...

Karn Wong
Text • Unknown

Data engineering toolset (that I use) glossary

Big data Spark: Map-reduce framework for dealing with big data, especially for data that doesn't fit into memory. Utilizes parallelization. Cloud AWS: Cloud platform for many tools used in software engineering. AWS Fargate: A task launch mode for ECS task, where it auto...

Karn Wong
Text • Unknown

Automatic Scrapy deployment with GitHub actions

Repo here Scrapy is a nice framework for web scraping. But like all local development processes, some settings / configs are disabled. This wouldn't pose an issue, but to deploy a scrapy project to zyte (a hosted scrapy platform) you need to run shub deploy, and if you run i...

Karn Wong
Text • Unknown

Elasticsearch with custom dictionary

Elasticsearch is a search engine with built-in analyzers (combination of tokenizer and filters), which makes it easier to set it up and get it running, seeing you don’t have to implement NLP logic from scratch. However, for some languages such as Thai, the built-in Thai anal...

Karn Wong
Text • Unknown

Shapefile to data lake

Background: we use spark to read/write to data lake. For dealing with spatial data & analysis, we use sedona. Shapefile is converted to TSV then read by spark for further processing & archival. Recently I had to archive shapefiles in our data lake. It wasn't rosy for...

Karn Wong
Text • Unknown

Spark join OOM fix

I have a big pipelines where one step performs crossjoin on 130K x 7K. It fails quite often, and I have to pray to the Rice God for it to pass. Today I found the solution: repartition before crossjoin.

The root cause is that the dataframe with 130K records has 6 partitions, so when I perform crossjoin (one-to-many) it's working against those 6 partitions. Total output in parquet is around 350MB, which means my computer (8 cores, 10GB RAM provisioned for spark) needs to be able to hold all uncompressed data in memory. It couldn't hence the frequent OOM.

So by increasing the partition size from 6 to 24, the current working dataframe size is smaller, which means things could pass along faster while not filling up my machine's RAM.

Karn Wong
Text • Unknown

Add Ghost content to Hugo

Ghost CMS is very easy to use, but the deployment overhead (maintaining db, ghost version, updates and etc) might be too much for some. Luckily, there's a way to convert a Ghost site to static pages, which you can later host on Github pages or something similar.

Setup

  • static site engine: Hugo
  • a Ghost instance

Usage

  1. Install https://github.com/Fried-Chicken/ghost-static-site-generator
  2. cd to static directory in your Hugo folder
  3. run
gssg --domain ${YOUR_GHOST_INSTANCE_URL} --dest posts --url ${YOUR_STATIC_SITE_DOMAIN_WITHOUT_TRAILING_SLASH} --subDir posts
  1. Update your hugo config to link to the above folder:
[[menu.main]]
    identifier = "posts"
    name       = "Posts"
    url        = "/posts"

All done! 🎉🎉🎉

Karn Wong
Text • Unknown

Hello Caddy

Since starting self-hosting back in 2017, I've always used apache2 since it's the first webserver I came across. Over time adding more services and managing separate vhost config is a bit tiresome. Enter Caddy. It's very simple to set up and configure. Some services where I ...

Karn Wong
Text • Unknown

Password auth with apache2 reverse-proxy

EDIT: see here for Caddy, also easier to set up too. Sometimes you found an interesting project to self-hosted, but it doesn't have password authentication built-in. Luckily, we need to reverse-proxy them anyway and apache2/ nginx / httpd happen to provide password auth with...