Sign up

Karn Wong

Not verified No WebSub updates No webmention support Not yet validated

Homepage and blog by Karn Wong

Generator
Zola
Public lists
I ♥ RSS
Fetched

Karn Wong
• Unknown

Cloudflare DNS management with Terraform

I self hosted a lot of services, sometimes I try out a few apps that would get deleted within the same day. All this requires setting up CNAME for reverse-proxy (because I want to make sure there's no funny reverse-proxy shenanigans going on, for future reference). I can alw...

Karn Wong
• Unknown

Deploy more efficiently with templating

You are building a website, it's a simple frontend that needs to call the database for [total lead drops this week]. Your website is still at an infancy stage, with only a few features. At this point, you contemplate whether you need a proper backend or not. But to deploy a ...

Karn Wong
• Unknown

IaC is cool, until someone specified the wrong machine size 💸

Back in the day, there was no cloud. If you want a lot of computing power, you need to build your own data center, and this is very expensive. Then cloud happened, and suddenly you can work with a lot of flexibility like you couldn't before. Want to try out a small deploymen...

Karn Wong
• Unknown

Deploy static site with branch preview via Cloudflare Pages

Updated 2023-02-20: update terraform code For frontends, if no server-side rendering is required, we can deploy it as a static site. If you already use GitHub, you might be familiar with GitHub Pages. One common use case is to deploy your personal landing page / blog via Git...

Karn Wong
• Unknown

Terraform with ECS task on EC2 backend

Previously I wrote about setting up ECS task on fargate backend. But we can also use EC2 as backend too, in some cases where the workload is consistent, ie scaling is not required, since EC2 would be cheaper than fargate backend, even more so if you have reserved instance on...

Karn Wong
• Unknown

Intro to Dagster Cloud

Imagine you have a few data pipelines to schedule. Simplest solution would be cronjob. Time goes by and next thing you know, you have around 50 pipelines to manage. The fun starts when you have to hunt down which pipeline doesn't run normally. And by then it would be super h...

Karn Wong
• Unknown

Intro to Pulumi

For IaC, no doubt that Terraform is the leader. But there are other alternatives too, one of them is Pulumi. Currently Pulumi provides fun challenges to get started with their services. Best of all, they give you swags too! We are going to create a simple Pulumi project for ...

Karn Wong
• Unknown

Data engineer archetypes

I have been working in the data industry since almost half a decade ago. Over time I have noticed so-called archetypes within various data engineering roles. Below are main skills and combinations I have seen over the years. This is by no means an exhaustive list, rather wha...

Karn Wong
• Unknown

Minimal ECS task with Fargate backend

To deploy a web application, there are many ways to go about it. I could spin up a bare VM and set up the environment manually. To make things easier, I could have package the app into docker image. But this still means I have to "update" the app manually if I add changes to...

Karn Wong
• Unknown

What SQL can't do for data engineering

I often hear people ask "if you can do data engineering with SQL, then what's the point of learning spark or python?" Data ingestion Let's circle back at bit. I think we all can agree that: there's a point in time where there's no data in the data warehouse (which DE-who-us...

Karn Wong
• Unknown

Use SSH key during Docker Build without embedding the key via ssh-agent

Imagine working in a company, and they have a super cool internal module! The module works great, except that it is a private module, which means you need to install it by cloning the source repo and install it from source. That shouldn't be an issue if you work on your loca...

Karn Wong
• Unknown

Use Pyspark locally with Docker

For data that doesn't fit into memory, spark is often a recommended solution, since it can utilize map-reduce to work with data in a distributed manner. However, setting up local spark development from scratch involves multiple steps, and definitely not for a faint of heart....

Karn Wong
• Unknown

Reduce Docker image size with Alpine

Creating scripts are easy. But creating a small docker image is not 😅. Not all Linux flavors are created equal, some are bigger than others, etc. But this difference is very crucial when it comes to reducing docker image size. A simple bash script docker image Given a Docke...

Karn Wong
• Unknown

Secrets management with SOPS, AWS Secrets Manager and Terraform

Correction 2023-07-06: I only recently realized SSM and Secrets Manager are not the same. At my organization we use sops to check in encrypted secrets into git repos. This solves plaintext credentials in version control. However, say, you have 5 repos using the same database...

Karn Wong
• Unknown

Run GitHub Actions faster with cache for Pipenv and Docker build

Update 2021-11-29 Recently we create more PRs, notice that there are a lot of redundant steps (env setup before triggering checks, etc). Found out you can cache steps in GitHub Actions, so I did some research. Got it working and turns out I reduce at least 60% actions time f...

Karn Wong
• Unknown

ecs-cli snippets 

ecs-cli configure profile \
  --access-key $KEY \
  --secret-key $SECRET \
  --profile-name $PROFILE

### launch mode: fargate
ecs-cli configure \
  --cluster $CLUSTER \
  --default-launch-type FARGATE \
  --config-name $NAME \
  --region ap-southeast-1

ecs-cli up \
  --cluster-config $NAME \
  --vpc $VPCID\
  --subnets $SUBNETID1, $SUBNETID2

### launch mode: ec2
ecs-cli configure \
  --cluster $CLUSTER \
  --region ap-southeast-1 \
  --default-launch-type EC2 \
  --config-name $NAME

ecs-cli up --keypair $KEYPAIR \
  --extra-user-data userData.sh \
  --capability-iam --size 1 \
  --instance-type t2.large \
  --cluster-config $NAME \
  --verbose \
  --force \
  --aws-profile $PROFILE

ecs-cli compose \
  --cluster-config $NAME \
  --file docker-compose.yml up \
  --create-log-groups

Karn Wong
• Unknown

Self-hosting primer

Updated: 2025-01-26 Self-hosting is a practice for running and managing websites / services using your own server. Some people do this because they are concerned about their privacy, or some services are free if they host it themselves. Below are instructions for how to do s...

Karn Wong
• Unknown

Python venv management

Update 2023-09-09: I now revert back to plain requirements.txt, since pipenv is very poor at resolving large dependencies list. Poetry still remains funky so that's off the table as well. Update 2024-01-20: I've been using poetry in production for a few months now and so far...

Karn Wong
• Unknown

Don't write large table to Postgres with Pandas

We have a few tables where the data size is > 3GB (in parquet, so around 10 GB uncompressed). Loading it into postgres takes an hour. (Most of our tables are pretty small, hence the reason why we don't use columnar database). I want to explore whether there's a faster way...

Karn Wong
• Unknown

Data engineering toolset (that I use) glossary

Big data Spark: Map-reduce framework for dealing with big data, especially for data that doesn't fit into memory. Utilizes parallelization. Cloud AWS: Cloud platform for many tools used in software engineering. AWS Fargate: A task launch mode for ECS task, where it auto...

Karn Wong
• Unknown

Automatic Scrapy deployment with GitHub actions

Repo here Scrapy is a nice framework for web scraping. But like all local development processes, some settings / configs are disabled. This wouldn't pose an issue, but to deploy a scrapy project to zyte (a hosted scrapy platform) you need to run shub deploy, and if you run i...

Karn Wong
• Unknown

Elasticsearch with custom dictionary

Elasticsearch is a search engine with built-in analyzers (combination of tokenizer and filters), which makes it easier to set it up and get it running, seeing you don’t have to implement NLP logic from scratch. However, for some languages such as Thai, the built-in Thai anal...

Karn Wong
• Unknown

Shapefile to data lake

Background: we use spark to read/write to data lake. For dealing with spatial data & analysis, we use sedona. Shapefile is converted to TSV then read by spark for further processing & archival. Recently I had to archive shapefiles in our data lake. It wasn't rosy for...

Karn Wong
• Unknown

Spark join OOM fix

I have a big pipelines where one step performs crossjoin on 130K x 7K. It fails quite often, and I have to pray to the Rice God for it to pass. Today I found the solution: repartition before crossjoin.

The root cause is that the dataframe with 130K records has 6 partitions, so when I perform crossjoin (one-to-many) it's working against those 6 partitions. Total output in parquet is around 350MB, which means my computer (8 cores, 10GB RAM provisioned for spark) needs to be able to hold all uncompressed data in memory. It couldn't hence the frequent OOM.

So by increasing the partition size from 6 to 24, the current working dataframe size is smaller, which means things could pass along faster while not filling up my machine's RAM.

Karn Wong
• Unknown

Add Ghost content to Hugo

Ghost CMS is very easy to use, but the deployment overhead (maintaining db, ghost version, updates and etc) might be too much for some. Luckily, there's a way to convert a Ghost site to static pages, which you can later host on Github pages or something similar.

Setup

  • static site engine: Hugo
  • a Ghost instance

Usage

  1. Install https://github.com/Fried-Chicken/ghost-static-site-generator
  2. cd to static directory in your Hugo folder
  3. run
gssg --domain ${YOUR_GHOST_INSTANCE_URL} --dest posts --url ${YOUR_STATIC_SITE_DOMAIN_WITHOUT_TRAILING_SLASH} --subDir posts
  1. Update your hugo config to link to the above folder:
[[menu.main]]
    identifier = "posts"
    name       = "Posts"
    url        = "/posts"

All done! 🎉🎉🎉

Karn Wong
• Unknown

Hello Caddy

Since starting self-hosting back in 2017, I've always used apache2 since it's the first webserver I came across. Over time adding more services and managing separate vhost config is a bit tiresome. Enter Caddy. It's very simple to set up and configure. Some services where I ...

Karn Wong
• Unknown

Password auth with apache2 reverse-proxy

EDIT: see here for Caddy, also easier to set up too. Sometimes you found an interesting project to self-hosted, but it doesn't have password authentication built-in. Luckily, we need to reverse-proxy them anyway and apache2/ nginx / httpd happen to provide password auth with...

Karn Wong
• Unknown

Buying tea when you have Celiac

Might come as a surprise to some of you, but tea can contain gluten from additives & cross-contamination, in which barley or malt is added for flavorings. Teavana is known for adding such additives in their tea (and I got glutented from it one time). Say, I'm interested ...

Karn Wong
• Unknown

Workarounds for archiving large shapefile in data lake

If you work with spatial data, chances are you are familiar with shapefile, a file format for viewing / editing spatial data. Essentially, shapefile is just a tabular data like csv, but it does thingamajig with geometry data type, where any gis tools like qgis or arcgis can ...

Karn Wong
• Unknown

Mongodb export woes

There's a task where I need to export 4M+ records out of mongodb, total uncompressed size is 17GB+ 26GB export methods mongoexport The recommended way to export is using mongoexport utility, but you have to specify the output attributes, which doesn't work for me because t...