Update 2021-11-29 Recently we create more PRs, notice that there are a lot of redundant steps (env setup before triggering checks, etc). Found out you can cache steps in GitHub Actions, so I did some research. Got it working and turns out I reduce at least 60% actions time f...

ecs-cli configure profile \
  --access-key $KEY \
  --secret-key $SECRET \
  --profile-name $PROFILE

### launch mode: fargate
ecs-cli configure \
  --cluster $CLUSTER \
  --default-launch-type FARGATE \
  --config-name $NAME \
  --region ap-southeast-1

ecs-cli up \
  --cluster-config $NAME \
  --vpc $VPCID\
  --subnets $SUBNETID1, $SUBNETID2

### launch mode: ec2
ecs-cli configure \
  --cluster $CLUSTER \
  --region ap-southeast-1 \
  --default-launch-type EC2 \
  --config-name $NAME

ecs-cli up --keypair $KEYPAIR \
  --extra-user-data userData.sh \
  --capability-iam --size 1 \
  --instance-type t2.large \
  --cluster-config $NAME \
  --verbose \
  --force \
  --aws-profile $PROFILE

ecs-cli compose \
  --cluster-config $NAME \
  --file docker-compose.yml up \
  --create-log-groups

Updated: 2025-01-26 Self-hosting is a practice for running and managing websites / services using your own server. Some people do this because they are concerned about their privacy, or some services are free if they host it themselves. Below are instructions for how to do s...

Update 2023-09-09: I now revert back to plain requirements.txt, since pipenv is very poor at resolving large dependencies list. Poetry still remains funky so that's off the table as well. Update 2024-01-20: I've been using poetry in production for a few months now and so far...

We have a few tables where the data size is > 3GB (in parquet, so around 10 GB uncompressed). Loading it into postgres takes an hour. (Most of our tables are pretty small, hence the reason why we don't use columnar database). I want to explore whether there's a faster way...

Big data Spark: Map-reduce framework for dealing with big data, especially for data that doesn't fit into memory. Utilizes parallelization. Cloud AWS: Cloud platform for many tools used in software engineering. AWS Fargate: A task launch mode for ECS task, where it auto...

Repo here Scrapy is a nice framework for web scraping. But like all local development processes, some settings / configs are disabled. This wouldn't pose an issue, but to deploy a scrapy project to zyte (a hosted scrapy platform) you need to run shub deploy, and if you run i...

Elasticsearch is a search engine with built-in analyzers (combination of tokenizer and filters), which makes it easier to set it up and get it running, seeing you don’t have to implement NLP logic from scratch. However, for some languages such as Thai, the built-in Thai anal...

Background: we use spark to read/write to data lake. For dealing with spatial data & analysis, we use sedona. Shapefile is converted to TSV then read by spark for further processing & archival. Recently I had to archive shapefiles in our data lake. It wasn't rosy for...

I have a big pipelines where one step performs crossjoin on 130K x 7K. It fails quite often, and I have to pray to the Rice God for it to pass. Today I found the solution: repartition before crossjoin.

The root cause is that the dataframe with 130K records has 6 partitions, so when I perform crossjoin (one-to-many) it's working against those 6 partitions. Total output in parquet is around 350MB, which means my computer (8 cores, 10GB RAM provisioned for spark) needs to be able to hold all uncompressed data in memory. It couldn't hence the frequent OOM.

So by increasing the partition size from 6 to 24, the current working dataframe size is smaller, which means things could pass along faster while not filling up my machine's RAM.

Ghost CMS is very easy to use, but the deployment overhead (maintaining db, ghost version, updates and etc) might be too much for some. Luckily, there's a way to convert a Ghost site to static pages, which you can later host on Github pages or something similar.

Setup

• static site engine: Hugo
• a Ghost instance

Usage

1. Install https://github.com/Fried-Chicken/ghost-static-site-generator
2. cd to static directory in your Hugo folder
3. run

gssg --domain ${YOUR_GHOST_INSTANCE_URL} --dest posts --url ${YOUR_STATIC_SITE_DOMAIN_WITHOUT_TRAILING_SLASH} --subDir posts

4. Update your hugo config to link to the above folder:

[[menu.main]]
    identifier = "posts"
    name       = "Posts"
    url        = "/posts"

All done! 🎉🎉🎉

Since starting self-hosting back in 2017, I've always used apache2 since it's the first webserver I came across. Over time adding more services and managing separate vhost config is a bit tiresome. Enter Caddy. It's very simple to set up and configure. Some services where I ...

EDIT: see here for Caddy, also easier to set up too. Sometimes you found an interesting project to self-hosted, but it doesn't have password authentication built-in. Luckily, we need to reverse-proxy them anyway and apache2/ nginx / httpd happen to provide password auth with...

Might come as a surprise to some of you, but tea can contain gluten from additives & cross-contamination, in which barley or malt is added for flavorings. Teavana is known for adding such additives in their tea (and I got glutented from it one time). Say, I'm interested ...

If you work with spatial data, chances are you are familiar with shapefile, a file format for viewing / editing spatial data. Essentially, shapefile is just a tabular data like csv, but it does thingamajig with geometry data type, where any gis tools like qgis or arcgis can ...

There's a task where I need to export 4M+ records out of mongodb, total uncompressed size is 17GB+ 26GB export methods mongoexport The recommended way to export is using mongoexport utility, but you have to specify the output attributes, which doesn't work for me because t...

I want to update my pc config to use a more powerful cpu (from Ryzen 1300x --> Ryzen 3600). Watched a few videos and it didn't look complicated enough, so I order the new CPU and plan to install it myself. Things didn't go exactly to plan. To sum up relevant things I didn...

You may have come across news headlines with something like "woman diagnoses herself with rare disease."

If you take this statement at face value, you might think "hmmm so this woman, out of the blue, just diagnose herself with a rare disease."

What most people don't often realize is that for a rare disease to be considered, she must have gone through many different doctors who diagnosed her with various diseases, but she feels that neither of them accurately describe her symptoms. She might do some research, and came across a rare disease that perfectly describe her symptoms.

A more accurate (but more verbose) headline would be "doctors failed to accurately diagnose woman with rare disease".

Takeaway? Think twice before taking anything at face value.

import geopandas as gpd import geoplot as gplt import matplotlib.pyplot as plt import numpy as np import pandas as pd from geoplot import polyplot from pythainlp.tokenize import syllable_tokenize, word_tokenize Data structure name: target region name geometry: spatial col...

Imagine having a dataset that you need to use for training a prediction model, but some of the features are missing. The good news is you don't need to throw some data away, just have to impute them. Below are steps you can take in order to create an imputation pipeline. fro...

2023 version: https://github.com/kahnwong/lyrics-analysis I listen to a lot of music, mostly symphonic heavy metal. What's interesting is that in this genre, each album often has different themes, also each band focus on different topics in terms of lyrics. For instance, Ni...

*ตัวโรมัน = a b c d e f g ... วันดีคืนดีชาวต่างชาติอยากมาเมืองไทย แต่จะมาเพราะอะไรนี่ก็ไม่ขอยุ่ง ไม่ได้สู่รู้เบอร์นั้น อนุมานว่ามาจากแถบยุโรป อเมริกาอะไรเทือกนี้ก็น่าจะเครื่องลงที่สุวรรณภูมิ ปัญหาที่จะเจออย่างแรกเลยก็คือ ชื่อป้ายชื่ออะไรไม่ได้ออกเสียงเหมือนที่ตัวโรมันเขียนไว...

Does it matter if you language has a lot of loanwords? To be considered a loan, it has to be of foreign origin, and speakers from the borrowing end can understand and use in various contexts. It's perfectly okay for the meaning to shift on the borrowing end, as semantic shif...

I read Harry Potter as a kid, and I couldn't shake the feeling that one character's name is a bit off. I couldn't quite pinpoint exactly why. Turns out, my hunch was right – the Thai translator used the wrong transliteration. In Thai, "Parvati Patil" is "ปาราวตี พาติล." This...

You often hear people say "you need to know x words in y language to be able to understand basic conversations.". This is a bit misleading, since the definition of "word" is varied. For example, in Chinese (yes, I know, Chinese as a language doesn't exist, but I digress) one...

One of my friends is a Syrian refugee, who was granted asylum in Sweden last year. I also want to try data analysis, so it fits that I should analyze something that's relevant to my friend. This is my first ever analysis in pandas, apologies for code abomination in advance. ...

More often than not, Thai loanwords are of Indic, Khmer, English or Chinese origin. Although loans from other languages do exist. For instance, องุ่น (grape). This word is of Persian origin (read: Iran), which is انگور, literally angur. In Thai it's angun. Notice that it's n...

You decided to visit Thailand for whatever reason I don't want to know. (It's not my business.) Assuming you're in from outside of Asia, you'll land at Suvarnabhumi Airport. The problem? The locals don't pronounce it that way. All the names and signs here are romanized using...