vladkens.me

Wanted a Dictation App, Got Three Libraries

2026-11-11T00:00:00+00:00

I swapped my M2 MacBook with the old design for a new fanless M5 Air. A new computer feels like a clean start. So instead of dragging all my old apps and settings along, I went looking for lighter and more native alternatives. For example, I replaced VS Code with Zed and Chrome with Helium.

To talk to agents, I used a dictation app called Handy. But I really didn't like that it's built on Tauri, and I wanted something native. I asked an agent to find an alternative, but nothing it found worked for me. So I decided to write my own.

People on X are rewriting everything in GPUI Kit right now, so I decided to try it too. The author of Handy had also moved the speech recognition engine into a separate project called transcribe.cpp. So the task looked easy, especially with agents.

Week 1. Picking a GUI and prototypes

So I took GPUI Kit, the one everyone on X was hyping, and quickly built the first version of the UI with agents. It came out pretty well: the app with an open window used about 80 MB of RAM. But compiling was painfully slow. Part of it was GPUI Kit itself, and part was linking transcribe.cpp, which took about a minute on every rebuild.

I also didn't like that GPUI Kit pulls in over 400 dependencies, just like Node.js. Who knows what malware might be hiding in there? Besides, GPUI from Zed isn't a finished library with stable releases. So GPUI Kit depends on a dev version from the Zed repo, and the API can change at any moment. Another problem is integration with the system, like a tray icon or a native context menu. GPUI Kit doesn't have it, so all of that has to be written by hand.

I decided to look at other options in the Rust ecosystem. Back in the pre-agent era, I played with egui for UIs, and it left a good impression. It really was faster and lighter on memory, and it pulled in fewer crates. But all the native integrations still have to be written by hand, and the components need a lot of customizing to look nice.

I also tried native controls: Rust bindings to the system UI libraries (AppKit on macOS, Win32 on Windows and GTK on Linux). The idea came from another project I saw on X, mygo, which builds native desktop UIs in Go. But in practice I had to write a huge pile of integration code and repeat it at least three times. The app looked native on each platform, but so different that some features couldn't be done properly. And that's for a dictation app, which is literally a settings menu, a list of models and a history.

The most annoying thing the whole time was rebuilding: every launch meant waiting almost a minute for transcribe.cpp to link. And I didn't really get why there should be C++ in my Rust toolchain at all. So I decided to look at what's going on inside.

Week 2. My own speech recognition engine

Inside transcribe.cpp there was ggml, which has separate code for each GPU backend (Metal, Vulkan and CUDA). Each one needs its own tools to build. Rust has wgpu, where shaders are written once in WGSL and run on any GPU. And everything builds with a plain cargo build.

I got curious whether I could run at least one model on WGSL, though I had no plans to rewrite anything yet. I took GigaAM v3, which I use most often, and made a small demo with it. To my surprise, the agent wrote the shader code easily, and the results were close to the original library.

That's when it became a challenge for me. I remembered optimizing math on the secp256k1 curve in ecloop and optimizing RIPEMD-160, and kept digging. The first model was GigaAM v3 e2e RNN-T, which outputs punctuation and capital letters. I also decided to add other models from the same family to see how hard it would be. They differ in the decoder. CTC recognizes each piece of audio on its own, while RNN-T also looks at the text it has already written, so it's more accurate but slower. Each of them also has a version with punctuation (e2e) and one without.

I dictate in two languages, Russian and English. Since adding Russian was easy, I thought English wouldn't be a problem either. NVIDIA's Parakeet v2 is a good fit for English because it's accurate and fast. But when I started adding it, I found out that models come in different architectures, and each one needs its own compute code.

I took the models already packed into GGUF by the author of transcribe.cpp. It turned out that GGUF stores only the weights and some metadata about the model: the architecture name, layer sizes and vocabulary. The computations aren't in the file, so they have to be written by hand. That's the difference from TensorFlow or Apple's Core ML, where the model file describes a graph of operations and the runtime just executes it.

So I had to sit down and figure it out by reading and watching YouTube videos about how speech recognition models work. In short, long audio is cut into chunks (Whisper, for example, uses 30-second windows). Each chunk goes through a fast Fourier transform (FFT) and becomes a spectrogram (which frequencies sound at each moment). The encoder (a stack of neural network layers that mostly multiply matrices) takes the spectrogram and turns it into a sequence of vectors. The decoder reads them and outputs tokens (pieces of words) one step at a time, and these tokens make up the text. Most of the work is matrix multiplication in the encoder layers, and that's what the GPU computes.

Once I understood how it works, I added support for Parakeet v2. Codex wrote the first version overnight, but the code was messy, so I threw it away and rewrote it cleanly with Claude Code. In the end I got a pretty fast speech-to-text program that beat transcribe.cpp in my tests. Since I was on a roll, I right away added Parakeet v3, a multilingual model from the same family (I barely had to write anything new). Right after it came Whisper, as a reference model to compare the others against.

Parakeet differs from GigaAM in plenty of ways. The model is bigger (24 layers vs 16, and each one is wider). It compresses audio twice as much (one vector per 80 ms instead of 40). It also has a different way to mark when each word was said. But the main difference is the decoder (TDT): along with each token, it predicts how many of the next audio frames it can skip.

The program was turning out so well that I decided to release it as a separate project in the style of ollama. I named it wavo, kind of from WAV, and the name was free. Besides, I had needed local audio transcription before, and there was nothing good for it in the terminal. I tried whisper.cpp, but it's slow and inconvenient (it has to be compiled from source, and the audio has to be cut into chunks by hand).

Before publishing, I decided to clean up the code. The rustfft crate was still slowing down the build. People can use wavo as a library, so everyone who builds it would get that delay too. Once I found the cause, I wrote my own FFT just for this task. It came out at 176 lines of code vs 55,000 in rustfft, and rebuilds went from 11.9 down to 2.1 seconds. The crate itself is good, it's just built for every case, and I needed only one.

The last step before publishing was testing. Luckily, I have a few Macs at hand, an N100 box with Linux and an x86 machine with a GeForce card and a good CPU. I had the agent run the final tests and compare the speed with other libraries. On Macs my library crushed everyone on speed, no question (I'm still shocked it was that easy). But on Windows with an NVIDIA card it was a big failure. Again I had to dig in, and it turned out there's a thing called flash attention.

Attention is the part of the model where each vector is compared with all the others in the same audio chunk. For Whisper that's 1,500 vectors per 30 seconds, so more than two million comparisons in each layer. Regular code writes this whole table to GPU memory and then reads it back. Flash attention computes the same thing in small pieces in fast memory next to the GPU cores and never stores the whole table.

On Macs, the agent had already made such shaders earlier, but for Apple's matrix units (hardware acceleration). NVIDIA's units have a different size, so on NVIDIA wavo computed everything with generic code. After I added shaders for NVIDIA's matrix units, wavo on Windows caught up with transcribe.cpp and in some cases beat it. A short Whisper turbo recording now takes 201 ms vs 226 ms for transcribe.cpp, down from 637.

I made all the measurements on an RTX 4060 and didn't test on AMD Radeon at all. So I'd be glad to see results from other GPUs in issues.

During testing I also found that on M5, wgpu uses only the old matrix units. The M5 got a neural accelerator in each GPU core, which multiplies matrices three times faster (0.60 ms vs 1.89 ms). But it's only available through Metal 4, and wgpu doesn't support it yet. I worked around this with a separate Metal shader just for M5 and opened an issue in wgpu.

I also taught wavo to run on the Neural Engine (a separate part of Apple chips for neural networks), but that's another story. At the end, I ran a final comparison of Parakeet v3 in English against other engines, and the numbers looked impressive.

Engine	Runs on	5 s phrase	1 h recording
wavo	Neural Engine	0.12 s	200×
FluidAudio	Neural Engine	0.13 s	175×
wavo	GPU	0.14 s	160×
transcribe.cpp	GPU	0.12 s	107×
parakeet.cpp	GPU	0.77 s	90×
parakeet-mlx	GPU	0.63 s	61×
sherpa-onnx	CPU	0.83 s	39×

The table shows the time for a 5-second phrase and how many times faster than real time a one-hour recording gets transcribed (MacBook Pro on M5). In Russian with GigaAM v3 the gap is similar: wavo is 330 times faster than real time, and transcribe.cpp is 262 times faster.

Week 3. My own renderer

While I worked on wavo, another agent was experimenting with a UI library in the background. Every approach still needed a lot of hand-written wrappers to talk to the OS, so I decided I could write the renderer myself too. I also had some experience with UIs in video games, so the task seemed simple.

I've long been a fan of Sublime Text, and especially Sublime Merge, and their approach to UI. I thought that if the authors of these apps and of the Zed editor could write their own UI framework, then I, armed with an agent, could too. Armed with strings and nm, we looked at which system functions Sublime Merge uses and figured out the idea behind it. It was an interesting one ;)

The text there is drawn by the system itself (CoreText on macOS, DirectWrite on Windows, Pango on Linux), so it looks native everywhere. That's unlike egui, for example, which draws glyphs on its own. The system draws each glyph once and it goes into a cache, and after that the screen gets ready glyphs from the cache. Context menus work the same way and come from the system (GPUI Kit draws them itself by default).

So it turned into a kind of Frankenstein. The edges are system stuff, and in the middle, like the filling in a sandwich, is my library, which lays out elements in the window and builds components from them.

Rendering uses immediate mode (the frame is built from scratch every time, without a complex event system). Regular apps don't have that much interactivity, and a settings window doesn't need to be redrawn 60 or 120 times per second. Besides, the frame is rebuilt only when something changes, and an idle window draws nothing.

I decided to draw the frame itself on the CPU. It's the simplest code, and it works the same everywhere, without shaders, drivers and other junk that has to be integrated and then tested too. The finished frame goes to the system's regular 2D window output. For an app without complex animations, that's more than fast enough.

3D acceleration pulls in extra libraries, and the program starts eating more memory (on Windows a simple window with D3D11 takes 27 MB instead of 5.1). But if it's ever needed, I can always add it later.

The renderer itself came out at about 3,000 lines of Rust, and the agent quickly put together the basic components on top of it (around 6,000 more lines). Everything worked, and it seemed like the job was almost done.

Next I needed a context menu and a small popup that shows the text being dictated. That's where the pain and suffering began, especially when I ran everything on real computers with Linux and Windows instead of VMs.

Now I understand why there were so few GUI frameworks in the pre-agent era. The hard part isn't writing a renderer and a component system at all. The hard part is wiring all of it together on different platforms so that it works. Each cross-platform feature can easily take several thousand lines for three systems:

windows (open them, drag them and draw in them): 5,000 lines;
text fields and IME: 4,000;
tray, hotkeys and autostart: 7,000;
accessibility on macOS: 3,000;
native context menu: 2,000.

Of course the agent writes the code, but someone still has to read it, check it and test it on each system, and that's a lot of work.

For example, a context menu can be drawn by the app's own renderer, and that's how GPUI in Zed does it. But it looks janky, and more importantly, such a menu can't go outside the window. I had never thought about this before, but once I started writing, I realized it matters. Sublime Merge can do it, and I give it credit for that.

As expected, Linux was the biggest pain. On macOS and Windows everything goes through one system API, but Linux is a damn zoo. At first I tried to draw separately on X11 and Wayland to make the program as optimized as possible, but the amount of code grew exponentially. So I made a compromise and took GTK 3, which is on almost every distro. Sublime Merge does the same.

I named the library nimbo. A simple counter window built with it pulls in 14 crates on macOS, 5 on Windows and 3 on Linux. Almost all of them are system bindings, like objc2 on macOS and windows-sys on Windows. It builds from scratch in 11 seconds, and the binary is 1.3 MB. For comparison, here is the same counter on other frameworks (the number of crates on each system, a clean release build on M2 and the binary size):

	macOS	Windows	Linux	Build	Binary
nimbo	14	5	3	11 s	1.3 MB
iced	152	159	237	59 s	10.9 MB
Tauri¹	217	223	265	64 s	9.9 MB
Xilem	157	177	257	65 s	12.2 MB
egui	177	178	279	74 s	14.3 MB
Slint	248	244	345	84 s	12.5 MB
GPUI	300	270	322	86 s	8.6 MB
GPUI Kit	411	395	518	144 s	19.4 MB

¹ Plus the system WebView, which isn't part of the binary.

In the era of personal software and endless hacks, I want my own foundation for this kind of program, without depending on other people's frameworks. So I decided to finish nimbo properly. There's no first release yet because I'm still working on the API, so I have no time to make cool videos for X. But if you need something minimal in Rust, you're welcome to try it.

The library is built in three layers. At the bottom is a common API for system stuff (windows, tray, context menu and so on), so the app code never has to touch it at all. Above it are headless components (behavior without styling), and any design can be built on top of them. On top is a base design in the style of system controls. The model for the library's API is SwiftUI.

Week 4. Release: signing, packaging, updates

The program is ready and tested, and all that's left is to ship a release. But that turned out to be not so simple either. There were three problems: signing, packaging and auto-update, and they took the last week.

Signing

In recent versions of macOS and Windows, unsigned apps just don't run. Windows is easier: SmartScreen shows a warning on the first launch, and once the user accepts it, the app starts. On macOS, though, the quarantine has to be removed from an unsigned app by hand with /usr/bin/xattr -dr com.apple.quarantine /Applications/Wavo.app, and a regular user is unlikely to do that.

I had to pay Apple $99 a year. And their payment form is straight from the early 2000s: I had to type in my card details by hand, and Apple charged the card on its own sometime later. In the end the charge failed on 3-D Secure. I had to temporarily lower the protection on my card and ask support to retry the payment, and the whole thing dragged on for several days. I can't remember the last time I saw something like that.

Packaging

A program for three systems needs installers for all three. On macOS the standard is a DMG. On Windows it's a setup.exe that installs the program, usually made with NSIS. On Linux it's an AppImage and a .deb package. Linux actually has more formats, but for now I picked these two.

At first it seemed easy to do with a Python script, so I gave that to the agent. There was one requirement: build and signing are separate. The build step downloads and compiles other people's code, so it must not have access to the signing keys. The second step only signs the finished files.

The script quickly grew to almost 2,000 lines. That surprised me, and I went looking for ready solutions, since after so many years someone must have solved this. I found a few (Velopack, cargo-packager and the Tauri bundler), but each has its own problems. Some can't separate build from signing. The worst part is that almost none of them can do everything on one machine (DMG and signing for macOS only on a Mac, deb and AppImage only on Linux, and Windows also needs NSIS). Velopack can build everything at once, but only on a Mac. In my opinion this is crazy. Packaging and signing should work on one computer, and ideally on any computer.

I looked through other people's repos, and from what I saw, only Mozilla does it properly. They package Firefox on Linux, even the DMG for macOS, but with wild hacks. For DMG they have their own fork of the C library libdmg-hfsplus, which has no prebuilt releases, and their CI builds it straight from the repo at a specific commit. To use it myself, I would have to pull an artifact out of someone else's unversioned build, and I didn't want that. Well, I had an agent, so I gave this job to it too.

At first I wanted to make it part of nimbo. Then I realized it's a completely separate program and moved it to its own crate, shipo.

The main idea of shipo is that all installers can be built on any machine, even on Linux. Everything that exists on only one system (hdiutil on macOS, makensis for Windows, apt packages on Linux) is replaced with its own code. The other tools are downloaded at a pinned version and hash.

For macOS, shipo writes the DMG itself, together with the HFS+ file system per Apple's spec. It even includes the Unicode tables HFS+ uses to normalize file names. It also writes the .DS_Store that places the app icon and the /Applications shortcut in the window. It builds the .icns icon itself too, without sips and iconutil. Signing and notarization outside a Mac go through rcodesign and Apple's notarization service.

For Windows, the installer is custom too. It's a small prebuilt setup.exe in Rust with an empty marker inside. The packager replaces the icon, version and manifest in the app's exe and compresses the files into one archive. Then it appends the archive to the end of setup.exe and writes into the marker where that archive is. The uninstaller is the same setup.exe, just without the files.

For Linux, AppImage (a prebuilt runtime with a squashfs image of the app appended to it) and deb are built right in memory.

Auto-update

The third thing actually has nothing to do with packaging, but I had to do it too. On macOS many apps update with Sparkle, but it works only on macOS. For Windows there's WinSparkle, which has a similar idea but is a different library. I wanted one thing for all systems, but didn't find anything here either. I decided that building and updating are close things and can live in one crate, so I made them together.

The idea of auto-update is simple. Each release also includes a small file that describes the latest version. The app reads it from time to time, and if the version is newer, it downloads and installs the update by itself.

In shipo it's one text file, update.txt, in the latest release on GitHub:

version 1.2.0
notes https://github.com/me/hello/releases/tag/v1.2.0
macos-aarch64 394788 <sha512> https://github.com/me/hello/releases/download/v1.2.0/Hello-v1.2.0-macos-aarch64.dmg
windows-x86_64 285930 <sha512> https://github.com/me/hello/releases/download/v1.2.0/Hello-v1.2.0-windows-x64-setup.exe
linux-x86_64 412204 <sha512> https://github.com/me/hello/releases/download/v1.2.0/Hello-v1.2.0-linux-x86_64.AppImage
sig <signature>

It has the version, a link to the release notes, and a line for each platform (size, SHA-512 hash and file URL). At the end is an Ed25519 signature of everything above. No separate update server is needed. The app just downloads the file from the latest GitHub release by a permanent link (releases/latest/download/update.txt).

The app checks the downloaded file by size and hash. It has two signing keys, a main one and a backup. If the main one leaks, the next release is signed with the backup.

The update uses the same installer as the first install. On macOS the library mounts the DMG, copies the app from it and replaces the old one. On Windows it runs setup.exe without a window. On Linux it replaces the AppImage file.

shipo has no separate HTTP library. Instead it has three short wrappers over the system ones (NSURLSession on macOS, WinHTTP on Windows, libcurl on Linux). So the updater has few dependencies: signature checking, errors and system bindings.

So in a week I got shipo, about 12,000 lines of code. Roughly 10,000 of them are the packager (with the setup.exe stub), and 2,400 are the library that goes into the app. Its packager builds installers for all three systems on one machine. The updater adds only 8–15 crates to the app depending on the system (Velopack and cargo-packager have over a hundred). In the end, releasing an app properly and setting up CI turned out to be harder than writing a UI renderer.

Wrap up

A program for a couple of days turned into a month of work. In that time there were about 850 commits and 167,000 changed lines (including prototypes and code I later threw away). Working with agents took about 90 active hours and 3.7 billion tokens, which is roughly $2,700 at API prices. The numbers are approximate because some agent logs got lost.

Most of the code was written by Claude Opus 5.5, and GPT-6 Astra did the reviews. Copilot on GitHub did about 390 more reviews until I hit its limit. Codex drew the graphics and images.

The work didn't end there. For anyone besides family and friends to hear about the program, I have to write about it on Reddit, X and other places, and that's a big separate job too. While doing it, I found a lot of similar programs. If I had seen them at the start, I probably wouldn't have built mine at all. But what's done is done. In the end I got my own cross-platform stack, and I hope to reuse it for other things.

All the programs from the post once more:

Wavo: local dictation for macOS, Windows and Linux;
wavo: CLI and Rust crate for speech recognition in the style of ollama;
nimbo: native UI library with almost no dependencies;
shipo: installers and auto-update for three systems from one machine.

I wanted to make one small dictation app, and got an app and three libraries for it.

Tracking Limits in Codex and Claude Code

2026-05-21T00:00:00+00:00

Still tweaking coding agents to fit my workflow. Last time it was about the work process, now a small thing — tracking limits via statusline.

I use two coding agents: Codex and Claude Code. Both work fine for me, and I switch between them freely — sometimes because one runs out of limits, sometimes just because I keep different tasks in different clients. Claude Code became my main one — that's where skills, MCP and all the other tooling showed up first. Over time a lot of settings piled up there, and I haven't moved most of them to Codex. I haven't figured out a good way to sync that either. If you've solved this kind of sync between Claude Code and Codex, drop a link in the comments.

When limits were introduced a month or so ago, constantly checking them with /status got annoying fast. Same with context — to see how much is left before autocompact you have to run /context separately. Not hard on its own, but doing it all the time gets old. I also wanted to see when limits or context were getting close so I could save things manually. After autocompact the model often gets dumb and forgets important things. So I went looking for a way to track this and found that both clients have a statusline. By default it's pretty minimal in both — mostly just shows the current model.

Codex has a decent setup for this. Just run /statusline and pick what to show in the interactive UI. Settings get saved to ~/.codex/config.toml.

I ended up with this format:

gpt-5.4 medium · ~/Code/blog · Context 73% left · 5h 93% · weekly 92% · 258K window · 1.2M used

Codex doesn't let you write a fully custom format string. You can only choose from a set of built-in blocks and set their order.

In the config it looks like this:

[tui]
status_line = ["model-with-reasoning", "current-dir", "context-remaining", "five-hour-limit", "weekly-limit", "context-window-size", "used-tokens"]
theme = "catppuccin-macchiato"

You can either click through the UI or just edit the file directly.

Claude Code also has /statusline. But instead of an interactive menu with blocks like Codex, they went the agent way: it tries to read your bash / zsh / fish configs and generate a statusline from them.

I didn't give it access to my files — I don't want my configs going to Anthropic. And I'm not sure why they'd need them anyway, GitHub is full of dotfiles repos.

In ~/.claude/settings.json you can point it to a command that runs on every statusline update. Like this:

{
  // ...
  "statusLine": {
    "type": "command",
    "command": "~/.claude/statusline.sh",
    "padding": 0,
  },
}

command is just a script that Claude Code runs on each update. There are quite a few implementations on GitHub. Like these: sirmalloc/ccstatusline, Owloops/claude-powerline, Haleclipse/CCometixLine, hagan/claudia-statusline, felipeelias/claude-statusline. I call them crazy because in a world where npm packages break every other week, installing extra software via npm for something this simple seems like a bad idea.

Still, the idea is worth taking. The command gets called on each update, receives JSON on stdin, and you can parse and format it however you want. Docs for available fields: Claude Code statusline docs

The statusline gets roughly this data:

{
  // shortened example; see docs for the full schema
  "session_id": "<session-id>",
  "transcript_path": "/Users/<user>/.claude/projects/<project>/<session-id>.jsonl",
  "cwd": "/Users/user/Code/blog",
  "session_name": "<session-name>",
  "model": { "id": "claude-sonnet-4-6", "display_name": "Sonnet 4.6" },
  "workspace": { "current_dir": "/Users/user/Code/blog" },
  "version": "2.1.126",
  "output_style": { "name": "default" },
  "context_window": {
    "total_input_tokens": 433,
    "total_output_tokens": 492,
    "context_window_size": 200000,
    "current_usage": {
      "input_tokens": 1,
      "output_tokens": 92,
      "cache_creation_input_tokens": 299,
      "cache_read_input_tokens": 14201,
    },
    "used_percentage": 7,
    "remaining_percentage": 93,
  },
  "exceeds_200k_tokens": false,
  "fast_mode": false,
  "effort": { "level": "high" },
  "thinking": { "enabled": true },
  "rate_limits": {
    "five_hour": { "used_percentage": 62, "resets_at": 1779296400 },
    "seven_day": { "used_percentage": 22, "resets_at": 1779750000 },
  },
}

I use starship.rs for terminal customization. They even have Claude Code integration: Statusline for Claude Code. But there are no blocks for hourly limits — after Claude added them, starship never caught up. There's already an issue on GitHub: starship/starship#7441. From what I can tell, the author has gone quiet, and the project has been mostly dead for months.

So I just asked Claude to write a bash script that outputs the same format as in Codex. Got something like this:

#!/usr/bin/env bash
set -euo pipefail

input="$(cat)"
get() { jq -r "$1" <<<"$input"; }

fmt_tokens() {
  awk -v n="$1" 'BEGIN {
    if      (n >= 1000000) printf "%.1fM", n / 1000000
    else if (n >= 1000)    printf "%.0fK", n / 1000
    else                   printf "%d",    n
  }'
}

model="$(get '.model.display_name')"
dir="$(get '.workspace.current_dir')"
dir="${dir/#"$HOME"/\~}"

context_remaining="$(get '(.context_window.remaining_percentage // 100)')% left"
five_hour="$(get '(100 - .rate_limits.five_hour.used_percentage) | round')%"
weekly="$(get '(100 - .rate_limits.seven_day.used_percentage) | round')%"
window="$(fmt_tokens "$(get '.context_window.context_window_size')")"
used="$(fmt_tokens "$(get '.context_window.context_window_size * .context_window.used_percentage / 100')")"

printf '%s · %s · Context %s · 5h %s · weekly %s · %s window · %s used\n' \
  "$model" "$dir" "$context_remaining" "$five_hour" "$weekly" "$window" "$used"

Both clients end up looking the same:

The only issue I noticed in Claude Code is that in new sessions the statusline does not show up immediately when the terminal opens. It appears only after the first interaction with the chat (you can also trigger it manually with Cmd+Shift).

That's it. Setting up the statusline and writing this post took about three hours.

Spec-Driven Development in Practice

2026-05-20T00:00:00+00:00

Tried two tools for Spec-Driven Development on my pet project — delta-farmer (51 files, 12k lines of code). Compared spec-kit and cc-thingz: ran each through two cases — security analysis and regular coding.

Tools

spec-kit (github) — open source toolkit for Spec-Driven Development. Helps you focus on product scenarios and predictable results instead of vibe coding from scratch. Works on top of different coding agents, generates separate files for each step.

cc-thingz (github) — a set of skills for Claude Code, installed via plugins / marketplace api. Basically a ready-made workflow inside Claude, not a separate wrapper around different agents.

Setup, workflow and artifacts

spec-kit is designed as a more universal thing for different agents, but you pay for that with a separate CLI wrapper, its own templates and its own file structure. The flow is a sequence of separate commands: /speckit-specify -> /speckit-plan -> /speckit-tasks -> /speckit-implement, plus /speckit-clarify for spec refinement. One task creates multiple files.

A separate problem with spec-kit is the .specify folder. This is its internal structure where it stores some data and workflow metadata. It's not clear what to add to git and what not to. I didn't find any clear recommendations or best practices for team work in the documentation.

spec-kit also inserts a link to the current task directly into CLAUDE.md / AGENTS.md. Any new Claude session immediately sees it and works in the context of that task — you can't just have a regular conversation about the project or plan something else in parallel.

cc-thingz is simpler in this sense: it's tied directly to Claude Code and installed in its ecosystem as a set of skills. The pipeline is also fixed: /brainstorm -> /plan:make -> /plan:review -> /plan:exec, but it feels simpler. One task gets one file.

Planning

In spec-kit, planning is split into several separate commands. First you run /speckit-specify <task-description>: the tool analyzes the task, looks at the source code, sometimes asks clarifying questions — you answer in chat by literally sending the letter A / B / C or typing your own answer (no interactive mode like in plan-mode).

After that spec-kit writes files to disk, and then you manually review them. You open the files, read them, check if the agent described the task correctly. If something is wrong, you write about it in chat — the tool generates an updated version.

Next step — /speckit-plan. The agent tries to write a technical spec based on the earlier description. Same process: generate, review, comment — until the result is good enough.

The last planning command is /speckit-tasks. It breaks the main task into subtasks. Same format. Basically the whole flow in spec-kit works the same way at each step: a command runs, the tool writes files to disk, and you have to validate them manually.

Inside specs/<task-name>/ you end up with files like spec.md, plan.md, tasks.md, research.md, data-model.md, checklists/requirements.md, and more. Each next step adds new files or updates old ones, so one task quickly grows into a lot of text that needs to be read and validated by hand.

There's also /speckit-clarify — to brainstorm a solution before writing the technical spec and before breaking it down.

cc-thingz works differently. It starts with /brainstorm: after you describe the task, the tool asks questions in interactive mode. You can pick a ready answer or type your own — it's not just chat, it's more like a menu with options (like plan-mode).

After /brainstorm the agent immediately suggests next steps: continue planning, review yourself, send to auto-review, or save. If you choose auto-review, other agents check the plan, find problems, and make fixes. So a big part of the review is built into the flow before you even look at it yourself.

Then you can either continue right in the same window, or save the file and come back later with /plan:make <file>. At this step the agent adds a technical plan and task list to the file. Everything goes into one Markdown file in docs/tasks/, and new info is just appended at the end.

After that the agent again offers options: run the plan through review again, save it, or start implementing. When the work is done, the file moves to docs/tasks/completed. Because everything lives in one file, it's easier to read diffs, easier to follow changes, and you don't have to think about which file to open.

Executing tasks

In spec-kit, implementation starts with /speckit-implement specs/<spec>, and then the agent starts working on the task and writing code.

In cc-thingz, implementation starts with /plan:exec. You can run it with a file in a new session or continue right after planning. Then the agent goes through the subtasks one by one, writes code, adds tests as it goes, runs them to check the result, and at the end runs everything through an additional review by another agent.

Task 1. Supply chain hardening

A Python trading automation project, distributed via git clone + uv run. Dependencies are described in pyproject.toml using >= ranges, uv.lock is not committed, so when cloning users get random versions from PyPI with no guarantees.

I needed to figure out how to protect such a project from supply chain issues, without external services like Renovate or Dependabot. Which files to commit, how to pin versions and how to update them later (using pnpm as a reference), how to be sure all deps are safe before release. Plus a reference to best practices.

Planning

First I gave both tools the same task description and ran the first pass. For cc-thingz it was /brainstorm, for spec-kit — /speckit-specify.

Then I ran the agent's proposed solution through a critic agent. spec-kit has /speckit-clarify for that, and cc-thingz after the first pass offered three options itself: commit the plan, review yourself, or send to another agent for review. I chose the last option.

cc-thingz produced a clear plan. The task came down to four changes: commit uv.lock — this gives reproducible installs and SHA256 verification; add exclude-newer = "7 days" — blocks packages published less than 7 days ago (protection against timing attacks like the LiteLLM case); enable only-binary = [":all:"] — prevents setup.py execution during install, wheels only; add uv audit for CVE scanning before releases. And separately — switch deploy to uv sync --locked for deterministic server installs.

spec-kit at the /speckit-clarify step started asking weird questions like "what to do if uv audit runs in an offline environment — skip the PR or not".

It also randomly suggested sending CVE notifications through the telegram integration in the code. That integration is used for trade start / stop — the bot saw a piece of code, didn't understand what it was for, and mixed up internal app logic with external security tooling.

Also, even though the project uses uv, it kept pulling pip-based solutions from the article instead of using proper uv tools. For example, it suggested uv run pip-audit, even though uv already has its own uv audit.

Then in planning mode it broke things into 21 subtasks — literally a separate item for every tiny thing.

Implementation

The task itself is simple, so both were able to implement it from the plan without much trouble.

The cc-thingz result was clean and easy to read. spec-kit used some unclear packages, so the result didn't even run. I didn't check if what it did actually works — I didn't trust the result and didn't bother running it.

Task 2. Real coding task

Task description (580 words, 5 kb): what to do, where to get the data, and how to verify the result. I intentionally didn't name specific files — left everything up to the developer, like in real life.

The short version: there are three data sources. The user's general trade history (1), the provider's trade history across all users (2), and the last few days of trades from the provider for the specific user (3). The task was to download sources 2 and 3, filter source 1 by them, cache that to disk, and plug it into the program's output instead of what's there now (which is basically just source 1).

Planning

The product task was already described well, so brainstorming wasn't really needed here. But I gave the same file to both tools anyway and asked them to start planning the technical implementation.

cc-thingz asked a few good questions about implementation, then proposed a plan. Then it offered two options: review the plan myself or send it to another agent. I chose the second — got the file for review, made a few small fixes, and committed the plan. The result was one file: 10.6 Kb, 158 lines, 1479 words, 7 tasks. It also asked about things I missed in the task description — like how to store the cache on disk and what the data types look like. The bug in this task was exactly that: two sources represent the same data differently — in one place it's 1000, in another 1000.0, and they don't map directly.

spec-kit didn't ask any additional questions on its own. After three commands (specify / plan / tasks) there was already a bunch of files to read and check: 40.5 Kb, 737 lines, 4761 words, 13 tasks. It's heavy to read even once, not to mention that these files change during the review process.

Implementation

cc-thingz ran not directly, but through ralphex: https://github.com/umputun/ralphex. Basically Claude / Codex in YOLO mode in Docker. My command was: ralphex --serve docs/plans/20260515-onyx-fills-filtering.md. It runs slow: a full pass takes an hour to an hour and a half.

spec-kit ran via /speckit-implement 002 (where 002 is the task number) in a new session.

When ralphex was already on the last task, my five-hour limit ran out. I had no idea what task spec-kit was working on at that point, because it didn't update its files during execution. With cc-thingz it was easier: it created a commit after each change, so the full history was there.

Resuming work in both tools is different. In spec-kit it's a regular Claude session: when the limit resets, you can keep writing in the same window. Or you can close it and reopen with --resume. With ralphex it's different: you can call it again with the same task file, and it figures out where it stopped last time because it marks checkboxes in the todo list as it goes. So it continues from the last stopping point. Obviously without ralphex — cc-thingz also works as a regular Claude session.

Result

Nothing to brag about here: both solutions didn't work out of the box, and both needed manual fixes.

cc-thingz had a problem where the agent didn't ask me about the full data structure during planning and didn't fetch a sample from the data source, even though I wrote that the data is public. As a result it interpreted the Time field as a timestamp instead of an ISO format string, so the data didn't parse. No other startup problems, but the numbers were wrong. After several dozen minutes of debugging in a separate Claude session (I don't know how to connect to ralphex logs, so I opened a new one) it turned out the bot called the API with wrong parameters. Because of that, part of the data couldn't be filtered correctly by source.

spec-kit had different errors at startup: it decided to calculate the report for 2025, even though I clearly said to start from January 1, 2026. It ended up downloading data for a very long time and hit the rate limit. Then there was a lock/unlock error when writing files. It decided to store data in a separate file for each date, even though I asked to save everything in one file and append to it when updating. After startup — same problem with wrong numbers: had to go to chat, enable debug logs, figure it out, and fix.

Both wrote tests. cc-thingz tests pass. spec-kit tests don't — it didn't run them and didn't check if the generated code was correct.

You can see the code, task files, and how changes are organized in the diffs: spec-kit (11 files, 1206 lines) and cc-thingz (9 files, 760 lines). spec-kit has more changes, but a big part is its own spec files, not code. cc-thingz worked more closely with the actual codebase.

Wrap up

Before testing both tools, I ran the second task myself — just in a Claude dialog in edit mode, without any spec. I didn't have a proper task description at that point: I just didn't understand where to get the data and how it was structured. So the whole investigation happened in chat — debugging together with the agent, looking at what came from the API, figuring out the structure. That took about two or three hours.

Once the task was clear, I could write a proper description — and that's exactly what I gave to both tools.

My final verdict:

spec-kit — tool with a clear idea, but annoying to work with. Generates a lot of files that you have to read and review manually. Planning took about an hour, then the agent ran for another hour, then another hour fixing what didn't work. Wrote tests but didn't run them. Left more artifacts than code. Overkill for a simple task.

cc-thingz — easier to work with. One file, interactive mode, built-in review. Same time pattern: an hour for the spec, an hour for implementation, an hour for fixes. Wrote tests and ran them. There were errors, but the reason is clear — the agent didn't clarify the data structure upfront and didn't fetch a sample from the public API. The result is more trustworthy, though it wasn't perfect either.

The whole thing — testing both tools on two tasks and writing this post — took me two days.

Full diffs for both tasks: spec-kit, cc-thingz.

A few final thoughts

Working with agents in this mode is generally fine, but the cognitive load is pretty high. You need to describe the task clearly upfront, carefully read everything the agent wrote during planning, and make sure it covered all the important parts. If that's done well, the agent works pretty reliably after that. In theory you could use that time to write a spec for the next task — but for now it's more of an illusion than a real parallel workflow.

Fully delegating code writing to the agent doesn't seem like a great idea to me. First, you end up poorly understanding what was actually written. Second, the agent makes bad decisions on its own — without explicit direction from a human it often makes logical mistakes or goes in the wrong direction. At least in my case. Tasks in dialog mode get done faster: you understand what's happening with the code, you control how the solution goes — and the agent makes fewer mistakes.

cc-thingz with ralphex is worth trying for really simple tasks. You write the task to a file, run it — and it goes. For that scenario it looks like a genuinely useful thing. But the app needs to be well-tested and have a clear structure — otherwise the agent has nothing to rely on, and you get what I got.

2025 in Review: A Quiet Year and a Dive into Crypto

2025-12-31T00:00:00+00:00

Another year is coming to an end, and following my new tradition, I would like to sum up some results in writing. Overall, the year felt rather passive for me. I don’t remember planning any big things, but still, previous years were more active for me in terms of creativity and beyond (open source, learning new skills, and so on).

Open Source

This year, I didn’t release any new open-source projects, although I had some work in progress. I’m still maintaining my older projects little by little, but also without much enthusiasm (for some of them, this was the main idea from the start — stability).

More details about the main projects:

twscrape (my most popular project) — it constantly requires dealing with new obstacles introduced by X;
apigen-ts — in the near future, the codegen will need to be rewritten using templates, because TypeScript was rewritten in Go, and there is a chance that AST primitives will be removed from the typescript package on npm. Also, for me personally, the project has become a bit less relevant due to using tRPC/NextJS in my main stack, but for non-JS backends it is still a must-have;
macmon — it is also quite difficult to maintain, because Apple releases new processors too often and in many different configurations, so it’s not always possible to get the required hardware for testing. I looked for online services that provide access to “bare metal” hardware via SSH for a limited time (a VPS-like model), but didn’t find anything suitable;
ecloop — I added SIMD optimizations for SHA-256 and RMD-160 to the library. I think this was my most interesting task this year. I wanted to play around with CUDA and even ordered a computer for that, but ended up not using it. As for crypto itself, I got into degen stuff (more on that later).

Public projects on GitHub — a total of 92 commits, 79 closed issues, 11 releases created, and more than 1k stars gained.

UPD: I remembered that at the end of 2024 / beginning of 2025, I worked on badges.ws in Rust. It looked like a fairly simple task, but turned into several months of work and debugging various APIs. In general, I probably should have dropped the project earlier, but I didn’t and pushed it through to a release — at the cost of my motivation to start new projects. What conclusion to draw from this, I’m not sure.

Blog

Overall, this year I wanted to write more posts, and I even have many drafts, but at some point I wanted to switch to a more convenient engine than Zola (which I currently use). However, I couldn’t find anything good and flexible enough.

Problems with Zola:

a strange templating language that lacks some features, while issues remain open for years;
GPT tools don’t handle this templating system well — every time I need to make a change, it’s hard even to google how to do it;
lack of flexibility in page navigation and file organization (I can’t arrange blog files the way I want and have to follow the engine’s structure);
broken redirects in content — Zola doesn’t understand relative links in text (all links must be written relative to the site navigation), so it’s not convenient to write and work with the blog locally: linking between my own posts and being sure that all links are valid. Right now, I have to solve this with a separate script.

As an engine, I would like something that uses JSX as a templating language, has Tailwind CSS out of the box, doesn’t have Zola’s navigation and linking issues, and is a simple and stable solution. Overall, Astro fits this category, and I tried migrating the blog to it, but it also has a number of its own problems:

duplicated logic and the difficulty of choosing between .astro and .jsx formats;
lack of proper MD/MDX integration (any customization is done via CSS classes instead of properly overriding the rendering of Markdown tags with JSX components);
no built-in Atom feed generation (only RSS is supported), which means it has to be implemented manually.

Overall, I did a test migration of the blog to Astro, but never released it, because the solution felt too complex. A blog is something you might not touch for months, and when you finally do, you want to immediately understand how things work, rather than having to google or ask GPT for solutions.

I also considered NextJS as an option, since it can be configured to fit your needs, but there are big concerns about its stability. I really don’t want to read changelogs every few months just to understand their new vision and figure out what needs to be updated in something that already worked, just to keep it working the same way.

That turned into a lot of text about the blog. In short, I’d like to write more while being less distracted by technical problems. The ideal solution for me right now would be MD/MDX with the ability to override elements. Maybe someone knows a good existing solution? (or maybe it’s an idea for a project on Bun)

Skills

In previous years, I explored and practiced DevOps, Rust, Go, LLMs, NestJS/NextJS, and other things, but this year felt neutral. I can’t quickly name anything new that I learned this year. Probably the main change in my work environment was the switch from iTerm2 to WezTerm. The reason was adding LLM features to the terminal and a security vulnerability.

Overall, throughout the year it felt scary to update software. A lot of things are now being built with AI, and the quality of releases has become frankly weaker. For example:

I’ve been using Chrome since 2009, and this year was the first time I saw rendering bugs, crashes, and similar issues;
Google Search was broken for several months, and they didn’t seem to care (if you type a query with a typo, Search suggests the correct version, you can click it and it appears in the search field, but if you then click into the field, it reverts back to the original typo — this might seem like a small issue to some, but it’s a very common use case for me);
on YouTube, for several days after releasing the new player, the volume couldn’t be increased above 60% via the UI: the slider just wouldn’t move. I don’t understand how something that wasn’t even properly tested in production could be released;
many similar small bugs that ruin the UX.

In short, something has gone wrong in the world of software development. That’s why updating feels risky — you never know how your workflow will break this time. My favorite app is Sublime Merge: updates there are purely cosmetic and happen only a couple of times a year.

Crypto aka Degen

This area deserves a separate mention. In January, I worked on a small project for bridging ERC-20 / SPL tokens between the Ethereum and Solana networks. Around the same time, Trump released his Meme Coin: those who were insiders or just fast made good money, while everyone else ended up at a loss. That got me interested in how crypto works in general — not the technical side, but the community itself.

As a result, I subscribed to various chats and other sources with related information (though it’s a huge stream, often full of noise). Later, I found the DoubleTop community, where I’m now a member and communicate regularly.

Overall, during the second half of the year I was moderately active in crypto — trying different areas to understand how things work. So far, my experience is this: it is possible to make money in crypto, but it’s really difficult and requires a lot of time and attention. Losing money is much easier. There are no huge multipliers on deposits, but steady percentages are possible.

Cases I participated in:

Sale Fleek on CoinList — rekt −70%, the team basically scammed;
Sale Enso on CoinList — good result, +70%;
Sale Plasma — x10 on allocation, but it was hard to get in, and funds were locked for 3 months;
Sale Pump.fun — x2 in 3 days, but as always, there were nuances. I was lucky to get part of the allocation through the website, while CEXs issued refunds;
Almanak on Legion — rekt −50%. Overall, it was a decent project, but the team delayed the launch and went live in a bad market. I forgot that I participated in this sale and didn’t sell the tokens in time;
Farm Paradex — x2 on spend. The project was expected to launch in summer, but the release was later postponed indefinitely. This was my first retrodrop-focused project, so my enthusiasm for farming other projects cooled down. In November–December, the Paradex team added an XP transfer option, which made it possible to sell points OTC, and I used that;
Farm Lighter — a lot of multiples here. I earned most of my points from hedge positions and didn’t farm aggressively on purpose. Given how bad the market is right now, the project literally gave everyone a New Year’s gift;
Binance Alpha — simple daily activity that required generating trading volume on Binance. It brought about $300–400 per month per account, but it was hard to multi-account;
Sales on BuidlPad — the platform provided stable returns for about half a year. It was easy to multi-account, but accounts were also often blocked. A lot of liquidity was required to participate, since allocations were small. On average, all sales ended up around x2 on spend;
Flash crash on Oct 10 — the whole crypto market dropped hard within a couple of hours and still hasn’t returned to previous levels. I didn’t have any perp positions, so I got lucky there, but I did have USDT/USDC loans backed by crypto assets that were liquidated;
Spot positions (bad market): BNB +30%, ETH +20%, SOL −20%, HYPE −40%.

Of course, there were more cases — I listed the main closed ones. In general, it is possible to make money in crypto, but it’s really hard and requires a systematic approach. The crypto market is alive only for a few months a year, when so many events happen that it’s impossible to participate in everything (not enough capital, energy, or time). For most of the year, nothing really happens — that’s when people go work at McDonald’s.

That quiet period is usually the time to farm retrodrops, but it’s genuinely difficult work: the results of your actions and money spent come only after 6–12 months, and psychologically that’s hard for me.

The current meta is Perp DEXes (Paradex / Lighter / Aster / Extended / Pacifica.fi / Ethereal / Trade.xyz / etc.) and opinion markets (Polymarket / Opinion / Limitless / etc.).

The hardest part of crypto for me is tracking balances and understanding whether I’m up or down overall (it’s basically real accounting work). Roughly speaking, I finished the year at about the same level I started, although at some points my deposit was up to +50%. It’s important to note that everything related to crypto should be done with proper risk control — I have a separate budget that I use only for experiments.

Anyway, this is a big topic, and I’ll probably need to talk about it separately.

Health

Still alive — and that’s good.

This year I quit smoking — I’m very happy about it, I feel better, and I’ve reduced the risk of various cancers. Overall, quitting wasn’t that hard: I watched a few YouTube videos to understand what to expect and what kind of “tricks” my brain would use to convince me to smoke again. For the first three weeks or so, you constantly think about smoking, and then it fades. Most of this “thinking” isn’t about smoking itself, but about habits and routines — like taking a break, looking out the window, having a coffee, and going for a smoke. Now, from time to time, I get short five-minute urges in the form of flashbacks.

I also started doing regular stretching/exercises — at least a couple of times a week (I track it in Sheets). Without this, I genuinely feel worse: it’s hard to work when your back hurts, for example. I periodically take body measurements, and probably the main change is my biceps: +2 cm.

Leisure

I bought an Xbox to relax by playing games (I had a PlayStation 4 in the past, so I wanted to try a different platform). I tried quite a lot of games via Game Pass — before that, I hadn’t played much for several years. From what I liked:

Clair Obscur: Expedition 33 — an obvious game of the year, 100% worth playing: the story, art, music, and combat are all excellent.
Mafia: Definitive Edition — a remake of a classic, slightly simplified. The racing mission still takes dozens of attempts — and that’s a good thing.
Indiana Jones and the Great Circle — a solid game. It was enjoyable to explore Rome and the Egyptian pyramids and solve local puzzles. The fistfights are also quite fun, and you don’t really want to use firearms.
Doom: The Dark Ages — the legendary demon-slaying series. I only played Doom 1 and Doom 3 before and skipped the entire modern lineup. I decided to start with The Dark Ages. Overall, the first half of the game is interesting: great locations, new weapons constantly unlocking, and so on. But the second half feels tedious — all locations turn into the same gray corridors. The robot titan missions were cool, though.
Assassin's Creed Mirage — at first it felt like just another Assassin’s Creed, but the more I played, the better it got. I have warm memories of Baghdad. In my opinion, it’s one of the strongest games in the series. What’s nice is that it’s short. I played it before the recent DLC; people say it’s even better now.
Escape from Duckov — a meme game in the extraction genre. I didn’t finish it because it’s quite long, but it’s worth trying to understand the genre. It’s addictive.
Lies of P — my first proper soulslike (before that, I only tried Sekiro, which was hell for me). I liked it: the bosses are manageable for a regular player, and there are many ways to beat them (shadow pulls, throwable jars with liquids). The story is simple and interesting, although the playtime feels a bit stretched toward the end.

Other media that stood out.

Movies / TV shows: Fly Me to the Moon (2024), The Fighting Sullivans (1944), American Beauty (1999), The Naked Gun (2025), How to Steal a Million (1966), Once Upon a Time... in Hollywood (2019), Bugonia (2025), Аватар: Путь воды (2022).

Music: Sabrina Carpenter, The Velvet Sundown (yes, AI slop), Alice in Chains.

Books: I didn’t read much; the most recent one was Ionesco’s Rhinoceros.

Happy New Year to everyone! 🎉

RIPEMD-160 55% Faster with ARM Neon SIMD

2025-05-08T00:00:00+00:00

I have a hobby project – ecloop – a Bitcoin key "calculator" designed to find Bitcoin puzzles, verify brain wallets, and so on. The mathematical chances of finding a private key from a used address tend toward zero. I'm interested in this program as a collection of tricks involving the elliptic curve (secp256k1) and as a way to practice low-level programming (fast 256-bit arithmetic), so I continue to develop it from time to time.

To calculate a Bitcoin address from a private key, several operations need to be performed:

Calculate the point on the elliptic curve (public key): P = G * PrivKey.
Compute SHA-256 from the compressed public key: (P.y % 2 == 0 ? 0x02 : 0x03) + P.x.
Compute RIPEMD-160 from the resulting SHA-256 hash.

The result is the so-called hash160, which is then encoded into a Bitcoin address using either base58 or bech32.

As slow as elliptic curve operations are, the slowest part of address generation is the RIPEMD-160 (RMD160) calculation. In fact, it takes about half of the total execution time (SHA-256 is hardware-accelerated on modern processors).

Almost all modern processors have SIMD support: AVX2 on amd64 and Neon on arm64. That's why I decided it would be worthwhile to speed up RMD160 by implementing it using parallel calculations. Moreover, I had never written SIMD code before, so I was interested in trying it out.

My main computer is a MacBook with Apple Silicon (M-series chips). Initially, I wanted to implement RMD160 SIMD using SVE (256-bit / 8-lane), but it turned out that Apple chips do not support SVE 🤦. (To clarify: the M2 chip implements the ARMv8.6 standard, while SVE was introduced in ARMv8.2 as an optional feature). So I had to use Neon (128-bit / 4-lane). If that's not the case and there is a way to run SVE instructions on M-series chips, I'd be happy to hear about it.

What is RMD160?

RIPEMD-160 (RMD160) is a cryptographic hash function that produces a 160-bit hash from arbitrary data. It was developed as a secure alternative to earlier algorithms such as MD5 and SHA-1. RMD160 is widely used in blockchain technology, particularly in Bitcoin, where it is used to generate wallet addresses: the public key is first hashed using SHA-256, and then hashed again with RMD160 to improve security and reduce length.

The RMD160 algorithm consists of five rounds. Each round includes basic logical functions, cyclic shifts (ROTL), and addition. A distinctive feature of RMD160 is that each round is executed in two parallel branches: the main (left) branch and the parallel (right) branch. These two branches use different constants, word orders, and logic functions, and their results are later combined.

There is a classic C implementation of RMD160 that uses a bunch of macros to define rounds, logical functions, and so on—but such code is hard to read. So I prefer the Golang implementation, which I previously ported to ecloop. I plan to continue the porting to Neon based on this codebase.

Stepping back a bit, SIMD instructions aren't extremely complex, but they lack "syntactic sugar" — so instead of writing a + b, you need to write something like vaddq_u32(a, b). There are special functions like this for every standard operation multiplied by the number of numeric types (u/i 8/16/32/64, f16/32/64).

RMD160 (like other hash functions) should not be too difficult to port to SIMD because their algorithms contain no branching. Essentially, the logic stays the same; it's just a matter of replacing all operations with SIMD-specific instructions.

Simple Neon Program

To understand how to write using Neon, we should start with the simplest possible program, such as multiplying 42 × 2. Since SIMD involves parallel computations, its operations are applied to the entire vector at once, and the result should be the same across all parts of the vector. To verify this, the result can be printed to the console.

#include <arm_neon.h>
#include <stdint.h>
#include <stdio.h>

void print_check(uint32x4_t *a) {
  uint32_t arr[4];
  vst1q_u32(arr, *a); // store 4x32-bit vector into a regular array
  for (int i = 0; i < 4; i++) {
    printf("%x%c", arr[i], i == 3 ? '\n' : ' ');
  }
}

int main() {
  uint32_t a = 42;
  uint32x4_t b = vdupq_n_u32(42); // load u32 to all 4 lanes (42, 42, 42, 42)
  printf("%x = ", a);
  print_check(&b); // out: 2a = 2a 2a 2a 2a

  a = a * 2;
  b = vmulq_n_u32(b, 2); // multiply each lane by 2
  printf("%x = ", a);
  print_check(&b); // out: 54 = 54 54 54 54

  return 0;
}

In general, I think the concept of how SIMD calculations work is clear. From here on, print_check will be used frequently to check the correctness of the hashing algorithm.

Basic Functions and ROTL

The RMD160 calculation uses 5 basic functions and ROTL; everything else is a shuffling of data in a specific order. GPT has rewritten these macros, and I have verified that they are correct:

// original functions
#define OLD_F1(x, y, z) ((x) ^ (y) ^ (z))
#define OLD_F2(x, y, z) (((x) & (y)) | (~(x) & (z)))
#define OLD_F3(x, y, z) (((x) | ~(y)) ^ (z))
#define OLD_F4(x, y, z) (((x) & (z)) | ((y) & ~(z)))
#define OLD_F5(x, y, z) ((x) ^ ((y) | ~(z)))
#define OLD_ROTL(x, n) (((x) << (n)) | ((x) >> (32 - (n))))

// simd functions
#define F1(x, y, z) veorq_u32(veorq_u32(x, y), z)
#define F2(x, y, z) vorrq_u32(vandq_u32(x, y), vandq_u32(vmvnq_u32(x), z))
#define F3(x, y, z) veorq_u32(vorrq_u32(x, vmvnq_u32(y)), z)
#define F4(x, y, z) vorrq_u32(vandq_u32(x, z), vandq_u32(y, vmvnq_u32(z)))
#define F5(x, y, z) veorq_u32(x, vorrq_u32(y, vmvnq_u32(z)))
#define ROTL(x, n) vorrq_u32(vshlq_n_u32(x, n), vshrq_n_u32(x, 32 - (n)))

void print_check(char *l, uint32_t c, uint32x4_t a) {
  printf("%s: %08x = ", l, c);
  uint32_t arr[4];
  vst1q_u32(arr, a); // store 4x32-bit vector into a regular array
  for (int i = 0; i < 4; i++) {
    printf("%08x%c", arr[i], i == 3 ? '\n' : ' ');
    // assert(arr[i] == c);
  }
}

uint32_t a1, b1, c1;
uint32x4_t a2, b2, c2;

a1 = 0x67452301, b1 = 0xefcdab89, c1 = 0x98badcfe;
a2 = vdupq_n_u32(a1), b2 = vdupq_n_u32(b1), c2 = vdupq_n_u32(c1); // loading vectors

// compare original and simd functions
print_check("F1", OLD_F1(a1, b1, c1), F1(a2, b2, c2));
print_check("F2", OLD_F2(a1, b1, c1), F2(a2, b2, c2));
print_check("F3", OLD_F3(a1, b1, c1), F3(a2, b2, c2));
print_check("F4", OLD_F4(a1, b1, c1), F4(a2, b2, c2));
print_check("F5", OLD_F5(a1, b1, c1), F5(a2, b2, c2));
print_check("RL", OLD_ROTL(a1, 12), ROTL(a2, 12));

// output:
// F1: 10325476 = 10325476 10325476 10325476 10325476
// F2: ffffffff = ffffffff ffffffff ffffffff ffffffff
// F3: efcdab89 = efcdab89 efcdab89 efcdab89 efcdab89
// F4: 67452301 = 67452301 67452301 67452301 67452301
// F5: 88888888 = 88888888 88888888 88888888 88888888
// RL: 52301674 = 52301674 52301674 52301674 52301674

Problem with Porting Golang Implementation

The Golang implementation consists of 5 large loops, each performing 16 left and right rounds. Inside each operation, the input data is mixed at index _n[i], and ROTL is performed at index _r[i]. The reference C implementation uses a series of consecutive macros, which makes it harder to read (in my opinion).

// Golang implementation
static const u8 _n[80] = { /* ... */ }; // Left DATA indexes
static const u8 _r[80] = { /* ... */ }; // Left ROTL indexes

// round 1
for (; i < 16; ++i) {
  // left branch
  alpha = a1 + F1(b1, c1, d1) + x[_n[i]];
  alpha = rotl32(alpha, _r[i]) + e1;
  beta = rotl32(c1, 10);
  a1 = e1, c1 = b1, e1 = d1, b1 = alpha, d1 = beta;
  // right branch
  // ...
}

// Reference C-implementation
#define F(x, y, z)        ((x) ^ (y) ^ (z))
#define FF(a, b, c, d, e, x, s)        {\
      (a) += F((b), (c), (d)) + (x);\
      (a) = ROL((a), (s)) + (e);\
      (c) = ROL((c), 10);\
   }

// round 1 - left branch
FF(aa, bb, cc, dd, ee, X[ 0], 11);
FF(ee, aa, bb, cc, dd, X[ 1], 14);
// ...
FF(bb, cc, dd, ee, aa, X[14],  9);
FF(aa, bb, cc, dd, ee, X[15],  8);

In general, if we rewrite the code directly using Neon instructions, the compilation crashes with an error stating that vshlq_n_u32 and vshrq_n_u32 require a known rotation value (the second argument) at compile time. See:

// Golang implementation (original)
#define F1(x, y, z) ((x) ^ (y) ^ (z))
#define rotl32(x, n) (((x) << (n)) | ((x) >> (32 - (n))))
alpha = a1 + F1(b1, c1, d1) + x[_n[i]];
alpha = rotl32(alpha, _r[i]) + e1;

// Golang implementation (SIMD)
#define F1(x, y, z) veorq_u32(veorq_u32(x, y), z)
#define ROTL(x, n) vorrq_u32(vshlq_n_u32(x, n), vshrq_n_u32(x, 32 - (n)))
alpha = vaddq_u32(a1, F1(b1, c1, d1));
alpha = vaddq_u32(alpha, X[_n[i]]);
alpha = vaddq_u32(ROTL(alpha, _r[i]), e1);
// err: argument to '__builtin_neon_vshlq_n_v' must be a constant integer
// err: argument to '__builtin_neon_vshrq_n_v' must be a constant integer

So, I have to use the macro version (C reference) because, in that case, the indexes are passed directly (the last argument in the FF macro) and expanded into constant values at compile time. Maybe this change is for the better (we will see why later).

Generic Round Macro

If we look at RMD160 rounds, the same actions are performed, but with the following changes: base function, constant, data index, and rotation. In general, the round macro was shown above (I took a slightly different version from GitHub). My goal is to port the round macro to SIMD.

In the round, we add up 4 variables, perform a ROTL + another addition, and a separate ROTL for another variable. Since the "+" operation is not available in SIMD, we need to use special instructions.

I added some macros for vector addition and described the round itself:

#define ADD2(a, b) vaddq_u32(a, b)
#define ADD3(a, b, c) vaddq_u32(vaddq_u32(a, b), c)
#define ADD4(a, b, c, d) vaddq_u32(vaddq_u32(vaddq_u32(a, b), c), d)

#define RN(a, b, c, d, e, f, x, k, r) \
  u = ADD4(a, f, x, vdupq_n_u32(k));  \
  a = ADD2(ROTL(u, r), e);            \
  c = ROTL(c, 10);

In the macro, a, b, c, d, e are the state variables, f is the value after the base function computation, x is the uint32 index data for the current iteration, k is a constant, and r is the rotation value for ROTL.

vdupq_n_u32(k) loads a constant into a vector (the same value in all 4 lanes). Earlier, we wrote code to multiply a vector by a number, and for this purpose, vmulq_n_u32 is used. Logically, the instruction to add a number to a vector should be vaddq_n_u32, but it does not exist. Instead, it should be written in vaddq_u32(vec1, vdupq_n_u32(2)) style (if anyone knows why this is the case – please leave a comment).

Then, based on this round macro, we can define left and right rounds. The code here is similar to any other macro implementation (except that I named the rounds as Li/Ri).

#define L1(a, b, c, d, e, x, r) RN(a, b, c, d, e, F1(b, c, d), x, 0, r)
#define L2(a, b, c, d, e, x, r) RN(a, b, c, d, e, F2(b, c, d), x, 0x5A827999ul, r)
#define L3(a, b, c, d, e, x, r) RN(a, b, c, d, e, F3(b, c, d), x, 0x6ED9EBA1ul, r)
#define L4(a, b, c, d, e, x, r) RN(a, b, c, d, e, F4(b, c, d), x, 0x8F1BBCDCul, r)
#define L5(a, b, c, d, e, x, r) RN(a, b, c, d, e, F5(b, c, d), x, 0xA953FD4Eul, r)
#define R1(a, b, c, d, e, x, r) RN(a, b, c, d, e, F5(b, c, d), x, 0x50A28BE6ul, r)
#define R2(a, b, c, d, e, x, r) RN(a, b, c, d, e, F4(b, c, d), x, 0x5C4DD124ul, r)
#define R3(a, b, c, d, e, x, r) RN(a, b, c, d, e, F3(b, c, d), x, 0x6D703EF3ul, r)
#define R4(a, b, c, d, e, x, r) RN(a, b, c, d, e, F2(b, c, d), x, 0x7A6D76E9ul, r)
#define R5(a, b, c, d, e, x, r) RN(a, b, c, d, e, F1(b, c, d), x, 0, r)

Now, using these macros, we can write the first iteration of the first round and compare it with the working implementation. If everything is OK, we can copy the entire round, check it, and then move on to the remaining rounds. I compared the result with the print_check function I created earlier.

First left round, first iteration:

// RMD160 initial constants
#define K1 0x67452301
#define K2 0xEFCDAB89
#define K3 0x98BADCFE
#define K4 0x10325476
#define K5 0xC3D2E1F0

void rmd160_block(uint32x4_t *s, const uint32_t x[4][16]) {
  // a1-e1 left rounds state, a2-e2 right rounds state, u - temp varible used in RD macro
  uint32x4_t a1, b1, c1, d1, e1, a2, b2, c2, d2, e2, u;
  // Load initial constants
  a1 = a2 = vdupq_n_u32(K1);
  b1 = b2 = vdupq_n_u32(K2);
  c1 = c2 = vdupq_n_u32(K3);
  d1 = d2 = vdupq_n_u32(K4);
  e1 = e2 = vdupq_n_u32(K5);

  uint32x4_t w[16]; // Load data to vector
  for (int i = 0; i < 16; i++) {
    // Load 4x32-bit integers from x[0][i], x[1][i], x[2][i], x[3][i]
    // w[i] = vsetq_lane_u32(x[0][i], w[i], 0);
    // w[i] = vsetq_lane_u32(x[1][i], w[i], 1);
    // w[i] = vsetq_lane_u32(x[2][i], w[i], 2);
    // w[i] = vsetq_lane_u32(x[3][i], w[i], 3);
    w[i] = vld1q_u32(((uint32_t[4]){x[0][i], x[1][i], x[2][i], x[3][i]})); // A bit faster
  }

  L1(a1, b1, c1, d1, e1, w[0], 11);
  print_check("a1", 0, a1);
  print_check("b1", 0, b1);
  print_check("c1", 0, c1);
  print_check("d1", 0, d1);
  print_check("e1", 0, e1);
}

uint32x4_t s[5] = {0}; // initial state
s[0] = vdupq_n_u32(K1);
s[1] = vdupq_n_u32(K2);
s[2] = vdupq_n_u32(K3);
s[3] = vdupq_n_u32(K4);
s[4] = vdupq_n_u32(K5);

uint32_t x[4][16] = {0}; // data block, filled with zeros
rmd160_block((uint32x4_t *)s, x);

It should be noted that hash functions are usually tested on zero data (for simplicity). Data in hash functions are processed in blocks. A block in RMD160 is 32×16 = 512 bits. rmd160_block can be called several times with the same state (which changes) and new data, for cases where you need to calculate the hash of a message larger than one round. In my task (address generation), all messages are placed in one block. The result of the first round compare to current implementation:

// a1: 1602f864 1602f864 1602f864 1602f864 vs c3d2e1f0
// a1: efcdab89 efcdab89 efcdab89 efcdab89 vs 1602f864
// a1: eb73fa62 eb73fa62 eb73fa62 eb73fa62 vs efcdab89
// a1: 10325476 10325476 10325476 10325476 vs eb73fa62
// a1: c3d2e1f0 c3d2e1f0 c3d2e1f0 c3d2e1f0 vs 10325476

In general, these values ± are similar to the values from the current version, differing by one offset. This is not a problem, since there are 5 variables, and the offsets will be aligned by the end. It's just a difference in the implementations.

I won't write all the rounds, as there are 80 on each side (160 total) – it would result in an overly large code block. Left and right rounds are independent of each other and can be calculated in any order: either first all left / all right, or alternating left / right, or alternating iterations within a round. This will not affect the final result.

Finalizing RMD160

At the end of RMD160 block, we need to combine the old state with the local state – this also involves three additions with index offsets.

void rmd160_block(uint32x4_t *s, const uint32_t x[4][16]) {
  // ... 160 rounds

  uint32x4_t t = s[0];
  s[0] = ADD3(s[1], c1, d2);
  s[1] = ADD3(s[2], d1, e2);
  s[2] = ADD3(s[3], e1, a2);
  s[3] = ADD3(s[4], a1, b2);
  s[4] = ADD3(t, b1, c2);
}

The final step is to change the endianness of the values (RMD160 uses a different endianness) and unload the values from the vector into the resulting array.

// ... init & rmd160_block

for (int i = 0; i < 5; ++i) {
  // swap32 for uint32x4_t (can it be shorter?)
  s[i] = vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(s[i])));
}

uint32_t r[4][5] = {0}; // result stored as 4x5 uint32_t
for (int i = 0; i < 5; i++) { // load it from uint32x4_t
  r[0][i] = vgetq_lane_u32(s[i], 0);
  r[1][i] = vgetq_lane_u32(s[i], 1);
  r[2][i] = vgetq_lane_u32(s[i], 2);
  r[3][i] = vgetq_lane_u32(s[i], 3);
}

That is all – r can be used further where needed (r[0], r[1], r[2], r[3] are the computed hashes — four at once).

To summarize this section, the complete parallel RMD160 algorithm looks like this:

Initialize a state of length 160 bits × 4 lanes (uint32x4_t s[5]).
Split the message (data) into blocks of 512 bits × 4 lanes (uint32_t x[4][16]).
Iterate RMD160 rounds until the data runs out (rmd160_block reads the data into the vector itself).
Change the endianness in the final state.
Unload the vector of the final state into a hash array (int32_t r[4][5]).

Performance of RMD160 SIMD

Now, it's time to measure the performance of this code – to compare the performance of the original and SIMD implementations. For this purpose, I created a small benchmark:

size_t tsnow() {
  struct timespec ts;
  clock_gettime(CLOCK_REALTIME, &ts);
  return ts.tv_sec * 1000 + ts.tv_nsec / 1e6;
}

void rmd160_simd() {
  uint32_t r[4][5] = {0};
  uint32_t x[4][16] = {0};

  size_t stime = tsnow();
  size_t iters = 1000 * 1000 * 32;
  for (size_t i = 0; i < iters; ++i) rmd160_4w(r, x);
  double dt = (tsnow() - stime) / 1000.0;
  double ir = iters / dt / 1000000;
  double hr = ir * 4; // 4 hash per iter
  printf("%.2fM it/s ~ %.2fM h/s ~ %.2fs\n", ir, hr, dt);

  printf("s[0]: %08x\n", r[0][0]);
  printf("s[1]: %08x\n", r[0][1]);
  printf("s[2]: %08x\n", r[0][2]);
  printf("s[3]: %08x\n", r[0][3]);
  printf("s[4]: %08x\n", r[0][4]);
}

void rmd160_naive() {
  uint32_t s[5] = {0};
  uint32_t x[16] = {0};

  size_t stime = tsnow();
  size_t iters = 1000 * 1000 * 32;
  for (size_t i = 0; i < iters; ++i) rmd160_1w(s, x);
  double dt = (tsnow() - stime) / 1000.0;
  double ir = iters / dt / 1000000;
  double hr = ir * 1; // 1 hash per iter
  printf("%.2fM it/s ~ %.2fM h/s ~ %.2fs\n", ir, hr, dt);

  printf("s[0]: %08x\n", s[0]);
  printf("s[1]: %08x\n", s[1]);
  printf("s[2]: %08x\n", s[2]);
  printf("s[3]: %08x\n", s[3]);
  printf("s[4]: %08x\n", s[4]);
}

I compiled both programs with -O3 and ran them (on a basic Apple M2):

❯ clang -O3 -march=native ./lib/rmd160.c && ./a.out # original
5.50M it/s ~ 5.50M h/s ~ 5.81s

❯ clang -O3 -march=native ./lib/rmd160s.c && ./a.out # neon
2.14M it/s ~ 8.55M h/s ~ 14.98s

The Neon version (128-bit / 4 lanes) is 55% faster. This is a great result, but it's unfortunate that M-chips don't have SVE for 256/512 bits (8/16 lanes), as that would make it even better!

One More Thing

While I was writing about the results above, I became curious to see what would happen if I changed the order of the rounds in RMD160. The original order of rounds was as follows: first all left rounds, then all right rounds. I thought this was good for the processor, because it seemed to require less "context switching".

I changed the order of rounds to alternating left and right rounds (L1 / R1, L2 / R2), and the performance increased significantly. Initially, I thought there was a data error, but print_check (as tests) confirms everything is fine.

I decided to try alternating iterations (L1_1 R1_1 L1_2 R1_2 R1_2, etc.). To be honest, rearranging 160 lines is not the most fun, but the result surprised me even more.

Comparison of different round/iteration placements (3 tests of RMD160 function itself and a full cycle of ecloop logic):

# L1_1 L2_2 .. L5_16 R1_1 R2_2 .. R5_16 (+56%)
2.25M it/s ~ 9.02M h/s ~ 14.19s
2.23M it/s ~ 8.93M h/s ~ 14.33s
2.23M it/s ~ 8.94M h/s ~ 14.32s
ecloop (addr33 x 8 core) ~ 19.53M it/s (+22%)

# L1_1-L1_16 R1_1-R1_16 L2_1-L2_16 .. (+165%)
3.70M it/s ~ 14.80M h/s ~ 8.65s
3.85M it/s ~ 15.42M h/s ~ 8.30s
3.87M it/s ~ 15.46M h/s ~ 8.28s
ecloop (addr33 x 8 core) ~ 22.46M it/s (+40%)

# L1_1 R1_1 L1_2 R1_2 .. L5_16 R5_16 (+175%)
3.96M it/s ~ 15.82M h/s ~ 8.09s
3.94M it/s ~ 15.78M h/s ~ 8.11s
3.94M it/s ~ 15.76M h/s ~ 8.12s
ecloop (addr33 x 8 core) ~ 24.83M it/s (+55%)

It's a mystery to me why it works this way, and maybe there is an even more effective arrangement. Who knows? Please, write in the comments.

Support for AVX2 (AMD64)

Initially, I didn't plan to implement this, but RMD160 algorithm using macros was quite abstract, and further porting to AVX2 seemed fairly simple. The main difference between AVX2 and Neon (aside from the different instruction sets) is the vector size – 256 bits vs 128 bits – which allows us to process 8 hashes in parallel (vs 4 in Neon).

Currently, the following Neon instructions are used directly in the algorithm code: vector type (uint32x4_t), state initialization via vdupq_n_u32, endian-swap, and load / dump data into the vector.

I'm moving these things to macros (just in case, I added RMD_ prefix to avoid conflicts with other files):

#define RMD_LEN 4                    // vector length
#define RMD_VEC uint32x4_t           // vector type
#define RMD_LD_NUM(x) vdupq_n_u32(x) // load same number into all lanes

#define RMD_SWAP(x) vreinterpretq_u32_u8(vrev32q_u8(vreinterpretq_u8_u32(x)))
#define RMD_LOAD(x, i) vld1q_u32(((uint32_t[4]){x[0][i], x[1][i], x[2][i], x[3][i]}))
#define RMD_DUMP(r, s, i)                                                                        \
  do {                                                                                           \
    r[0][i] = vgetq_lane_u32(s[i], 0);                                                           \
    r[1][i] = vgetq_lane_u32(s[i], 1);                                                           \
    r[2][i] = vgetq_lane_u32(s[i], 2);                                                           \
    r[3][i] = vgetq_lane_u32(s[i], 3);                                                           \
  } while (0);

And update the current code to something like this:

void rmd160_block(RMD_VEC *s, const uint32_t x[RMD_LEN][16]) {
  RMD_VEC a1, b1, c1, d1, e1, a2, b2, c2, d2, e2, u;
  a1 = a2 = RMD_LD_NUM(RMD_K1);
  b1 = b2 = RMD_LD_NUM(RMD_K2);
  c1 = c2 = RMD_LD_NUM(RMD_K3);
  d1 = d2 = RMD_LD_NUM(RMD_K4);
  e1 = e2 = RMD_LD_NUM(RMD_K5);

  RMD_VEC w[16];
  for (int i = 0; i < 16; i++) w[i] = RMD_LOAD(x, i);
  // ... rounds and iterations
}

// new function to process full single block
void rmd160_batch(uint32_t r[RMD_LEN][5], const uint32_t x[RMD_LEN][16]) {
  RMD_VEC s[5] = {0}; // load initial state
  s[0] = RMD_LD_NUM(RMD_K1);
  s[1] = RMD_LD_NUM(RMD_K2);
  s[2] = RMD_LD_NUM(RMD_K3);
  s[3] = RMD_LD_NUM(RMD_K4);
  s[4] = RMD_LD_NUM(RMD_K5);

  rmd160_block((RMD_VEC *)s, x);                     // round
  for (int i = 0; i < 5; ++i) s[i] = RMD_SWAP(s[i]); // change endian
  for (int i = 0; i < 5; ++i) RMD_DUMP(r, s, i);     // dump data to array
}

Of course, it's already quite magical, but there aren't really many changes. Now, I should add redefined macros for AVX2. In addition, I wrapped a series of architecture-specific macros in #ifdef. In fact, I have a single codebase for RMD160 algorithm, and the necessary macros are included depending on which processor the program is being compiled for.

#if defined(__aarch64__) && defined(__ARM_NEON)
  #include <arm_neon.h>

  #define RMD_LEN 4                    // vector length
  #define RMD_VEC uint32x4_t           // vector type
  // ... move all current Neon related macros here
#elif defined(__x86_64__) && defined(__AVX2__)
  #include <immintrin.h>

  #define RMD_LEN 8
  #define RMD_VEC __m256i
  #define RMD_LD_NUM(x) _mm256_set1_epi32(x)

  #define RMD_SWAP(x)                                                                              \
    _mm256_shuffle_epi8((x), _mm256_setr_epi8(3, 2, 1, 0, 7, 6, 5, 4, 11, 10, 9, 8, 15, 14, 13,    \
                                              12, 19, 18, 17, 16, 23, 22, 21, 20, 27, 26, 25, 24,  \
                                              31, 30, 29, 28))

  #define RMD_LOAD(x, i)                                                                           \
    _mm256_set_epi32(x[0][i], x[1][i], x[2][i], x[3][i], x[4][i], x[5][i], x[6][i], x[7][i])

  #define RMD_DUMP(r, s, i)                                                                        \
    do {                                                                                           \
      r[0][i] = _mm256_extract_epi32(s[i], 0);                                                     \
      r[1][i] = _mm256_extract_epi32(s[i], 1);                                                     \
      r[2][i] = _mm256_extract_epi32(s[i], 2);                                                     \
      r[3][i] = _mm256_extract_epi32(s[i], 3);                                                     \
      r[4][i] = _mm256_extract_epi32(s[i], 4);                                                     \
      r[5][i] = _mm256_extract_epi32(s[i], 5);                                                     \
      r[6][i] = _mm256_extract_epi32(s[i], 6);                                                     \
      r[7][i] = _mm256_extract_epi32(s[i], 7);                                                     \
    } while (0);

  #define _mm256_not_si256(x) _mm256_xor_si256((x), _mm256_set1_epi32(0xffffffff))
  #define RMD_F1(x, y, z) _mm256_xor_si256(x, _mm256_xor_si256(y, z))
  #define RMD_F2(x, y, z) _mm256_or_si256(_mm256_and_si256(x, y), _mm256_andnot_si256(x, z))
  #define RMD_F3(x, y, z) _mm256_xor_si256(_mm256_or_si256(x, _mm256_not_si256(y)), z)
  #define RMD_F4(x, y, z) _mm256_or_si256(_mm256_and_si256(x, z), _mm256_andnot_si256(z, y))
  #define RMD_F5(x, y, z) _mm256_xor_si256(x, _mm256_or_si256(y, _mm256_not_si256(z)))

  #define RMD_ROTL(x, n) _mm256_or_si256(_mm256_slli_epi32(x, n), _mm256_srli_epi32(x, 32 - (n)))
  #define RMD_ADD2(a, b) _mm256_add_epi32(a, b)
  #define RMD_ADD3(a, b, c) _mm256_add_epi32(_mm256_add_epi32(a, b), c)
  #define RMD_ADD4(a, b, c, d) _mm256_add_epi32(_mm256_add_epi32(a, b), _mm256_add_epi32(c, d))
#else
  #error "Unsupported arch for RIPEMD-160 (AVX2 or NEON required)"
#endif

The main differences are:

A different header file.
A different vector type (8 lanes instead of 4 in Neon) and different names for intrinsics.
AVX2 has no Bitwise NOT, so I had to add it separately as _mm256_not_si256.
No separate function for endian-swap, but there is a more generalized function to rearrange bits in a given order: _mm256_shuffle_epi8 (the first argument specifies where to rearrange the bits, and the second specifies how to rearrange them).
_mm256_set_epi32 makes it more convenient to load data into different lanes, while in Neon we had to use a temporary array (the variant with setting each lane separately (vsetq_lane_u32) is slower).

The rewriting of the macros was mostly handled by GPT; I just checked the correctness once again.

Performance of AVX2 Version

I have a small fanless PC running Linux on an Intel N100, which I use for native application testing. I ran the benchmark written earlier on it and got the following results:

❯ clang -O3 -march=native ./lib/rmd160.c && ./a.out # original (on Intel N100)
4.26M it/s ~ 4.26M h/s ~ 7.51s

❯ clang -O3 -march=native ./lib/rmd160s.c && ./a.out # avx2 (on Intel N100)
2.25M it/s ~ 17.96M h/s ~ 14.25s

8 lanes of AVX2 and the correct arrangement of rounds in the algorithm (zebra iterations) result in a 320% increase in the number of hashes per second compared to the original code. What's interesting is that AVX2 on the Intel N100 runs 20% faster than Neon on the Apple M2 (mainly because of the vector size). Overall ecloop speedup: 5.45M it/s vs 7.73M it/s (+42%).

Fallback Implementation for Older Processors / VMs

In the #ifdef above, I left the #else section with #error to prevent compilation on unsupported systems. In general, this is not ideal, and I would like the program to work everywhere (mainly for potential runs in VMs). Since the whole algorithm is already written in macros, adding a new implementation is not difficult. I simply redefine all macros to use a vector size of 1 and uint32_t as the "vector" type. In reality, the program will work with a single array, which, from a memory perspective in C, is essentially the same as just using uint32_t.

// ... #ifdef for neon / avx2
#else
  #warning "Fallback RIPEMD-160 implementation used. AVX2 or NEON required for SIMD"

  #define RMD_LEN 1
  #define RMD_VEC uint32_t
  #define RMD_LD_NUM(x) x

  #define RMD_SWAP(x) __builtin_bswap32(x)
  #define RMD_LOAD(x, i) x[0][i]
  #define RMD_DUMP(r, s, i) r[0][i] = s[i]

  #define RMD_F1(x, y, z) ((x) ^ (y) ^ (z))
  #define RMD_F2(x, y, z) (((x) & (y)) | (~(x) & (z)))
  #define RMD_F3(x, y, z) (((x) | ~(y)) ^ (z))
  #define RMD_F4(x, y, z) (((x) & (z)) | ((y) & ~(z)))
  #define RMD_F5(x, y, z) ((x) ^ ((y) | ~(z)))

  #define RMD_ROTL(x, n) (((x) << (n)) | ((x) >> (32 - (n))))
  #define RMD_ADD2(a, b) (a + b)
  #define RMD_ADD3(a, b, c) (a + b + c)
  #define RMD_ADD4(a, b, c, d) (a + b + c + d)
#endif

I ran it and checked it – it works correctly, except that the speed became a bit faster due to the new order of rounds. It is not difficult to add an implementation for AVX512 (16 lanes) in the same way, but I don't have a processor to test it, so I didn't do it. Also, the article was already getting quite long, so I’m wrapping it up here.

Fin

SIMD programming turned out to be easier than it seemed. The necessary intrinsics can be "Googled" in GPT; it is difficult to search for them yourself because there are thousands of combinations. RMD160 itself is mostly used in cryptocurrencies (at least I don't know of other popular use cases), so the usefulness of the obtained code outside of the learning factor is questionable.

SIMD calculations give a good boost to execution speed, but, of course, you should take into account the specifics of the task: you need to have a lot of data to process, and these data should have the same size. It makes no sense to use SIMD if some data to be hashed fits into one block and others into 100 (for example, when processing files). Also, the main program should be able to process data in batches.

The final code as a single file is available on Github.

Why I Moved from iTerm2 to WezTerm

2025-01-09T00:00:00+00:00

For many years, I used iTerm2 as my main terminal emulator and probably spent hundreds of hours in it. Overall, I was satisfied with it, despite some strange recent updates like adding AI features, KeyChain integration, and security vulnerabilities.

In recent years, new terminal emulators have appeared. I tried using them mainly for testing macmon. A couple of years ago, I tried switching to kitty, which was faster due to GPU acceleration. However, it required too much customization and still looked very non-native for macOS. GPU acceleration was added to iTerm as well, so I stayed with it.

I also tried Alacritty, but it is very basic and lacks the features I need. Recently, I tested Ghostty, which has gained huge attention – it has nice defaults, but its RAM usage is concerning (around 250MB per empty tab). Currently, it also lacks buffer search, which makes the terminal useless for me.

I heard a lot of positive feedback about WezTerm from bobuk. So I decided to try it out and use it for some time – and I am very satisfied with it.

The next part of the article will be about how to make WezTerm understandable for iTerm users to try it or switch completely.

Initial

WezTerm can be installed from Homebrew:

brew install --cask wezterm

When you first open WezTerm, it looks plain, non-native, and not very functional. But actually, it has everything you need. This is because WezTerm is cross-platform, unlike iTerm2, so the author left minimal default settings, leaving the rest of the configuration to the user. My goal is to achieve a similar experience in WezTerm as I had in iTerm.

WezTerm does not have a graphical interface for configuration, and the setup is done through a Lua file (this is unusual for me, but Vim users are familiar with it). The configuration file can be located at ~/.wezterm.lua or ~/.config/wezterm/wezterm.lua. I prefer the second path because it is where Fish and starship store configs too.

Create a configuration file:

mkdir -p ~/.config/wezterm/ && touch ~/.config/wezterm/wezterm.lua

And then add the boilerplate code for configuration:

local wezterm = require "wezterm"
local config = wezterm.config_builder()
local action = wezterm.action

-- (here will be added actual configuration)

return config

Appearance

First, I want to change the default font, its size, and disable ligatures (because it's important to see the actual characters in case of some logs).

config.font = wezterm.font {
  family = 'JetBrains Mono',
  weight = 'Medium',
  harfbuzz_features = { 'calt=0', 'clig=0', 'liga=0' }, -- disable ligatures
}
config.font_size = 14.0
config.line_height = 1.0

Save the file, and the changes should be applied immediately (yes, no reload required!).

Next, I want to change the theme and set it to follow the current system theme automatically. For the past year, I have been using Catppuccin themes, and I am very happy with them. They rarely have issues where some text colors blend with the background. To make the theme switch automatically based on the current system theme, we need to write a small Lua function (I copied it from the documentation):

local function scheme_for_appearance(appearance)
  if appearance:find "Dark" then
    return "Catppuccin Macchiato"
  else
    return "Catppuccin Latte"
  end
end

config.color_scheme = scheme_for_appearance(wezterm.gui.get_appearance())

I forgot to mention that the Catppuccin theme, like many other popular themes, is pre-installed in WezTerm, so you don't need to install it separately like in iTerm, which is very cool. Of course, you can also set up your own colors.

The default padding also seems a bit large, so let's make it smaller. I also had a Blinking Line Cursor (I don't remember if this is the default or not in iTerm). Let's set both configurations:

config.window_padding = { left = '0.5cell', right = '0.5cell', top = '0.5cell', bottom = '0.5cell' }
config.default_cursor_style = 'BlinkingBar'

Window style

In iTerm, I used the Minimal theme to save vertical space for tabs. We can do the same in WezTerm (the default tabs don't look very good). Also possible to add some transparency and blur to the window. I didn't use this in iTerm, but let's try it now:

config.window_decorations = 'RESIZE|INTEGRATED_BUTTONS'
config.window_background_opacity = 0.96
config.macos_window_background_blur = 20

Maximize on start-up

In iTerm, I usually set large values for Cols/Rows to make the start window bigger. This approach is a bit outdated, so let's make the window open in full screen when the terminal starts.

-- https://github.com/wez/wezterm/issues/3299#issuecomment-2145712082
wezterm.on("gui-startup", function(cmd)
  local active = wezterm.gui.screens().active
	local tab, pane, window = wezterm.mux.spawn_window(cmd or {})
  window:gui_window():set_position(active.x, active.y)
  window:gui_window():set_inner_size(active.width, active.height)
end)

Key bindings

Finally, the most important part is setting up hotkeys – it's hard to give up what you've been used to for years. An important thing for me is Split Panels with Cmd + D / Cmd + Shift + D. Cmd + W should close the current active panel, not the whole tab. Cmd + K – clear the current screen. Cmd + ← / Cmd + → for start/end of the line. I think these are all defaults from iTerm, so let's make them the same in WezTerm:

config.keys = {
  { key = 'd', mods = 'CMD|SHIFT', action = action.SplitVertical { domain = 'CurrentPaneDomain' } },
  { key = 'd', mods = 'CMD', action = action.SplitHorizontal { domain = 'CurrentPaneDomain' } },
  { key = 'k', mods = 'CMD', action = action.ClearScrollback 'ScrollbackAndViewport' },
  { key = 'w', mods = 'CMD', action = action.CloseCurrentPane { confirm = false } },
  { key = 'w', mods = 'CMD|SHIFT', action = action.CloseCurrentTab { confirm = false } },
  { key = 'LeftArrow', mods = 'CMD', action = action.SendKey { key = 'Home' } },
  { key = 'RightArrow', mods = 'CMD', action = action.SendKey { key = 'End' } },
  { key = 'p', mods = 'CMD|SHIFT', action = action.ActivateCommandPalette },
}

WezTerm also has a Command Execution menu. I haven't used it much yet, but just in case, I bind it to the same keys as in VSCode / Zed – Cmd + Shift + P.

Note: In the configuration, the key value is case-sensitive, so { key = 'd', ... } and { key = 'D', ... } are different things (basically key='D' is the same as key='d', mods='SHIFT'). I initially got confused with this.

Fin

Overall, I am very satisfied with WezTerm and have fully switched to using it daily. One cool feature I want to mention is the quick connect to servers in Known hosts via SSH (Command Palette → type server name → Enter OR Nav Bar → Shell → Click on server name). WezTerm also has Workspaces – an alternative to tmux, but I haven't used it actively yet.

Here a gist with the final config from this article.

2024 in Review: Go to Rust and 1,394 Stars

2024-12-31T00:00:00+00:00

I have never written an annual report on my work before, but since I also spend time on open source projects, I decided to write one (maybe it will become a tradition).

Programming Languages and Tools

The most noticeable shift here was Rust, so it makes sense to start with it before the rest.

Rust

The main change in my work was the active use of Rust in the second half of the past year. I mainly write in Rust where I used to write in Go. Overall, the language is more complex than Go and sometimes confusing with the borrow checker, but in general, it is quite convenient to write – I can't say that the speed of writing code significantly decreases compared to other languages – in general, the approaches that need to be used are roughly the same for all languages.

What I like about Rust:

crates ecosystem – many libraries available and can be installed in one command
single binary file and low memory consumption
ability to write applications of various directions (web-app, tui, gui, parsers, etc.)

What I don't like about Rust:

slow Docker image builds (my solution)
a large number of sub-dependencies for web projects (any project on axum will contain 200+ dependencies)
use of C-binding in "core" libraries like rust-tls (ring) – complicates cross-compilation because you have to maintain a C toolchain as well (fixed through Zig build, but it looks monstrous)
lack of finished solutions for serde (default values, partial initialization)

It's nice to write various parsers in Rust, and then they can be coolly parallelized (with rayon or tokio), in Go I had a problems with this due to immutable strings and lack of GC control.

Main languages

The main languages remained the same: Python, NodeJS/TypeScript. I also actively used NextJS on several projects, overall I was satisfied – it's faster and easier to write frontends, but it's quite difficult to find problem's solutions sometimes (many Github issues, often they are open, and the comments contain answers that no longer work).

The main fail of the year in the NodeJS ecosystem is the update of ESLint from 8 -> 9 – completely incompatible API with all existing plugins, the configuration file became larger and more complex – it's not clear why the authors made this diversion. I also tried Binome as an alternative to ESLint / Prettier, but it's completely different and you need to redo a lot, and not everything you need is available at the moment, I continue to monitor this project.

I also tried Zig, but didn't write anything serious. Personally, I didn't understand why this language is needed. At the time I tried it, there was no even package manager yet. Obviously Zig is a competitor to C (not Rust), but C is now much richer in its toolkit and support in other languages. The only thing I can point out is zig build which can build C projects too and does it with less problems.

About LLM

I want to separately note about use of GPTs in work. I have an opportunity to do business tasks in the format of parsing and analyzing texts. What a couple of years ago was done with a bunch of regex, etc., can now be done through GPT and it works great! I tried using local Llama2, but it would have taken weeks to process my data on M2 chip. I redo the solution to use ChatGPT API and processed the data in a couple of days. It's good that all LLMs support openai-client API in Python, so the code almost didn't need to be rewritten. Among the disadvantages of LLama-like aka local running, I also want to note the lack of support for Function calling (to guarantee to bring the LLM response to the desired structure, in my case). The last time I did something serious was more than six months ago, maybe a lot has changed. I also tested gemma2, the inference speed was impressive, and the quality of the answers was not much worse.

As local runners, I used llama.cpp and mlx – the second one uses Mac resources better (checked through macmon), but new models come out a bit slower on it. Some people use ollama, but I didn't understand why, considering that it has an incompatible client with openai-client API and slower performance.

Blog

Initially, I ran my blog only on Medium, but it's not comfortable to write text there – it's much easier to write and edit in the same editor where I write code (plus my editor has Copilot). Therefore, I decided to move my blog to a static page generator. As an engine, I chose Zola (Rust-based, simple, and fast); I also tried Astro, but it's too big and complex for such a simple task. Overall, Zola is good, but it lacks some filters in templating, not enough flexibility in file structure, and not all popular languages are supported in code highlighting.

I left Medium as a fallback for old readers. I also made a cross-post on dev.to, but there are very few views. Medium is disappointed with the lack of the ability to automate cross-posting (they have a deprecated API, but it does not support updating articles and code highlights). Most likely, it's possible to reverse engineer how they save articles through their editor (changes are sent as diffs). Does anyone else need this besides me?

New projects

Last year I launched several new projects (in chronological order):

timewiz.cc (react, webapp) – a tool to find common working hours for distributed teams (an alternative to worldtimebuddy.com, I don't remember exactly why I made my own version). Score: fail.
ecloop (c, cli) – secp256k1 elliptic curve implementation for Apple chips. Mainly an educational project to understand cryptography, SIMD optimizations, etc. Added several standard use cases for finding Bitcoin keys by brute force, but mathematically this is pointless, so as I said earlier, this is more of an interesting educational project (and I wanted to publish a series of articles on cryptography, but didn't have time last year, I hope to do it this year). Some people use this project in hopes of finding something. I also had ideas to add Metal API support – but that takes a lot of time, so I switched to other projects. Score: okay.
macmon (rust, tui) – TUI program for monitoring MacOS performance (CPU/GPU/ANE usage, RAM, temps). My first public Rust project, a lot of reverse engineering of how powermetrics in MacOS works, C-bindings. The project is used both by users in TUI mode and by other projects in headless mode (JSON metrics). Score: success.
ghstats (rust, axum, htmx, webapp) – a small self-hosted service that collects and saves GitHub repo statistics, has several built-in views and an API to retrieve all statistics. The project started well, but then kind of fizzled out in terms of issues and feature requests, most likely users are satisfied with the current functionality. I would like to add more visualizations, but I haven't had time for that yet. The main thing for now is that the statistics are collected and saved, so something can be done with this in the future. Technically, I like that the service consumes only 10 MB of RAM. Score: okay.
ogp (rust, axum, api-service) – a small project for generating OGP articles preview as a service. Used in this blog (you can check by sending this article to your friends in a messenger or social network 😀). Worked with the SVG standard (I thought it was more adequate) and generating PNG from SVG. Score: more of fail.
compose-updater (python, api-service) – a project for updating the version of a docker image from CI in a closed infrastructure. I didn't find anything ready to solve this task. There is watchtower, but it's more about keeping up-to-date docker images, rather than updating a specific one, and also I couldn't run it with ECR repos – the author made a super minimalist image in which I couldn't enter through sh to debug. The second alternative is umputun/updater, but it's more about running different shell commands through a webhook. I chose Python because I didn't find a Rust crate for Docker API (most likely it exists, I just didn't understand which one is really good). Score: – (I didn't promote the project anywhere, I use it for my own purposes for now).

Existing projects

twscrape (python, lib) – a library for parsing data from X/Twitter. Mainly the project is in maintenance mode, I check its functionality every few months, and there have been no requests for new features.
apigen-ts (typescript, lib) – a library for generating TypeScript client OpenAPI spec. I continue to use it in my projects; I also see that there are a small number of other users. My relevance has slightly decreased due to the NextJS/TRPC stack, but in other configurations (e.g. FastApi + React), it is still relevant, so use it. Periodically there are requests to fix something or add a feature.

Abandoned projects

I also have several projects that I didn't finish and abandoned. I'll write about them here, maybe someone needs something from this, then I can return to on it.

anypost – cross-posting between static blog, Medium, Dev.to, Substack, and collecting joined article statistics. I abandoned the project due to the lack of an API at Medium – the only solution I found was through protocol reverse and session cookie.
bcloop – a program to walk Bitcoin-like blockchain and compile the final UTXO list. I abandoned it because I couldn't figure out how to effectively create a HashMap-like structure with external memory support to run on computers with small RAM (less than 32GB, LOL).
tg-watcher – a program that listens to all incoming messages in Telegram (chats & channels) and saves their history in a separate database with the ability to search through these messages later. I have the first version that just saves messages in json files, but I didn't develop the idea further.

Devices

I'm not a big tech geek, but I want to say about a few devices which made my life easier:

MeLE Fanless Quieter 4C – a small fanless PC for homelab, I use it as a native Linux via SSH for testing various programs and keep some cron jobs on it for data collection. It's cool that it's silent because it has no fan, but it heats up a lot under load. The performance of the N100 chip surprised me – only 2x slower than M2 chip (tested with ecloop bench, single core).
Promate HDMI-PD100 – nice cable format to connect a monitor and my Mac. One cable is needed to charge and transmit the video stream. Before that, I used hubs, but it's a big block. I'm glad I see fewer wires.

At Glance

Over the year, 8 articles were published, totaling 9039 words (~45 minutes to read), and gained 5203 views. Top 3 articles:

Table sorting and pagination with HTMX – 1714 views
macmon – Mac Usage Monitor in Terminal – 953 views
Setting up WireGuard VPN at AWS with Terraform – 642 views

Data aggregated from this site (with umami), Medium, and Dev.to. Script.

Public GitHub projects – totaling 312 commits, 49 issues closed, 43 releases created, and 1394 stars gained. Top-3 repos:

twscrape – +802 stars / +52515 views
macmon – +420 stars / +6899 views
ghstats – +128 stars / +2757 views

Data aggregated with ghstats.

I hope I will see a different top with new projects this year 😀

Donations

In twscrape repository I was asked to make it possible to send coffee as donations. Over the year I received 38 cups of coffee from 10 people. I spend these coffees on an annual Parallels subscription, which allows me to test some apps on Windows. Also, I partially paid from this fund for MeLE and VPS, which I used to test apps as well. Many thanks for supporting my work!

Creating OG Images for Static Pages with Rust

2024-11-30T00:00:00+00:00

OGP (Open Graph Protocol) is an HTML meta-tags specification for websites to make links look nice in third-party services such as Facebook, X, LinkedIn or chat apps. OGP also allows to specify a link to page thumbnail that will be shown as preview when link shared, which can increase clickability.

Some blog engines have built-in functions for this. I use Zola (static page generator) to create pages for this blog. Unfortunately, Zola does not have an ability to generate OG images. This can be achieved with external tools or scripts during the build process. Another approach to use edge functions (such as vercel/og) to generate thumbnails on the fly. However, this approach relies on a specific service provider, which I want to avoid.

I thought, why not use OG image generation as a service? I searched around and found og-image-generator by sagarhani.

The functionality of og-image-generator worked well for me, and I used it for this blog for some time. However, the project seems abandoned, and I'm unsure how the author feels about additional load on their service. I decided it shouldn't be too hard to create a similar service myself — a single endpoint for generating images with search params. Moreover, when I looked into the source code of og-image-generator, I noticed that it launches Chrome to render an image from HTML, which felt like significant overhead to me.

Coding

I need an endpoint that accepts: article title, URL/domain, author name and avatar via search parameters. Lately, I've been using Rust for most of my pet projects, so I decided to build this service with Rust as well. I added a route in axum and implemented a handler that generates SVG images, using some basic math to calculate the positioning of elements. The size of the OG image is predefined as 1200x630, following the specefication. SVG allows the use of a <style> tag, making it easy to customise default font-family.

The only challenge I encountered was creating a mask for the image. I initially thought SVG might have some syntactic sugar for this, but it essentially works the same way as masks in Photoshop.

Example of SVG generation (maud is an HTML template engine on Rust macros):

use maud::{html, Markup};

fn render_svg(title: &str, author: &str, url: &str, photo: &str) -> Markup {
  let (w, h) = (1200, 630);

  let pic_r = 50;
  let pic_y = h - 128;
  let l1y = pic_y - (pic_r - 10) / 2 + 48 / 2 - 6;
  let l2y = pic_y - (pic_r - 10) / 2 + 32 / 2 + 6 + 32;

  html!(svg xmlns="http://www.w3.org/2000/svg" viewBox=(format!("0 0 {} {}", w, h)) width=(w) height=(h) {
    style { ("text { font-family: 'Open Sans', Arial, sans-serif; }" ) }

    rect x="0" y="0" width=(w) height=(h) fill="black" stroke="white" stroke-width="16" {}
    text x=(120) y=(200) font-weight="700" font-size="72" fill="white" { (title) }
    text x=(200) y=(l1y) font-weight="700" font-size="48" fill="white" { (author) }
    text x=(200) y=(l2y) font-weight="400" font-size="32" fill="white" { (url) }
    (circle_avatar(photo, 128, pic_y, pic_r))
  })
}

fn circle_avatar(url: &str, cx: u32, cy: u32, radius: u32) -> Markup {
  html!({
    defs {
      clipPath id="circle-photo" { circle cx=(cx) cy=(cy) r=(radius) {} }
    }

    g clip-path="url(#circle-photo)" {
      image href=(url) x=(cx-radius) y=(cy-radius) width=(radius * 2) height=(radius * 2) {}
      circle cx=(cx) cy=(cy) r=(radius) stroke="white" stroke-width="4" fill="none" {}
    }
  })
}

And then the SVG image is generated in the Axum handler:

use axum::extract::{Query, Request};
use axum::http::{header, StatusCode};
use axum::response::IntoResponse;
use std::collections::HashMap;

async fn svg_handler(req: Request) -> Res<impl IntoResponse> {
  let qs: Query<HashMap<String, String>> = Query::try_from_uri(req.uri())?;
  let title = qs.get("title").cloned().unwrap_or("This is default article title".to_string());
  let author = qs.get("author").cloned().unwrap_or("Author".to_string());
  let url = qs.get("url").cloned().unwrap_or("https://example.com".to_string());
  let photo = qs.get("photo").cloned().unwrap_or("https://gravatar.com/avatar/".to_string());

  let svg = render_svg(&title, &author, &url, &photo);
  Ok((StatusCode::OK, [(header::CONTENT_TYPE, "image/svg+xml")], svg))
}

This code will generate the following SVG:

The endpoint can accept title, author, url and photo in the search params, and these values will be passed to SVG template. That's pretty much it, I thought, deployed the service to fly.io, updated the blog links, went to check how new preview looked, and the thumbnail didn't display.

It turned out that OG images cannot be SVG (which makes sense, because then different mobile clients would need to be able to render it, which is harder then showing prepared image).

SVG to PNG

It's not a big deal, I thought. A lot things has already been rewritten in Rust, and there's probably a ready to use solution for SVG rendering, especially since I found several popular crates for this. But it turned out to be more complicated than expected. Rendering SVG to PNG is quite a complex task, and the SVG specification is huge, so libs not fully implement the specification. However, I only need basic SVG support. I tried several crates and settled on resvg. I wrote a rendering function:

use resvg::{tiny_skia, usvg};

fn render_png(svg: &str) -> Vec<u8> {
  let tree = {
    let mut opt = usvg::Options::default();
    opt.fontdb_mut().load_system_fonts();
    usvg::Tree::from_str(svg, &opt).unwrap()
  };

  let size = tree.size().to_int_size();
  let mut pixmap = tiny_skia::Pixmap::new(size.width(), size.height()).unwrap();
  resvg::render(&tree, tiny_skia::Transform::default(), &mut pixmap.as_mut());
  pixmap.encode_png().unwrap()
}

And updated handlers:

// in some reason axum not allow to pass Request as ref – code will not compile, so req.uri() used
async fn prepare_svg(uri: &axum::http::Uri) -> Res<String> {
  let qs: Query<HashMap<String, String>> = Query::try_from_uri(uri)?;
  let title = qs.get("title").cloned().unwrap_or("This is default article title".to_string());
  let author = qs.get("author").cloned().unwrap_or("Author".to_string());
  let url = qs.get("url").cloned().unwrap_or("https://example.com".to_string());
  let photo = qs.get("photo").cloned().unwrap_or("https://gravatar.com/avatar/".to_string());
  Ok(render_svg(&title, &author, &url, &photo).into_string())
}

async fn svg_handler(req: Request) -> Res<impl IntoResponse> {
  let svg = prepare_svg(req.uri()).await?;
  Ok((StatusCode::OK, [(header::CONTENT_TYPE, "image/svg+xml")], svg))
}

async fn png_handler(req: Request) -> Res<impl IntoResponse> {
  let svg = prepare_svg(req.uri()).await?;
  let png = render_png(&svg);
  Ok((StatusCode::OK, [(header::CONTENT_TYPE, "image/png")], png))
}

I run it, and generally, everything works, but the avatar doesn't show up.

Loading remote images

resvg itself cannot load images from remote URLs used in SVG, but it can render images from a base64-encoded data string. Okay, I'll load the image and convert it myself:

use base64::Engine;
use reqwest::header::{HeaderMap, HeaderValue};

async fn load_base64_image(url: &str) -> Res<String> {
  let ua = format!("{}/{}", env!("CARGO_PKG_NAME"), env!("CARGO_PKG_VERSION"));

  let mut headers = HeaderMap::new();
  headers.insert("User-Agent", HeaderValue::from_str(&ua)?);

  let client = reqwest::Client::builder()
    .default_headers(headers)
    .read_timeout(std::time::Duration::from_secs(10))
    .build()?;

  let rep = client.get(url).send().await?;

  let max_size = 1024 * 1024 * 5; // 5MB
  match rep.content_length() {
    None => return AppError::new("Image size is unknown"),
    Some(len) if len > max_size => return AppError::new("Image is too large"),
    _ => {}
  };

  let allowed = ["image/jpeg", "image/png", "image/webp", "image/svg+xml"];
  let mime = match rep.headers().get(header::CONTENT_TYPE) {
    None => return AppError::new("Content type is unknown"),
    Some(ct) if !allowed.contains(&ct.to_str().unwrap()) => {
      return AppError::new(&format!("Content type is not allowed: {:?}", ct))
    }
    Some(ct) => ct.to_str().unwrap().to_string(),
  };

  let rep = rep.bytes().await?;
  let rep = base64::engine::general_purpose::STANDARD.encode(rep);
  Ok(format!("data:{mime};base64,{rep}"))
}

async fn prepare_svg(uri: &axum::http::Uri) -> Res<String> {
  // ...
  let photo = qs.get("photo").cloned().unwrap_or("https://gravatar.com/avatar/".to_string());
  let photo = load_base64_image(&photo).await?;
  Ok(render_svg(&title, &author, &url, &photo).into_string())
}

I check generation again – everything is fine, the avatar is present in the final render:

Docker image

So, the service is ready. Now, just needed to pack code into a Docker image. It's pretty much the same as usual, except that fonts files should be added to image too. I usually use Alpine images, and fonts can be added via apk:

FROM alpine:latest
RUN apk add --no-cache ttf-opensans
WORKDIR /app
# ... copy and run bin file

Fin

I added a template and copied themes from sagarhani/og-image-generator. I also created a preview page where you can experiment with the generation. This is enough for my current needs, but if anyone needs more, Issues/PRs are welcome.

The service is available at: https://ogp.fly.dev
The code is on GitHub: https://github.com/vladkens/ogp

The service is currently running without any restrictions. There is also the option to run self-hosted version with ghcr.io/vladkens/ogp image.

To integrate this service into a static website, you just need to add corresponding <meta> tags with service link and replace token with your data.

<meta
  property="og:image"
  content="https://ogp.fly.dev/v0/png?title={title}&author={author}&photo={photo}&url={url}&theme={theme}"
/>

Usage example for this blog: link generation, meta tags. To see link sharing at work click on links below ⬇️💀

Deploy FastAPI application with SQLite on Fly.io

2024-10-21T00:00:00+00:00

Cloud solutions are good for medium and large projects, but too heavy for small personal projects. If you want to launch something small (a few api endpoins and a small repository), there are three options:

Use the same approaches as for "big" projects (AWS ECS/EKS, RDS), but they are redundant, and infrastructure code can be larger than code of the actual project. Also it's expensive (~$100).
Use serverless solutions (Lambda, Vercel). Most cloud providers have such solutions, but these services have difficulties with simple databases – they provide cheap vendor solutions (AWS) or require a managed database, which again is expensive (mostly nothing for serverless, ~$20 for DB)
Use VPS with Docker. It is cheap (~$5 for small machine) and almost no need to manage infrastructure, but deployments sucks (needs private or self-hosted registry, SSH access from CI).

I usually write my small applications using SQLite, it's a handy little single file database that works in any programming language and can be copied to local machine to analyze data for example. So I was looking for some middleware solution that combines the serveless approach, ease of deployment and ability to use SQLite and found Fly.io.

Setup

If you don't have an account in Fly.io – you need to create it. Also CLI tool called flyctl required to manage projects. Fly.io can be deployed both locally and from CI.

flyctl makes deploy from project's root folder from Dockerfile, which is cool, because same Dockerfile can be used in other systems. For play with Fly.io, I prepared a simple FastAPI project that stores state in database – generic url shortener with click counting.

Dockerfile:

FROM python:3.13-alpine
WORKDIR /app

COPY ./requirements.txt .
RUN pip install --no-cache-dir --upgrade -r requirements.txt

COPY . /app

ENV HOST=0.0.0.0 PORT=8080
EXPOSE ${PORT}
CMD uvicorn main:app --host ${HOST} --port ${PORT}

main.py:

import asyncio
import random
import string
from urllib.parse import urlparse

import aiosqlite
from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import RedirectResponse

DB_PATH = "/data/app.db"

app = FastAPI()

async def get_db() -> aiosqlite.Connection:
    if db := getattr(get_db, "_db", None):
        if db.is_alive:
            return db

    db = await aiosqlite.connect(DB_PATH, loop=asyncio.get_event_loop())
    db.row_factory = aiosqlite.Row

    qs = """
    CREATE TABLE IF NOT EXISTS links (
        created_at INTEGER DEFAULT (strftime('%s', 'now')),
        short_code TEXT PRIMARY KEY,
        full_url TEXT NOT NULL,
        clicks INTEGER DEFAULT 0
    )
    """

    await db.execute(qs)
    await db.commit()

    setattr(get_db, "_db", db)
    return db

def random_code(length=8) -> str:
    alphabet = string.ascii_letters + string.digits
    return "".join(random.choice(alphabet) for x in range(length))

def is_valid_url(url: str) -> bool:
    try:
        parts = urlparse(url)
        return all([parts.scheme, parts.netloc])
    except ValueError:
        return False

@app.post("/")
async def shorten(url: str, req: Request):
    if not is_valid_url(url):
        raise HTTPException(status_code=400, detail="Invalid URL")

    host = req.headers.get("host")
    if host is None:
        raise HTTPException(status_code=500, detail="Missing host header")

    short_code = random_code()
    db = await get_db()
    qs = "INSERT INTO links (short_code, full_url) VALUES (?, ?)"
    await db.execute(qs, (short_code, url))
    await db.commit()

    return f"https://{host}/{short_code}"

@app.get("/")
async def list_links():
    db = await get_db()
    qs = "SELECT short_code, full_url, clicks FROM links ORDER BY created_at DESC"
    async with db.execute(qs) as cursor:
        return await cursor.fetchall()

@app.get("/{short_code}")
async def redirect(short_code: str):
    db = await get_db()
    qs = """
    UPDATE links SET clicks = clicks + 1 WHERE short_code = ?
    RETURNING full_url
    """

    async with db.execute(qs, (short_code,)) as cursor:
        if row := await cursor.fetchone():
            return RedirectResponse(row["full_url"])

    raise HTTPException(status_code=404)

requirements.txt:

aiosqlite
fastapi
uvicorn

Deploy

To deploy our code, first we need to create a Fly.io project. This can be done either in the web interface or with flyctl. To create proejct with CLU tool in root folder (where code located) flyctl launch should be runned. This command will offer to select desired hardware and will create fly.toml file:

fly launch --build-only

You can modify project in future by changing parameters in this file or via web ui. The basic fly.toml looks fine, but SQLite requires Storage, which can be created with:

fly volumes create sqlite_data -s 1 -r ams

where -s 1 sets volume size to 1 GB (default is 3 GB), and -r is region in which volume will be created (use same region in which Fly.io project is created). You can always change storage size later.

The last thing to do is to add a mounts section to fly.toml, which attaches the volume to the application:

[mounts]
source = "sqlite_data"
destination = "/data"

sqlite_data is the name of the storage, /data is the path where volume will be connected. This is essentially same as docker run --mount source=sqlite_data,target=/data or corresponding Docker Compose section.

SQLite cannot be writable from more than one app, and Fly.io by default creates 2 instances for an app, so we can specify the number of replicas as one just in case:

fly scale count 1

All configurations are done now and we can deploy our app with command:

fly deploy

The app should boot successfully and the public DNS name will be printed to console. Now we can check it out by posting some url to shortener:

❯ curl -X POST 'https://fly-fastapi-sqlite.fly.dev/?url=https://example.com'
https://fly-fastapi-sqlite.fly.dev/8ydmfAcK

Then we can visit this link, it should redirect to https://example.com. Finally, we can check that clicks are updated:

❯ curl -s 'https://fly-fastapi-sqlite.fly.dev/' | jq
[
  {
    "short_code": "8ydmfAcK",
    "full_url": "https://example.com",
    "clicks": 1
  }
]

To check that database state saved between deployments, we can perform new deployment with fly deploy and check that links list remained same as above (1 link, 1 click).

Migrations

If you are using an external solution for migrations, rather than running them from code when app starts, then only way to run migration is to put it in Dockerfile as part of the RUN command.

Backup

We can connect to machine with fly ssh console and then in /data folder interact with database file. Also we can copy database file to local machine with:

fly ssh sftp get /data/app.db ./app-backup.db

Conclusion

Fly.io is a simple and convenient service for deploying applications. Deploy works from Docker Containers, additional services include PSQL, Redis, S3 like storage (unlike Vercel). It's cheap, the cheapest service costs 3 dollars (1 shared CPU / 256 MB) – maybe even less, if you have little traffic – container shuts down after a few minutes without activity and automatically turns on when traffic appears.

On downside, there is no built-in solution for scheduled tasks – instead, the official solution is to set up a separate server with crontab and run tasks from it – it's kind of creepy.

Setting up WireGuard VPN at AWS with Terraform

2024-10-08T00:00:00+00:00

All resources in AWS work inside private VPC. Sometimes you may need to access these resources from local computer (e.g. to interact with database). Some resources, like RDS, have the option to enable public access to them – but this is unsecure. Of course you can configure Security Group to allow access to public resource only from allowed IPs to make this setup a bit better, but still in this case all your colleagues must to have static IPs, which is not always true.

Objective

We need to provide a quick and easy solution to give access to internal AWS VPC resources for our team. In general, there are two ways to do this:

Bastion / Jump Host – EC2 instance with SSH access for each member
Private VPN – EC2 instance with VPN config for each member

In both cases we need to to setup a custom EC2 instance, so second variant seems more convenient to me.

Preparation

In addition to the traditional Terraform client, we will need WireGuard CLI tools to generate a key pair. You can install WireGuard tools on macOS via brew:

brew install wireguard-tools

Initial Terraform

First, as usual, we need to configure Terraform Provider and get VPC and Public Subnet data where we want to host our VPN server. On this step you can export VPC / Public Subnet from another layer or pass them via variables. I will access my VPC and Public Subnet by name.

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.70" }
  }
}

provider "aws" {
  profile = "default"   # Change this to your AWS profile
  region  = "us-east-1" # Change this to your AWS region
}

# MARK: Getting VPC and Subnets

locals {
  vpc_name = "my_vpc" # Change this to yor VPC name
}

data "aws_vpc" "vpc" {
  filter {
    name   = "tag:Name"
    values = [local.vpc_name]
  }
}

data "aws_subnets" "public" {
  filter {
    name   = "vpc-id"
    values = [data.aws_vpc.vpc.id]
  }

  tags = { Tier = "Public" } # This is custom tag
}

WireGuard Server

To setup WireGuard server we need to know AMI ID of OS will be running on EC2 instance, prepare key pairs for the server and for client (peers), create a security group to control network access, create and assing Elastic IP to be sure that the server IP will be same.

Generate key pairs

First we need to create key pairs which can be generated with this command:

bash -c 'priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"'

In output first line is privkey, second is pubkey.

I will generate it twice, one for server, second for first client (peer). Then for each new peer you will need to generate new key pairs. I will same generated keys to secrets.yaml file, but you can use different ways to store secrets.

Generate secrets for server and first peer (do not use this keys in your setup):

❯ bash -c 'priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"'
0McsAe5Y/oupwJeju+94ZFl1bmp2KIX2bBQGk0cSWXQ=
KeV+BlZMfCgRhdfaHqpdPp23Nu/otir+6VR02DER7lU=

❯ bash -c 'priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"'
SK71fHXpTo2cGKvlLLTlu+0AqpwaNWG30rzGXFIryk0=
wf3j1JLhXw/u6iDZf9Qhq0lU34exF8mrHdGzz/u+xFU=

Then save generated keys to secrets.yaml

wg_server:
  privkey: 0McsAe5Y/oupwJeju+94ZFl1bmp2KIX2bBQGk0cSWXQ=
  pubkey: KeV+BlZMfCgRhdfaHqpdPp23Nu/otir+6VR02DER7lU=

wg_peers:
  - name: user1
    addr: 172.16.16.1
    privkey: SK71fHXpTo2cGKvlLLTlu+0AqpwaNWG30rzGXFIryk0=
    pubkey: wf3j1JLhXw/u6iDZf9Qhq0lU34exF8mrHdGzz/u+xFU=

Setup script

Then we need create WireGuard setup script which will be runned during EC2 creation. User data will be generated from this script and populated from secrets.yaml file. So create templates directory with two files in it: wg-init.tpl (general setup script) and wg-peer.tpl (peer info).

#!/bin/bash
sudo apt update && apt -y install net-tools wireguard

# https://github.com/vainkop/terraform-aws-wireguard/blob/master/templates/user-data.txt
sudo mkdir -p /etc/wireguard
sudo cat > /etc/wireguard/wg0.conf <<- EOF
[Interface]
PrivateKey = ${wg_privkey}
ListenPort = ${wg_port}
Address = ${wg_cidr}
PostUp = sysctl -w -q net.ipv4.ip_forward=1
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o ENI -j MASQUERADE
PostDown = sysctl -w -q net.ipv4.ip_forward=0
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o ENI -j MASQUERADE

${wg_peers}
EOF

# Make sure we replace the "ENI" placeholder with the actual network interface name
export ENI=$(ip route get 8.8.8.8 | grep 8.8.8.8 | awk '{print $5}')
sudo sed -i "s/ENI/$ENI/g" /etc/wireguard/wg0.conf

sudo wg-quick up wg0
sudo systemctl enable wg-quick@wg0

[Peer]
# ${peer_name}
PublicKey = ${peer_pubkey}
AllowedIPs = ${peer_addr}

Next we need to prepare user data script in terraform file from this templates.

# MARK: Config

locals {
  wg_cidr = "172.16.16.0/20" # CIDR of WireGuard Server
  wg_port = 51820            # Port of WireGuard Server
  secrets = yamldecode(file("secrets.yaml"))

  userdata = templatefile("templates/wg-init.tpl", {
    wg_cidr    = local.wg_cidr
    wg_port    = local.wg_port
    wg_privkey = local.secrets.wg_server.privkey
    wg_peers = join("\n", [
      for p in local.secrets.wg_peers :
      templatefile("templates/wg-peer.tpl", {
        peer_name   = p.name
        peer_pubkey = p.pubkey
        peer_addr   = p.addr
      })
    ])
  })
}

Server setup

Then we need to select Ubuntu AMI, create Security Group and Elastic IP.

# MARK: Server dependencies

data "aws_ami" "ubuntu" {
  owners      = ["099720109477"] # Canonical
  most_recent = true
  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-arm64-server-*"]
  }
}

resource "aws_eip" "wireguard" {
  tags = { Name = "wireguard" }
}

resource "aws_security_group" "wireguard" {
  name        = "${local.vpc_name}-wireguard"
  description = "SG for WireGuard VPN Server"
  vpc_id      = data.aws_vpc.vpc.id
  ingress {
    from_port   = local.wg_port
    to_port     = local.wg_port
    protocol    = "udp"
    cidr_blocks = ["0.0.0.0/0"]
    description = "Allow WireGuard Traffic"
  }
  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Next we can setup a server and assing Elastic IP to it:

resource "aws_instance" "wireguard" {
  count                       = 1
  ami                         = data.aws_ami.ubuntu.id
  instance_type               = "t4g.nano"
  subnet_id                   = data.aws_subnets.public.ids[0]
  vpc_security_group_ids      = [aws_security_group.wireguard.id]
  user_data                   = local.userdata
  user_data_replace_on_change = true
  tags                        = { Name = "wireguard" }
}

resource "aws_eip_association" "wireguard" {
  instance_id   = aws_instance.wireguard[0].id
  allocation_id = aws_eip.wireguard.id
}

Client configuration

To access to WireGuard server we need to have WireGuard client app and valid client config with .conf extension.

We can generated this configs with Terraform Templates as well. So first, lets add new template file called client-conf.tpl:

[Interface]
Address = ${client_addr}/32
PrivateKey = ${client_privkey}
DNS = ${client_dns}

[Peer]
Endpoint = ${server_addr}
PublicKey = ${server_pubkey}
AllowedIPs = ${client_routes}
PersistentKeepalive = 25

And add this code to generate client config to out terraform file:

# MARK: Export client configuration

locals {
  # https://docs.aws.amazon.com/vpc/latest/userguide/AmazonDNS-concepts.html#AmazonDNS
  client_dns = ["169.254.169.253", "1.1.1.1", "1.0.0.1"]
  client_routes = [
    data.aws_vpc.vpc.cidr_block, # All IPs in the VPC
    "169.254.169.253/32",        # AWS DNS
    # "0.0.0.0/32",                # Route all traffic through VPN (uncomment if you want this)
  ]
}

resource "local_file" "peer_conf" {
  for_each = { for index, p in local.secrets.wg_peers : p.name => p }
  filename = "generated/${each.value.name}.conf"
  content = templatefile("templates/client-conf.tpl", {
    server_addr    = "${aws_eip.wireguard.public_ip}:${local.wg_port}",
    server_pubkey  = local.secrets.wg_server.pubkey,
    client_addr    = each.value.addr,
    client_privkey = each.value.privkey,
    client_dns     = join(",", local.client_dns),
    client_routes  = join(",", local.client_routes),
  })
}

Execute

Finally we can run terraform init & terraform apply. In less then a minute VPN server will be created and user client will be exported to generated/user1.conf. Then we can share this file with our teammates.

To check VPN connection working fine you can export .conf into WireGuard client. Then for example I check my RDS instance before VPN enabled and after.

❯ dig +short my-rds.4oexsiqydp8q.us-east-1.rds.amazonaws.com
ec2-34-226-xx-xx.compute-1.amazonaws.com.
34.226.xx.xx

❯ dig +short my-rds.4oexsiqydp8q.us-east-1.rds.amazonaws.com
ec2-34-226-xx-xx.compute-1.amazonaws.com.
10.10.20.80

Adding clients

Adding new client configuration is quite simple. We need to generate new key pair:

❯ bash -c 'priv=$(wg genkey); pub=$(echo $priv | wg pubkey); printf "$priv\n$pub\n"'
ALqgaUiWd0rtcLza2K143x5RwS0Y5zwh3YwHx/L7nV0=
u1yQmh/G7n+S8aCp0PRuRuAuacmqYNzHLCPOcl9aS28=

Then update secrets.yaml file:

wg_peers:
  # ...
  - name: user2
    addr: 172.16.16.2
    privkey: ALqgaUiWd0rtcLza2K143x5RwS0Y5zwh3YwHx/L7nV0=
    pubkey: u1yQmh/G7n+S8aCp0PRuRuAuacmqYNzHLCPOcl9aS28=

And run terraform apply again. Server will be re-created (!) with new configuration. Note that existing clients will be disconnected during deploy.

Summary

WireGuard provides easy solution for creating VPN server. We created cheap and fast setup to access to internal AWS VPC with ability to create different clients configurations, which can be shared with out team members.

PS. Source files of this article can be found here.

Multi-arch Docker Builds for Rust 3.8x Faster

2024-10-04T00:00:00+00:00

Rust allows programs to run very fast and memory efficient, but it has a cost – compile time. In web development, it is standard practice to deliver programs as Docker images and then run them in Kubernetes / Amazon ECS / Docker Compose / etc. After the popularity of ARM processors in recent years, programmers have been faced with the additional step of preparing multi-arch Docker images (meaning that the same image should be able to run natively on both x86-64/amd64 and aarch64/arm64 processors).

Preparing Docker images for interpreted languages is usually not a problem – Docker Buildx is used, which in turn runs inside the emulator and builds the image for each architucture natively. NodeJS / Python just install dependencies via npm / pip, copy the project code, does a little polishing and that's pretty much it. Even in compiled Go this approach works fine because Go has an extensive standard library. In the case of Rust, compiling even a simple web application is a rebuilding of the “universe” – almost any web application requires: HTTP-server, HTTP-client (which in turn requires a library to work with cryptography for https), asynchronous runtime (tokie, etc), serialization/desirialization (serde, serde_json); which in Rust should be installed as external libraries (crates) and should be compiled each time when program builded.

Although Rust compiler has a lot of work to do, it can do it quickly. Even not strongest CI, will build an average project in a couple of minutes. However, this is true only in case of building on native architecture (for example building amd64 binary on an amd64 processor). As soon as we need to build a multi-arch image, we have to do emulation and the compilation speed drops dramatically. For example, on my simple public project – ghstats, building multi-arch image from scratch took about 50 minutes, when same build for native architecture takes 2-3 minutes.

Building time can be optimized by proper usage of Docker layers, so that the step with rebuilding dependencies occurs only when they are actually changed. So only the first build will be long and following builds will be fast. Unfortunately, Rust infrastructure have a problem in this point – any change in Cargo.toml file (for example version number) will invalidate Docker layer and triggers full rebuild of all dependencies.

Problem definition

So there are two problems with building multi-arch Docker images for Rust projects:

Layer invalidation and full rebuild on any change in Cargo.toml
Very slow multi-arch build due to emulation

Better dependencies building

Easiest way to solve the first problem is to use cargo-chef which was created exacly for it. cargo-chef converts Cargo.toml & Cargo.lock into a special recipe.json file that will remain unchanged until project dependencies are unchanged. Then we can use this json file to cache expensive Docker layer that compiles dependencies.

Dockerfile with cargo-chef will use multi-stage build splited to 5 parts:

Installing cargo-chef and build dependencies (OpenSSL, linux-headers, etc)
Preparing recipe.json file with project dependencies description
Installing & builiding project dependencies from recipe.json
Building whole project
Copy binaries to runtime image and final polishing

Example of Dockerfile with cargo-chef and multi-stage build (I use alpine in my projects, but you can use other base-images).

# (1) installing cargo-chef & build deps
FROM rust:alpine AS chef
WORKDIR /app
RUN cargo install --locked cargo-chef
RUN apk add --no-cache musl-dev openssl-dev

# (2) prepating recipe file
FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

# (3) building project deps, cache magic happen on COPY command
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --recipe-path recipe.json --release

# (4) actual project build
COPY . .
RUN cargo build -r

# (5) runtime image, you can use any base image you want
FROM alpine:latest AS runtime
WORKDIR /app
COPY --from=builder /app/target/release/prog /app/prog
CMD "/app/prog"

This approach will speed up Docker image build many times while project dependencies are not changed. Because Docker builds stages separately, planner stage will be executed each time (but cargo-chef is fast!) and builder stage will be partially cached until recipe.json file are same.

If you need single architecture build this will work fine. But if you need multi-arch images, this approach still be slow.

Multi-arch image with cross-compilation

Multi-arch builds using Docker Buildx run very slow due to QEMU emulation. If we get rid of emulation, compilation will be at full speed. Rust has built-in cross-compilation for other architectures, so we adapt it in Docker build.

Cross-compilation in Rust itself works fine, but some crates are based on C libraries (OpenSSL, SQLite, etc). Compiling and linking C code is quite complicated and not always clear (usually you have to look for error codes somewhere on Stack Overflow or Github Issues until you get the right set of compilers/header files added). There is another tool that surprisingly solves the problem of cross-compiling C code very well – Zig (actually this is programming language, but they have build toolchain as well).

To connect Zig build toolchain with Rust I will use cargo-zigbuild crate. I other Docker file looks pretty same to our cargo-chef variant, expect I added second target in Cargo to build and cargo build replaced with cargo zigbuild.

# (1) this stage will be run always on current arch
# zigbuild & Cargo targets added
FROM --platform=$BUILDPLATFORM rust:alpine AS chef
WORKDIR /app
ENV PKG_CONFIG_SYSROOT_DIR=/
RUN apk add --no-cache musl-dev openssl-dev zig
RUN cargo install --locked cargo-zigbuild cargo-chef
RUN rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-musl

# (2) nothing changed
FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

# (3) building project deps: need to specify all targets; zigbuild used
FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --recipe-path recipe.json --release --zigbuild \
    --target x86_64-unknown-linux-musl --target aarch64-unknown-linux-musl

# (4) actuall project build for all targets
# binary renamed to easier copy in runtime stage
COPY . .
RUN cargo zigbuild -r \
    --target x86_64-unknown-linux-musl --target aarch64-unknown-linux-musl && \
    mkdir /app/linux && \
    cp target/aarch64-unknown-linux-musl/release/prog /app/linux/arm64 && \
    cp target/x86_64-unknown-linux-musl/release/prog /app/linux/amd64

# (5) this staged will be emulated as was before
# TARGETPLATFORM usage to copy right binary from builder stage
# ARG populated by docker itself
FROM alpine:latest AS runtime
WORKDIR /app
ARG TARGETPLATFORM
COPY --from=builder /app/${TARGETPLATFORM} /app/prog
CMD "/app/prog"

In general that all. Such build approach will work faster. For my project:

initial build 50 min → 13 min (3.8x)
code updated 7 min → 3 min (2.3x)

Additional performance can be achieved by moving first step into separate base image, then our main Docker image will not need to build cargo-zibguild and cargo-chef (-2 min from initial build).

Note: If you already know your server architecture (if you build your docker for a specific server), you can remove everything link to the other architecture. This will lead you to have performance improvement in both initial build and updates.

https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide

apigen-ts – Simple TypeScript API Client Generator

2024-09-04T00:00:00+00:00

OpenAPI / Swagger is a good way to describe API contracts between backend / frontend or different services. Nowadays frontend is almost always TypeScript + some framework (React, Vue, Angular, etc). Backend in turn can be in NodeJS (and then in principle you can share contracts with code reuse), or in another language (e.g. Python / Go / Scala / Rust) and then code reuse and API synchronization becomes a problem.

OpenAPI can solve this problem through codegen – backend should qualitatively describe the available API methods and data types, after that you can use the client generation utility and get ready typed TypeScript code.

OpenAPI has a standard tool for such generation, but it has a number of problems:

Tool is written in Java, which requires special software to be installed on dev computer / CI
Generate client is extensive and consists of many .ts files, which makes it not user friendly (in my opinion)

In JavaScript ecosystem it is common to use npm (or yarn / pnpm – whatever you like more) as a package manager. But the point is the same: packages are written in JavaScript, installed from one place, versioned.

In npm you can find several alternative apigen utilities written in TypeScript which provides a better development experience. I used one of those too (swagger-typescript-api), until they changed interfaces a few times and I had to fix my several projects. I tried to find an alternative in npm, but all available generators (as of late 2023) also have their own problems: generate a lot of files, are difficult to configure, break compatibility between versions, require to install additional packages.

Meet apigen-ts

I have a couple of projects in NestJS / FastAPI in support mode, for which I just need a stable API client generation utility that is simple and won't change often. So I wrote my own TypeScript HTTP API generation package from Swagger specification – apigen-ts.

Originally I wrote apigen-ts for personal projects, but then I share it to my friends and they also started to use apigen-ts in their projects. Now I want to share it with the community, maybe it will be useful for someone too. So library features:

supports Swagger / OpenAPI v2 / v3 / v3.1 (parsed with redocly)
generates a typed TypeScript client
single output file, single api class
exports data models for use in your code
automatic code formatting with prettier (peer dependency)
ability to automatic dates parsing

Usage

Usage is as simple as possible and is divided into two steps: generating a client from specification (local file or remote url; yaml or json), using the generated typed client in the app.

Generation of an arbitrary client can be performed by command:

npx -y apigen-ts https://petstore3.swagger.io/api/v3/openapi.json ./api-client.ts

Or you can install apigen-ts as a dev dependency (npm i -D apigen-ts) and create a task to generate the client in package.json:

{
  "name": "my-frontend",
  "scripts": {
    "apigen": "apigen-ts http://127.0.0.1:3000/docs-json ./api-client.ts --parse-dates && tsc --noEmit"
  }
}

Then use: npm run apigen or yarn apigen.

Generated file contains a client class named ApiClient and TypeScript types used in API endpoints. Next you can use them in app code in the following way:

import { ApiClient, Pet } from "./api-client";

const api = new ApiClient({
  baseUrl: "https://example.com",
  headers: { Authorization: "secret-token" },
});

// GET https://example.com/pet/findByStatus?status=sold
await api.pet.findPetsByStatus({ status: "sold" }); // -> Pet[]

Generator tries to create method names in format like {domain}.{operation}. The types used in Swagger spec can be imported directly into code (in example above Pet is generated type). This approach makes it easy to track changes in contract and fix the application code where types are updated or methods are changed (of course if you describe your Swagger well). Just run new api generation and check frontend code for errors afterwards with TypeScript.

Also apigen-ts works well in CI pipelines, when on new commits to the backend part you run frontend api client generation and check code compatibility.

The generated client uses native fetch API (does not require additional packages) and can be used both in browser and NodeJS environment. If you use prettier in your project, the generated client will be automatically formatted according to project settings.

More configuration options and customization abilities can be in readme.

Conclusion

Backend ↔ frontend (or backend ↔ backend) communication is monotonous and repetitive work from project to project. Do not waste time and effort on manual writing of api clients. Instead use simple tools to automate process and keep code quality.

You can support this project with a star ⭐️ or leave an issue / feature request on Github page.

Table sorting and pagination with HTMX

2024-08-18T00:00:00+00:00

I recently tried HTMX for my new project – ghstats – a dashboard of Github repository traffic in a single interface for longer than 14 days. This project was planned as a self-hosted service, so I was thinking about a really simple and memory-less tech stack. Last time I played with Rust, so I decided to use it instead of NodeJS / Python. Of course, if I'm generating static HTML on the server side, I have two options for implementing table sorting and pagination: use query parameters and do everything on backend, or use some JavaScript to call data from the backend and render the table on the client side. But HTMX offers a new, third way: write all the logic on the backend and replace the necessary parts of HTML with just a few tag attributes. Let's see how it works.

Initial setup

To demonstrate table sorting and pagination, I'll be using the same stack I used for my service: axum as a backend framework and maud for HTML templating (it's a DSL over Rust templates).

Let's create new project with:

cargo init htmx-example && cd-html-example

Then install dependencies:

cargo add tokio axum maud --features tokio/full,maud/axum

and do initial setup in src/main.rs file:

use axum::{response::IntoResponse, routing::get, Router};
use maud::html;

async fn index_page() -> impl IntoResponse {
  html! {
    h1 { "Hello, World!" }
  }
}

#[tokio::main]
async fn main() {
  let service = Router::new().route("/", get(index_page)).into_make_service();
  let listener = tokio::net::TcpListener::bind("127.0.0.1:8080").await.unwrap();
  axum::serve(listener, service).await.unwrap();
}

Then we cant run this with cargo watch -x run and open http://127.0.0.1:8080 in the browser. "Hello, World!" should be displayed.

Creating a static table

Let's add crates to generate random data:

cargo add fake rand rand_chacha --features fake/derive

and define simple Contact struct (like our application is simple CMS).

use fake::faker::company::en::*;
use fake::faker::internet::en::*;
use fake::faker::name::en::*;
use fake::faker::phone_number::en::*;
use fake::{Dummy, Fake, Faker};
use rand::prelude::*;
use rand_chacha::ChaCha8Rng;

#[derive(Debug, Clone, Dummy)]
struct Contact {
  id: u64,

  #[dummy(faker = "Name()")]
  name: String,

  #[dummy(faker = "CompanyName()")]
  company: String,

  #[dummy(faker = "FreeEmail()")]
  email: String,

  #[dummy(faker = "PhoneNumber()")]
  phone: String,
}

fn get_contacts(n: usize) -> Vec<Contact> {
  let mut items = Vec::with_capacity(n);
  let mut rng = ChaCha8Rng::seed_from_u64(42);
  for _ in 0..n {
    items.push(Faker.fake_with_rng(&mut rng));
  }
  items
}

async fn index_page() -> impl IntoResponse {
	let contacts = get_contacts(100);
	println!(">> {:?}", contacts[0]);
	println!(">> {:?}", contacts[1]);

  html! {
    h1 { "Hello, World!" }
  }
}

Now on each request same random contacts will be generated, so we can use this data in our table. get_contacts function emulates database query in real environment.

I will also add base layout for page with PicoCSS framework to have nice look:

fn base(html: maud::Markup) -> impl IntoResponse {
  html! {
    html {
      head {
        meta charset="utf-8" {}
        title { "Contacts" }

        link rel="stylesheet" href="https://unpkg.com/@picocss/pico@2.0" {}
        script src="https://unpkg.com/htmx.org@2.0" {}
      }
      body class="container" {
        { (html) }
      }
    }
  }
}

And finally update index_page to render contact table:

async fn index_page() -> impl IntoResponse {
  let contacts = get_contacts(100);

  base(html! {
    table {
      thead {
        tr {
          th { "ID" }
          th { "Name" }
          th { "Company" }
          th { "Email" }
          th { "Phone" }
        }
      }
      tbody {
        @for contact in contacts {
          tr {
            td { (contact.id) }
            td { (contact.name) }
            td { (contact.company) }
            td { (contact.email) }
            td { (contact.phone) }
          }
        }
      }
    }
  })
}

We can request again http://127.0.0.1:8080 and see our table with random contacts (which on each request at same).

Sorting table

Now is the time to add sorting to our table. We will use HTMX to send requests to the server and get sorted HTML back. First, we need to add sorting logic to our backend. We will use query parameters to pass sorting column and direction.

To make it easier, I will add struct TableFilter which will represent sorting and pagination parameters, for code simplicity I will parse Request manually rather than using Query extractor and serde crate:

use std::collections::HashMap;
use axum::extract::{Query, Request};

#[derive(Debug)]
struct TableFilter {
  sort: String,
  order: String,
  page: u32,
  per_page: u32,
}

fn parse_table_filter(req: &Request) -> TableFilter {
  let qs: Query<HashMap<String, String>> = Query::try_from_uri(req.uri()).unwrap();
  let sort = qs.get("sort").unwrap_or(&"name".to_string()).to_string();
  let order = qs.get("order").unwrap_or(&"desc".to_string()).to_string();
  let page = qs.get("page").unwrap_or(&"1".to_string()).parse().unwrap();
  let per_page = qs.get("per_page").unwrap_or(&"10".to_string()).parse().unwrap();
  TableFilter { sort, order, page, per_page }
}

Next, we need to create a function to render th with HTMX attributes to handle sorting:

fn th(title: &str, qs: &TableFilter) -> maud::Markup {
  let id = title.to_lowercase();
  let order = if qs.sort == id && qs.order == "desc" { "asc" } else { "desc" };
  let url = format!("/?sort={}&order={}&page=1&per_page={}", id, order, qs.per_page);

  html! {
    th scope="col" style="cursor: pointer;"
      hx-trigger="click"
      hx-get=(url)
      hx-target="#contacts_table"
      hx-swap="outerHTML"
    {
      (title)
      @if qs.sort == id {
        span style="margin-left: 0.5em;" {
          @if qs.order == "asc" { "↑" } @else { "↓" }
        }
      }
    }
  }
}

Update get_contacts to accept TableFilter and sort items / do pagination:

fn get_contacts(n: usize, qs: &TableFilter) -> (Vec<Contact>, u32) {
  let mut items: Vec<Contact> = Vec::with_capacity(n);
  let mut rng = ChaCha8Rng::seed_from_u64(42);
  for _ in 0..n {
    items.push(Faker.fake_with_rng(&mut rng));
  }

  items.sort_by(|a, b| {
    let cmp = match qs.sort.as_str() {
      "id" => a.id.cmp(&b.id),
      "name" => a.name.cmp(&b.name),
      "company" => a.company.cmp(&b.company),
      "email" => a.email.cmp(&b.email),
      "phone" => a.phone.cmp(&b.phone),
      _ => a.id.cmp(&b.id),
    };

    if qs.order == "asc" {
      cmp
    } else {
      cmp.reverse()
    }
  });

  let pages = (items.len() as f64 / qs.per_page as f64).ceil() as u32;
  let range_s = (qs.page as usize - 1) * qs.per_page as usize;
  let range_e = qs.page as usize * qs.per_page as usize;
  let items = items[range_s..range_e].to_vec();

  (items, pages)
}

And finally, we can update our index_page to use TableFilter and th:

async fn index_page(req: Request) -> impl IntoResponse {
  let qs = parse_table_filter(&req);
  let (contacts, pages) = get_contacts(100, &qs);

  base(html! {
  	// id added to tell HTMX where to put new data
    table id="contacts_table" {
      thead {
        tr {
          (th("ID", &qs))
          (th("Name", &qs))
          (th("Company", &qs))
          (th("Email", &qs))
          (th("Phone", &qs))
        }
      }
      // same as before
    }
  })
}

That's it! Now we can sort our table by clicking on the column header. HTMX will send a request to the server with sorting parameters and replace the table with the new sorted data. Let's me explain HTMX attributes:

hx-trigger - event which will trigger the request, in our case it's click
hx-get - URL to send the request and HTTP method (can be GET or POST)
hx-target - CSS selector to replace with new data
hx-swap - how to replace the target, in our case it's outerHTML which will replace the whole table (by default it's inserting new data into the target)

All this attributes handled automatically by HTMX, so we just need to include HTMX script in <head>.

Pagination

In previous step we already parse pagination filters to struct, so lets add code to render pagination links. I will use code from my another react project, which originally from StackOverflow answer:

// https://stackoverflow.com/a/70263913
fn calc_pagination(page: usize, total: usize, len: usize) -> Vec<Option<usize>> {
  use std::cmp::{max, min};

  let len = if len == 0 { 5 } else { len };
  let total = max(total, page);
  let start =
    max(1, min(page as isize - ((len - 3) as isize / 2), total as isize - len as isize + 2))
      as usize;
  let end = min(total, max(page + (len - 2) / 2, len - 1));

  let mut result = Vec::new();

  if start > 2 {
    result.push(Some(1));
    result.push(None);
  } else if start > 1 {
    result.push(Some(1));
  }

  for i in start..=end {
    result.push(Some(i));
  }

  if end < total - 1 {
    result.push(None);
    result.push(Some(total));
  } else if end < total {
    result.push(Some(total));
  }

  result
}

Let's define two more helper function which will render pagination link and delimiter (like ...):

fn pagination_link(qs: &TableFilter, i: u32) -> maud::Markup {
  let url = format!("/?sort={}&order={}&page={}&per_page={}", qs.sort, qs.order, i, qs.per_page);
  html! {
    li {
      a href=(url)
        hx-trigger="click"
        hx-get=(url)
        hx-target="#contacts_table"
        hx-swap="outerHTML"
      { (i) }
    }
  }
}

fn pagination_delim() -> maud::Markup {
  html! {
    li {
      span { "..." }
    }
  }
}

And finally we can write pagination function, which will return list of links and delimiters:

fn pagination(pages: u32, qs: &TableFilter) -> maud::Markup {
  let items = calc_pagination(qs.page as usize, pages as usize, 5);

  html!(
    nav {
      ul {
        @for item in items {
          @if let Some(i) = item {
            (pagination_link(qs, i as u32))
          } @else {
            (pagination_delim())
          }
        }
      }
    }
  )
}

async fn index_page(req: Request) -> impl IntoResponse {
	// ...
	base(html! {
		// ..
		// add pagination call after table
		(pagination(pages, &qs))
	})
}

Partial HTML updates

Before this step, we always replaced the whole page with new HTML content. This can be done without HTMX at all, so what the point of using it? It's right, we have very simple layout before. Let's make it more complex to show nice feature of HTMX – partial updates.

First of all let's move table generation into separate function:

fn get_contacts_table(contacts: Vec<Contact>, qs: &TableFilter, pages: u32) -> maud::Markup {
  html! {
  	// id moved from table to div
  	div id="contacts_table" {
	    table {
	      thead {
	        tr {
	          (th("ID", &qs))
	          (th("Name", &qs))
	          (th("Company", &qs))
	          (th("Email", &qs))
	          (th("Phone", &qs))
	        }
	      }
	      tbody {
	        @for contact in contacts {
	          tr {
	            td { (contact.id) }
	            td { (contact.name) }
	            td { (contact.company) }
	            td { (contact.email) }
	            td { (contact.phone) }
	          }
	        }
	      }
	    }

	    (pagination(pages, &qs))
   	}
  }
}

Now we can call this code in any other place and have contacts table. Now let's add some navigation and other content to our main page:

async fn index_page(req: Request) -> impl IntoResponse {
  let qs = parse_table_filter(req);
  let (contacts, pages) = get_contacts(1000, &qs);

  base(html! {
    nav {
      ul {
        li { a href="/dashboard" { "Dashboard" } }
        li { a href="/" { "Contacts" } }
        li { a href="/settings" { "Settings" } }
      }
    }

    h1 { "Contacts" }
    (get_contacts_table(contacts, &qs, pages))
  })
}

Right now in the application we have a simple navigation. But when we interact with the table, a full page will be returned, and the result will look like a nested page in a nested page... This is wrong. Let's fix this with HTMX (and this is actually where HTMX comes power).

When browser loads the page it load as regular page with default headers. When request send by HTMX it will set extra headers like HX-Request and HX-Target. We can use this headers to detect which part of HTML should be returned. Let's add this code to our index_page:

fn get_hx_target(req: &Request) -> Option<&str> {
  match req.headers().get("hx-target") {
    Some(x) => Some(x.to_str().unwrap_or_default()),
    None => None,
  }
}

async fn index_page(req: Request) -> impl IntoResponse {
  let qs = parse_table_filter(&req);
  let (contacts, pages) = get_contacts(1000, &qs);

  match get_hx_target(&req) {
    Some("contacts_table") => return get_contacts_table(contacts, &qs, pages).into_response(),
    _ => {}
  }

  base(html! {
  	// ... this part is the same
  })
  .into_response() // added into_response() to relax rust type inference
}

So in this code we check if hx-target header is present. If not – we return full page. If it's present and equal to contacts_table – we return only table. This is how partial updates works in HTMX. We can split page to smaller parts and update it independently.

Conclusion

HTMX is good utility for simple application which allows to add some interactivity without writing a lot of JavaScript code. It easy to use and can be integrated with any backend language, because not require any special server-side support. It's also very lightweight and can be used in any project without any additional dependencies.

PS. Source files of this article can be found here.

How to get macOS power metrics with Rust?

2024-08-13T00:00:00+00:00

In this post, I’m going to talk about how macmon works, specifically where it gets system metrics.

MacOS has a built-in powermetics utility that can show the current CPU core frequencies, utilisation, and power consumption. This program should take these values from the system somewhere. So the first thing to check is what shared libraries and function calls (symbols) are used.

There are several programs can do this:

otool — show list of shared libraries
nm — see the list of used symbols
strings — just prints all strings from a binary file, but it may be useful to understand what is going on in the program.

> otool -L /usr/bin/powermetrics
 /usr/lib/libIOReport.dylib (compatibility version 1.0.0, current version 1.0.0)
 /usr/lib/libpmsample.dylib (compatibility version 1.0.0, current version 2.0.0)
 /usr/lib/libpmenergy.dylib (compatibility version 1.0.0, current version 2.0.0)
 /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
...

> nm -a /usr/bin/powermetrics
/usr/bin/powermetrics (for architecture arm64e):
  ...
  U _IOPMCopyPowerStateInfo
  U _IORegistryEntryCreateCFProperties
  U _IORegistryEntryCreateCFProperty
  U _IORegistryEntryFromPath
  ...

This data provides some starting point. Next I just did a search on Github to see how these libs might have been used in other repositories.

Getting CPU / GPU Usage

The first thing I found is that macOS has an IOReport shared library that returns a lot of hardware information. Including it can do sampling and return CPU/GPU & energy usage.

IOReport operates on a subscription basis. It is possible to select interesting data channels (or all available) and perform sampling on it in order to obtain actual metrics. The channel has a group / subgroup name (sometimes may not exist). I’m interesting in “Energy Model”, “CPU Stats” / “CPU Core Performance States” and “GPU Stats” / “GPU Performance States”. These values were found by comparing available channels names with the list of those strings used in powermetrics (strings /usr/bin/powermetrics).

To subscribe to a channel and receive updates, it is first necessary to get internal channel struct as CFDictionary using IOReportCopyChannelsInGroup and merge all the dicts into one using IOReportMergeChannels. Then merged dictionary can be used to create an subscription using IOReportCreateSubscription. Once subscription object obtained, updates can be received with IOReportCreateSamples and IOReportCreateSamplesDelta. All together it looks something like this:

// simplified code without error handling
use std::ptr::null;

let mut channels = vec![];
channels.push(IOReportCopyChannelsInGroup("Energy Model", null(), 0, 0, 0));
channels.push(IOReportCopyChannelsInGroup("CPU Stats", "CPU Core Performance States", 0, 0, 0));

let chan = channels[0];
for i in 1..channels.len() {
  IOReportMergeChannels(chan, channels[i], null());
}

let size = CFDictionaryGetCount(chan);
let chan = CFDictionaryCreateMutableCopy(kCFAllocatorDefault, size, chan);

let mut subs: MaybeUninit<CFMutableDictionaryRef> = MaybeUninit::uninit();
IOReportCreateSubscription(std::ptr::null(), chan, s.as_mut_ptr(), 0, std::ptr::null());
subs.assume_init();

let sample1 = IOReportCreateSamples(subs, chan, null());
std::thread::sleep(std::time::Duration::from_millis(100)); // 100ms
let sample2 = IOReportCreateSamples(subs, chan, null());

let rs = IOReportCreateSamplesDelta(sample1, sample2, null());
CFShow(rs);

// rs is CFDictionary, so we can read metrics by keys from it

The basic dict functions are exported from core_foundation crate. IOReport functions are C-bindings to a private macOS API:

#[link(name = "IOReport", kind = "dylib")]
extern "C" {
  fn IOReportCopyChannelsInGroup(a: CFStringRef, b: CFStringRef, c: u64, d: u64, e: u64) -> CFDictionaryRef;
  fn IOReportMergeChannels(a: CFDictionaryRef, b: CFDictionaryRef, nil: CFTypeRef);
  fn IOReportCreateSubscription(a: CVoidRef, b: CFMutableDictionaryRef, c: *mut CFMutableDictionaryRef, d: u64, b: CFTypeRef) -> IOReportSubscriptionRef;
  fn IOReportCreateSamples(a: IOReportSubscriptionRef, b: CFMutableDictionaryRef, c: CFTypeRef) -> CFDictionaryRef;
  fn IOReportCreateSamplesDelta(a: CFDictionaryRef, b: CFDictionaryRef, c: CFTypeRef) -> CFDictionaryRef;
}

Full implementation can be found here.

Note: For CPU I get values separately for each core. In general, there is also a group “CPU Complex Performance States” to get values for the whole cluster at once, but the group results sometimes show 100% load of the cluster, when in reality there is no load. This is caused by some internal bug. For GPU there is no information for each core separately, so the load information is obtained for the whole GPU at once — good luck no errors were noticed.

Parsing CPU / GPU Usage

IOReportCreateSamplesDelta returns an object with one IOReportChannels field, which containts array of metrics objects. Each metrics object have group and subgroup (on which subscription was created), and extra fields: channel name, value unit and value itself.

The values for “CPU Stats” / “GPU Stats” are returned in array of tuples (string, int):

string: frequency level name (first always IDLE)
int: time been on this level (should be in nanoseconds, but not sure)

In general, internally CPU can work at different frequencies and voltages. The values of these frequencies are known in advance and are always the same. Under load CPU works at higher levels (higher frequency, higher voltage). During idle work it constantly jumps between modes (in any case OS always has some background activity). Simple tasks are mostly executed on E-cluster, more complex ones are switched to P-cluster.

Each cluster has different DVFS config (Dynamic Voltage and Frequency Scaling), but all cores within same cluster uses same DVFS config. Same approach is used for GPUs.

Usually GUI interfaces show single number as current core frequency, but in reality it is calculated as average of time each core work on DVFS level. So the average frequency of a core can be calculated like this:

// predefined DVFS values for E-cluster
let frequencies = [600, 912, 1284, 1752, 2004, 2256, 2424]; // 7 items
// time was on each level (first value is idle)
let residencies = [17563069, 0, 5787276, 156965, 419037, 100647, 106031, 41735]; // 8 items

let total_time = residencies.iter().sum::<u64>();
let usage_time = residencies.iter().skip(1).sum::<u64>();

let mut freq = 0f64;
for i in 0..frequencies.len() {
 let percent = residencies[i + 1] as f64 / usage_time as f64;
 freq += percent * frequencies[i] as f64;
}

println!("E-cluster freq: {:.2} MHz", freq);

DVFS values for target system can be read from AppleARMIODevice. Full code can be found here. For M1/M2/M3 processors looks names are: voltage-states1-sram for E-Cluster, voltage-states5-sram for P-Cluster and voltage-states9 for GPU. For Max / Ultra chips is also exists voltage-states11-sram or something like this, because this chips have two or more P-Clusters but fortunately this values are same for all clusters of same group (at least for now).

Parsing Energy Usage

In the channel earlier, we also asked for metrics for the “Energy Model” group. These are easier to read because it’s just a number. Value unit can be mJ / uJ / nJ, so it just needs to be converted to Watts. This is easily done using the formula: P(W) = E(J) / t(s), where s is sampling time. Full code here.

let joules_raw = IOReportSimpleGetIntegerValue(metric_dict, 0) as f32;
let joules = match metric_unit.as_str() {
  "mJ" => joules_raw / 1e3f32,
  "uJ" => joules_raw / 1e6f32,
  "nJ" => joules_raw / 1e9f32,
  _ => Err(format!("Unknown energy unit: {}", metric_unit).into()),
};

let sec = (duration as f32 / 1000.0);
let watts = joules / sec;

Getting RAM Usage

Getting RAM / SWAP usage is done via libc in principle I think similar to any UNIX-like system. Nothing specific here, except to figure out how to calculate the value in bytes, since internally memory is measured in pages.

MacOS uses the vm_statistics64 structure, which has quite a few fields. By experimentation I found a value that is similar to what Activity Monitor shows. To get current memory host_statistics64 need to be called. For total memory available and SWAP info sysconf call is enough. Full code can be found here. Short example with libc crate:

let mut count: u32 = libc::HOST_VM_INFO64_COUNT as _;
let mut stats = std::mem::zeroed::<libc::vm_statistics64>();

libc::host_statistics64(
  libc::mach_host_self(),
  libc::HOST_VM_INFO64,
  &mut stats as *mut _ as *mut _, // cast to void (untyped) pointer
  &mut count,
);

let page_size_kb = libc::sysconf(libc::_SC_PAGESIZE) as u64;
let usage = page_size_kb * (
  + stats.active_count as u64
  + stats.inactive_count as u64
  + stats.wire_count as u64
  + stats.speculative_count as u64
  + stats.compressor_page_count as u64
  - stats.purgeable_count as u64
  - stats.external_page_count as u64);

Getting Temperature Values

This appeared to be the most complicated part of the program. Initially I wanted to get values for each core separately and show them in a separate tab in the interface. On macOS there are GUI programs (Stats, Macs Fan Control) that can do this, but after a detailed study I realised that the values in these programs are shown by guessing which sensor is responsible for which core.

Temperature value can be obtained from two places: IOHID for M1 on macOS 12–13 and SMC for M1/M2/M3 on macOS 14–15.

IOHID approach is simpler — it’s just a function call that returns a CFDictionary with values. There seems only one person in open-source who understands how it works, because all other repositories I found referenced it.

SMC (System Management Controller) is more complicated. It works through something similar to RPC calls with large structures where fields are optionally filled in depending on which RPC method called. SMC looks like a more classic approach, but it looks like Apple didn’t have time to implement all the functionality during migration to ARM architecture. To make macmon works everywhere, I had to implement both approaches.

IOHID implementation can be found here, there is really nothing to talk about technically, except that the parameter values are found "magically".

In SMC, it is possible to obtain a sensor’s value using a key. The key itself uses FourCC notation, which is essentially int32 but appears human-readable in code (for example, Tp01 -> 0x54703031, Tp02 -> 0x54703032, etc.). This key is used to obtain KeyInfo, from which KeyData can be retrieved. Therefore, to get the sensor value, two RPC calls are needed, which is inefficient and costly. Hence, it is necessary to cache the KeyInfo value beforehand.

The SMC initialisation process looks like: obtain all possible keys using the special key #KEY (0x234b4559), then filter out only those KeyInfo that start with Tp (CPU related) and Tg (GPU related), and then make requests only to them – resulting in one RPC call per sensor.

Since I do not know which sensors correspond to which core or other sensors on the chip, the program simply calculates average of all values per type.

SMC implementation can be found here (utility to perform RPC calls), here (initialisation) and here (actual metrics gathering).

Combine all together

There’s nothing special about this part. Once per second, the current metric values are obtained from sources described above. The only thing worth noting is that SMC is used by default, if its value is not available, then IOHID is used (fallback for M1 on macOS 13). Then these values are combined into a single structure and return to caller.

Then these values are aggregated in the "frontend" code and output to the console using ratatui crate or in plain text in raw mode. But this is already a material for a separate article.

You can try macmon with Homebrew:

brew install vladkens/tap/macmon

macmon – Mac Usage Monitor in Terminal

2024-08-06T00:00:00+00:00

When MacBook moved to ARM architecture (M1 processor), old system APIs stopped working and with them existing monitoring tools. However, macOS has a built-in console utility called powermetrics. It can show real-time resource consumption, but it does it in plain text.

asitop is the first utility that allows users to view resource and energy consumption visually in a terminal. But it works on top of powermetrics which requires sudo to work. The program is written in Python and has compatibility issues with new M-processors.

macmon attempts to solve these issues. Coded in Rust as a single binary file it uses an undocumented Apple API (the same API that uses powermetrics, but doesn’t require sudo) to get metrics on system utilization, power consumption and temperature.

Installation

brew install vladkens/tap/macmon

Features

No Sudo Required: Run seamlessly without administrative privileges.
Real-Time Metrics: Monitor CPU/GPU/ANE power usage live.
Detailed Utilisation: Track CPU usage per cluster and RAM/Swap usage.
Historical Data: View historical charts with average and max values.
Customizable Interface: Switch between six color variants.
Compact Design: Fits neatly in a small terminal window.
Built with Rust: Ensuring performance and reliability.

Interface Overview

The top line shows the chip model, number of Effective (E) and Performance (P) cores, number of GPU cores, and RAM.

The second line shows the chart of E and P cluster utilization of the CPU. MacOS uses E-cores for normal tasks, but when something complex needs to be done, process switches to P-cores.

The third line shows memory usage and GPU loading (during gaming, video processing or LLM running 😈).

And finally, the fourth line, the most interesting one, shows the power consumption for compute tasks, by system in general and separately for CPU, GPU and ANE (Apple Neural Engine). It also shows the average temperature of CPU and GPU.

macmon is an open-source program. You can view, star, request features or report bugs on Github.

REST API Deployment on AWS Lambda with Terraform

2023-10-12T00:00:00+00:00

In this article I will show the way how to deploy AWS Lambda using Terraform, with support for code updates in a single workflow.

The problem with AWS Lambda is that at the time of its creation we need to have prepared code in zip archive or Docker image uploaded to ECR, unlike ECS / Kubernetes service, where we just specify the path to the repository, which can be empty at the creation stage. So to make an AWS Lambda deploy using Terraform we will need to prepare the code in the same Terraform recipe.

In this article I will use the Docker image as source for AWS Lambda, as it is essentially the only one normal way to run any complex applications in Lambda that have external dependencies (libraries). There is a way to package the dependencies in a zip archive too, but if the architecture of the computer where the dependencies were built is different from the one where AWS Lambda runs, there will be problems. That’s why Docker is the most clear and reliable option.

So this article will explore the creation of such resources:

Simple HTTP server on Python with FastAPI
Docker image to that application
Creating ECR storage
Build & push Docker image in Terraform file (null_resource)
Creating AWS Lambda
Creating access to Lambda from public url
Updating Lambda on code change

Preparation

As usual, we start our work by creating a new project and declaring terraform dependencies:

mkdir lambda-demo; cd lambda-demo; touch main.tf

Describe Terraform dependencies in main.tf file

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "5.17.0" }
  }
}

provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

Creating HTTP application with FastAPI

Let’s create subfolder for our code and create 3 files:

mkdir code; touch code/app.py code/requirements.txt code/Dockerfile

Next, in the app.py file add the application code:

import os
import uvicorn
from fastapi import FastAPI
from mangum import Mangum


app = FastAPI()


@app.get("/")
def index():
    return "Hello, from AWS Lambda!"


handler = Mangum(app, lifespan="off")

if __name__ == "__main__":
    uvicorn_app = f"{os.path.basename(__file__).removesuffix('.py')}:app"
    uvicorn.run(uvicorn_app, host="0.0.0.0", port=8000, reload=True)

This code runs the FastAPI service, so we can write basically as much complex API as we want. Mangum is a wrapper for the Lambda API, it does all the request processing logic in Lambda internally, allowing FastAPI to work normally. The code in the __name__ == "__main__" section is needed to run the application during development — we can write our API locally as usual.

The following library should be added to requirements.txt:

fastapi==0.103.2
mangum==0.17.0
uvicorn==0.23.2

And finally the Dockerfile to build an image for our application:

FROM public.ecr.aws/lambda/python:3.11
ENV PYTHONUNBUFFERED=1

COPY requirements.txt ./
RUN pip3 install -r requirements.txt

COPY app.py ./
CMD [ "app.handler" ]

Creating ECR

We have the application ready, now we need to run it in AWS Lambda. First of all, we need to create a resource for ECR image. All the work below continues in the main.tf file.

# --- ECR ---

resource "aws_ecr_repository" "api" {
  name                 = "lambda-api"
  image_tag_mutability = "MUTABLE"
  force_delete         = true

  image_scanning_configuration {
    scan_on_push = true
  }
}

Build & Push Docker image

The next step is to build the image directly in the Terraform recipe. For this purpose null_resource will be used.

# --- Build & push image ---

locals {
  repo_url = aws_ecr_repository.api.repository_url
}

resource "null_resource" "image" {
  triggers = {
    hash = md5(join("-", [for x in fileset("", "./code/{*.py,*.txt,Dockerfile}") : filemd5(x)]))
  }

  provisioner "local-exec" {
    command = <<EOF
      aws ecr get-login-password | docker login --username AWS --password-stdin ${local.repo_url}
      docker build --platform linux/amd64 -t ${local.repo_url}:latest ./code
      docker push ${local.repo_url}:latest
    EOF
  }
}

data "aws_ecr_image" "latest" {
  repository_name = aws_ecr_repository.api.name
  image_tag       = "latest"
  depends_on      = [null_resource.image]
}

The local-exec part is a classic bash script that logs into ECR, builds and flushes the image. We just wrote it in a Terraform file directly.

The more interesting part is the triggers. It is needed to rebuild the image every time the files in the code folder are changed. The files are selected by mask, so if you need other extensions you should add them to the list.

And the last block is to get information about the last loaded image in ECR. We will need this next to update AWS Lambda.

Creating Lambda resource

As is typical in AWS, we need permissions to perform various actions. Lambda is no exception and needs a separate role as well. Let’s create a new role with access to Lambda functions and the ability to write logs.

# --- IAM Role ---

resource "aws_iam_role" "lambda" {
  name = "lambda"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect    = "Allow"
      Action    = "sts:AssumeRole"
      Principal = { Service = "lambda.amazonaws.com" }
    }]
  })
}

resource "aws_iam_role_policy_attachment" "lambda_logs" {
  role       = aws_iam_role.lambda.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

Next, we create a resource for the Lambda function:

# --- Lambda ---

resource "aws_cloudwatch_log_group" "api" {
  name              = "/aws/lambda/api"
  retention_in_days = 14
}

resource "aws_lambda_function" "api" {
  function_name    = "api"
  role             = aws_iam_role.lambda.arn
  image_uri        = "${aws_ecr_repository.api.repository_url}:latest"
  package_type     = "Image"
  source_code_hash = trimprefix(data.aws_ecr_image.latest.id, "sha256:")
  timeout          = 10

  environment {
    variables = {}
  }

  depends_on = [
    null_resource.image,
    aws_iam_role_policy_attachment.lambda_logs,
    aws_cloudwatch_log_group.api,
  ]
}

Lambda public endpoint

Finally we need to create public endpoint. CORS block required to access API from browser. In this article all values is wildcard, but you can change it as your requirements, like: allow_origins = ["https://my-domain.com"]

# --- Lambda Endpoint ---

resource "aws_lambda_function_url" "api" {
  function_name      = aws_lambda_function.api.function_name
  authorization_type = "NONE"

  cors {
    allow_credentials = true
    allow_origins     = ["*"]
    allow_methods     = ["*"]
    allow_headers     = ["date", "keep-alive"]
    expose_headers    = ["keep-alive", "date"]
    max_age           = 86400
  }
}

output "api_url" {
  value = aws_lambda_function_url.api.function_url
}

Deploying

Initialise terraform and apply the configuration:

terraform init
terraform validate && terraform apply

Once this is complete, we will see a link to AWS Lambda:

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Outputs:

api_url = "https://xxx.lambda-url.us-east-1.on.aws/"

Let’s try:

https $(terraform output -raw api_url)
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 25
Content-Type: application/json
Date: Thu, 12 Oct 2023 13:19:40 GMT
X-Amzn-Trace-Id: root=1-6527e150-3c0aed5e335681cb7e120cde;sampled=0;lineage=1da281dc:0
x-amzn-Remapped-content-length: 25
x-amzn-RequestId: a560d83b-40c4-4c27-a381-0bb630547a4a

"Hello, from AWS Lambda!"

It’s works!

Updating code and deploy new version

Since we have prepared all the necessary invalidations above, we simply update the app.py file and re-run terraform apply. I’ll just add v2 to the welcome text. Check it:

New application version available on same URL!

Note: https is binary from httpie.

Next, depending on requirements, we can add AWS Lambda to Route53 (so that there is a link to our domain), configure restriction policy. But these are topics for another article. Good luck with your AWS journey!

AWS ECS Cluster on EC2 with Terraform

2023-09-15T00:00:00+00:00

This short guide will describe how to create an AWS ECS Cluster on EC2 in 2023. Basically there are already articles and code samples on the Internet, but some products in AWS are becoming obsolete (like Launch Configuration) and Terraform is undergoing API changes. So the purpose of this article is to show how to run ECS Cluster on EC2 with Terraform today.

This article will look at how to create a Terraform configuration to provide such resources:

VPC with public subnet
Internet Gateway to connect to the global Internet
Security groups for EC2 Node & ECS Service
Auto-scaling group for ECS cluster with Launch Templates
Publish image to ECR
ECS cluster with task and service definition
Load Balancer to public access & scale ECS Service

Preparation

To get started, you need to create a new Terraform project. Let’s create new directory and main.tf file.

mkdir ecs-ec2-demo; cd ecs-ec2-demo; touch main.tf

In the Terraform file, it is required to declare the necessary dependencies and initialise the project.

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "5.17.0" }
  }
}

provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

Creating a VPC

Let’s make a new VPC & two public subnets.

# --- VPC ---

data "aws_availability_zones" "available" { state = "available" }

locals {
  azs_count = 2
  azs_names = data.aws_availability_zones.available.names
}

resource "aws_vpc" "main" {
  cidr_block           = "10.10.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true
  tags                 = { Name = "demo-vpc" }
}

resource "aws_subnet" "public" {
  count                   = local.azs_count
  vpc_id                  = aws_vpc.main.id
  availability_zone       = local.azs_names[count.index]
  cidr_block              = cidrsubnet(aws_vpc.main.cidr_block, 8, 10 + count.index)
  map_public_ip_on_launch = true
  tags                    = { Name = "demo-public-${local.azs_names[count.index]}" }
}

Then create Internet Gateway:

# --- Internet Gateway ---

resource "aws_internet_gateway" "main" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "demo-igw" }
}

resource "aws_eip" "main" {
  count      = local.azs_count
  depends_on = [aws_internet_gateway.main]
  tags       = { Name = "demo-eip-${local.azs_names[count.index]}" }
}

And after it public route table:

# --- Public Route Table ---

resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
  tags   = { Name = "demo-rt-public" }

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }
}

resource "aws_route_table_association" "public" {
  count          = local.azs_count
  subnet_id      = aws_subnet.public[count.index].id
  route_table_id = aws_route_table.public.id
}

Now it is possible to run terraform apply and a new VPC with two public subnets will appear in AWS. Each subnet will placed in different Availability Zone (AZ) and have access to the Internet via IGW.

Note: Load Balancer requires at least two subnets created in different Availability Zones (AZ).

Creating a scalable ECS Cluster

The next step is to create a complete ECS Cluster.

# --- ECS Cluster ---

resource "aws_ecs_cluster" "main" {
  name = "demo-cluster"
}

The code above will create an empty cluster, but in order for the cluster to run containers it needs EC2 instances on which it will run these containers.

AWS can automatically create these EC2 instances and scaling them based on load, but for this we need to define Launch Template & Autoscaling Group. To implement this we need to create such resources:

IAM Role for ECS Node
Security Group (SG) for ECS Node
Launch Template (describes EC2 instance)
Autoscaling Group (AG)
Capacity Provider to connect ECS Cluster & Autoscaling Group

IAM Role & Security Group for ECS EC2 Node

Let’s start from creating IAM Role for ECS Node:

# --- ECS Node Role ---

data "aws_iam_policy_document" "ecs_node_doc" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "ecs_node_role" {
  name_prefix        = "demo-ecs-node-role"
  assume_role_policy = data.aws_iam_policy_document.ecs_node_doc.json
}

resource "aws_iam_role_policy_attachment" "ecs_node_role_policy" {
  role       = aws_iam_role.ecs_node_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"
}

resource "aws_iam_instance_profile" "ecs_node" {
  name_prefix = "demo-ecs-node-profile"
  path        = "/ecs/instance/"
  role        = aws_iam_role.ecs_node_role.name
}

Then create simple Security Group for ECS Node which allow outgoing traffic (its required to pull image to start service later)

# --- ECS Node SG ---

resource "aws_security_group" "ecs_node_sg" {
  name_prefix = "demo-ecs-node-sg-"
  vpc_id      = aws_vpc.main.id

  egress {
    from_port   = 0
    to_port     = 65535
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Launch Template

Next step is to create Launch Template for EC2 Instance.

# --- ECS Launch Template ---

data "aws_ssm_parameter" "ecs_node_ami" {
  name = "/aws/service/ecs/optimized-ami/amazon-linux-2/recommended/image_id"
}

resource "aws_launch_template" "ecs_ec2" {
  name_prefix            = "demo-ecs-ec2-"
  image_id               = data.aws_ssm_parameter.ecs_node_ami.value
  instance_type          = "t2.micro"
  vpc_security_group_ids = [aws_security_group.ecs_node_sg.id]

  iam_instance_profile { arn = aws_iam_instance_profile.ecs_node.arn }
  monitoring { enabled = true }

  user_data = base64encode(<<-EOF
      #!/bin/bash
      echo ECS_CLUSTER=${aws_ecs_cluster.main.name} >> /etc/ecs/ecs.config;
    EOF
  )
}

A couple of comments on the code above:

ecs_node_ami is the recommended image for ECS (but it possible to use another image by passing different ami)
In user_data you is required to pass ECS cluster name, so AWS can register EC2 instance as node of ECS cluster

Autoscaling Group

Next step is to create autoscaling group (ASG) and connect it with Launch Template:

# --- ECS ASG ---

resource "aws_autoscaling_group" "ecs" {
  name_prefix               = "demo-ecs-asg-"
  vpc_zone_identifier       = aws_subnet.public[*].id
  min_size                  = 2
  max_size                  = 8
  health_check_grace_period = 0
  health_check_type         = "EC2"
  protect_from_scale_in     = false

  launch_template {
    id      = aws_launch_template.ecs_ec2.id
    version = "$Latest"
  }

  tag {
    key                 = "Name"
    value               = "demo-ecs-cluster"
    propagate_at_launch = true
  }

  tag {
    key                 = "AmazonECSManaged"
    value               = ""
    propagate_at_launch = true
  }
}

This ASG will create from 2 to 8 instances of EC2 using Launch Template. Name tag is optional and used here just to simplify to recognise instances connected to cluster. AmazonECSManaged tag is required by AWS.

At this point we can run terraform apply again and make sure the ASG is working. AWS should create two EC2 instances. The can be seen at AWS Console > EC2 > Instances. If you manually Terminate any of these instances, AWS will automatically create a new instance to match the configuration of ASG (in our case minimum 2).

Capacity Provider

The last step is to connect the ECS Cluster to the ASG group so that the cluster can use EC2 instances to deploy containers.

# --- ECS Capacity Provider ---

resource "aws_ecs_capacity_provider" "main" {
  name = "demo-ecs-ec2"

  auto_scaling_group_provider {
    auto_scaling_group_arn         = aws_autoscaling_group.ecs.arn
    managed_termination_protection = "DISABLED"

    managed_scaling {
      maximum_scaling_step_size = 2
      minimum_scaling_step_size = 1
      status                    = "ENABLED"
      target_capacity           = 100
    }
  }
}

resource "aws_ecs_cluster_capacity_providers" "main" {
  cluster_name       = aws_ecs_cluster.main.name
  capacity_providers = [aws_ecs_capacity_provider.main.name]

  default_capacity_provider_strategy {
    capacity_provider = aws_ecs_capacity_provider.main.name
    base              = 1
    weight            = 100
  }
}

Run terraform apply again and after it we should see Capacity Provider & EC2 instances connect to ECS cluster on AWS Console > ECS > demo-cluster > Infrastructure tab.

Creating ECS Service

The last step is to run the service on the ECS and configure heath check and autoscaling for it. Plan of Action:

Create Elastic Container Registry (ECR) & push image
Create IAM Role for ECS Task
Create ECS Task Definition
Create Security Group for service
Create ECS Service
Create Load Balancer

Elastic Container Registry (ECR)

The example will use helloword-http server, which we will run in our private ECR.

# --- ECR ---

resource "aws_ecr_repository" "app" {
  name                 = "demo-app"
  image_tag_mutability = "MUTABLE"
  force_delete         = true

  image_scanning_configuration {
    scan_on_push = true
  }
}

output "demo_app_repo_url" {
  value = aws_ecr_repository.app.repository_url
}

Let’s run terraform apply again. We should see output with repository URL in AWS. Now push helloword-http to ECR.

# Get AWS repo url from Terraform outputs
export REPO=$(terraform output --raw demo_app_repo_url)
# Login to AWS ECR
aws ecr get-login-password | docker login --username AWS --password-stdin $REPO

# Pull docker image & push to our ECR
docker pull --platform linux/amd64 strm/helloworld-http:latest
docker tag strm/helloworld-http:latest $REPO:latest
docker push $REPO:latest

After this image should be available in AWS Console > AWS ECR > demo-app and it can be used as source for ECS Task Definition.

Note: In case you are using ECS Nodes on ARM processors, use --platform linux/arm64

IAM Role for ECS Task

Next required step is to defined IAM roles for ECS task. Roles required to have access ECR, Cloud Watch, etc.

# --- ECS Task Role ---

data "aws_iam_policy_document" "ecs_task_doc" {
  statement {
    actions = ["sts:AssumeRole"]
    effect  = "Allow"

    principals {
      type        = "Service"
      identifiers = ["ecs-tasks.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "ecs_task_role" {
  name_prefix        = "demo-ecs-task-role"
  assume_role_policy = data.aws_iam_policy_document.ecs_task_doc.json
}

resource "aws_iam_role" "ecs_exec_role" {
  name_prefix        = "demo-ecs-exec-role"
  assume_role_policy = data.aws_iam_policy_document.ecs_task_doc.json
}

resource "aws_iam_role_policy_attachment" "ecs_exec_role_policy" {
  role       = aws_iam_role.ecs_exec_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}

Cloud Watch Logs

Next small step is to create Cloud Watch logs group to be able to see container logs.

# --- Cloud Watch Logs ---

resource "aws_cloudwatch_log_group" "ecs" {
  name              = "/ecs/demo"
  retention_in_days = 14
}

ECS Task Definition

At this point, we simply describe from where and how to launch the docker container.

Note: Task Definition is created per AWS account, not per ECS Cluster. So the family name must be unique.

# --- ECS Task Definition ---

resource "aws_ecs_task_definition" "app" {
  family             = "demo-app"
  task_role_arn      = aws_iam_role.ecs_task_role.arn
  execution_role_arn = aws_iam_role.ecs_exec_role.arn
  network_mode       = "awsvpc"
  cpu                = 256
  memory             = 256

  container_definitions = jsonencode([{
    name         = "app",
    image        = "${aws_ecr_repository.app.repository_url}:latest",
    essential    = true,
    portMappings = [{ containerPort = 80, hostPort = 80 }],

    environment = [
      { name = "EXAMPLE", value = "example" }
    ]

    logConfiguration = {
      logDriver = "awslogs",
      options = {
        "awslogs-region"        = "us-east-1",
        "awslogs-group"         = aws_cloudwatch_log_group.ecs.name,
        "awslogs-stream-prefix" = "app"
      }
    },
  }])
}

ECS Service

In order for the ECS service to be available within the cluster and have access to the Internet, a new Security Group must be created. ecs_svc allows all incoming and outgoing traffic. The service itself will be inside the subnet, without public ip, so it will not be accessible from the Internet. SG allows access to the service only for VPC network members.

# --- ECS Service ---

resource "aws_security_group" "ecs_task" {
  name_prefix = "ecs-task-sg-"
  description = "Allow all traffic within the VPC"
  vpc_id      = aws_vpc.main.id

  ingress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = [aws_vpc.main.cidr_block]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_ecs_service" "app" {
  name            = "app"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.app.arn
  desired_count   = 2

  network_configuration {
    security_groups = [aws_security_group.ecs_task.id]
    subnets         = aws_subnet.public[*].id
  }

  capacity_provider_strategy {
    capacity_provider = aws_ecs_capacity_provider.main.name
    base              = 1
    weight            = 100
  }

  ordered_placement_strategy {
    type  = "spread"
    field = "attribute:ecs.availability-zone"
  }

  lifecycle {
    ignore_changes = [desired_count]
  }
}

The ordered_placement_strategy will try to make sure that each service instance is equally distributed across Availability Zones. You can use a different placement strategy, read the AWS documentation about it.

If we run terraform apply at this step it will create & run new service. But because this service has no Public IP, we can't access to it from public Internet. But service accessible from internal VPC. So if we create a Bastion Host we can ping it.

To make the service available from the public network, as well as to be able to scale the service depending on the load we need to create Load Balancer.

Load Balancer (ALB)

ALB also needs a Security Group. We will allow any incoming traffic on ports 80 and 443. Next, the ALB will listen on port 80 and forward the traffic to our ECS Service. This is the simplest ALB setup sufficient for this article.

# --- ALB ---

resource "aws_security_group" "http" {
  name_prefix = "http-sg-"
  description = "Allow all HTTP/HTTPS traffic from public"
  vpc_id      = aws_vpc.main.id

  dynamic "ingress" {
    for_each = [80, 443]
    content {
      protocol    = "tcp"
      from_port   = ingress.value
      to_port     = ingress.value
      cidr_blocks = ["0.0.0.0/0"]
    }
  }

  egress {
    protocol    = "-1"
    from_port   = 0
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_lb" "main" {
  name               = "demo-alb"
  load_balancer_type = "application"
  subnets            = aws_subnet.public[*].id
  security_groups    = [aws_security_group.http.id]
}

resource "aws_lb_target_group" "app" {
  name_prefix = "app-"
  vpc_id      = aws_vpc.main.id
  protocol    = "HTTP"
  port        = 80
  target_type = "ip"

  health_check {
    enabled             = true
    path                = "/"
    port                = 80
    matcher             = 200
    interval            = 10
    timeout             = 5
    healthy_threshold   = 2
    unhealthy_threshold = 3
  }
}

resource "aws_lb_listener" "http" {
  load_balancer_arn = aws_lb.main.id
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.app.id
  }
}

output "alb_url" {
  value = aws_lb.main.dns_name
}

The output alb_url displays the ALB URL provided by AWS. We can ping this url to prove that ALB is working. If we try to ping alb_url right now it will respond with 503 code — that expected because ALB Target Group is empty right now.

Connect ECS Service to ALB

At the final stage, we need to update the aws_ecs_service configuration by adding a section with load_balancer.

# --- ECS Service ---

resource "aws_ecs_service" "app" {
  # ...

  depends_on = [aws_lb_target_group.app]

  load_balancer {
    target_group_arn = aws_lb_target_group.app.arn
    container_name   = "app"
    container_port   = 80
  }
}

Finally run terraform apply. ECS Service will be started and we can check it on ALB URL provided by AWS:

curl $(terraform output --raw alb_url) # Hello from ip-10-10-10-XXX
curl $(terraform output --raw alb_url) # Hello from ip-10-10-11-YYY

If we run CURL multiple times, it should write "Hello from HOSTNAME", the hostname will be different between calls, indicating that ALB is working and redirecting requests to different copies of the ECS Service.

Bonus: ECS Service Auto Scaling

# --- ECS Service Auto Scaling ---

resource "aws_appautoscaling_target" "ecs_target" {
  service_namespace  = "ecs"
  scalable_dimension = "ecs:service:DesiredCount"
  resource_id        = "service/${aws_ecs_cluster.main.name}/${aws_ecs_service.app.name}"
  min_capacity       = 2
  max_capacity       = 5
}

resource "aws_appautoscaling_policy" "ecs_target_cpu" {
  name               = "application-scaling-policy-cpu"
  policy_type        = "TargetTrackingScaling"
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageCPUUtilization"
    }

    target_value       = 80
    scale_in_cooldown  = 300
    scale_out_cooldown = 300
  }
}

resource "aws_appautoscaling_policy" "ecs_target_memory" {
  name               = "application-scaling-policy-memory"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_appautoscaling_target.ecs_target.resource_id
  scalable_dimension = aws_appautoscaling_target.ecs_target.scalable_dimension
  service_namespace  = aws_appautoscaling_target.ecs_target.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ECSServiceAverageMemoryUtilization"
    }

    target_value       = 80
    scale_in_cooldown  = 300
    scale_out_cooldown = 300
  }
}

Here is the end of ECS cluster configuration. Next it is possible to add a domain name, SSL certificate, https termination, routing traffic between different ECS services — but all this is stuff for other articles.

How to still scrape millions of tweets with twscrape

2023-07-02T00:00:00+00:00

Twitter is a great place to gather data and assess various trends. Many analytics teams have used this source for their models.

In February 2023, Twitter set unrealistic prices for its API, giving away crumbs of data for big bucks. Some started using libraries such as snscrape, which used web public APIs. But in April 2023, Twitter closed that option as well — making search only for authorized accounts.

But data can still be collected in much the same way as before using the authorised account approach.

Introduction to twscrape

Released in May 2023, it is a tool for scraping data from tweets. It collects data such as user profiles, follower lists and follower lists, likes and retweets, as well as keyword searches.

Getting Started with twscrape

Requirements: Python 3.10 or higher

Installing twscrape

pip install twscrape

Or development version with latest features:

pip install git+https://github.com/vladkens/twscrape.git

Adding accounts

twscrape needs Twitter accounts to work. Each account has a rather small limit on the use of APIs, after which some time is no way to make requests through that account. twscrape is designed to switch accounts when one of them is not available. In this way the data flow looks continuous to the user, although in fact the requests come from different accounts internally.

Accounts can be added in two ways, via software API or CLI command. Let’s use the CLI command:

# twscrape add_accounts <file_path> <line_format>
# line_format should have "username", "password", "email", "email_password" tokens
# tokens delimeter should be same as an file
twscrape add_accounts accounts.txt username:password:email:email_password

Note: It is possible to register a new account or buy on special websites, e.g. here.

You then have to go through the login procedure to get the tokens to request the API. It’s not a quick process, but it’s needed once after adding new accounts. Then the token is stored in the SQLite database and reused for subsequent queries.

twscrape login_accounts

Note: Not all accounts can pass authorisation because of the antifraud system. You can try logging into these accounts again later.

Using twscrape

You can use twscrape in two ways.

Using an CLI (terminal) and receive JSON object
Using Python API (usuful for custom data collection scripts)

Lets get some tweet details from CLI:

twscrape tweet_details 1674894268912087040

Result:

{
  "id": 1674894268912087000,
  "id_str": "1674894268912087040",
  "url": "https://twitter.com/elonmusk/status/1674894268912087040",
  "date": "2023-06-30 21:34:46+00:00",
  "user": {
    "id": 44196397,
    "id_str": "44196397",
    "url": "https://twitter.com/elonmusk",
    "username": "elonmusk",
    "displayname": "Elon Musk",
    "created": "2009-06-02 20:12:29+00:00",
    // ...
    "_type": "snscrape.modules.twitter.User"
  },
  "lang": "en",
  "rawContent": "This platform hit another all-time high in user-seconds last week"
  // ...
}

It’s that simple. The data format is almost the same as it was in snscrape. So if you already have some scripts to process the data, you can continue to use them without too much trouble.

Scrapping tweets from a text search query by Python API

Using the code below, we are scraping 5000 tweets between January 1, 2023, and May 31, 2023, with the keywords "elon musk". Then printing in console tweet id, tweet author and content.

import asyncio
from twscrape import API, gather
from twscrape.logger import set_log_level

async def main():
    api = API()

    q = "elon musk since:2023-01-01 until:2023-05-31"
    async for tweet in api.search(q, limit=5000):
        print(tweet.id, tweet.user.username, tweet.rawContent)


if __name__ == "__main__":
    asyncio.run(main())

A general execution time for the entire code could be anywhere between 5 mins — 10 mins, depending on the number of tweets fetched by your username or keyword query.

Working with raw API reponses

If you don’t have enough data provided by Tweet & User objects or want to get more insights from the data, then there is an option to use raw Twitter responses. Each method has a _raw version that returns the original data.

import asyncio
from twscrape import API, gather
from twscrape.logger import set_log_level

async def main():
    api = API()

    q = "elon musk since:2023-01-01 until:2023-05-31"
    async for rep in api.search_raw(q, limit=5000):
        # rep is httpx.Response object
        print(rep.status_code, rep.json())

if __name__ == "__main__":
    asyncio.run(main())

Or same from CLI:

twscrape search "elon musk since:2023-01-01 until:2023-05-31" --raw

List of available functions

search — just regular search by keywords
tweet_details — information about specific tweet
retweeters — list of users who retweet specific tweet
favoriters — list of users who like specific tweet
user_by_login — get user profile by login
user_by_id — get user profile by id
user_tweets — list of tweets of specific user (max 3200 tweets)
user_tweets_and_replies — list of tweets and replies of specific user
followers — list of followers of specific user
following — list of users the user is subscribed to
list_timeline — get all tweets of list

Found a bug or need new feature? Fill free to open an issues

More examples of use can be found on the project's Github page.

Better forms handling with react-use-form

2023-04-04T00:00:00+00:00

React-use-form is a powerful library that simplifies form handling in React applications. Its intuitive API and robust feature set make it an excellent choice for developers.

One of its standout features is the ability to work with custom components while minimizing re-renders of the main component, which is undeniably cool. This is achieved through the use of the Controller component, although its syntax may not be the most convenient.

In this article, I’ll demonstrate how to create a generic wrapper that makes it easy to use your own components with the full power of Controller.

Imagine we have a cool input component:

import { zodResolver } from "@hookform/resolvers/zod";
import { FC, useId } from "react";
import { Controller, useForm } from "react-hook-form";
import { z } from "zod";

type SuperInputProps = {
  label: string;
  value: string;
  onChange: (value: string) => void;
  error?: string;
};

const SuperInput: FC<SuperInputProps> = (props) => {
  const { label, value, onChange, error } = props;
  const id = useId();

  return (
    <div>
      <label htmlFor={id}>{label}</label>
      <input id={id} value={value} onChange={(e) => onChange(e.target.value)} />
      {error && <span>{error}</span>}
    </div>
  );
};

const schema = z.object({
  firstName: z.string().min(1, { message: "Required" }),
  lastName: z.string().min(1, { message: "Required" }),
});

const SuperForm: FC = () => {
  const { control } = useForm<z.infer<typeof schema>>({
    resolver: zodResolver(schema),
  });

  return (
    <form>
      <Controller
        control={control}
        name="firstName"
        render={({ field: { onChange, value }, fieldState: { error } }) => (
          <SuperInput label="First name" value={value} onChange={onChange} error={error?.message} />
        )}
      />
      <Controller
        control={control}
        name="lastName"
        render={({ field: { onChange, value }, fieldState: { error } }) => (
          <SuperInput label="Last name" value={value} onChange={onChange} error={error?.message} />
        )}
      />
    </form>
  );
};

The part about the Controller looks verbose. If you have a lot of forms in your application, it’s tedious to write it every time. Let’s make it better!

To do this, we first need to write some TypeScript helpers to get a list of fields in dot notation for a particular data type.

import { Primitive } from "react-hook-form";

type NestedImpl<K extends string | number, V, T> = V extends T
  ? K
  : V extends Primitive | Array<infer V>
    ? never
    : `${K}.${NestedByType<V, T>}`;

export type NestedByType<O, T> = {
  [K in keyof O]-?: K extends string ? NestedImpl<K, O[K], T> : never;
}[keyof O];

// test code

type A = { abc: string; xyz: number; nested1: { abc: number; xyz: string } };
type B = A & { arr: number[] };
type a1 = NestedByType<A, number>; // "xyz" | "nested1.abc"
type a2 = NestedByType<A, string>; // "abc" | "nested1.xyz"
type a3 = NestedByType<A, boolean>; // never
type a4 = NestedByType<B, number[]>; // "arr"
type a5 = NestedByType<B, string>; // "abc" | "nested1.xyz"

Now we need to make a small utility to comfortably retrieve errors from the form state for specific fields.

const getCtrlError = (formState: { errors: Record<string, any> }, name: string) => {
  const tokens = name.split(".");

  let value = formState.errors;
  for (const token of tokens) {
    if (value[token]) value = value[token];
  }

  return value && value.message ? (value.message as string) : undefined;
};

Finally, we can make our Controlled wrapper for custom components.

import { FC } from "react";
import { Control, Controller, FieldValues } from "react-hook-form";

const makeCtrl = <T extends Pick<T, "value" | "onChange"> & { error?: string }>(
  Component: FC<T>,
  options: { errorKey?: string } = {},
) => {
  // react-hook-form has FieldPathByValue but it slow, so we use NestedByType
  const Wrapped = <D extends FieldValues, P extends NestedByType<D, T["value"]>>(
    props: Omit<T, "value" | "onChange" | "error"> & {
      control: Control<D>;
      name: P;
    },
  ) => {
    const { control, name, ...rest } = props;
    const errorKey = options.errorKey ? `${name}.${options.errorKey}` : name.toString();

    return (
      <Controller
        control={control}
        // @ts-expect-error because of comment above
        name={name}
        render={({ field: { value, onChange }, formState }) => {
          const error = getCtrlError(formState, errorKey.toString());
          const props = { ...rest, value, onChange, error } as unknown as T;
          return <Component {...props} />;
        }}
      />
    );
  };

  return Wrapped;
};

I assume that the component for which we do wrapper has properties: value, onChange, error. These properties will be removed from the resulting component and automatically inserted into child component. But if you want to “remove” other fields, you can add them to generic type of makeCtlr function, e.g.: makeCtrl<SomeType, "field_to_remove_1" | "field_to_remove_2">(...).

And now it can be used somehow like this:

const SuperInputCtrl = makeCtrl(SuperInput); // create new component

const SuperForm: FC = () => {
  const { control } = useForm<z.infer<typeof schema>>({
    resolver: zodResolver(schema),
  });

  return (
    <form>
      <SuperInputCtrl control={control} name="firstName" label="First name" />
      <SuperInputCtrl control={control} name="lastName" label="Last name" />
    </form>
  );
};

Now it looks much shorter, but still safe (the wrapper only allows fields that match the value type in the input). The validation error is automatically filled in. Only the input is re-rendered during data entry, not the entire form.

Better date parsing for REST API with TypeScript

2022-12-15T00:00:00+00:00

A little trick in TypeScript to make it easier to work with date strings in the API. Let’s say we have a model like this on the client:

type User = {
  id: number;
  username: string;
  createdAt: Date;
};

And we transfer this data in textual protocol in JSON format, e.g. via REST API or Websockets. And we get data in a format similar to this:

GET /api/users/current
{
  "id": "1234567890",
  "username": "johndoe",
  "createdAt": "2020-01-01T00:00:00.000Z"
}

After that, you have to write data converters from the protocol model to the data model used in the application.

type UserDto = {
  id: number;
  username: string;
  createdAt: string;
};

const decodeUser = (dto: UserDto): User => {
  return {
    id: dto.id,
    username: dto.username,
    createdAt: new Date(dto.createdAt),
  };
};

const getCurrentUser = async (): Promise<User> => {
  const user = await get<UserDto>("/api/users/current");
  return decodeUser(user);
};

const u = await getCurrentUser();
console.log(typeof u.createdAt, u.createdAt instanceof Date); // string, false

Basically, if the API only has a few methods, it’s not a problem to write decoders from DTO models. But usually the API is much bigger. And why should we do it if we can do not? :)

In general, if we think about JSON format, it represents all basic types except date. So if we avoid the moment with string to date conversion, we won’t need any decodes at all. Let’s fix it.

import { parseISO } from "date-fns/esm";

const ISODateFormat = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d*)?(?:[-+]\d{2}:?\d{2}|Z)?$/;

const isIsoDateString = (value: unknown): value is string => {
  return typeof value === "string" && ISODateFormat.test(value);
};

const handleDates = (data: unknown) => {
  if (isIsoDateString(data)) return parseISO(data);
  if (data === null || data === undefined || typeof data !== "object") return data;

  for (const [key, val] of Object.entries(data)) {
    // @ts-expect-error this is a hack to make the type checker happy
    if (isIsoDateString(val)) data[key] = parseISO(val);
    else if (typeof val === "object") handleDates(val);
  }

  return data;
};

And now we can update our http client to automatically turn date strings into a native date object.

const http = <T>(url: string, config?: RequestInit) => {
  return fetch(url, config).then((x) => handleDates(x.json()) as Promise<T>);
};

const get = <T>(url: string) => {
  return http<T>(url);
};

const getCurrentUser = (): Promise<User> => {
  return http<User>("/api/users/current");
};

const u = await getCurrentUser();
console.log(typeof u.createdAt, u.createdAt instanceof Date); // object, true

// or if you use axios you can make it event better

axios.interceptors.response.use((rep) => {
  handleDates(rep.data);
  return rep;
});

const getCurrentUser = async (): Promise<User> => {
  const rep = await axios.get<User>("/api/users/current");
  return rep.data;
};

Docker Compose templating using YAML merge feature

2022-12-13T00:00:00+00:00

Useful trick when you have many of the same services in the docker-compose.yml file.

Imagine that we have some data processing application which has a master service and several different workers. All applications are written in the same language and use a common base docker container. In general this is a fairly common configuration. The docker-compose.yml file will look something like this:

version: "3.8"

services:
  master:
    build:
      context: .
    ports:
      - "8080:8080"
    volumes:
      - ".:/app"
    command: python run_master.py

  worker-foo:
    build:
      context: .
    depends_on:
      - master
    volumes:
      - ".:/app"
    environment:
      - MASTER_HOST=master:8080
    command: python run_foo_worker.py

  worker-bar:
    build:
      context: .
    depends_on:
      - master
    volumes:
      - ".:/app"
    environment:
      - MASTER_HOST=master:8080
    command: python run_bar_worker.py

It looks verbose. A lot of code duplication. Let’s fix it with YAML Anchors!

version: "3.8"

x-base: &base
  build:
    context: .
  volumes:
    - .:/app

x-worker: &worker
  <<: *base
  depends_on:
    - master
  environment:
    - MASTER_HOST=master:8080

services:
  master:
    <<: *base
    ports:
      - "8080:8080"
    command: python run_master.py

  worker-foo:
    <<: *worker
    command: python run_foo_worker.py

  worker-bar:
    <<: *worker
    command: python run_bar_worker.py

It looks much better! Such a file is easier to maintain, expand and edit in the future.

The x-field is a special property in the docker-compose.yml file. In the Docker compose documentation they are called extension fields. You can put any valid YAML code in them.

Merge problems

Note that YAML does merge at the top level and do not deep merge. So this code will not work as expected:

x-base: &base
  environment:
    - SUPER_SECRET=42

x-worker: &worker
  <<: *base
  environment:
    - MASTER_HOST=master:8080

The worker service will completely overwrite environment of base and will contain only MASTER_HOST. Unfortunately, this array combining cannot be achieved directly in the docker-compose file. Therefore, in such cases, you can use extends.

Solution for single variable

However, if you want to make only one value common, you can put that value into a variable, and use that variable as an array element.

x-super-secret: &super-secret SUPER_SECRET=42

x-base: &base
  environment:
    - *super-secret

x-worker: &worker
  <<: *base
  environment:
    - *super-secret
    - MASTER_HOST=master:8080

This can be useful for volumes, depends_on and other array-like directives.

Solution for single object

As for environment, you can put the shared variables in a separate object and combine them in the same way we did with services. 🙃

x-env: &env
  SUPER_SECRET: 42

x-base: &base
  environment:
    <<: *env

x-worker: &worker
  <<: *base
  environment:
    <<: *env
    MASTER_HOST: master:8080

That’s all for now. If this information was helpful to you, don’t forget to subscribe to receive notifications of new posts.

asdf: One Version Manager for Python, Node and More

2022-12-10T00:00:00+00:00

When you develop or maintain many projects or use a microservice architecture, over time you have a zoo of different programming languages and their versions on your computer. Each service has its own dependencies. After a while it becomes hellishly difficult to work comfortably.

Initially, I controlled runtimes versions through homebrew, switching them when needed. Later I switched to nvm for NodeJS, pyenv for Python. If you add to this tools automatic version selection on cd to project folder, it becomes almost comfortable. But this solution in the terminal is visually slow.

I was looking for alternatives and found an amazing tool with a strange name – asdf. This is single cli-tool for managing multiple runtimes and their versions. Each runtime is added via plugins and there are already about 450 of them.

asdf itself knows how to install different versions, can switch runtime when directory changed and works fast.

Installation

On MacOS you can install asdf with homebrew (for other OS follow official guide):

brew install asdf

After that we can set the necessary runtimes:

# Install NodeJS
asdf plugin-add nodejs
asdf install nodejs latest:16
asdf install nodejs latest:14

# Install Python
asdf plugin-add python
asdf install python latest:3.10
asdf install python latest:3.11

# Also possible to get list of available versions for inslation
asdf list all nodejs
asdf list all python

# Or list of already installed versions
asdf list nodejs
asdf list python

# Get current selected runtimes
asdf current

# Set global runtime (will create a .tool-versions in root of user directory)
asdf global nodejs latest:16
asdf global python latest:3.10

# Or set local runtime (will create a .tool-versions file in current directory)
asdf local nodejs latest:14
asdf local python latest:3.10

Version select on directory change

asdf can also automatically select the correct version depending on the current directory and will do this on each cd. It does this by going up in the hierarchy until it finds version of required runtime in the .tool-versions file.

This allows flexible control over multiple versions of runtimes in each folder. For example you have an old project where tooling is written in python 3.8, you want to start code a new microservice in python 3.11, and globally in the system you are using version 3.9. asdf will handle it for you!

To enable this auto version select feature add next line to your shell rc-file (for more options or other shells see official guide):

# For zsh (~/.zhsrc)
. $(brew --prefix asdf)/libexec/asdf.sh

# For fish-shell (~/.config/fish/config.fish)
source "(brew --prefix asdf)"/libexec/asdf.fish

That’s all. Now just make a cd to the project folder and set the required runtime version. Now when you navigate to the project folder, asdf will automatically select python of this version. You can check selected versions with:

# for example install global python version
asdf global python latest:3.11

# go to project folder
cd ~/some_project

# install reqired python version (will create or update .tool-versions file)
asdf local python latest:3.10

# see current active runtime versions (should be 3.10)
asdf current

# check python version (should be same)
python --version

# go back to user folder
cd ~

# check python version again (should be 3.11)
asdf current
python --version

Also you can commit the .tool-versions file directly into GIT. So then all team members will also know what versions of runtimes is in use for that service.

One more life story

Using asdf you can manage not only runtime language versions, but also cli tools such as aws-cli, terraform, kubectl.

So if you have different versions of Kubernetes clusters, even the official documentation does not recommend using more than one version of difference.

Using multiple versions via brew is not convenient (you have to switch versions manually all the time). But these cli tools can also be put in .tool-versions.

asdf plugin-add kubectl https://github.com/asdf-community/asdf-kubectl.git

asdf install kubectl latest:1.20
asdf install kubectl latest:1.25

asdf global kubectl latest:1.25

cd ~/legacy_project
asdf local kubectl latest:1.20

That’s all for now. If this information was helpful to you, don’t forget to subscribe to receive notifications of new posts.

How to See Env Variables of a Running Docker Container

2022-12-06T00:00:00+00:00

TL;DR

# get all
cat /proc/1/environ | tr '\0' '\n'

# get by name
cat /proc/1/environ | tr '\0' '\n' | grep ABC

Details

When you develop services, sometimes you need to know what environment variables a process in a container is running with.

For example if you use kubernetes and the environment variables come from several places or if you debug a production container and want to know what environment it is running with.

Well, that's easy. Normally your process in the docker is running under PID=1. You can check this by running the command:

docker exec <CONTAINER_ID> ps aux
# output:
PID   USER     TIME  COMMAND
    1 root      0:02 node src/main.js

Now we can see the environment variables that this process uses:

docker exec <CONTAINER_ID> cat /proc/1/environ | tr '\0' '\n'
# output:
NODE_VERSION=16.18.1
HOSTNAME=544193567d05
YARN_VERSION=1.22.19
SHLVL=2
HOME=/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/app

This approach is also useful if you need to get environment variables of kubernetes pod:

# connect to pod
kubectl exec --stdin --tty node-app-abc123 -- /bin/sh

# get process envs
cat /proc/1/environ | tr '\0' '\n'
# output: list of environment variables

Keep Type Guards valid after refactoring in TypeScript

2022-12-03T00:00:00+00:00

TypeScript is a great language for writing applications with type safety checks. Extending or refactoring code in TypeScript is much easier than in plain JavaScript.

TypeScript has a nice built-in functionality for interface narrowing — Type Guards. But they don’t always protect against errors during code extension / refactoring, especially if you have a large project.

Problem description

For example, we have a tea store. We sell two kinds of tea: loose and bagged. We show all the goods in one list. Each type of product has humanised link, and in the name of the product we want to specify the number of grams or the number of tea bags in a pack.

import React from "react";

type Tea = { id: number; name: string; price: number };

type LooseTea = Tea & { weight: number };
type BaggedTea = Tea & { size: number };

const isLooseTea = (x: Tea): x is LooseTea => "weight" in x;
const isBaggedTea = (x: Tea): x is BaggedTea => "size" in x;

const getLink = (x: Tea): string => {
  if (isLooseTea(x)) return `/loose-tea/${x.id}`;
  if (isBaggedTea(x)) return `/bagged-tea/${x.id}`;
  throw new Error("Unknown tea");
};

const getTitle = (x: Tea): string => {
  if (isLooseTea(x)) return `${x.name} / ${x.weight}g`;
  if (isBaggedTea(x)) return `${x.name} / ${x.size} teabags`;
  throw new Error("Unknown tea");
};

const TeaItems: React.FC<{ items: Tea[] }> = ({ items }) => {
  return (
    <ul>
      {items.map((x) => (
        <li key={x.id}>
          <a href={getLink(x)}>{getTitle(x)}</a>
        </li>
      ))}
    </ul>
  );
};

Later, we got large tea bags (for the teapot) and changed our data model:

type BaggedTea = Tea & { bags: number; weightPerBag: number };

After that we get an error in getTitle function and fix it:

// before:
const getTitle = (x: Tea): string => {
  if (isLooseTea(x)) return `${x.name} / ${x.weight}g`;
  // Property 'size' does not exist on type 'BaggedTea'. ts(2339)
  if (isBaggedTea(x)) return `${x.name} / ${x.size} teabags`;
  throw new Error("Unknown tea");
};

// after:
const getTitle = (x: Tea): string => {
  if (isLooseTea(x)) return `${x.name} / ${x.weight}g`;
  if (isBaggedTea(x)) return `${x.name} / ${x.bags} teabags ~ ${x.weightPerBag}`;
  throw new Error("Unknown tea");
};

Everything seems to be fine and TypeScript doesn’t found any error. But in fact during the rendering process an "Unknown tea" error will occur, because the Type Guard in the getLink function is no longer pass in isBaggedTea guard.

Problem solution

The problem with our Type Guard is that internally we check for the presence of a field in the object. This is a JavaScript operation and it is not typed in any way.

Well, let’s handle that. I propose to use a factory function to create Type Guard functions based on the data-model fields.

type AnyObject = Record<string, any>;

// https://stackoverflow.com/a/52991061
type RequiredKeys<T> = {
  [K in keyof T]-?: AnyObject extends Pick<T, K> ? never : K;
}[keyof T];

const createShapeGuard = <T extends AnyObject>(...keys: RequiredKeys<T>[]) => {
  return (obj: unknown): obj is T => {
    if (typeof obj !== "object" || obj === null) return false;

    for (const key of keys as string[]) {
      if (!(key in obj)) return false;
    }

    return true;
  };
};

Now let’s rewrite our Type Guards in a new way:

const isLooseTea = createShapeGuard<LooseTea>("weight");
const isBaggedTea = createShapeGuard<BaggedTea>("bags", "weightPerBag");

Everything works fine now. If we change the data structure again in the future, TypeScript will check and tell us that there is now an error in our Type Guard:

type BaggedTea = Tea & { bags: number; bagWeight: number };

// Argument of type '"weightPerBag"' is not assignable to parameter of type 'RequiredKeys<BaggedTea>'.ts(2345)
const isBaggedTea = createShapeGuard<BaggedTea>("bags", "weightPerBag");

That’s all for now. If this information was helpful to you, don’t forget to subscribe to receive notifications of new posts.

Sync all Git branches with remote

2022-12-02T00:00:00+00:00

Sometimes there is no good internet connection, but you can still work at your laptop (merge PR, etc). It’s very handy to have a local full local copy of the Git repository of a project on your computer for this purpose.

Unfortunately, Git itself doesn’t have out-of-the-box functionality to sync your local repository with a central server (like Github).

I searched for a solution on stackoverflow, but surprisingly there are only separate commands to do different parts of this task.

So I made this little script in bash to be able to do this synchronisation.

#!/usr/bin/env bash

# Debug: Print all commands with expanded variables
# set -x

# Get all remote refs & remove outdated local refs
git fetch --all --prune

# Create local copy of remote branch & load latest changes
for rb in `git branch --remotes | grep -v 'HEAD'`; do
  lb=${rb#origin/}
  echo "$rb -> $lb"
  git branch --track $lb $rb 2>/dev/null
  git fetch --update-head-ok origin $lb:$lb
done

# Remove all merged local branches
git branch --merged | xargs echo | grep -v '^*\|^main$\|^develop$|^stage$' | xargs git branch -d

Note: On line 18 you can specify a list of branches that should always remain local.

This script will also be useful if you often have to check other branches manually before merging them.

Installation & Run:

curl https://gist.githubusercontent.com/vladkens/5261c703111fd1d0a870c3bf5e53c698/raw/698af0fdac327bcb3b7d8777a4593a4d66321bb6/git-sync.sh > git-sync.sh
chmod +x git-sync.sh

./git-sync.sh

Colima: A Free Docker Desktop Alternative for macOS

2022-07-22T00:00:00+00:00

Starting in February 2022, Docker Desktop became a paid service for large teams. This may be a reason to look for alternatives to Docker Desktop.

In my case one day Docker Desktop just stopped opening. The default solution of uninstalling and re-installing didn’t work. I didn’t have enough time to figure out the problem. So I looked for alternatives to running containers on macOS locally.

One of the projects I looked at is Colima. I found it minimalistic to install and easy to use. Colima supports M1 & Intel chips, port forwarding, volume mounts, docker-compose & kubernetes. Basically everything you need for local development.

How to install Colima?

Install docker & docker-compose client (just clients to interact with Colima containers runtime):

brew install docker
brew install docker-compose

Then install Colima:

brew install colima

Usage

Start Colima is easy:

colima start

That’s it. Now you can run your docker-compose files and everything works. To stop Colima:

colima stop

Colima also support different profiles, kubernetes, etc. You can check start options in docs or by:

colima start --help

Customising

By default Colima runs with 2 CPUs, 2GiB memory and 60GiB storage. This is not enough for me so these settings can be changed:

# stop VM before apply changes
colima stop
# this will change configuration of exits VM
colima start --cpu 2 --memory 4
# or by changing config file
colima start --edit

Possible problems

If you are using utilities such as dry or dive, they may not start with the error like “docker.sock not found”. This is easy to fix:

sudo ln -sf ~/.colima/docker.sock /var/run/docker.sock

That’s all for now. I hope that article was useful for you. If yes, don’t forget to follow me to get new updates.

How to Make a Node.js Docker Image 4x Smaller

2022-01-16T00:00:00+00:00

Hi. Today I want to talk about how to reduce the size of the Docker Image with NodeJS inside.

In my company project we use NodeJS as a backend of one of the services. I noticed that there is a big delay between when the project is built in CI and when the new version of the service actually starts working.

I started researching this topic and noticed that the NodeJS Docker Image size takes up 1.3Gb of space. So on each deploy these data first uploaded to registry and then downloaded back to run in Kubernetes cluster.

To check the size of your Docker need to run few commands:

> docker build -t app .
> docker images | grep app
app     latest    62dec8181ae0   30 seconds ago   1.28GB

Well, let’s fix that problem.

Smaller Docker Base Image

The most basic and simple solution would be to change the basic Docker Image to the alpine version. To do this, change in the first line of the Dockerfile FROM node:16 to FROM node:16-alpine. This was already in my Dockerfile, so this optimization won’t help me. The original Docker file looks like:

FROM node:16-alpine

WORKDIR /app
COPY . .

RUN yarn install
RUN yarn app:build

EXPOSE 3000
CMD node dist/app.js

Multi-Stage Builds

The process of creating a Docker container can be divided into several steps, where different parts of the application are prepared individually. And then the necessary files from the intermediate stages can be copied into the final image.

This makes sense, because during the build a lot of temporary files are created, there are dev dependencies in package.json and other things that are not really needed for the final application to work.

I divided the build process of container into three steps. The first step builds the application (compiling the TypeScript, validation schemes, etc.). The second stage installs only those dependencies that are needed for the production of the application. Finally, the third stage copies the compiled application from the first stage and the dependencies from the second stage and runs the application.

This approach requires a little more time to build the container, but it also reduces its size considerably. For example, the TypeScript dependency alone takes up over 80 mb of space. The final Dockerfile looks like:

FROM node:16-alpine as dist
WORKDIR /app
COPY . .
RUN yarn install
RUN yarn app:build

FROM node:16-alpine as deps
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install --production --frozen-lockfile

FROM node:16-alpine
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY --from=dist /app/package.json /app/.env ./
COPY --from=dist /app/dist/src ./src

EXPOSE 3090
# CMD du -sh ./node_modules/* | sort -nr | grep '\dM.*'
CMD node src/app.js

In line 19 I comment a sh command to print size of depencencies in node_modules. So you can check what dependencies remained in the final build and possibly remove some of them, or replace them with the optimal version. For example, if you use AWS S3 in your application, you can use a dependency only for S3 and not for all AWS services.

Final size of the Docker Image is 322Mb, which 4x times smaller than original image.

> docker images | grep app
app     latest    b6f283f00bb2   12 seconds ago   322MB

Conclusion

Using simple things you can reduce the size of the image of your application and speed up the time between building and running the application in the production.

That’s all for now. I hope that article was useful for you. If yes, don’t forget to follow me to get new updates.