Josh Huang's Engineering Notes
Case studies from 18+ years of chip design and infrastructure engineering — the debugging process, not just the result. Unlock individual articles for $5, or subscribe for $9/month unlimited — see Pricing.
A VLESS+REALITY handshake failed against one cover domain and succeeded against another, with identical keys and config on both — the difference turned out to be measured in bytes, not cryptography.
Weeks of 'the model just isn't very good' numbers from an AI-agent benchmark turned out to be measuring nothing at all — the grading script itself was silently crashing on every single run, including the ones that actually succeeded.
A wireless controller kept disconnecting on the same in-game trigger, and a live network scan seemed to hand over the smoking gun — until the identical channel numbers and suspiciously sequential MAC addresses gave away that the scan data itself was fabricated.
A ported CLI authenticator produced a plausible-looking code for every account — right length, right character set — for one specific account type it would never actually have worked, because it silently skipped that type's own non-standard algorithm entirely.
A dashcam-to-YouTube upload pipeline failed with the exact same quota error on three separate debugging passes — and each time it was a genuinely different bug wearing the same symptom, including one caused by the previous fix itself.
Standing up three chatbot personas on a self-hosted multi-agent gateway, all three went silent after their first message with no visible error — and each one turned out to be broken for a completely different reason, at a different layer of the same system.
OpenSSH kept rejecting a private key as insecure no matter how the file's Windows ACLs were rewritten — because the check it was actually running had nothing to do with the ACL at all.
A Photos API call kept returning 'insufficient authentication scopes' with the exact right scope confirmed present — because no scope could have ever worked, for a reason that had nothing to do with the request itself.
How I root-caused a hierarchical LEC failure everyone else had given up on, and shipped a FULL-EQ verdict with zero manual pin-constraints.
What changes when you point an autonomous coding agent at real chip sign-off work instead of boilerplate — and what 20+ iterations on a production ISP module actually looked like.
Selectively routing one console's streaming-app traffic through a Japan exit node, without touching its own login/matchmaking traffic — and the two independent kernel-level bugs that made it look impossible.
The actual engineering reasoning behind picking a specific proxy protocol stack for personal infrastructure — and why the choice was about detection resistance, not encryption strength.
Deploying a VLESS+REALITY node on Fly.io's global anycast platform — the free-tier tradeoffs, the IPv6-only gotcha, and why the deployment model actually matters more than the region you pick.
A 24/7 home camera → YouTube Live pipeline that looked healthy in every way that mattered, except it never actually produced a single frame — and the YouTube API lifecycle bugs waiting on the other side once it did.
Centralizing several different cameras' RTSP feeds behind one restreaming layer solved the fan-out problem cleanly — until one camera with no audio track at all ran into a YouTube health check nobody expects to fail on video grounds.
A remote-desktop client's 'cannot create tunnel' error came with genuinely compelling MTU/MSS evidence pointing at a network black hole — evidence that was real, reproducible, and completely unrelated to the actual failure.
A router firmware commit that failed identically every single time, with an error message giving no useful detail — until running `strings` on the vendor's own closed-source binary turned up the exact internal state machine causing it.
A single, textbook NAT rule silently rewrote loopback DNS queries' source address, and every application on the machine lost name resolution at once — including the tool being used to push the fix.
Restoring IPv6 reachability behind a NAT'd home connection via a tunnel broker was the easy part. The mistake that made every device on the network feel sluggish was one boolean flag, and understanding why required being honest about what the tunnel was actually for.
An automation's two-factor login kept failing with 'input is empty' even when the correct code was submitted a second later — because a second later was already too late, and the actual fix was in the order of two API calls, not their content.