You’re all doing these amazing things, whether it’s for work, open source, or some personal project, and that’s great and all except for the fact that we can’t see what you are doing!
So what are you working on this week? Maybe are you implementing an exciting feature, fixing mundane bugs (or writing them?), or are you caught in a massive yak shave? Share!
The “Commander” architecture (Bobby Calderwood’s talk) is just what I need, to coordinate a system that’s increasingly big.
Bobby uses Kafka at a big company. I mainly use Google’s cloud with 0 devops team. (Well, I have access to a devops team, but refuse to live on their schedule.) What messaging system do I use?
tiny brain: Google PubSub
stirring brain: prepackaged Kafka from Google’s marketplace (like Bitnami)
expanded brain: move to AWS for Kinesis
transcendent brain: PostgreSQL!
Gonna try out Bitnami Kafka. Then maybe Google PubSub if I’m not happy. But tbh I see myself using PostgreSQL to write an ad-hoc, informally-specified, bug-ridden, slow implementation of half of Kafka.
So I went with kafka. The tip I’d give: install’s a problem? Print debug logging when you make a kafka-poking client app.
Example: I didn’t give my 1 kafka VM the firewall exception to access itself. Oddly, my producers created topics, but not actually write nor read records.
Oh, my analysis:
Kafka: Semantics: what I want. Ops: my loads are so low, unless Kafka’s surprisingly crappy, little Ops burden.
Google PubSub: my system’s designed fundamentally as a batch system, with fanciness to support incremental updates. (Similar arguments as Nathan Marz uses to justify Lambda Architecture’s batch vs speed layers.) Running everything from scratch implies a commit log somewhere — increasingly a virtual Kafka.
Amazon Kinesis: 7 days retention? Pfft. My load’s so small, I’ll use Kafka as storage. Full retention.
PostgreSQL: NOTIFY/LISTEN seems brittle. I’d have to store consumer offsets: increasingly a virtual Kafka. I don’t need to query over commands/events (and if I did, it’s trivial to make a view over immutable commands/events).
Fulcro-Bulma library (work in progress):