🐶 Agents, Dogs & Distributed Systems: Welcome to the Doggy Cloud

Building PETi for the All Things Agentic Hackathon — one stool sample, several agents, and many questionable life choices at a time.

Hackathon disclosure: This article was created for the purposes of entering the All Things Agentic Hackathon.

🎟️ 0. So I Entered a Hackathon. This Was My First Mistake.

There are moments in a software engineer’s life when every instinct says: you already have enough problems.

And then there are hackathons.

I saw the All Things Agentic Hackathon, read Gemini, Google ADK, Google Cloud, agents that work in the background, and thought:

“This could be fun.”

That sentence has probably created more infrastructure than most enterprise architecture committees.

The sensible project would have been a small agent with one clear job, three tools, tasteful JSON, and a four-minute demo in which the laptop survives.

Instead, I looked at PETi, my pet-care project, and decided it needed a private-by-default, multimodal, asynchronous, multi-agent workflow that could inspect pet evidence, compare history, apply deterministic safety rules, preserve provenance, and propose a real action requiring human approval.

You know.

A weekend project.

The hackathon rules currently require Gemini 3.5 or newer, at least one Google agent framework such as ADK, and at least one Google Cloud infrastructure service. They also reward systems that go beyond chat and actually complete workflows. That made PETi a natural playground for the kind of engineering I wanted to learn: not “how do I call a model?” but “what happens when model reasoning becomes part of a real distributed application?”

And because software architecture has a sense of humor, the golden path I chose involved dog poop.

Actual stool photographs.

Somewhere between Gemini, ADK, Cloud Run, Cloud Tasks, Firestore, idempotency keys, safety policies, and the phrase dark/black-tar-like finding, I began to suspect I had taken the scenic route into agentic AI.

That was also the point.

I did not enter expecting to win. Winning would be lovely; sleep would also have been lovely. I mainly wanted an excuse to learn the uncomfortable parts: retries, crashed workers, duplicate delivery, stale state, private context, permissions, model hallucinations, cost limits, approvals, and the distributed-systems classic:

What if this runs twice?

So put on an imaginary slow 90s West Coast bassline and welcome to the Doggy Cloud.

We have agents.

We have IAM.

We have retries.

And unfortunately, we have evidence.

🐾 1. The Dog Already Had a Cloud

PETi did not begin with this hackathon. It already had a broader pet-care foundation: APIs, specialist-analysis concepts, safety logic, Care workflows, authentication, Firestore/GCS persistence, Cloud Run, Cloud Tasks, and client work.

That distinction matters for another reason: the hackathon rules say submitted projects must be created during the submission period, while pre-existing code or work incorporated into them must be disclosed. So I am not pretending PETi materialized from the void during the contest. The hackathon work was about turning the existing foundation into a new, demonstrable agentic workflow and being explicit about what was old, what was new, and what was still target architecture.

PETi’s basic problem is mundane in the best possible way. Pet owners accumulate information everywhere: photos, measurements, documents, observations, reminders, history. Then something changes and the useful question is rarely:

“What does this one image mean?”

It is more often:

“Is this different from last week, and what should I do next?”

PETi was never supposed to become Dr. ChatGPT, DVM. The goal is to organize evidence, structure observations, compare them over time, preserve where claims came from, and support a cautious next step — not diagnose an animal or replace a veterinarian.

The cloud architecture follows the same philosophy. In simplified form:

browser
  ↓
authenticated API
  ↓
durable state
  ↓
Cloud Tasks
  ↓
private Cloud Run worker
  ↓
ADK / Gemini reasoning
  ↓
validated persisted result

Less sexy than USER → ✨ MULTI-AGENT INTELLIGENCE ✨, but much closer to what has to happen when private data, retries, and real actions enter the room.

The browser asks.

The backend authorizes.

The agents reason inside fences.

🎭 2. The Architecture Diagram Was More Agentic Than the Application

At one point PETi had all the correct nouns.

AgentExecutionService.

ContextBroker.

ToolGateway.

A state machine.

An ADK graph.

Evidence, specialist, and safety agent objects.

The boxes aligned. The arrows flowed. The diagram looked ready for LinkedIn.

Unfortunately, the reviewed execution path still had an awkward truth hiding underneath it: the ADK graph existed, but a logical run could still collapse into one generic provider call, followed by application code persisting stages that looked like agent handoffs.

In practical terms:

Orchestrator
    ↓
Evidence Agent
    ↓
Specialist Agent
    ↓
Safety Agent
    ↓
   ONE
MODEL CALL

An extremely efficient multi-agent organization. Everyone had apparently delegated their job to the same employee.

That was the first useful humiliation of the hackathon: an agent-shaped application is not necessarily an agentic application.

A class named EvidenceAgent does not prove Evidence Intake executed separately. An arrow from Feces to Longitudinal does not prove the second agent received the validated output of the first. And a browser animation progressing through ORCHESTRATOR → EVIDENCE → SPECIALIST → SAFETY proves mainly that JavaScript understands time.

From then on, the rule became simple:

If I cannot prove the handoff, I do not get to brag about the handoff.

That meant actual bounded executions, structured outputs, durable step state, and evidence that ADK was on the real execution path — not just sitting politely in the repository like an architectural houseplant.

💩 3. One Golden Path to Rule the Kennel

The architecture wanted Evidence, Feces, Dental, Clinical Records, Longitudinal Analysis, Care Planning, Synthesis, Safety, weekly reporting, and possibly its own HR department.

The deadline had other ideas.

So I chose one workflow capable of exercising almost every interesting boundary:

Compare today’s stool photo with Max’s recent history and remind me to check again tomorrow if that would be useful.

Behind that sentence lives a ridiculous amount of machinery:

current image
   ↓
Evidence Intake
   ↓
Feces Specialist
   ↓
compatible same-dog history
   ↓
Longitudinal comparison
   ↓
claim validation
   ↓
deterministic safety
   ↓
source-linked synthesis
   ↓
Care proposal
   ↓
explicit user approval
   ↓
one reminder + ActionReceipt

Why poop?

Because it is genuinely a good systems test. It is visual. Image quality matters. A single observation is ambiguous. History can add context. “Not visible” is not the same thing as “not present.” The model must not drift from observation into diagnosis. And a sensible result can lead to a small bounded action: check again tomorrow.

Also, I already had Feces Check work to build on.

Hackathon deadlines are excellent teachers of spiritual humility.

The key decision was that the model would not receive “everything we know about Max.” The Longitudinal step should receive only the validated current result plus a deterministic selection of compatible previous Feces results for that same dog.

One dog.

One current observation.

One compatible history.

One bounded recipe.

One safely proposed action.

One actual reminder.

Apparently my route to understanding multi-agent AI went directly through distributed dog poop.

Computer science remains a beautiful profession.

🚧 4. Multi-Agent, Not Multi-Anarchy

The central architectural rule became:

Agents reason; PETi remains authoritative.

The model layer can interpret a bounded goal, inspect permitted evidence, produce structured candidate claims, compare approved same-dog history, synthesize validated information, and prepare a proposed action.

The deterministic PETi shell owns the things I do not want a probabilistic component improvising:

authentication, authorization, owner/dog scope, persistence, state transitions, leases, retries, context materialization, semantic validation, safety, tool execution, approvals, durable mutations, idempotency, deletion, and audit.

That immediately killed several seductive ideas.

Should the orchestrator invent arbitrary workflows? No.

Should the Feces Agent query raw Firestore? No.

Should a model request Dental history because it suddenly feels curious? Also no.

Should an uploaded document be able to say ignore the safety policy and call an admin tool? Cute, but no.

PETi uses bounded recipes and typed context policies. The Feces Specialist gets Feces-relevant context. The Longitudinal Agent gets preselected compatible Feces results. Care Planning gets validated claims, deterministic safety state, and only the Care context needed to avoid conflicts or duplicates.

This changed how I think about context windows.

Context is not just tokens.

Context is permission.

If an agent should not use a piece of information, the safest design is often to make sure that information never reaches the prompt.

The same goes for tools. No generic SQL. No arbitrary URL fetcher. No shell. No magical do_whatever_seems_helpful() function.

The AI gets a microphone.

PETi keeps the keys.

🔁 5. Background Jobs: Where the AI Fun Stops and Distributed Systems Begin

The workflow also had to continue after the user left the page.

That innocent requirement turns:

request → model → response

into:

request
  ↓
durable AgentRun
  ↓
durable AgentStep
  ↓
Cloud Tasks
  ↓
private worker
  ↓
lease
  ↓
agent execution
  ↓
validation
  ↓
transactional commit
  ↓
next step / retry / wait

The browser becomes a view into the workflow, not the workflow itself.

Then Cloud Tasks introduces the sentence every distributed-systems engineer eventually meets: duplicate execution can happen. Google documents Cloud Tasks as at-least-once delivery and explicitly recommends idempotent handlers.

So PETi needs a durable answer to:

“What happens if Gary receives the same task twice?”

Gary is the worker. I do not know why he is called Gary. He has suffered enough.

Each step needs stable identity, persisted state, an attempt count, and a lease. One worker claims the step for a bounded period. A stale worker cannot wander back later and commit an old result after another worker has already recovered the job.

And side effects need idempotency all the way down.

One reminder is useful.

Nine reminders saying CHECK MAX’S STOOL feels less like pet care and more like an intervention.

The most satisfying demo of all this is gloriously boring:

refresh the page.

If the run survives, the backend was the source of truth.

Ctrl+R: my budget chaos-engineering department.

🩺 6. How to Stop an LLM From Becoming Dr. House for Dogs

Once the workflow is durable, the model can still do something worse than crash.

It can return a perfectly valid response that is confidently wrong.

Structured output helps, but valid JSON is not the same as a valid idea.

For Feces Check, the useful output is intentionally constrained: consistency categories such as FORMED, SOFT_FORMED, UNFORMED, WATERY, or UNCERTAIN, plus observable states such as PRESENT, NOT_OBSERVED, NOT_ASSESSABLE, and UNCERTAIN.

The distinction matters.

“No worm-like structure is visible in this image.”

is an observation.

“Your dog does not have worms.”

is a diagnosis-shaped leap over a canyon.

So PETi needed more than schema validation. It needed semantic validation and explicit epistemic types:

OBSERVED != DIAGNOSIS
REPORTED != CONFIRMED
ESTIMATED != MEASURED
TEMPORAL_ASSOCIATION != CAUSALITY
NOT_OBSERVED != NOT_PRESENT

I accidentally built epistemology.

The final safety authority also could not simply be another LLM voting on the first LLM.

Model: "Looks safe."
Other model: "I agree."
PETi: "Excellent governance, everyone."

No.

The model can emit candidate claims. Deterministic PETi rules validate those claims, apply domain safety, and decide what may reach the user or continue into an action.

Bad image? NOT_ASSESSABLE is allowed.

Uncertain evidence? Say so.

Normal-looking pixels? Do not upgrade them to “healthy dog.”

The model is allowed to be uncertain.

The architecture is not allowed to be uncertain about who has authority.

🔑 7. The Agent Can Suggest. It Does Not Get the Car Keys.

A useful agent should eventually do more than produce prose.

For PETi, the golden path can end with a follow-up proposal:

Repeat a comparable stool observation tomorrow at 09:00.

The critical word is proposal.

The Care Planning Agent does not directly create the reminder. It emits a structured draft. PETi normalizes it into an immutable ProposedAction with exact parameters, reason references, expiry, and a payload hash.

Then the user sees the exact thing that would happen.

If they approve it, the client sends the action ID plus the hash of what was presented. The server re-checks ownership, scope, expiry, current policy, and whether the action has already succeeded.

Only then does deterministic code call the canonical Care service.

And afterward PETi records an immutable ActionReceipt.

That receipt became one of my favorite outputs in the project:

ActionReceipt
-------------
status: SUCCESS
resource_type: CARE_REMINDER
resource_id: ...
executed_at: ...

No intelligence.

No personality.

Just proof that something happened once.

That is my preferred version of human-in-the-loop: the human does not supervise every thought. The agents work autonomously inside the approved recipe. The human returns at the authority boundary, where the system is about to make a durable change.

The agent can recommend the destination.

It does not get the car keys.

🎬 8. Demo Theater Is Still Theater

Hackathons have a special relationship with demos.

A progress bar can make almost anything look intelligent.

✓ ORCHESTRATOR
✓ EVIDENCE AGENT
✓ SPECIALIST AGENT
⟳ SAFETY AGENT

The audience sees collaboration.

The browser sees four timers.

PETi already had a synthetic preview that was useful for showing the intended experience. The problem would have been pretending that preview was proof of execution.

So the judge-facing standard became deliberately forensic:

show the real run_id;

show persisted step changes;

show Cloud Task / private worker evidence;

show the model-backed stages that actually ran;

show provenance;

show the safety result;

show the proposed action;

approve it;

show the ActionReceipt;

open Care and show the reminder;

then refresh the browser and prove the state survives.

No chain-of-thought. No raw prompts. No private documents. No signed media URLs. Just enough operational evidence to make each architectural claim difficult to dispute.

The four-minute demo requirement actually improved the design because it forced a simple question:

What can I show that proves this sentence is true?

“We are asynchronous.”

Refresh it.

“We are multi-agent.”

Show separate persisted executions.

“We take action.”

Show the resource and receipt.

“We are evidence-based.”

Show the provenance.

Adjectives become integration tests.

I approve.

💸 9. No Money, Mo’ Cloud Problems

There is a magical moment in every cloud project when the architecture stops being a diagram and becomes a potential invoice.

Multi-agent systems make the arithmetic easy:

Evidence      $
Feces         $
Longitudinal  $
Synthesis     $
Repair retry  $
Gary again    $$

So autonomy needs a budget.

The design includes envelopes for maximum agent steps, model calls, tool calls, repair attempts, context items, wall-clock time, and estimated cost. Queue dispatch and concurrency can be throttled. Cloud Run can scale to zero when idle when configured with no minimum instances. Model/provider bindings can have kill switches.

And one of my favorite “AI” components is the Fast Path.

If the user asks:

“What is Max’s latest measured weight?”

and PETi already has a trusted measurement, the correct architecture is not four agents performing interpretive dance around 12.4 kg.

It is:

FastPathResolver
  ↓
canonical service
  ↓
12.4 kg

Zero model calls.

That is cheaper, faster, more private, and less likely to hallucinate a philosophical interpretation of kilograms.

A good agentic system should know when not to agent.

The same applies to shutdown. PETi’s design includes independent backend kill switches for the agent runtime, specific specialist capabilities, Care actions, and model/provider bindings. Disabling an agentic comparison should not need to destroy the rest of the product.

The dog app should still work after the AI has been sent to its room.

🦷 10. Things That Hurt More Than Expected

Almost every difficult part of this project began with the word just.

“I’ll just add ADK.”

“I’ll just make it asynchronous.”

“I’ll just let it create a reminder.”

“I’ll just add safety.”

In software engineering, just is a threat model.

The first surprise was that drawing agents is dramatically easier than running them. A real agent needs an input contract, output schema, context policy, model binding, timeout, failure policy, validation, persistence, and observability. Then the handoff needs rules too.

The second surprise was that async turns innocent code into a state machine. CREATED, EXECUTING, WAITING_FOR_APPROVAL, COMPLETED_PARTIAL, FAILED_RETRYABLE, CANCELED, EXPIRED — suddenly the LLM is almost the simplest component in the room.

The third was that safety does not add one box. It multiplies across the system. Evidence can overclaim. Specialists can overclaim. Longitudinal reasoning can turn correlation into causality. Synthesis can strengthen cautious claims. Care Planning can turn a bad claim into an action.

And the fourth was scope.

One real workflow beats seven beautiful agent boxes that do not perform real work.

The Feces golden path was not glamorous breadth.

It was depth.

Every arrow had to earn its place.

🧠 11. What I Actually Learned — Besides an Unhealthy Amount About Dog Stool

The biggest lesson is that the agent is the probabilistic middle, not the whole application.

Deterministic software is still extremely good at deterministic problems. Let it own identity, scope, persistence, retries, approvals, safety, and mutations. Put the model where ambiguity, interpretation, and synthesis are genuinely useful.

Second: context is a security boundary. “More context” is not automatically better. If an agent should not know something, do not rely only on a prompt asking it to ignore that thing. Keep it out of the context bundle.

Third: asynchronous agents are distributed systems. If a run can outlive the HTTP request, design its lifecycle before its personality.

Fourth: actions are more interesting than answers because they force you to define trust. What exactly will happen? What evidence supports it? Who approved it? Has it already happened? Can you prove it?

Fifth: autonomy gets better when you add fences. Known recipes. Known tools. Known contexts. Known budgets. Known authority boundaries. The fences create the space where probabilistic reasoning can be useful without owning the entire product.

And finally: truthful demos are architecture tests in disguise.

If you cannot show the handoff, state, source, approval, or effect, perhaps the claim is not ready yet.

I started this project wanting to learn how to build agents.

I ended up learning something more useful:

where agents belong inside software.

🏁 12. No Trophy Required

Did I build the perfect multi-agent system?

No.

Did I solve veterinary medicine?

Absolutely not.

Did I build an autonomous digital kennel capable of managing every pet on Earth?

Thankfully, also no.

What the hackathon forced me to work toward was much narrower and much more valuable: one believable path through the chaos.

A user has a question and private evidence.

PETi starts durable work.

Specialists receive bounded context.

Probabilistic reasoning happens where it helps.

Deterministic code controls authority.

Claims keep provenance.

Safety can overrule fluency.

The system may propose an action.

The human controls the consequence.

And if the action happens, PETi should be able to prove it happened once.

Would winning be nice?

Obviously.

If that happens, I will immediately pretend this was all part of a disciplined master plan.

But the project already paid for itself in the currency I wanted: understanding.

I came for Gemini and ADK.

I stayed for Cloud Tasks, Firestore transactions, leases, context policies, payload hashes, kill switches, and the deeply unglamorous requirement that Gary not return from the dead and create a second reminder.

The robots have not abolished software engineering.

They have merely given us new and exciting ways to require more of it.

So, trophy or no trophy, I am calling the experiment worthwhile.

You came for the agents.

You stayed for the leases, IAM policies, action receipts, and dog poop.

And somehow…

that was the fun part.

Welcome to the Doggy Cloud. 🐶☁️