What is actually deciding when your job runs.

Two pieces, and only one is yours to run

The split is deliberate: everything that learns or decides lives in the cloud, everything that touches your hardware lives on your box and is small enough to audit.

A night in the life

1. 02:00 UTC

   Prices land, then everything replans

   The optimizer pulls the day's price curve for every location from the Hungry Machines Energy feed, stores it, and replans every user against their own location. This is one job: prices first, planning second, so a plan is never made against yesterday's numbers by accident.

2. within a minute

   Your controller notices

   It polls for a schedule newer than the version it holds. A new version means a new plan; it adopts it and acknowledges. Schedules are immutable — a replan publishes version n+1 rather than editing what you already have.

3. when the window opens

   The job runs, instrumented

   The controller resolves the workflow through your local job catalog, starts the profiler, and blocks on the adapter until the work finishes. Power, utilization, temperature, and clocks are sampled at 1 Hz for the whole run.

4. immediately after

   The record goes back

   Duration, measured energy, power statistics, and framework-reported work units are reported upstream. If your connection is down it spools to local disk and drains when the link returns — nothing is lost.

5. tomorrow

   The next plan is better

   That run becomes training data. Tomorrow the prediction for this workload is the median of its real measured runs instead of the guess you typed in.

6. 04:00 UTC

   Raw traces age out

   The per-second sample traces are deleted after 90 days. The run records and their aggregate statistics are kept — those are what the predictor actually uses.

How a slot gets chosen

The optimizer knows three things about each job: how much energy it is likely to draw, how long it is likely to take, and the window it is allowed to occupy. It prices every candidate placement in that window and takes the cheapest one that satisfies every constraint.

grid_kw   = max(0, job_avg_kw - solar_kw_in_slot)
slot_cost = grid_kw × price_per_kwh × slot_hours

Solar beyond what the job draws is free headroom, not a credit — we do not model export payments. The rules the result must satisfy are worth stating outright, because they are what keeps this safe to automate:

• Every placement sits inside its own [earliest_start, deadline]. No exceptions, no "close enough".
• No two scheduled windows overlap — one job at a time per box.
• A job that cannot fit comes back feasible: false with a reason attached. It is never dropped from the plan to make the plan look clean, and never run late to make it cheaper.
• The same inputs always produce the same schedule. No randomness, so a surprising plan can be traced to a changed input.

How energy gets measured

A scheduler that estimates its own savings is grading its own homework, so the controller measures instead. It samples the GPU once a second for the length of the run, and reports where the energy number came from:

Where the framework reports how much work it did — completion tokens, typically — that comes back too, as work_units. Energy per token is the thing that eventually makes predictions sharp for workloads whose size varies night to night.

Why it gets better

Each job request is reduced to a fingerprint — a stable hash of what makes the work expensive, with volatile inputs like seeds and timestamps deliberately excluded, so two runs that cost the same hash the same. The predictor looks up successful past runs with that fingerprint and takes the median.

Where the prices come from

The curves come from Hungry Machines Energy, which maintains the tariff and day-ahead market feeds — retail time-of-use schedules plus wholesale day-ahead prices across a set of market zones. Async Energy pulls every location's curve nightly and stores it; your plan is built against whichever location you selected in your settings.

That is the only connection between the two systems. It is a data feed, not an account — you never authenticate to Hungry Machines Energy, and it never sees your workloads, your runs, or your machine.

What happens when something breaks

Every schedule carries an expiry and a fallback policy, so the controller knows what to do without asking. If the API is unreachable it keeps following the last plan it holds; past that plan's expiry it applies the policy baked into it. Run records queue on local disk and drain on reconnect. A failed job triggers an immediate replan so its deadline can still be met by a later attempt.