The shift from deterministic software to probabilistic, agentic AI systems breaks the foundational assumptions of interaction design. For decades, we designed and optimized software for human control and predictability. Now non-deterministic systems are designing non-deterministic systems for users whose mental models of software haven't changed in 20 years. This talk examines what changes in practice when both the tool and the artifact it produces are probabilistic. Drawing on experience building and deploying autonomous agents at Amazon, we address four interconnected challenges: (1) how to surface agent behavior so users can calibrate trust without being overwhelmed (observability), (2) how to design for both success and failure when the same input can yield meaningfully different results (variability), (3) how to structure human oversight and agent confidence so both scale toward complete autonomy (human-in-the-loop), and (4) how to verify that the agent understands the user's intent across open-ended, long-horizon tasks (alignment).