Buildings are increasingly equipped with smart appliances that allow a fine grained adaption to personal comfort requirements. Such comfort adaption should be based on a human-feedback loop and not on a centralized comfort model. We argue that this feedback-loop should be achieved through local interaction with smart appliances. Two issues stand out: (1) How to impose logical locality when interacting with a smart appliance? (2) How to mediate conflicts between several persons in a room, or between building-wide policies and user preferences? We approach both problems by defining a general model for human-smart appliance interaction. We present a prototype implementation with an off-the-shelf smart lighting and heating system in a shared office space. Our approach minimizes the need for location metadata. It relies on a human-feedback loop (both sensor based and manual) to identify the optimal setpoints for lights and heating. These setpoints are determined by considering individual comfort preferences, current user location and a global goal of minimizing energy consumption.