Does propane "leak" if the tank is one and a device is off?

Propane can leak at any point in the system - the tank valve, the hose connection at the valve, a faulty hose, the connection at the appliance, or the valve at the appliance. Shutting off the valve at the tank will minimize the chances of a leak.