Can failure to clean my panels harm the panels?

Yes it can in some cases. These cases are the exception thankfully, but it can occur. As noted to us by experts, “improperly directed spray or random spray, can create a hotspot.” This can cause reverse polarity which will affect all the panels that are in the same string. It can also eventually kill the solar panel.
Another source of problems can occur for ground mounts where there is an irrigation system nearby. Typically an irrigation system is not going to be using soft water. Hard water from the irrigation system or by simply using a hose if you have hard water will leave spots if you do not immediately dry the panels. Similar to hard water spotting on cars, but this will buildup and will result in a reduction of output and possible damage to the panel. If you are using hard water, make sure to dry or squeegee it off as soon as possible. Also avoid spraying water in the heat of the day where the sun will evaporate the water and leave spots before you can dry the panels.P1030176