As Hardoff says, you can get a long way without needing any math at all. Maths is just a helper, a language to visualise other complicated ideas with.
I suggest Steven E Smiths "Scientist and Engineers guide to DSP"
http://www.dspguide.com/
The (almost complete) PDF is avilable free online, but I'm sure after reading it you'l want to get the paperback version. It's nicely written, introduces concepts in the right order and doesn't really use much math.
Level 1 study also includes:
- Dodge and Jerse "Computer Music" - expensive hardback but fundamental
- Miranda "Computer Sound Design" - easy going and general
- Roads "The Computer Music Tutorial" - extensive
Level 2 study is
- Miller Puckettes book "Theory and techniques"
- Julius Smiths pages on filters, waveguides and modelling
- Dave Benson "Music, a mathematical offering"
- Perry Cook "Real time sound synthesis"