Oh no, definately do not give up the two languages!
I can't dig up the info tonight, but it is being proven over and over again, that being bilingual usually strengthens an individuals ability to learn any language!
Meaning, a child learning two, will learn their first language better then if they just studied the ONE language.
Even with your sons speech problems, I would not recommend dropping a language at this point!
My daughter was a late talker. We put her into a French Daycare, having barely learned any french prior to it! She came out two months later, still speaking almost no french, but bursting in English! I think it helped her.
I'll try and dig up more tommorow, but the only thing I might recommend differently, is make sure you are not "mixing" the two languages, and perhaps try as best you can, to "pick a time or place" to use one language or the other. That way, he knows walking in the room or up to you, what language is most likely going to be spoken! For example, "Mom is in the Kitchen cooking... This is Spanish time". Or, "We are shopping, we are talking in English".