For most people, just communicating with teammates the way you would acquaintances sitting in the same room with you is all you need to not get punished, but if you actually need a breakdown of what to do, here it is.
*Note: I don't include intentional feeding, leaving, or any other active attempts to ruin the game in this analysis. This analysis is based on the assumption that you don't understand what's getting you in trouble. In those circumstances, you know exactly what the issue is.
**In a neutral game (no one is already tilted, argumentative, trolling, or anything of the sort):**
0. Avoid hate speech. This really should be obvious, and is completely unambiguous.
1. Avoid pointless criticism. Examples of pointless criticisms include: Stop feeding, should have warded, jungler never ganks, etc. This sort of criticism adds no value to the game and annoys the player being criticized.
2. If you wish to lead, communicate in concise, tone neutral directives. Examples: Let's baron, Ward red, No flash mid, gank please.
3. As an addendum to #3, if people don't follow you, LET IT GO. Clearly you haven't established yourself as "the leader" of your team. No amount of chatting at them will fix this.
**In a game that has gone negative:**
0. NEVER ARGUE WITH SOMEONE IN GAME OVER ANYTHING. DOESN'T MATTER IF YOU'RE RIGHT OR WRONG.
1. Make 1 (one, singular) attempt to diffuse the situation. Example: Oops, my bad, should have tried to gank sooner.
2. If #2 fails, say you've muted them, but don't actually do it. Often they'll stop if no one is listening. They might have something positive to say later on.
3. If #3 fails, actually mute them.
4. If you had to do more than #2, report after the game. Don't mention you're reporting. Don't ask for reports. Just do it.
If you follow these simple instructions, I guarantee you won't get penalized.