You can't earn XP from playing a custom game no matter how many people you have in that custom game. Before the preseason 8 update, you were able to earn IP from custom games. Here are some of the rules for custom games from before the preseason 8 update according to [(http://leagueoflegends.wikia.com/wiki/Custom_game)]
* Leaving a custom game will not count towards your game's left counter.
* Custom games do not count towards champion kill/death/assist ratings.
* Just because it is a custom game does not exclude players from the summoners code.
* No EXP/IP is rewarded If the number of players on each team is not equal.
* No EXP/IP is rewarded if the number of human players is less than half the maximum players for the map. (Need 3 players on Twisted Treeline, 5 players on Howling Abyss or Summoner's Rift)
* If you are summoner level 15 or higher, you may earn EXP/IP from up to 120 minutes per a day in custom games. Additional game time rewards no EXP or IP. (This timer is shared with co-op vs. AI games as well)
* Custom games do not award matchmade IP, and from summoner level 15 upwards, award 25% less game time IP than do normal games on the same map.
If all the real players in a custom game quit, the custom game ends and players can no longer reconnect. However, neither IP nor EXP is rewarded to the players.
* On certain days, when there is high traffic on the server, custom games would require at least 2 real players to start the game. This is done to lessen the burden on the system hosting multiple single player games. (Since the start of Season One the minimum number of player required can get up to 10, if the server is too overloaded with matches)
Notice that none of the rules above say that you don't earn IP for playing custom games (The one that says you don't earn matchmade IP means that you don't earn the same IP that you would from playing a normal game), so why do we not earn XP for playing custom games?
1) Start a custom game with 9 other people and have both teams full (play a 5v5 match)
2) Let the match last for at least 20 minutes (let the custom function just like a normal game)
3) End the custom game.
Once you get to the end-game screen, you will notice that you have not earned XP from playing that custom game. This has happened in every custom game that I've played with 9 other people. I'm sorry that I don't have a picture or video, but I don have friends who can testify with me.
I feel like it might be important to mention that I posted something mentioning this in the General Discussion, but that probably wasn't the right spot to post it, and it didn't get much attention.