Koulibaly was dismissed in the 81st minute after he was first booked for a foul on Matteo Politano and was then shown a second yellow card for sarcastically applauding the referee's decision.
"We asked three times to do something but it just went on".
Koulibaly and his manager Carlo Ancelotti were furious after persistent racist chanting. "It's not good for us and it's not good for Italian football".
"I'm disappointed by the defeat, but above all at leaving my brothers", Koulibaly wrote on Twitter after the game.
'Maybe we have to take matters into our own hands next time and stop play ourselves.
The Senegalese was subjected to monkey chants at the San Siro with Carlo Ancelotti insisting the events shamed Italian football. It remains to be seen whether this latest episode will lead Koulibaly to seek a career outside Italy. Napoli would demand a transfer fee over £70million. And, in an unrelated incident, four Gli Azzurri supporters were stabbed about an hour before kick off in Milan, during violent clashes between the two sides' ultras. The 35-year-old was reported to have been struck by a van.
The Napoli fans got out of the van and fought back, with several receiving knife wounds, before getting back into the vehicle on Via Novara - one of the main roads toward San Siro - and driving off, knocking down the Inter fan on their way.
At a news conference, the Chief Constable of Milan, Marcello Cardona, said two arrests had already been made and a third person had been identified and is being sought in relation to the clashes, which he said involved "around 100 ultras" including some from Varese and Nice, who had "come on objective for the fights".
Lega Serie A announced on Thursday that Inter would have to play their next two home games behind closed doors, while the Curva Nord section of San Siro will be shut for a third match. The championship will not stop. Those who do not understand this history do not stand with us'.
The mayor of Milan, Guiseppe Sala, apologised to Koulibaly on his Facebook page, calling the abuse "a shameful act against a respected athlete".