That could prove fatal for the Alliance, with the Liberal and Center parties repeatedly ruling out a deal with the far-right.
And whoever wins, the rise of populist, nationalist and anti-immigrant parties is changing Europe's political landscape, with centrists adopting far-right language and policies to defend their vote.
Sweden's economy is booming but many voters are concerned that housing, healthcare and welfare services have come under pressure from a wave of immigration during the 2015 migrant crisis.
Moderate MEP Christofer Fjellner predicted Sweden "will be governed by the Alliance... there are three blocs and then you sit in the middle and govern".
The SD, widely tipped to make gains in the election, won about 18 percent support, jumping almost five percentage points since the last elections four years ago.
Jimmie Åkesson, Sweden Democrats leader, promised to wield "real influence" in parliament after the modest gains. While the Social Democrats initially welcomed immigrants with open arms, party officials have recently hinted at support for restricting immigration.
Overseas votes, which could make a slight change to the outcome, have not yet been counted and the final result is not expected until Wednesday.
Prime minister and party leader of the Social Democrat party Stefan Löfven waves at an election party in Stockholm, on Sunday.
The party, with roots in the neo-Nazi movement, called the arrival of immigrants to Sweden a threat to the country's culture, European news network The Local reported. The Sweden Democrats hotly deny these claims. Sweden now looks set for some complicated talks about which side can form a workable goverment.
The party also wants to hold a referendum on Sweden's European Union membership along the lines of the Brexit vote in the U.K. Despite a clear victory, Sweden Democrats seem to have gained less than one-fifth of votes and remain the third-largest party.
The Sweden Democrats rise comes after widespread discontent over immigration, particularly the massive influx of refugees which started in 2015.
The Sweden Democrats will likely continue to influence the country's politics as talks begin for forming the next government.
But centre-left and centre-right parties face an uphill battle in trying to form a viable coalition government, following decades of acrimony between the two blocs that have defined Swedish politics for decades.
The election had been watched closely for signs about the extent to which a cascade of anti-immigrant fear could hit even Sweden, which has always been one of Europe's most open nations toward refugees.
"It's not that they are shy voters, but that they are distrustful of the polling agencies", said Henrik Ekengren Oscarsson, a professor in political science at Gothenburg University.
Mattias, a Stockholm resident at an election night party in the city, said he was "extremely concerned" about the far right's steady climb since it entered parliament in 2006 with 5.7 percent. If he loses that, the speaker of parliament presents a new candidate to become prime minister. Among other things, the party no longer advocates for reintroducing the death penalty or for limiting the adoption of non-Nordic children.
At the same time, the SD is a one-issue movement, nearly exclusively focusing on immigration - and demanding a much stricter approach to it.
Many voters are also concerned about violence. Instead, it told a more subtle but increasingly familiar tale now seen across a variety of European parliamentary systems and perhaps further afield, too - that of increasing political fragmentation and the slow decline of dominant political parties.