The Araria Lok Sabha seat had fallen vacant following the death of RJD MP Mohammed Taslimuddin.
In Bihar, RJD is leading by 23,187 with 3,33,050 votes, while BJP trailing with 3,09,863 votes for Araria constituency.
The balloting in Jehanabad and Bhabua followed the death of RJD and BJP legislators respectively.
RJD leader Tejashwi Yadav.
In Bihar's Araria parliamentary by election 57 per cent polling was registered.
Former Bihar Chief Minister and Hindustani Awami Morcha chief Jitan Ram Manjhi celebrates after RJD party's success in Araria and Jehanabad Assembly elections in Patna on Wednesday.
RJD leaders and workers celebrated across the state by distributing candies and bursting fire crackers.
In Jehanabad, RJD's Suday Yadav, son of deceased MLA Mundrika Singh Yadav is ahead of his rival from JD (U) Abhiram Sharma, according to latest trends.
West Bengal Chief Minister Mamata Banerjee also congratulated Lalu for his party's win in the by-polls.
Party leaders said Tejashwi had a foolproof strategy to reach out to his party's traditional voters in both the constituencies. Now, the JD (U) is in alliance with the BJP and has put up its candidate to take on the RJDs nominee.
In Phulpur, SP's Pratap Singh Patel is leading by 29,474 with 218,963 votes, while BJP's Kaushlendra Singh Patel is second with 189,489 votes after 20 rounds of counting.
Rinki is the widowed wife of Anand Pandey, the former elected MLA of the constituency.
While, BJP retained the Bhabua assembly seat, RJD was able to hold on the the Jehanabad assembly seat. There is a sympathy wave, he said.
The bypolls were necessitated after Adityanath vacated the Gorakhpur Lok Sabha seat and Deputy Chief Minister Keshav Prasad Maurya the Phulpur seat following their election to the UP legislative council. Sukhdeo Paswan won on BJP ticket in 2004. "This ideology would break the arrogance of the saffron party", he said in another tweet.