An AIDS epidemic in the impoverished southern African country has orphaned more than one million children.
In her ruling, Judge Esimie Chombo warned against celebrity adoptions, saying they could lead to child trafficking.
"Anyone could come to Malawi and quickly arrange for an adoption that might have grave consequences on the very children that the law seeks to protect," she said.
Madonna's lawyer, Alan Chinula, said she would lodge an appeal with the Supreme Court on Friday. Her London spokeswoman was not immediately available for comment.
Madonna has entertained millions around the world with sexy high-energy performances and songs like "Material Girl" and "Papa Don't Preach," and created controversies along the way.
In 1989, the video for "Like A Prayer," with its links between religion and eroticism, was condemned by the Vatican and caused Pepsi-Cola to cancel a sponsorship deal with the star.
Madonna surprised fans in February by dedicating another of her hits, "Like a Virgin," to the pope at a concert in Rome.
Malawian rights groups, who accused the government of skirting residency laws when Madonna adopted David Banda in 2006, also opposed the latest adoption attempt.
Prominent rights activist Mavuto Bamusi defined said the decision was a "defining moment for child protection" but urged the pop star to continue her charity work.
"We finally appeal to Madonna to take this positively and continue to help the children of this country," he told Reuters.
Malawian Information Minister Patricia Kaliati said on Thursday that Madonna had helped in the country and was a worthy mother who was supporting over 25,000 orphans here.
Madonna's charity, Raising Malawi, plans to build a multi-million dollar school for girls in Chikhota village, about 12 km (7 miles) outside Lilongwe.
Madonna, accompanied by David, arrived in Malawi on Sunday ahead of the court examination of her application.
The star, who was divorced last year from British film director Guy Ritchie, is one of the most successful singers of all time, with album sales of more than 200 million.
(Reporting by Mabvuto Banda; Writing by Michael Georgy; Editing by Louise Ireland)
COPYRIGHT: (c) Reuters 2009. All rights reserved. Republication or redistribution of Reuters content, including by caching, framing or similar means, is expressly prohibited without the prior written consent of Reuters. Reuters and the Reuters sphere logo are registered trademarks and trademarks of the Reuters group of companies around the world.