We can, however, state with certainty that Christmas-themed music has been produced since the fourth century -- although it wasn't part of most religious services until much later.
Until St. Francis of Assisi in the 12th century, Christmas music wasn't typically used in religious services. And even after that, the presence of Christmas music in church wasn't continuous. Throughout the centuries, some felt Christmas music was inappropriate for what was supposed to be a solemn holiday, and so carols were sung on streets more than in churches (Oliver Cromwell even banned Christmas carols in the U.K. during his 17th century reign).
Regardless, the oldest Christmas songs we know about are religious. St. Hilary of Poitiers composed the Latin carol "Jesus refulsit omnium" ("Jesus illuminates all") in the fourth century, presumably after the first recorded Christmas celebration (336 A.D.). Also in the 4th century, the Roman Christian poet Prudentius composed "Corde natus ex Parentis." These, however, are more hymns than carols.
The French version of "The Friendly Beasts" (written about the animals surrounding Christ at the nativity) traces back to the 12th century, making it a strong contender for the oldest Christmas carol that people still sing regularly today. Another French oldie, "Entre le bœuf et l'âne gris" (English: "Between the Ox and the Grey Ass"), is placed by scholars as far back as the 13th century. The German/Latin carol "In Dulci Jubilo" (later used by Johann Sebastian Bach) also dates back to the Middle Ages.
In terms of carols originally composed in English, the history is more recent, but it's similarly uncertain. While "Adam lay ybounden" has been traced back to the 15th century by scholars, the song -- which is mostly about Adam from Genesis -- hardly resembles a Christmas carol in terms of lyrical content (and the original music has been lost). Other 15th century English carols "This Endris Night" and "Sir Christmas" are more in the holiday spirit, but mostly forgotten.
Fast forward to the 16th century, though, and we've got a song with household name recognition: "God Rest Ye Merry, Gentlemen." While the standard lyrics come from the 19th century, variations on the song go back to at least the 1650s. A century later, in 1739, we have the first published version of "Hark! The Herald Angels Sing." By the 1800s, a number of the Christmas carols we know and sing today start cropping up.
In short, the title "World's Oldest Christmas Carol" doesn't really have one easy answer ready for holiday party trivia. Like much music history prior to the Guttenberg press, the vast majority has been lost. So for all we know, traveling minstrels were singing songs along the lines of "All I Want For Christmas Is You" back in the Middle Ages.