I think the show you’re referring to is Big Cat Diary (I loved that show!). However, in regards to it being “tail-up” behavior- it’s not. A domestic cat’s tail-up behavior is when the cat holds its tail straight up.
While the leopard’s signal looks more like this:
As far as I know, the “leopard telling prey that they aren’t hunting” theory is also not valid. If you think about it, it makes no sense; why would the leopard evolve a signal to tell prey it isn’t hunting? Or, in other words, if the prey successfully interpreted such a signal, why wouldn’t the leopard switch to using it all the time to trick prey into thinking it isn’t hunting?
Both intraspecific and interspecific signals need evolutionary incentive to stay “honest,” otherwise they would never work. For example, in the three-spined stickleback, females prefer to mate with males that have redder bellies, because males with fewer parasites are able to produce brighter colors. The males cannot fake this signal, otherwise every male would try to be as red as possible and it would lose its meaning.
Back to the leopard- it wouldn’t benefit the leopard to make its prey think it isn’t hunting, because the leopard probably would love for its prey to think that all the freakin’ time. However, there might be someone else that the leopard wants to tell that it isn’t hunting, someone that could harm the leopard enough to make it evolutionarily worth it to express an I’m-not-hunting signal. That someone would be any other large predator.
I’m not talking about a cost in terms of “other predators would try to eat the leopard,” I’m talking about kleptoparasitism. Both Asian and African leopards live in habitats rife with large predators, from lions to tigers to hyenas and even to wild dogs. These predators can all be opportunistic and will try to steal a leopard’s kills when possible. It would certainly benefit the leopard to signal to these creatures when it would do them no good to follow it, especially if they are large, hungry lions or tigers.
But the above is actually something I just came up with as an idea; there’s no actual scientific evidence I could find for what the leopard’s tail-curl actually means, just speculation. Another theory is that it’s used primarily to signal cubs when it is safe to follow; this is probably actually the theory that makes the most sense.
I’ve poked around the literature and found surprisingly little on leopard body language and social behavior, so if anyone has any good sources on hand I’d be excited to read them!
Refs and further reading:
Bakker, T.C.M. and Mundwiler, B., 1994, “Female mate choice and male red coloration in a natural three-spined stickleback (Gasterosteus aculeatus)”, Behavioural Ecology 5: 74
Karanth, K. U., & Sunquist, M. E. (2000). Behavioural correlates of predation by tiger (Panthera tigris), leopard (Panthera pardus) and dhole (Cuon alpinus) in Nagarahole, India. Journal of Zoology, 250(02), 255-265.