Panjoya transforms video dubbing with the AI | Festina Lente - Your leading source of AI news | Turtles AI

Israeli startup Panjaya is revolutionizing video dubbing with a generative technology that allows you to recreate voices and movements of speaking speakers in different languages. The new platform, Bodytalk, aims to make multimedia content more natural and engaging, with the aim of improving the global use of videos in the B2B field.

Key points:

• Panjaya launched Bodytalk, a dubbing platform based on AI which synchronizes the voice and movements of the speaker’s body with the translation into 29 languages.
• The startup, founded by Israeli Deep Learning experts, has recently collected 9.5 million dollars of funding.
• The Lip Sync engine is developed internally to offer superior quality, with particular attention to the precision of facial movements and expressions.
• Panjaya turns to the B2B market and is expanding the use of technology in sectors such as education, health and marketing, with applications in almost real time.

In the panorama of emerging technologies for translation and video dubbing, Panjoya is making it talk about itself with an innovative approach that combines and Deep Learning to offer more natural and realistic multimedia experiences. The Israeli startup has in fact developed a platform called BodyTalk, which uses the AI ​​to translate not only the words, but also the voice and physical movements of the speaker, creating dubbed videos that maintain a high degree of authenticity. This hyper -realistic dubbing technology, which is able to faithfully replicate the original voice of a person in a different language, changes in almost real time also the movements of the lips and other facial expressions to adapt to the new speech, avoiding misalignments between audio and video. After a long period of research and development under the radar, the startup recently launched the first version of the product, attract the market with an external funding of 9.5 million dollars, which will serve to consolidate and expand its offer.

Panjoya is the result of Hilik Shani and Ariel Shalom’s ingenuity, two Deep Learning specialists who, after having worked for years for the Israeli government, have decided to make the big leap into the world of startups. In 2021, together with CEO Guy Piekarz, a former founder of Matcha, acquired by Apple in 2013 to contribute to the development of the video streaming platform of the Cupertino company, gave birth to this initiative with the aim of applying the potential of the Deep Video translation Learning. Although not founder of Panjaya, Piekarz brought with him a long experience in the media technologies and streaming sector, which had a positive impact on the evolution of the startup. His knowledge of digital media and his experience with Apple TV are reflected in the ambitions of Panjaya to reach a global audience with a highly innovative product.

The Bodytalk platform is able to perform an automatic translation in 29 different languages, and stands out for the ability to redesign the audio and video part of a multimedia content so that the translation is natural, with the item that imitates that of the speaker original. The changes to facial movements are also synchronized with the new language, obtaining an extremely realistic dubbing, which has nothing to do with traditional subtitles or standard dubbed versions. The entire translation and adaptation process to the new spoken schemes is made in an almost automatic way, starting from the video uploaded on the platform, where the user can also make small changes through an integrated editing dashboard. Although the technology is still in the aging phase, the platform is able to return high quality results in a few minutes, and the company is working to offer a version that allows real -time processing. Panjaya, in fact, does not limit itself to using third -party linguistic models, but is also developing its algorithms, in particular as regards the part relating to the Lip Sync, an aspect that requires a very high level of precision, often difficult to obtain with existing technologies.

Currently, Panjaya turns to the B2B market, with prestigious customers such as Ted and Jfrog. The choice to focus on this market segment allows startups to have greater control over the use of its technology, which could have important implications in terms of ethics and safety. Not surprisingly, one of the main objectives of Panjaya is to prevent its platform from being used improperly, as in the case of deepfakes. Although video synthesis technologies such as that of Panjaya can evoke, in some cases, distorted or disturbing images, Piekarz is firmly intending to distinguish its product from what is associated with deepfakes, a term that has a negative connotation now. The startup is implementing technological "guardrails" to monitor the use of the platform and prevent disinformation, including tracking measures such as watermark to identify when a video has been modified.

In a world where video content are increasingly global and where multilingual communication is fundamental, Panjoya technology could be the key to responding to an increasing question of authentic and convincing translations. If its intuition proves correct, the offer of content in native language, made with the support of advanced dubbing technologies, could exceed the traditional use of subtitles. It is no coincidence that one of the first customers of the startup, Ted, recorded a significant increase in views (115%) for its voids dubbed with Bodytalk technology, reporting a huge potential for the adoption of this type of innovation in the corporate sectors and training.

The growth of the automatic video and dubbing translation market, combined with the possibility of an even more immediate and natural use of the contents, marks the beginning of a new era in global digital communication.