Current progress of MediaCoder transcoding farm (Stanley posted on December 8th, 2007 )

Right I am spending day and night on the implementation of the transcoding farm (multiple computer transcoding). As the initial goal is to improve HDTV transcoding speed effectively, I started from implementing the farm for x264. The MiniWeb project which I registered on in 2004 became active again this time as it will act a quite important role in the new transcoding system.
The idea is straight-forward. Assuming there are 3 computers in the LAN. PC1 acts like a controller and PC2 and PC3 work as agents. Both controller and agent have MiniWeb built in. For controller, MiniWeb is for collecting encoded data from agents as well as serving original file for agents. For agent, MiniWeb is for receiving and responding to controller commands. Now a huge 1080i HDTV file is to be transcoded. PC1 connects to PC2 and PC3 telling them they need to transcode part of a file. It will serve the file with MiniWeb through HTTP protcol. On getting the command, PC2 and PC3 start to process the file via HTTP from different given positions. During the procedure, their output will be posted back via HTTP to PC1. PC1 is also processing part of the whole file. After the 3 parts are transcoded, they are joined on PC1. PC2 and PC3 don’t need to have any temporary storage and all they need to have is a fast CPU.
Up to now I have finished following tasks:

  1. add file uploading feature to MiniWeb
  2. patch x264 to give it the ability to push data to web server instead of writing to local file
  3. implementing MediaCoder agent (a CLI program, more like a standalone web server which can respond to HTTP requests and doing transcoding)
  4. add controller features to current MediaCoder so that it can send commands, serve streams and collect encoded data.

A usable prototype will come out by the end of this year. Please stay tuned.

COMMENTS: 12 Comments »

12 Responses

  1. HTM says:

    cant wait.. 😀

  2. SirAuron says:

    Great work! I figured you worked on this^^

    Tell me if you need an alpha tester (right now I can serve a Core2Duo @2,54 ghz and a notebook with a Pentium M 1,6 ghz). In future I may also use the EEE PC which has a 900 mhz Processor (would this one suffice as controller?).

  3. JJC says:

    Greetings, Stanley:
    Just to ensure your ‘cred’ isn’t unduly tarnished: correct the spelling of “Brief” on your Home page.
    Amazing work (otherwise).

  4. stanley says:

    Thanks, the typo is corrected.

  5. stanley says:

    Hey SirAuron, EEE PC is really cute. I really want buy one too. 😀
    The controller will transcode a portion of the file and the whole of audio and is responsible for joining all the agent encoded parts togother. So I am afraid EEE PC isn’t sufficient for the job. 😉

  6. SirAuron says:

    Oh well, I want the EEE PC primarily for its portablity and cheapness (and because I can use Ubuntu and Windows XP with it). I’ll buy a small 2.5″ harddrive for it and propably use it in school/uni (although I am not sure what I want to do at the university:( ). So it will be my portable word/internet/movie station.

    Hope I will be able to get one as they seem to be very rare in Germany.

  7. kehn says:

    Thanks a lots! You and all contributors are doing such a great job:)
    I’m a newbie to the field of video/audio converting!
    And, for the first time I use MediaCoder, I find it very easy to use with friendly interface.
    Besides, I see the difference in SPEED while transcoding from files.WMV to files .VOB (standard DVD PAL).It takes much of time, the speed is about 22-23fps.When transcoding to files .avi, the speed is faster of about 55-56fps!
    Can I adjust the speech of transcoding or anything I can to do to speed up the transoding process to files .vob?
    I can say that: MediaCoder is completely nice!
    I also use VirtualDub, I see this program is good too.Is there any chance MediaCoder will have more effects like adding logo or watermark?

  8. milanbb55 says:

    i am fascinated what can one person 🙂 i trust that mediacoder has very BIG future 🙂

  9. Nitrocosm says:

    This is a great application and I am using it heavily on multiple machines right now to preserve all 30 of my T-120 VHS tapes full of home videos as h.264 avi’s.

    This works very well and I plan on making MediaCoder a permanent part of my video studio’s arsenal of tools. I look forward to the transcoding farm tools!

    Keep up the good work; you’re developing the best video encoding tool I’ve ever come across.

  10. lo says:

    Just wanna let you know that it makes no longer sense to bundle the neroAACenc_SSE.exe:

    Even before it was slower than the normal updated one, according to one of the nero developers.

    Besides it segfaults and quits on AMD machines.

  11. Brian says:

    Thank Goodness!! I’ve been WAITING for a cluster/farm transcoder for what seems like ever. I can hardly wait. Thanks for working on this. Media Coder is a powerhouse, and this will make it a juggernaut!

  12. Trevor says:

    I have done a lot of work building, implementing and tweaking render farms and HPCCs. If you have any questions, please feel free to email me.

    The main things you have to think about are:
    1. MPI – Message Passing Interface
    2. File System Sharing, CIFS or NFS work fine for this kind of work
    3. Scheduler

    For some ideas on this, you might want to take a look at Platform Rocks, which is a “light” version of Platform LSF.

    Best of luck!!! I am looking forward to being able to utilize all of my systems to transcoding!