x264farm

A distributed video encoder

Copyright © 2006 Reed Wilson ("Omion")
ce​di​ll LOREM IPSUM DOLOR SIT AMET a ACCENT gm CONSECTETUR ADIPISICING ELIT ail DONT​ cHmm.. it looks like you don't have CSS. My email will be very hard to decipherSED DO EIUSMOD TEMPOR INCIDIDUNT UT LABORE ET DOLORE MAGNA ALIQUAom

Abstract

x264farm is a program which utilizes x264 in a distributed environment to improve encoding speed. It is designed mainly to be portable and relatively easy to use.

License

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. Please see gpl.txt for more details. Source code for this program should be available from the same place as the compiled version.

Installing

x264farm uses two distinct parts: the agent, which runs x264 on each encoding computer, and the controller, which gives commands and data to each of the agents.

Installing the agents

Each computer that you want to run an agent on must have a copy of x264. Similarly, in order to use the nice function, you must have the "nice" program installed to a location on the path. I have made a Windows version of nice which can be downloaded on my site; most Unix-ish operating systems already have it.

  1. Copy the contents of the agent directory somewhere. You must, of course, use the version that has been compiled for your architecture.
  2. Create a temporary directory for storing the results of each x264 run.
  3. Edit the contents of config.xml to reflect your computer's configuration:
    1. <temp> should be set to the directory you made in step 2.
    2. <port> can be just about anything greater than 1056 and less than 65535. <port> specifies a range of ports to use, in case one of them is tied up with something else. Specifying too many ports will make it difficult for the controller to connect if the agent gets reset. Specifying too few ports will, oddly enough, create the same problem.
    3. <x264> should be set to the name of the x264 executable. It will default to "x264" if this line is not present.
    4. <nice> is used to change the priority of the x264 process. 0 is normal priority, 20 is idle. On Windows, 0-3 is "normal", 4-11 is "below normal", 12-20 is "low".
    5. <number> currently does nothing in the agent's config, but in the future it may set a maximum number of jobs being worked on. Currently, the agent simply accepts all of the jobs that the controller sends it. The agent will even work on jobs from multiple different controllers at the same time.
    6. You may also add a <base> parameter to attempt agent-based encoding (see encoding methods)
    7. <agentpipe> can be set to 1 to indicate that the agent should pipe data from avs2yuv even in agent-based encoding. If you have a 64-bit version of x264 and a 32-bit version of avisynth, you need to set <agentpipe> to 1 in order to get agent-based encoding to work. Otherwise, there's really no reason to use it.
    8. <compression> should only be changed if you have a fast controller computer connected to a slow network. The order in which the compression types appear is the arder which they are tried. Therefore, if <type>0</type> is at the top of the list, compression is turned off. The currently supported compression types are:
      • 0: No compression
      • 1: Paeth / Huffman. Tends to get 3:1 - 4:1 compression on most movies
      <passcompression> may be used instead of <compression> for specifying different compression settings for the first and second passes. Within the <passcompression> segment, set the <first> and <second> settings the same way as the above <compression> segment. For example, to use compression on only the first pass, you may use the following:
      <passcompression>
       <first>
        <type>1</type>
       </first>
       <second>
        <type>0</type>
       </second>
      </passcompression>
      or simply leave out the <second> block.
    9. If using ad-hoc agent discovery, add an <adhoc> element specifying the ports to use, like so: <adhoc controller="12345" agent="23456"/>.
    10. Also with ad-hoc, the <name> element will broadcast the name for the agent to be associated with.
  4. As of 1.09, the agent will clean up the temporary directory of any files older than a certain amount. By default, it will delete files which were created over a week ago, but you may use the --stale command-line option to specify the number of seconds before temp files are considered abandoned. Use --stale 0 to turn off old file deletion.

Installing the controller

The controller should be installed on the most powerful Windows-based machine on the network. The controller needs to have access to the avs2yuv program and all the source files and filters that any encode needs.

  1. Copy the contents of the controller directory to somewhere that makes sense (like c:\program files\x264farm\controller)
  2. Create a temporary directory for storing the work returned from the agents
  3. Edit the contents of controller\config.xml to reflect your network configuration (note that the config file has changed since 1.02):
    1. <temp> should point to the temporary directory you made in step 2.
    2. Make one <agent> for each computer you want running an agent. Multi-core computers only need one agent running.
    3. <name> may be anything, and will only influence the output messages.
    4. <ip> is the IP address of each agent. You may use 127.0.0.1 to refer to the computer running the controller.
    5. <port> should be set to whatever you set <port> to in the agent's config.xml file.
    6. <number> is the number of jobs to send out to the agent. This basically specifies how many encodes to do at one time. It should be set to the number of processor cores in the agent's computer.
    7. Use <number pad="1"> to send one additional job to the agent during the second pass. Since the second pass jobs are so short (generally a few seconds) it is a good idea to have something else working in the background when a job gets done. This setting has no bearing on the first pass.
    8. If you want to use ad-hoc agent discovery, add an <adhoc> element specifying the ports to use, like so: <adhoc controller="12345" agent="23456"/>. Note that this line should be the same in both controller and agent config.xml files in order to work properly.

Running

  1. Start up each of the agents with the command agent. If your config file is not called config.xml, use agent --config other_config_file.xml.
  2. The agent should respond with something that looks like this: x264farm version 1.03-152
    Using config file ".\config.xml"
     Temp dir: "E:\x264farm\temp\agent"
     Ports: 50700 - 50703
     x264: "x264.bat"
     nice: 20
     agents: 2
     agent base: ""
    trying port 50700
    bound to port 50700
    listening...
    working!
  3. Start the controller. See the controller options section for details.

Controller options

The minimum options that should be given is --first, --second, --bitrate, --avs, and --output. (or their equivalent short forms -1, -2, -B, -i, and -o)

General x264 parameters

-B / --bitrate "100%"
The second pass bitrate. May either be specified in kbps (-B 789kbps) or as a percentage of the first pass bitrate (-B 100%)
-1 / --first ""
The options to use during the first pass, enclosed in quotes. DO NOT specify an input file, output file, bitrate, or any option listed in the assorted parameters section. An example of this switch is --first "--crf 19 --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 5 --ref 8 --me umh --bframes 7 --weightb --b-pyramid --analyse all"
-2 / --second ""
The options to use during the second pass. Like the --first option, don't specify an input file, output file, bitrate, or any option listed in the assorted parameters section. An example of this switch is --second "--trellis 2 --bime --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 7 --ref 12 --me umh --bframes 7 --weightb --b-pyramid --analyse all --8x8dct"
-i / --avs
The input AVS to use for encoding.
--firstavs
An alternate AVS for encoding the first pass. Many options such as postprocessing do not need to be in the first pass, so you can pass a simplified script with this option. It must have the same dimensions and framerate as the --avs file. It will default to whatever --avs is.
--fastavs
This file is used for split detection during the first pass. It will not affect quality at all. It must be the same framerate as the --avs and --firstavs files, but it may have different dimensions. As long as the output of this file changes significantly when a scene change occurs, it may be as simple as you can make it.
-o / --output "output.mkv"
The output file. Only Matroska (MKV) files are created. Specifying a different file extension will result in an MKV file with the wrong extension.

x264farm control parameters

--preseek 0
How many frames to pre-render before each job. The only time I've encountered needing this option was using telecide mode=2, when --preseek needs to be set to 5.
--force
This will re-encode the file even if the controller has determined that it has been encoded before. Generally this will cause it to restart at the beginning of the second pass.
--restart
Delete all intermediate files and start over from scratch.
--config "config.xml"
This is the location of the configuration file. It will default to config.xml, and will be searched for in the current directory, then the directory the controller executable is in.
--savedisk
When x264farm gets done encoding, all the scenes are located in separate files in the controller's temp directory. The last pass is to merge them into a single file. By default, the temp files are only deleted after all of them are merged, which will use about twice the disk space of the completed file. Using --savedisk will delete each file right after it is merged, which does not use too much extra disk space. However, if the controller is restarted during merging, the deleted files will have to be re-encoded.
--nocomp
Turns off compression for controller-based encodes, even if agents request it. Use this if you experience errors or corruption with compression.

First pass split parameters

--batch 5000
The minimum number of frames to send out in one job at the beginning, assuming there are this many contiguous unencoded frames. If this number is smaller than the -I / --keyint option passed to x264 (250 by default), then it may take a long time to do anything..
--batchmult 0.5
How much smaller the jobs at the end will be. 0.5 means that, after most frames have been encoded, the jobs will be around 1/2 what the --batch parameter is set to. This is to prevent a slow computer from getting a big job at the end of the encode, although it currently doesn't work too well.
--split 250
The maximum number of frames to look for a split point after --batch frames have been found. This value should be larger than x264's -I / --keyint option.
--thresh 20.0
This is sort of the carefulness of the splitter. A higher number will result in more frames tested before a split point determination is made. No more than --split frames will ever be tested, though. See the split point section for more details.
--rethresh 10.5
This uses the same units as the --thresh option. Increasing this number will cause more jobs to be temporarily overlooked if they have a weak starting point. Using --rethresh 1.0 will revert to the way 1.12 and before handled jobs. If not specified, it will default to (--thresh + 1) / 2.

Selective third-pass parameters

--3thresh 0.8
The maximum accuracy to consider a scene eligable for a re-encode, from 0.0 (re-encode nothing) to 1.0 (everything is eligable).
--3gops 1073741823
The maximum number of scenes to re-encode.
--3ratio 0.05
The maximum ratio of re-encoded scenes to total scenes. The default value will re-encode a maximum of 5% of the scenes.
--rerc 0
The number of scenes to re-encode before the ratecontrol is run again. 0 means the total number of scenes in the file.

Assorted x264 parameters

These are the same as the x264 parameters of the same names. They should never be used inside the --first or --second parameters. They apply to both the first and second passes (except the ratecontrol parameters which don't make sense during the first pass).

Other parameters

-h / --help
Show a help message and exit
(anything else)
Every non-option is taken to be a file which contains more options. This is useful if you have a set of common encoding settings which you don't want to type in every time. The file must have one option per line, with blank lines and lines starting with "#" ignored. Here is an example:
# The -B and the 100% MUST be on separate lines
-B
100%

# First pass settings
# "--crf 19" is the argument to --first
# Therefore "--crf 19" must be on ONE line
--first
--crf 19

# Here is the second pass line:
--second
--trellis 2

--batch
#230 - This line is ignored
3400

How it works

The agent program is quite simple: encode the data recieved from the controller, and send out the encoded data. The controller is quite a bit more complicated, and can be divided into 2 parts: the first pass, and everything after that.

Controller: the first pass

The first pass is also made up of two parts, each of which run simultaneously: the splitter and the workers

First pass: splitter

It is the splitter's job to take any unencoded parts of the video and turn them into ranges for the workers to take. The length of the ranges is determined by the --batch, --split, and --thresh options. Once a range has been split, it is put in a queue for the workers to pick up.

First pass: workers

The workers pick out ranges from the splitter and stream the raw video data of the range out to the agents. They then recieve the stats file produced by the agent and process it. If the split point chosen by the splitter was found to be bad, the worker will signal the splitter to re-split the end of the range.

When all the workers are done and the splitter can't find any frames to encode, the resultant stats file is written and the first pass ends.

Controller: everything after the first pass

When the first pass ends, the controller reads the stats file and breaks it up into GOPs. Each GOP starts with an "I" frame and goes to (but does not include) the next "I" frame. This represents the smallest group of frames which can be calculated independantly. After the stats file is split into GOPs, they are sent to the ratecontrol equation to figure out the optimal bitrate of each GOP. Each GOP is sent to an agent to be encoded into a video fragment. When the video fragment comes back, it is analyzed to see how many bits it actually used. The ratecontrol used is designed to be as close to the ratecontrol done by x264 itself. Therefore, defining your own ratecontol in the second pass will not have much effect at all, and will only result in more re-encoded frames.

Once all the GOPs have been processed, the ratecontrol is done again on the (hopefully) more accurate stats of the second pass. Here, the issue of accuracy should be mentioned: the closer the optimal number of bits (specified by the ratecontrol) and the actual number of bits, the more accurate the GOP is, and the less it needs re-encoding. The actual equation is: max(optimalBits,actualBits) / min(optimalBits,actualBits).

After the second pass is finished, each GOP is processed from the least accurate to the most accurate. When a GOP is finished for the third pass, it is put back in the same list, and therefore may be processed multiple times before the total encoding is finished. The number of GOPs processed is determined by the --3thresh, --3gops, and --3ratio options. --3gops determines the total number of GOPs to recode after they have been allprocessed. --3ratio is the same thing, but in terms of the percentage of total GOPs (i.e. 0.1 will process at most 10% of the total GOPs again). --3thresh indicates a maximum accuracy to encode before deciding that the film is "good enough." Basically, if the accuracy of all GOPs is greater than --3thresh, then encoding ends. If any of those three conditions are met, then the encoding is over and the final video file is spliced together.

Ad-hoc agent discovery

As of version 1.15, the agents and the controller will attempt to find each other over the local network. When an agent starts up, it will send a broadcast to all the computers on the local network. Controllers will also broadcast messages every minute to search for new agents on the local network. These features allow agents to be dynamically added to the encoding process. In 1.15, this feature is disabled by default. Enable it by adding the appropriate line to the controller's and agents' config.xml file. The agent will listen on the "agent" port and send on the "controller" port. Conversely, the controller will listen on the "controller" port and send on the "agent" port. The "agent" port must not be the same as the "controller" port, but both ports must be consistant in each config.xml file (that is, the line itself should just be copied and pasted in each file).

Encoding methods

There are currently two encoding methods available to the agents: controller-based encoding and agent-based encoding.

Controller-based encoding

Controller-based encoding is the default encoding method. The controller starts up avs2yuv and sends the agent all of the rendered data across the network. This is the most portable method of encoding, as the input files need only be available to the computer with the controller. However, it is also the least scalable method of encoding, since the controller's computer is rendering all of the data.

Agent-based encoding

Agent-based encoding is used if the agent uses the <base> option in its config.xml file, and the agent determines that the AVS file is available to the agent. In this case, the agent will start up x264 directly, and does not need to be sent the data across the network. This method is extremely scalable, but it is more difficult to set up. Each agent using this method needs access to AVIsynth, all the filters, and the AVS file itself.

Setting up agent-based encoding

The <base> option in the agent's config.xml file determines where to search for the AVS file. If no <base> option is specified, or if it points to a nonexistant directory, controller-based encoding is used instead. If you use <base/> or <base></base>, it is interpreted to mean only search exactly where the controller reports it. This is useful on the controller's computer, especially if you plan on encoding from multiple hard drives. If the directory specified by <base> is valid, the AVS file is searched from the deepest controller directory fragment within the <base> directory. For example:

Controller's AVS file: D:\movies\working\somewhere\file.avs
Agent's <base> directory: F:\temp\x264farm

The agent's AVS file is searched for in the following locations, in this order:
F:\temp\x264farm\D:\movies\working\somewhere\file.avs (can't exist, but is checked anyway)
F:\temp\x264farm\movies\working\somewhere\file.avs
F:\temp\x264farm\working\somewhere\file.avs
F:\temp\x264farm\somewhere\file.avs
F:\temp\x264farm\file.avs

The first of those AVS files that matches the controller's AVS file is used as the input to x264. If it is not found in any of those places, controller-based encoding is used instead.

How split points are chosen

During the second pass, the video is split into GOPs and sent out to be encoded. But the first pass does not know where the GOPs are, so it guesses based on a few parameters.

The splitter will break down un-processed parts of the movie into pieces which are larger than --batch, but smaller than --batch + --split. The splitter will compare every pair of frames starting at --batch, in order to determine the frame which is most likely to be an "I" frame. This is the basic loop that occurs:

Find the difference between the next two frames
Update the average difference
Update the current threshold to be between --thresh at the first frame analyzed, to 1 at the last (--split) frame to be analyzed
If any frame difference is more than "threshold" times the current average difference, the largest frame is used as the split point
Otherwise, the next pair of frames are analyzed

The threshold changes in order to take advantage of the properties of the video. The alternative would be to read a set number of frames and pick the two consecutive frames which are the most different from each other as the split point. However, there are two problems with the static approach:

  1. If there is an obvious candidate early on, then the rendering of all the other frames has gone to waste
  2. If all the frames are more or less the same, it may be wise to analyze more frames to see if there is a better frame later on

To summarize, the options which affect the split points are as follows:

--batch
This is the minimum number of frames to have in one batch. The loop given above starts this many frames from the beginning
--split
This is the maximum number of times to run the loop. Generally, this is never reached thanks to...
--thresh
Increasing this will increase the initial cautiousness of the splitter. It will process more frames before making a decision. The initial threshold is set to this number and goes down to 1

Caveats:

Bugs:

Tips:

Changelog:

1.15 (20071025):
Added ad-hoc agent discovery (although disabled by default). Agents may be added while the encoding is running, even if they are not specified in the controller's config.xml file.
Ability to specify log file location on command line for both agent and controller.
Faster controller-based compression.
Changed the bucket-based agent cache to a single circular buffer.
Changed the default port from 50700 to 40700. This will only affect you if you re-use one of my example config files.
1.14 (20071003):
Fixed an inconsistency with running x264 with spaces in the path.
Added the capability to change the avs2yuv executable in the config.xml file.
Add timestamps to the error messages and when the display was last updated (counting from when the program was started).
Show "~" for errors if there are aren't enough to display, and made the default number of errors displayed 4.
Doesn't update the FPS on the second pass if the encoding failed.
Properly handle dead agents on the second pass.
1.13 (20070703):
Added the --batchmult option to help with one agent checking out a big job at the end and making everything else wait for it.
Added previous split points to the 1st-pass temp file. This means that encodes will start up faster after having been stopped.
Optimized the splitter to try to get the jobs out faster in the beginning.
1st-pass agents will now try to avoid working on jobs which may have incorrect start points, in order to minimize the number of re-encoded frames.
Put ETA display back for both passes. Note that the ETA given in the first pass measures the end of the first pass only, and is incredibly inaccurate.
Second pass GOPs are no longer encoded chronologically.
Completely redesigned first pass to be easier to work on.
1.12-173 (20070530):
Added a heartbeat thread to each agent when agent-based encoding starts. Agents will send a signal to the controller every 10 seconds. If the controller does not receive a signal within 30 seconds, that agent is given up for dead. This should greatly reduce on the amount of stalled encodes.
Controller output is much more organized, and no longer prints huge amounts of scrolling garbage. The out-dump.txt files are just as garbage-filled as before, though.
1.11-168 (20070428):
Fixed an overflow in the Matroska timecode calculation.
Second-pass equality check takes bitrate into account.
Changed the GOP layout from B+trees to red-black trees (not that anybody would notice the difference).
1.10-164 (20070425):
Implemented per-pass compression, so that the first and second passes have different compression priorities.
More precise file size (tunable with the --sizeprec option)
Ratecontrol handles extreme bitrates better.
The --keeptemp option actually works now...
Minor sanity checks when writing the files.
1.09-163 (20070310):
Agent now deletes temp files which are older than a specified amount of time (by default 1 week).
Agent now uses niceness when both nice and piped input are being used.
2nd pass bitrate should be more accurate.
1.08-161 (20070207):
Redesigned first pass. It should pick jobs more intelligently. The occasional problem with credits taking a very long time to encode is minimized.
More verbose garbage printed to the screen regarding the new first pass mode. It will disappear shortly.
Second pass should recover from x264 complaining about the minimum bitrate. Some jobs may fail a few times, but they will encode eventually.
The agent and config.xml format are unchanged.
1.07-158 (20070105):
Fixed an issue with the controller computer rejecting network connections after an hour or so
Made some of the more pointless messages log-only
1.06-156 (20061225):
2nd pass resuming works again (sorry, guys!)
Plugged the hole in the first pass which could cause frames which are being worked on to be re-encoded
Compression should be faster. There are no more checksums, and the compression loop is faster.
Fixed up the re-encode part to ask for confirmation instead of just quitting, and will just automatically restart if the old output file does not exist anymore.
Fixed a potentially major bug when the output files aren't written if the requested bitrate is below what x264 thinks is the minimum bitrate.
1.05-154 (20061212):
Added optional compression for controller-based encodes. Currently extremely slow, though, in order to check for validity.
Added --nocomp switch to force compression off.
Added <pipe> setting to agent's config.xml file to locally pipe data to x264 in agent-based encodings.
Added tables in 2nd pass to show the performance of the various agents.
Controller now searches its own directory for avs2yuv.
Hid a bug involving the 1st pass splitter re-splitting ranges which are already allocated to agents not using controller-based encoding.
1.04-153 (20061124):
Made agent-based encoding.
User-selectable number of agent cache buckets.
Added many ratecontrol parameters to the controller, and pass them to x264.
Lowered latency and memory usage due to sending data across the network.
1.03-152 (20061120):
Fixed an error when the number of frames per second was an integer.
1.02-151 (20061117):
MAJOR rewrite that honestly shouldn't have needed to happen.
Made the agent per-computer instead of per-core.
Fixed a bug with remote computers not getting enough data.
Controller only connects once per session, instead of once per job.
The code itself is much better-written.
Config file syntax completely changed.
1.01-150 (20061109):
Made the --rerc option in order to change the re-ratecontrol frequency during the third pass.
Made the network more resilient to delayed-ACK problems.
Removed some of the more verbose printing (although everything is still in out-dump.txt).
Added simple FPS display (although it's buried in the debug info).
Fixed the batch length (now 5000, was 25000 but claimed to be 2500).
1.00-148 (20061104):
Initial release. Rampant bugs and programmer hacks predominate.