x264farm is a program which utilizes x264 in a distributed environment to improve encoding speed. It is designed mainly to be portable and relatively easy to use.
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. Please see gpl.txt for more details. Source code for this program should be available from the same place as the compiled version.
x264farm uses two distinct parts: the agent, which runs x264 on each encoding computer, and the controller, which gives commands and data to each of the agents.
Each computer that you want to run an agent on must have a copy of x264. Similarly, in order to use the nice function, you must have the "nice" program installed to a location on the path. I have made a Windows version of nice which can be downloaded on my site; most Unix-ish operating systems already have it.
agent directory somewhere. You must, of course, use the version that has been compiled for your architecture.config.xml to reflect your computer's configuration:
<temp> should be set to the directory you made in step 2.<port> can be just about anything greater than 1056 and less than 65535. <port> specifies a range of ports to use, in case one of them is tied up with something else. Specifying too many ports will make it difficult for the controller to connect if the agent gets reset. Specifying too few ports will, oddly enough, create the same problem.<x264> should be set to the name of the x264 executable. It will default to "x264" if this line is not present.<nice> is used to change the priority of the x264 process. 0 is normal priority, 20 is idle. On Windows, 0-3 is "normal", 4-11 is "below normal", 12-20 is "low".<number> currently does nothing in the agent's config, but in the future it may set a maximum number of jobs being worked on. Currently, the agent simply accepts all of the jobs that the controller sends it. The agent will even work on jobs from multiple different controllers at the same time.<base> parameter to attempt agent-based encoding (see encoding methods)<agentpipe> can be set to 1 to indicate that the agent should pipe data from avs2yuv even in agent-based encoding. If you have a 64-bit version of x264 and a 32-bit version of avisynth, you need to set <agentpipe> to 1 in order to get agent-based encoding to work. Otherwise, there's really no reason to use it. <compression> should only be changed if you have a fast controller computer connected to a slow network. The order in which the compression types appear is the arder which they are tried. Therefore, if <type>0</type> is at the top of the list, compression is turned off. The currently supported compression types are:
<passcompression> may be used instead of <compression> for specifying different compression settings for the first and second passes. Within the <passcompression> segment, set the <first> and <second> settings the same way as the above <compression> segment. For example, to use compression on only the first pass, you may use the following:<passcompression>
<first>
<type>1</type>
</first>
<second>
<type>0</type>
</second>
</passcompression>
or simply leave out the <second> block.<adhoc> element specifying the ports to use, like so: <adhoc controller="12345" agent="23456"/>.<name> element will broadcast the name for the agent to be associated with.--stale command-line option to specify the number of seconds before temp files are considered abandoned. Use --stale 0 to turn off old file deletion.The controller should be installed on the most powerful Windows-based machine on the network. The controller needs to have access to the avs2yuv program and all the source files and filters that any encode needs.
controller directory to somewhere that makes sense (like c:\program files\x264farm\controller)controller\config.xml to reflect your network configuration (note that the config file has changed since 1.02):
<temp> should point to the temporary directory you made in step 2.<agent> for each computer you want running an agent. Multi-core computers only need one agent running.<name> may be anything, and will only influence the output messages.<ip> is the IP address of each agent. You may use 127.0.0.1 to refer to the computer running the controller.<port> should be set to whatever you set <port> to in the agent's config.xml file.<number> is the number of jobs to send out to the agent. This basically specifies how many encodes to do at one time. It should be set to the number of processor cores in the agent's computer.<number pad="1"> to send one additional job to the agent during the second pass. Since the second pass jobs are so short (generally a few seconds) it is a good idea to have something else working in the background when a job gets done. This setting has no bearing on the first pass.<adhoc> element specifying the ports to use, like so: <adhoc controller="12345" agent="23456"/>. Note that this line should be the same in both controller and agent config.xml files in order to work properly.agent. If your config file is not called config.xml, use agent --config other_config_file.xml.
x264farm version 1.03-152
Using config file ".\config.xml"
Temp dir: "E:\x264farm\temp\agent"
Ports: 50700 - 50703
x264: "x264.bat"
nice: 20
agents: 2
agent base: ""
trying port 50700
bound to port 50700
listening...
working!
The minimum options that should be given is --first, --second, --bitrate, --avs, and --output. (or their equivalent short forms -1, -2, -B, -i, and -o)
-B / --bitrate "100%"-B 789kbps) or as a percentage of the first pass bitrate (-B 100%)-1 / --first "" --first "--crf 19 --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 5 --ref 8 --me umh --bframes 7 --weightb --b-pyramid --analyse all"-2 / --second "" --first option, don't specify an input file, output file, bitrate, or any option listed in the assorted parameters section. An example of this switch is --second "--trellis 2 --bime --sar 10:11 --no-psnr --direct auto --mixed-refs --subme 7 --ref 12 --me umh --bframes 7 --weightb --b-pyramid --analyse all --8x8dct"-i / --avs--firstavs--avs file. It will default to whatever --avs is.--fastavs--avs and --firstavs files, but it may have different dimensions. As long as the output of this file changes significantly when a scene change occurs, it may be as simple as you can make it.-o / --output "output.mkv" --preseek 0 --preseek needs to be set to 5.--force--restart--config "config.xml" --savedisk--savedisk will delete each file right after it is merged, which does not use too much extra disk space. However, if the controller is restarted during merging, the deleted files will have to be re-encoded.--nocomp--batch 5000 -I / --keyint option passed to x264 (250 by default), then it may take a long time to do anything..--batchmult 0.5--batch parameter is set to. This is to prevent a slow computer from getting a big job at the end of the encode, although it currently doesn't work too well.--split 250 --batch frames have been found. This value should be larger than x264's -I / --keyint option. --thresh 20.0 --split frames will ever be tested, though. See the split point section for more details.--rethresh 10.5 --thresh option. Increasing this number will cause more jobs to be temporarily overlooked if they have a weak starting point. Using --rethresh 1.0 will revert to the way 1.12 and before handled jobs. If not specified, it will default to (--thresh + 1) / 2.--3thresh 0.8 --3gops 1073741823--3ratio 0.05--rerc 0These are the same as the x264 parameters of the same names. They should never be used inside the --first or --second parameters. They apply to both the first and second passes (except the ratecontrol parameters which don't make sense during the first pass).
--zones ""--seek 0--frames 0--qcomp 0.6--cplxblur 20.0--qblur 0.5--qpmin 10--qpmax 51--qpstep 4--ipratio 1.4--pbratio 1.3-h / --help# The -B and the 100% MUST be on separate lines
-B
100%
# First pass settings
# "--crf 19" is the argument to --first
# Therefore "--crf 19" must be on ONE line
--first
--crf 19
# Here is the second pass line:
--second
--trellis 2
--batch
#230 - This line is ignored
3400The agent program is quite simple: encode the data recieved from the controller, and send out the encoded data. The controller is quite a bit more complicated, and can be divided into 2 parts: the first pass, and everything after that.
The first pass is also made up of two parts, each of which run simultaneously: the splitter and the workers
It is the splitter's job to take any unencoded parts of the video and turn them into ranges for the workers to take. The length of the ranges is determined by the --batch, --split, and --thresh options. Once a range has been split, it is put in a queue for the workers to pick up.
The workers pick out ranges from the splitter and stream the raw video data of the range out to the agents. They then recieve the stats file produced by the agent and process it. If the split point chosen by the splitter was found to be bad, the worker will signal the splitter to re-split the end of the range.
When all the workers are done and the splitter can't find any frames to encode, the resultant stats file is written and the first pass ends.
When the first pass ends, the controller reads the stats file and breaks it up into GOPs. Each GOP starts with an "I" frame and goes to (but does not include) the next "I" frame. This represents the smallest group of frames which can be calculated independantly. After the stats file is split into GOPs, they are sent to the ratecontrol equation to figure out the optimal bitrate of each GOP. Each GOP is sent to an agent to be encoded into a video fragment. When the video fragment comes back, it is analyzed to see how many bits it actually used. The ratecontrol used is designed to be as close to the ratecontrol done by x264 itself. Therefore, defining your own ratecontol in the second pass will not have much effect at all, and will only result in more re-encoded frames.
Once all the GOPs have been processed, the ratecontrol is done again on the (hopefully) more accurate stats of the second pass. Here, the issue of accuracy should be mentioned: the closer the optimal number of bits (specified by the ratecontrol) and the actual number of bits, the more accurate the GOP is, and the less it needs re-encoding. The actual equation is: max(optimalBits,actualBits) / min(optimalBits,actualBits).
After the second pass is finished, each GOP is processed from the least accurate to the most accurate. When a GOP is finished for the third pass, it is put back in the same list, and therefore may be processed multiple times before the total encoding is finished. The number of GOPs processed is determined by the --3thresh, --3gops, and --3ratio options. --3gops determines the total number of GOPs to recode after they have been allprocessed. --3ratio is the same thing, but in terms of the percentage of total GOPs (i.e. 0.1 will process at most 10% of the total GOPs again). --3thresh indicates a maximum accuracy to encode before deciding that the film is "good enough." Basically, if the accuracy of all GOPs is greater than --3thresh, then encoding ends. If any of those three conditions are met, then the encoding is over and the final video file is spliced together.
As of version 1.15, the agents and the controller will attempt to find each other over the local network. When an agent starts up, it will send a broadcast to all the computers on the local network. Controllers will also broadcast messages every minute to search for new agents on the local network. These features allow agents to be dynamically added to the encoding process. In 1.15, this feature is disabled by default. Enable it by adding the appropriate line to the controller's and agents' config.xml file. The agent will listen on the "agent" port and send on the "controller" port. Conversely, the controller will listen on the "controller" port and send on the "agent" port. The "agent" port must not be the same as the "controller" port, but both ports must be consistant in each config.xml file (that is, the line itself should just be copied and pasted in each file).
There are currently two encoding methods available to the agents: controller-based encoding and agent-based encoding.
Controller-based encoding is the default encoding method. The controller starts up avs2yuv and sends the agent all of the rendered data across the network. This is the most portable method of encoding, as the input files need only be available to the computer with the controller. However, it is also the least scalable method of encoding, since the controller's computer is rendering all of the data.
Agent-based encoding is used if the agent uses the <base> option in its config.xml file, and the agent determines that the AVS file is available to the agent. In this case, the agent will start up x264 directly, and does not need to be sent the data across the network. This method is extremely scalable, but it is more difficult to set up. Each agent using this method needs access to AVIsynth, all the filters, and the AVS file itself.
The <base> option in the agent's config.xml file determines where to search for the AVS file. If no <base> option is specified, or if it points to a nonexistant directory, controller-based encoding is used instead. If you use <base/> or <base></base>, it is interpreted to mean only search exactly where the controller reports it. This is useful on the controller's computer, especially if you plan on encoding from multiple hard drives. If the directory specified by <base> is valid, the AVS file is searched from the deepest controller directory fragment within the <base> directory. For example:
Controller's AVS file: D:\movies\working\somewhere\file.avs
Agent's <base> directory: F:\temp\x264farm
The agent's AVS file is searched for in the following locations, in this order:
F:\temp\x264farm\D:\movies\working\somewhere\file.avs (can't exist, but is checked anyway)
F:\temp\x264farm\movies\working\somewhere\file.avs
F:\temp\x264farm\working\somewhere\file.avs
F:\temp\x264farm\somewhere\file.avs
F:\temp\x264farm\file.avs
The first of those AVS files that matches the controller's AVS file is used as the input to x264. If it is not found in any of those places, controller-based encoding is used instead.
During the second pass, the video is split into GOPs and sent out to be encoded. But the first pass does not know where the GOPs are, so it guesses based on a few parameters.
The splitter will break down un-processed parts of the movie into pieces which are larger than --batch, but smaller than --batch + --split. The splitter will compare every pair of frames starting at --batch, in order to determine the frame which is most likely to be an "I" frame. This is the basic loop that occurs:
Find the difference between the next two frames
Update the average difference
Update the current threshold to be between --thresh at the first frame analyzed, to 1 at the last (--split) frame to be analyzed
If any frame difference is more than "threshold" times the current average difference, the largest frame is used as the split point
Otherwise, the next pair of frames are analyzed
The threshold changes in order to take advantage of the properties of the video. The alternative would be to read a set number of frames and pick the two consecutive frames which are the most different from each other as the split point. However, there are two problems with the static approach:
To summarize, the options which affect the split points are as follows:
<base> directory in agent-based encoding is any sort of secure sandbox. The agent can still encode things outside the directory if you tell the controller to use something like c:\movies\working\..\..\movies\working\something.avs. This will, at one point, search in <base>\..\..\movies\working, which may be a completely different directory from the <base> directory.SetMemoryMax(64) (or similar) at the beginning of the AVS file. All the frame accesses are in-order, so there is no noticable perforance hit. <nice> option to give priority to the controller.sudo sysctl -w net.inet.tcp.delayed_ack=0. Note that you will need to type this every time you reboot, or put it into a startup script.--batchmult option to help with one agent checking out a big job at the end and making everything else wait for it.--sizeprec option)--keeptemp option actually works now...--nocomp switch to force compression off.<pipe> setting to agent's config.xml file to locally pipe data to x264 in agent-based encodings.--rerc option in order to change the re-ratecontrol frequency during the third pass.