Taken from the handbrake.fr forum:
This information is up-to-date for HandBrake 0.9.5. It may no longer apply to future versions.
HandBrake 0.9.5 does not natively support x264 presets and tunes. However, nothing prevents you from re-creating them manually.
You can specify most x264 options (nearly all options parseable by libx264’s x264_param_parse function) via HandBrake’s advanced panel:
* –option value becomes option=value
* Options are separated by a colon.
* Options that do not require a parameter can be specified as either option or option=1
The MacGUI’s advanced panel will automatically add =1 if you don’t specify a value.
* In most cases, no-<something>=1 and <something>=0 are equivalent.
This isn’t true for deblock – to disable it, you must use no-deblock=1; deblock=0 is the same as deblock=0,0 (the default)
A list of all available x264 options can be found on MeWiki.
PRESETS
Formatted for HandBrake’s advanced panel (getting rid of redundant options; e.g. –bframes 0 implies –b-adapt 0), we get:
ultrafast
ref=1:bframes=0:cabac=0:8x8dct=0:weightp=0:me=dia:subq=0:rc-lookahead=0:analyse=none:trellis=0:aq-mode=0:no-deblock=1:scenecut=0:mbtree=0
superfast
ref=1:mixed-refs=0:weightp=1:subq=1:rc-lookahead=0:analyse=i8x8,i4x4:trellis=0:mbtree=0
veryfast
ref=1:mixed-refs=0:weightp=1:subq=2:rc-lookahead=10:trellis=0
faster
ref=2:mixed-refs=0:weightp=1:subq=4:rc-lookahead=20
fast
ref=2:weightp=1:subq=6:rc-lookahead=30
slow
ref=5:b-adapt=2:direct=auto:me=umh:subq=8:rc-lookahead=50
slower
ref=8:b-adapt=2:direct=auto:me=umh:subq=9:rc-lookahead=60:analyse=all:trellis=2
veryslow
ref=16:bframes=8:b-adapt=2:direct=auto:me=umh:merange=24:subq=10:rc-lookahead=60:analyse=all:trellis=2
placebo
ref=16:bframes=16:b-adapt=2:direct=auto:me=tesa:merange=24:subq=10:rc-lookahead=60:analyse=all:trellis=2:no-fast-pskip=1
–slow-firstpass can be set by disabling “Turbo First Pass” in the Video panel.
The official x264 presets are meant to be good speed / compression efficiency tradeoffs. The options that are most useful to compression efficiency are enabled first; the less useful options are enabled in the slowest presets (i.e. once all the most useful options are already enabled).
If you’re using settings slower than the placebo preset, you’re doing it wrong (the difference won’t be noticeable).
Note: if you’re starting with an official HandBrake preset for your device, be sure to add these settings before the preset’s options (so that the HandBrake options override the x264 preset – the other way around will likely break compatibility with your device).
Some settings should be safe to override with the x264 preset values (no guarantee, however):
* b-adapt (Adaptive B-Frames)
* direct (Adaptive Direct Mode)
* me (Motion Estimation Method)
* subme/subq (Subpixel ME & Mode Decision)
* merange (Motion Estimation Range)
* aq-strength (Adaptive Quantization)
* psy-rd (Psychovisual Rate Distorsion, Psychovisual Trellis)
* partitions/analyse (Partition Types)
* trellis (Trellis)
* deblock (Deblocking)*
* rc-lookahead
* no-fast-pskip
* Some very weak devices may require that deblocking be disabled
The following settings are NOT safe to override:
* ref (Reference Frames)
* bframes (Maximum B-Frames)
* cabac (CABAC Entropy Coding)
* 8x8dct (8×8 Transform)
* weightp (Weighted P-Frames)
* b-pyramid (Pyramidal B-Frames)
* weightb
* mixed-refs
* vbv-bufsize and vbv-maxrate
(meaning that you should not enable them if they are disabled in the HandBrake preset, or increase the value)
The official x264 tunes for r1834 are:
–tune Tune the settings for a particular type of source or situation
Overridden by user settings.
Multiple tunings are separated by commas.
Only one psy tuning can be used at a time.
TUNES
In HandBrake’s advanced panel:
* film: optimizes settings for film sources
Set Deblocking to -1,-1
Set Psychovisual Trellis to 0.15 (leaving Psychovisual Rate Distorsion untouched)
* animation: optimizes settings for cartoons and other animated content
Set Deblocking to 1,1
Set Psychovisual rate Distorsion to 0.4 (leaving Psychovisual Trellis untouched)
Set Adaptive Quantization to 0.6
Optional: if B-Frames are enabled, add 2 B-Frames to Maximum B-Frames*
Optional: if using more than one Reference Frame(s), double the number of Reference Frames*
* While these changes are part of the tune, they’re likely to break device compatibility
Leave B-Frames and Reference Frames untouched unless you know what you’re doing
* grain: optimizes settings to preserve grain
Set Deblocking to -2,-2
Set Psychovisual Trellis to 0.25 (leaving Psychovisual Rate Distorsion untouched)
Enable No DCT Decimation
Add the following options to the option string:
deadzone-inter=6:deadzone-intra=6:ipratio=1.1:pbratio=1.1:qcomp=0.8
* stillimage: optimizes settings for videos made of multiple still frames lasting several seconds
Set Deblocking to -3,-3
Set Adaptive Quantization to 1.2
Set Psychovisual Rate Distorsion to 2.0 and Psychovisual Trellis to 0.7
* psnr: optimizes the settings for best PSNR (hurts perceived quality, not useful for encodes you intend to watch
Add the following options to the option string:
aq-mode=0:psy=0
* ssim: optimizes the settings for best SSIM (hurts perceived quality, not useful for encodes you intend to watch)
Add the following options to the option string:
aq-mode=2:psy=0
* fastdecode: optimizes settings for faster decoding on weak devices
Disable CABAC Entropy Coding
Disable Weighted P-Frames
Add the following options to the option string:
weightb=0:no-deblock=1
* zerotatency: minimizes encoding latency (useful for live streaming, which HandBrake doesn’t do)
Set Maximum B-Frames to 0
Add the following options to the option string:
mbtree=0:rc-lookahead=0:sync-lookahead=0:sliced-threads=1
CFR (Constant Framerate) is set automatically by HandBrake when required
Using –force-cfr will likely cause audio/video sync issues
bytebin 