File Format Specification, edition of January 10, 2008
This document defines the Extensible ROMhack Interchange Format (XRIF), a general-purpose and extensible binary patching format.
The website for XRIF is at zerosoul.arc-nova.org/Technology/XRIF; failing that, use a search engine.
There is a format called IPS; whether this stands for International Patching System or Intelligent Patching System is a matter of some disagreement among users. (I myself hold that neither title accurately describes it.) It is a simple, straightforward patching format, with data arranged into blocks, each of which contains an address into the target file at which to begin overwriting data, the amount of data to write there, and finally the data itself. IPS is well-supported and widely used in the ROM hacking scene, but years of experience have revealed its limitations.
Foremost among these is that it is remarkably unintelligent about handling shifted data, for example if some data is inserted into the middle of the modified data. In this case, all IPS can do is perform meaningless comparisons between the data at that location and the unrelated data that used to be at that location, and record all the (numerous) differences. This gets the job done, but is not at all elegant, intelligent or efficient.
There are some other concerns as well, such as the fact that it is limited to 24-bit addressing, which is quite enough for most purposes, but has become a much more visible limitation with the advent of Game Boy Advance hacking: it is not uncommon to see 16 MB ROM images, which spans the entire addressing range of IPS. Also, the end of the file is marked by a special signature. However, this signature occupies the space where the next block's address field would normally be found. Because of this, it's possible that a patch of a large file might have a block whose address field happens to correspond to the end-of-file signature, which will cause problems (a legitimate address being misinterpreted as the end-of-file signature). Although the problem can be worked around, the fact the problem exists shows a lack of foresight in the format's design.
And, IPS doesn't allow for embedded annotation (the title of the work, who made it, and so on), or for starting a checksum of the target data to help ensure the correct file is being patched.
Instead of consisting almost entirely of a sequence of blocks, an XRIF file can contain many different types of data, which are encapsulated in chunks, each with a label to distinguish one type of data from another. There are many different potential kinds of data (pure overwriting data, data shifting instructions, annotation, and so forth), and they are all cleanly separated from other types of data in the file itself.
IPS uses 24-bit addressing; that fact is hard-wired into the format. XRIF can use up to 63-bit addressing, and each patch can choose its own addressing size, from 8-bit, 16-bit, 31-bit, 32-bit and 63-bit. The latter two are not required to be implemented by patchers (though they won't even be necessary for patches of files under 2 GB), but are made available in the format structure in case they will be useful.
XRIF can contain instructions that copy some range of data from one place to another. This is a quick and efficient way to implement insertion of data into the middle of the target data: take the data after the insertion point and copy it, all at once, to its new location, then go in and write the inserted data in the usual way.
XRIF can store annotation (for example, who created the hack), in a manner that is fully internationalized, by recording the natural language (such as English) of this annotation— and allowing for annotation in multiple languages— and recording this annotation in Unicode, by way of UTF-8. There are two methods of recording annotation in XRIF— a patch can use either method, or both— one of which is simple and straightforward, while the other is much more expressive.
XRIF can also record multiple patches in a single file, with data common to all of them being recorded as such. Such variations can be used for many purposes, such as accomodating different versions of the target data, or for distributing an upgrade to a previous patch that includes the original patch too, or simply conveying multiple patches in a single XRIF file.
Perhaps XRIF's most notable and defining characteristic is its ability to invoke Extensions. The Core XRIF format, as defined by this document, is meant to be general-purpose, but XRIF can be adapted to highly specialized purposes by means of Extensions. These are invoked by name, and are defined separately from the Core XRIF format, and inherit XRIF's features.
XRIF has a number of exotic features, most of which are summarized above. However, being mindful of how the format might be implemented, many of these features are declared as optional: meaning that programs that implement XRIF are not required to implement some of the more unusual parts of the format. If a patch uses any of those optional features, it will say so in the file header, so that a program can see if the patch uses a feature the program doesn't provide, and if so, to exit gracefully, without having to pre-scan the file and/or to abort in the middle of the patching process.
Of course, I do not control others: they are able to write programs that do not actually implement everything marked as “required”. The purpose, then, of marking features as “required” is so that patch creators can feel free to use those features, with the reasonable expectation that patchers will support them. Conversely, a patch creator should weigh the benefits of using optional features against the fact that they cannot be assumed to be widely supported. Thus, the “required” features comprise a lowest common denominator format that all implementations of XRIF are expected to support.
In this way, a programmer who just wants to put a quick and simple patcher together, and a programmer who is willing to commit to writing a comprehensive patcher, can both be in compliance with the XRIF specification: either program will be able to apply a relatively simple patch. At the other end of the spectrum: even though some functionality is merely “optional”, nevertheless it is well-defined, so that two independently-developed comprehensive patchers will both be able to apply patches that use the same optional features.
The area of the header that says what features are used, also indicates which of those features are merely advisory (meaning, if the patcher doesn't recognize the feature, it can safely ignore it) and which are mandatory (meaning the patcher can't safely ignore it). This helps ensure compatibility of current programs with future versions of XRIF, in a way much more flexible and meaningful than using version numbers.
For example, suppose XRIF did use version numbers. Version 1 is published, and it is good. Version 2 adds some complex feature that might be useful in some instances but would be difficult to support. Version 3 adds a nice, useful feature that makes everyone's life easier. In this case, you couldn't add support for version 3's nice and easy feature without having to support version 2's obtrusive feature along with it. With XRIF's current approach, the two features would be marked independently: you could support just the nice and easy feature.
Today's XRIF format originated from my attempts to address NES cartridge file patching. The predominant NES cartridge file format is called iNES (after the emulator that introduced it), and it is a simple format. However, a more flexible and expressive format called UNIF has been defined. The only major problem, for the purposes of ROM hacking, is that it does not store ROM data in the same location within the file (because it uses a chunk-based structure)... both as compared to iNES, and possibly even to other UNIF files of the same game. However, the IPS format relies on the data being at the same location in the source file as in the target file, which UNIF cannot guarantee, so IPS, as designed, clearly does not work with UNIF.
My first attempt at addressing this problem was a format I called UNIF-IPS, which addressed UNIF chunks by name. Although this would work well with UNIF itself, it was not suitable for iNES, and so the idea was abandoned. Later, I created a format I called UNRIF, which was meant to be format-neutral. However, although I even got so far as to make a public proposition for the format, it was never implemented in code.
At one point, I had visions of a whole family of patching formats based on the idea of UNRIF, each tailed to a particular platform (NES, Super NES, and so forth). Although UNRIF was eventually abandoned, the idea of a family of formats was expanded upon, and I devised a new, general-purpose format that could be “extended” to serve specialized needs. This format was called XRIF. The details of the format were worked out, and eventually it was first published on January 3, 2006, soon thereafter followed by the XRIF+NES specification (an Extension of the format), both of which were implemented in code not long afterward (by myself, no less).
0x12345678, this is represented as the byte sequence 78 56 34 12.NN means a byte with the value NN as expressed in hexadecimal (base 16). NN NN NN means a byte sequence. Byte, as used in this document (and, overwhelmingly, as used elsewhere too), means an 8-bit unit, an “octet”.00, which serves to mark the end of it. This is commonly used for text data.An XRIF file consists of chunks, which are self-contained units of data with an associated label specifying the purpose and format of the chunk's contents. (You might think of the chunk label as analogous to an XML or HTML element name, or to a filename extension.)
Each chunk consists of the following, in this order:
TREP. The name must consist entirely of upper- or lower-case letters and digits, with one exception: “out-of-band” chunks start with a . (period / full stop).0x7FFFFFFF).For example, here's a complete example chunk:
58 41 50 50 06 00 00 00 4E 45 53 00 00 00
\_________/ \_________/ \_______________/
| | |
Label Length Contents
The label; in this case, is “XAPP”. This indicates that it's an XAPP chunk.
The length contains the value 0x00000006 (in little-endian byte order), defining the length of the contents.
Chunks are laid out in the file sequentially, with no intervening bytes: the last byte of a chunk is followed by the first byte of the label field of the next. An XRIF file consists entirely of chunks, start to finish.
These are the Core XRIF chunks, and thus which comprise the set of chunks available to Core XRIF files, and inherited by Extensions.
XRIF chunk: File header(Note: this refers to a chunk with the label “XRIF”.)
This is the first chunk in an XRIF file. It specifies some basic information about the patch, such as structural parameters, as well as flags indicating what features are used by the patch. Also, the fact that it is present at the very start of an XRIF file can serve as a way to quickly filter out things that aren't XRIF files.
The first two bytes of the chunk's contents are always, without exception, 7F 00. This serves as a “magic number” of sorts: every valid XRIF file will have this signature.
The next byte indicates the size of address-type fields, found in payload chunks such as TREP:
00 | 8-bit addressing (0 through 0xFF) |
|---|---|
01 | 16-bit addressing (0 through 0xFFFF) |
02 | 31-bit addressing (0 through 0x7FFFFFFF, but see below) |
03 | 63-bit addressing (0 through 0x7FFFFFFFFFFFFFFF) |
The next byte is the same, but defines the size of length-type fields instead.
Address-type and length-type fields are collectively referred to as block values.
32-bit integers are, by default, limited to the positive range of signed 32-bit integers (0 through 0x7FFFFFFF). For addresses, this yields an address range of 2 GB, which is more than enough for the vast majority of applications. For example, any two entire CD images could fit quite comfortably within that range; or, if the average size of a disc doesn't exceed about 682 MB, then it could contain three CD images.
If that isn't enough, patches can request that 32-bit integers be allowed to use their full unsigned range (up to 0xFFFFFFFF), thereby doubling the addressable range to 4 GB, which could address any byte across about six CD images, and the better part of many DVD images.
If even that isn't enough, then a patch can use 63-bit addressing. This gives an addressable range of no less than 8,589,934,592 GB (8 EB), which can address about 170 million full-size (50 GB) Blu-ray discs, or could probably address most, if not all, the static content of the entire Internet, including the Internet Archive and search engine databases, at the individual byte level.
So, 63-bit addressing will be enough. The format allows for it as a natural extension of 4-byte (32-bit) addressing, because there are potential practical applications that exceed 32-bit range, such as dealing with DVD images— not to mention newer technologies like HD DVD and Blu-ray. It is by no means required to be supported, though. Also, 8-byte (64-bit) integers are always, without exception or recourse, limited to their positive signed range (that is, up to 0x7FFFFFFFFFFFFFFF); the figures above take this into account.
Besides being more than will be needed by all conceivable and realistic uses of the format (or any other patching format), it can also be troublesome to actually implement. In the C language for example, there is not universal agreement on how to obtain a 64-bit datatype to begin with (the use of <stdint.h> notwithstanding), and then there is the issue of using filesystem functions capable of handling files larger than 2 GB; after all, there's no point in using 64-bit— or even “merely” full-32-bit— addressing if only the first 2 GB are addressed!
And, on some platforms, obtaining even just an unsigned 32-bit datatype can be difficult if not impossible. Because 2 GB is more than enough for the vast majority of applications, it is not required to support the full unsigned range of 32-bit integers.
Even if unsigned 32-bit datatypes are available, it is still easier to work with “31-bit” integers, for example by using unsigned 32-bit datatypes to hold values, and you can add any two positive “31-bit” integers and store the result in a 32-bit integer without worrying about overflow. This can be useful for checking that a TFSM operation does not exceed its boundaries, for example.
In any event, it is always required to support the full, unsigned range of 8-bit and 16-bit block values. Furthermore, the above rules regarding “signed” and “unsigned” are merely concessions to environments where unsigned datatypes may be unavailable: in XRIF, integers are never to be interpreted as being signed.
All the above notes apply to block length sizes too, except that— as of the 2006-04-26 edition of XRIF, it is no longer required to support 32-bit lengths at all (positive signed range or otherwise). This is to allow implementations to allocate a 64 KB copy buffer, for example (and 64 KB is not much to ask for), and use that, and not have to worry about handling blocks that might exceed the buffer size.
Once and for all, what is and is not required to be supported:
| Size | Addressing | Block lengths |
|---|---|---|
| 8-bit | Optional[1] | Required |
| 16-bit | Required | Required |
| 31-bit | Required | Optional[2] |
| 32-bit | Optional | Optional |
| 63-bit | Optional | Prohibited[3] |
[1] As of March 1, 2008. No new edition is published to this effect, but I figured that 8-bit addressing is kinda silly anyway (it would only allow addressing the first 256 bytes). So, if you'd rather not bother with it, then don't.
[2] As of the 2006-04-26 edition of XRIF. Prior to that, it was required.
[3] If lengths were 64-bit, then a given length-type field would either be:
Besides, the choice of length size is merely one of efficiency: any length size will get the job done, it is only a question of how many data blocks it'll take to do it. So, prohibiting certain block length sizes does not reduce the format's capabilities.
After the block address size and block length size bytes, are at least two bytes specifying flags that inform the patcher what features are used by the patch: each bit of these bytes is a flag. Of these bytes, ones at even numbered offsets (starting with the first flag byte) are advisory, and those at odd numbered offsets are mandatory. The difference is in how a patcher is expected to handle unrecognized flags:
The first of these bytes (5th byte of the XRIF chunk contents overall) is advisory:
·······X (01)0=Yes, 1=No)······X· (02)0) or after (1) the variation proper.·····X·· (04)1=Yes). An explanation of this is given below.····X··· (08)TFSM is used. (1=Yes)···X···· (10)I” (such as “ILOC”) are used. (1=Yes)··X····· (20)0) or each variation can be independently enabled/disabled (1)·X······ (40)i” (such as “iLOC”) are used, and whether all chunks meeting that description are considered to be annotation. (1=Yes)Note that those first two flags, as well as the TFSM flag, should be treated as mandatory; these flags were assigned to this byte before I devised the alternate-between-advisory-and-mandatory scheme. However, these flags have been part of the specification since the beginning, therefore they are not “unrecognized”, so this shouldn't be a problem.
If the target data is streamable, it just means the patcher could potentially apply the patch to the target data in a sequential manner, such as by reading a target file from the standard input or decompressing it on-the-fly, and/or writing it sequentially to the standard output or recompressing it on-the-fly. This could be crucial for patches of very large files, so that the patcher does not need to load the entire file into memory.
Specifically, this means the target data is modified sequentially: after an access (read or write) takes place, subsequent accesses shall only occur after this point. Read-modify-write operations (such as TXOR) are not allowed to be used at all.
It is not required for patchers to implement streaming: this flag is simply a hint to the patcher that it could stream the target data if it wanted to. Also, it is not required— but is of course recommended— that a patch generator mark an eligible patch as streamable.
In short: if nobody paid any attention to this flag, or if some people did and others did not, everything would still work.
Extensions of XRIF can define additional rules as to whether a given patch is or is not eligible for this.
The second of the flag bytes (6th byte of the XRIF chunk contents overall) contains mandatory flags:
·······X (01)······X· (02)TCFS.·····X·· (04)TADD.····X··· (08)TROT.···X···· (10)TREP for example).··X····· (20).AKA.·X······ (40)TAND, TIOR, TXOR). This flag invokes all three at once.X······· (80)INFO and VARY. (Refer to the definitions of those chunks for what exactly this means.)The third of the flag bytes (7th byte of the XRIF chunk contents overall) is advisory, but there aren't any flags assigned to it yet. However, because these flag bytes are defined as alternating between advisory and mandatory, compliant programs will recognize this byte as such.
The fourth of the flag bytes (8th byte of the XRIF chunk contents) contains more mandatory flags:
·······X (01)0x00 would be interpreted as meaning 0x100 (one higher than 0xFF, the normal maximum). This only applies to 8-bit and 16-bit lengths, not addresses or larger lengths.······X· (02)TAND, TIOR, TXOR) use a TREP-style format.Future editions of XRIF may specify additional flags; more bytes can be added to the XRIF chunk to accomodate them (alternating between advisory and mandatory, as has been described) if necessary; patchers shall allow for these extra bytes (or lack thereof).
The XRIF chunk shall never, ever, exceed 0xFF (255) bytes in size, so that a patcher does not have to accomodate anything larger, while still remaining in full compliance with this Specification. This also means that the upper 24 bits of the XRIF chunk's length header will always be 00 00 00, thereby adding yet another “magic number” to the header.
Therefore, in a valid XRIF file, the first ten bytes will always be like so:
58 52 49 46 xx 00 00 00 7F 00...
\_________/ \_________/ \____...
| | |
Label Length Content
Where the length is between 6 and 0xFF (inclusive).
This allows programs to determine fairly confidently that a file is, or is not, an XRIF file, without even having to actually implement it (with mime.magic files, for example).
A summary of the content bytes:
| Offset | Function |
|---|---|
00 | Magic number: must always be 7F |
01 | Magic number: must always be 00 |
02 | Addressing size: 00 (8-bit), 01 (16-bit), 02 (31-bit), or 03 (63-bit) |
03 | Block length size: as above |
04+ | Flags. Bytes at even offsets (04, 06...) are advisory, the others mandatory |
XAPP chunk: Extension DeclarationOne of XRIF's defining features is its ability to invoke Extensions, which are formats that build on XRIF. XRIF itself is meant to be a general-purpose format; Extensions can give it specialized functionality or adapt it for a specialized purpose.
The XAPP chunk invokes such an Extension. It consists of the name of the Extension being invoked, which is nul terminated, and optionally followed by zero or more bytes giving flags or parameters that can be used by the Extension, in much the same manner as the XRIF chunk. However, whether the Extension's parameter bytes alternate between advisory and mandatory— or for that matter, whether it even uses them as flags at all— is a decision left to the designer of the Extension.
An Extension inherits all the functionality that is available to the XRIF format that it extends. For example, if an Extension is defined, then later on the Core XRIF format is updated to add new features, those features are inherited by the Extension, and should be taken into account by the patcher. (A properly written patcher, though, will be able to distinguish which unrecognized features are safe to ignore and which are not.)
Remember, Extensions are built on top of XRIF: they are not self-contained, complete, standalone formats unto themselves.
A patch can have zero or more XAPP chunks, all of which must occur immediately after the XRIF chunk. If there is more than one XAPP chunk, they must all be contiguous (occuring one after the other with no other chunks between them).
A given XAPP chunk operates on whatever came before it: for example, if there are two XAPP chunks, the first one extends XRIF itself, the second one extends the first Extension.
Originally, XRIF was supposed to be able to invoke not only Extensions, but also Supersets (which might be thought of as “advisory Extensions”, where if a patcher doesn't recognize a Superset, it can fall back to Core-compatible functionality) and Platform Declarations (which do not affect functionality at all, but merely declare the intended platform of the target data; for example, a patch for a Game Boy Advance game— which would only require Core XRIF functionality— could be declared with an appropriate Platform Declaration).
Extensions, Supersets and Platform Declarations are (or were) collectively referred to as Applications of XRIF. When I was first designing what became today's XRIF format, the chunk that invoked an Application was labeled “XAPP”. Even though I later dispensed with the idea of Supersets and Platform Declarations, the name of the chunk— XAPP— remained.
In order to distinguish these types of Applications, the Application name started with a sigil identifying what type of Application it was: for Extensions this was “+”, which is where the “XRIF+FOO” naming convention came from. (For Supersets it was “/”; for Platform Declarations, “=”.)
No patcher is required to support any Extension, unless it purports to. It is up to the patcher to recognize an Extension: if it does not, then it should exit gracefully; and if it does recognize an Extension, it should use it in accordance with that Extension's specification.
As with the XRIF chunk, the size of an XAPP chunk shall never exceed 255 bytes. The idea is, the XAPP chunk itself should only contain flags that serve to indicate what functionality is required, so that a patcher can see if it'll be called upon to do something it can't do, and exit gracefully if so. If the Extension needs bulk data, it can be conveyed in separate chunks defined by the Extension (XRIF+NES for example has a separate CSIZ chunk instead of adding this information to its XAPP), perhaps with its XAPP chunk only indicating whether or not this data is present.
There are a number of chunks which perform the actual operation of modifying the target data, which is of course the fundamental purpose of an XRIF file, or indeed of any patch format.
The data in these chunks is arranged into blocks, each of which performs a single operation, containing fields such as addresses and data. The size of the block values (address and length fields) is determined by the XRIF header, described above. Each block within these chunks occurs immediately after the previous block, with no intervening bytes.
Be aware that it is not allowed to modify (read or write) data outside the range of the target data. If this needs to be done, then a TFSM chunk will increase the size of the target data, and the extra space can then be addressed. The reason for this is, unlike IPS, XRIF does not assume the target data is a file on disk: perhaps it will be applied to a set of data in memory (such as by an emulator that can perform on-the-fly XRIF patching), and unlike files on disk, memory doesn't magically expand to accomodate extra data if you put more into it than has been allocated.
TREP chunk: ReplacementThis modifies the target data by replacing parts of it with new data. This is the most basic method of binary patching, and is comparable to IPS's native— and only— format. The bulk of most XRIF files will consist of TREP chunks.
Each block consists of the following:
For example, if the patch uses 31-bit addressing and 16-bit lengths, then this block would write the bytes 6C 6F 6C 7A at offset 0x000158BF:
BF 58 01 00 04 00 6C 6F 6C 7A
\_________/ \___/ \_________/
| | |
Address Length Data
Here, the Address is 0x000158BF, in little-endian byte order. The Data is 4 bytes long, therefore the Length is set to the value 0x0004.
TRLE chunk: Repeated replacementThis is similar to TREP, except that the data is written multiple times successively; each such time is called an iteration. This can be used to fill in a region of the target file with a repeated pattern (including a one-byte pattern) while having to record the pattern itself just once.
Each block consists of:
For example, if— for some reason— you want to write “SourceSourceSource” onto the target file, you can repeat “Source” three times. “Source” is six bytes long, so Data Length = 6. You write it three times, so Iteration Count = 3. If you write it at 0x158BF, again using 31-bit addressing and 16-bit lengths:
BF 58 01 00 06 00 03 00 53 6F 73 72 63 65
\_________/ \___/ \___/ \_______________/
| | | |
Address Length Iter. Data
Here, the Data is 6 bytes long, so Length is set to the value 0x0006.
If you want to fill in some region with a single byte value (such as 00), you would have a Data Length of 1, the Iteration Count being how many bytes to overwrite (that is, the number of times to repeat the one-byte Data). For example, to zero out everything from 0x10000 to 0x11FFF:
00 00 01 00 01 00 00 20 00
\_________/ \___/ \___/ \/
| | | |
Address Length Iter. Data
Here, the Data is one byte, so Data Length is set to the value 0x0001. There are 0x2000 bytes to overwrite, so Iteration Count is set to that value.
TRLE will likely be frequently used for just that: to “zero out” parts of the target data. In this case, the Data is a single byte (with Iteration Count specifying how many bytes to write), so implementations are advised to optimize for the case of Data Length being 1, for example by reading the data byte and using memset() as found in the Standard C Library:
memset ( targetdata or buffer, data byte, iteration count );
As of the 2008-01-10 edition of XRIF, it's not allowed for Data Length × Iteration Count to exceed 65536 (0x10000) bytes, unless block lengths (as defined in the XRIF chunk) are 31-bit or greater.
TCFR chunk: Copy from ResultThis copies a group of data from one location to another, all at once. This will mainly be of use if the target data is to have some data inserted into the middle of it:
TFSM to increase the target data size to the required amount.TCFR to take the data after the insertion point and copy it to its new location.TRLE to zero-out the insertion area.TREP to write the new data to the insertion area.Each block of TCFR consists of:
TREP).For example, if the target data size is 0x30000 bytes, and you want to insert 0x10000 bytes of data after the first 0x20000 bytes:
TFSM to set the target data to its new size of 0x40000 bytes.TCFR to copy 0x10000 bytes (Data Length) from 0x20000 (Source Address) to 0x30000 (Target Address):
00 00 02 00 00 00 03 00 00 00 01 00
\_________/ \_________/ \_________/
| | |
Source Destination Amount
It is legal for the two ranges to overlap, but not to be identical (which would be useless anyway). If the ranges overlap, the implementor should take care to ensure that, in the process of copying data, it doesn't clobber data it will need to read later; such as by using memmove() rather than memcpy().
TCFS chunk: Copy from SourceThis is like TCFR, except the data is copied from the original target data, the way it existed before any modifications that had been made to it by the patch. The block format is exactly the same; but unlike TCFR, it is perfectly allowable for the Source Address and Target Address to be the same. (This is because the target data is no longer being used as its own source; it's copying from one set of data to the same offset of another set of data.)
In order to implement this, a patcher will need to make arrangements to have the original target data available throughout the patching process. In general, this is a good idea anyway (for example, if something goes wrong in the middle of applying the patch, the target data won't have been irreversibly modified). However, making these arrangements can be impractical, such as if the target data is very large.
As such, it is not required for patchers to implement TCFS; but, it is recommended that they do.
TAND, TIOR, TXOR chunks: Bit twiddlersThese perform “bit twiddling” operations: instead of simply overwriting data wholesale, they tweak individual bits, leaving others unchanged. They work by comparing the data in the patch to the target data, bit by bit, and outputting the result of the comparison.
The block structure is the same for all three. However, there are two formats available: either the default structure, or a TREP-style structure available by setting a flag in the XRIF header (the latter being available as of the 2008-01-10 edition of XRIF).
| Default structure | TREP-style structure |
|---|---|
|
|
| In this case, a single byte is used to twiddle a number of bytes in the target data. | This works like TREP, except the target data is twiddled rather than simply overwritten. |
Either way: for each byte of the target data, that byte is compared to the respective patch byte, bit by bit, and the result written back to the target file at the same location. The three chunks differ in the specific comparison performed:
TAND1, the result is 1, else 0.TIOR1, the result is 1, else 0.TXOR0, else 1.These correspond to the C language operators “&”, “|” and “^”, respectively.
You might also think of them in this manner:
TAND0, the result is 0, else unchanged.TIOR1, the result is 1, else unchanged.TXOR1, the result is inverted (1<=>0), else unchanged.These chunks are not required to be supported. However, if any of them are, then all three must be. It is not required to support both of the above formats (original and TREP-style); a patcher can support just one just the other if it wishes, or it can support both (or neither).
TADD chunk: Mathematic additionThis will perform mathematic addition on integers in the target file. This is not required to be supported.
·····XXX (07)000 | 8-bit |
|---|---|
001 | 16-bit |
010 | 24-bit |
011 | 32-bit |
100 | 40-bit |
101 | 48-bit |
110 | 56-bit |
111 | 64-bit |
TADD itself is). Also, those are the only allowed values, unless the patch uses 63-bit addressing, in which case they're all allowed.
·X······ (40)X······· (80)0 means Little-endian, 1 means Big-endian. Both must be supported.If the Method Flag (above) is 0, then the Data (below) is one integer (of the size/endianness given above), which is repeated the number of times given in this field. For example, you would use this if you want to adjust a number of fields all by the same amount.
If the Method Flag is 1, then the Data is this many integers (of the size/endianness given above). You would use this if you want to adjust a number of fields each by their own amount (without needing separate blocks for each of them).
Both of these must be supported.
0, this is one integer; otherwise the number of integers is given by Data Amount. The integers are stored in the same byte order as specified in the flags byte, above.For example: if you want to add the value 0x0005 to a list of 0xFF (255) 16-bit addresses starting at 0x28002:
02 80 02 00 01 FF 00 05 00
\_________/ \/ \___/ \___/
| | | |
Address Flags Count Data
In this case, the operation starts at 0x00028002 (Address), adding the value 0x0005 (Data) to a list of 0x00FF (Data Amount) addresses, which are 16-bit and little-endian (Flags).
TROT chunk: Shifting and RotationThis performs shifting or rotation of bits within bytes (or, more precisely, byte groups). This is not required to be supported.
·····XXX (07)···X···· (10)0) or right (1). “Left” means multiplication by 2; “right” means division by 2.··X····· (20)0) or rotate (1). If a bit is rotated off the end of a group, it reappears at the other end; if shifted, it is discarded.·X······ (40)X······· (80)0) or big-endian (1). When a bit is moved off the end of a byte, this flag controls what byte it will go to. This is unnecessary if the group size is 1 byte.TFSM chunk: Data size modificationThis changes the size of the target data. No chunk is allowed to read, write, or otherwise address anything past the end of the target data. If that data needs to be increased in size, a TFSM chunk will make it so.
This is not required to be supported.
Prior to the 2006-01-22 edition of XRIF, it was required to be supported, except in situations where it was not possible to change the target data size (for any reason). It is for this reason that the TFSM flag in the XRIF header is in an advisory byte rather than a mandatory one, even though, now, it ought to be in the latter.
The chunk consists of a 00 byte, followed by a value of the Address type, which specifies the new size of the target data.
Actually, the byte before the Size field can take on one of these values:
00 | The target data is set to exactly Size bytes (as above). |
|---|---|
01 | The target data is increased by Size bytes. |
02 | The target data is decreased by Size bytes. |
But, as of the 2008-01-10 edition of XRIF, the relative size methods are formally deprecated: new patches should not use them. They would only be useful if the patch could do things such as addressing n bytes from the end of the data, which, at the moment, it can't. Therefore, the only useful method is to explicitly state the exact final size.
If the data winds up larger than before, 00s are added to the end to fill out the new space. If the data winds up smaller, data is truncated off the end.
There is no format-imposed upper limit on the target size: if it's a valid Address value, it's a valid size, as far as XRIF is concerned. It is up to the patcher to determine and respect the relevant platform, filesystem or memory size limitations. If such limits are surpassed, the patcher should do its best to handle the situation gracefully.
As of the 2008-01-10 edition of XRIF, it is required— in the sense of “can be assumed to be widely supported”— that if TFSM is present, it be used before any (other) payload chunks such as TREP. This is so a program can check what the requested size will be, and see if it's possible to carry it out, before it starts changing the target data.
TCRC chunk: ChecksumsThis records one or more checksums of the intended target data, computed at the time the patch was created. The patcher can compute the checksum of the actual target data (when it goes to apply the patch) and compare it to this: if they match, it's reasonable to assume the patch is being applied to the correct, intended target data. It is not required for patchers to implement this.
Multiple checksums can be recorded in a single TCRC chunk (for example, if a number of different— yet known— versions of the target data are compatible with the patch). If there is more than one checksum in a chunk, then if any of them matches, the target data is considered to have passed the CRC check. If there are no TCRC chunks at all, the target data passes unconditionally.
Prior to the 2008-01-10 edition of XRIF, multiple checksums were done by recording each in its own separate TCRC chunk, but this behavior has been deprecated in favor of recording them all in a single chunk, so that an implementation can deal with them all and pass judgement in a single operation, rather than having to keep track of prior results and possibly having to pre-scan the patch to know how many checksums there are.
TCRC can be used in tandem with VARY to auto-select a variation, based on which variation is the first to pass the CRC check.
The actual CRC algorithm is Adler-32, with the difference that the result is recorded in little-endian order in the TCRC chunk (for consistency with the rest of the XRIF format), rather than big-endian as in Adler-32 proper.
The algorithm works like so: let there be two counters, call them A and B. A starts with the value 1, B with 0. For each byte of the target data, add that byte to A, then add A to B. When done with the target data, take both values modulo 65521 (that is, the remainder of dividing them by 65521); the value of A forms the lower 16 bits of the result, B the high 16 bits, yielding a 32-bit value: this value is then written, in little-endian byte order, into the TCRC chunk.
Here is some sample code in the C language that will do this. I wrote this myself; for the purposes of licensing, consider this to be in the public domain:
uint32_t adler32 (
unsigned char * data,
unsigned int length
)
{
uint32_t a = 1, b = 0;
uint16_t counter = 5803;
while (length--)
{
if (!counter--)
{
a %= 65521;
b %= 65521;
counter = 5802;
}
a += *(data++);
b += a;
}
a %= 65521;
b %= 65521;
return (b << 16) | a;
}
Here, uint32_t means an unsigned 32-bit datatype; uint16_t means an unsigned 16-bit datatype. Both of these are available if you #include <stdint.h>, or you can replace them with whatever is appropriate for your system (such as unsigned int and unsigned short int respectively) if that header is unavailable.
The value 65521 is the highest prime number less than 216; the value 5802 is the maximum number of bytes that can be safely calculated before there is a possibility that B will overflow its datatype. So, after doing that many iterations, the code will modulo both counters in order to be safe (and reset the iteration counter to 5802 instead of 5803, because the next iteration is performed immediately afterward). The point of all this is to delay the modulo operation as long as possible, to avoid doing it on every iteration.
If A and B are implemented as signed 32-bit datatypes, then the maximum number of safe iterations (before B could surpass 0x7FFFFFFF) is 4103 rather than 5803.
If they are implemented as unsigned 64-bit datatypes, then the value becomes 380,368,696 iterations.
The checksum algorithm chosen is not meant as a security measure, but as a convenience measure. It's meant to catch the user applying the patch to the wrong game entirely, or to the wrong version of it (such as a prior hack), not to prevent someone composing a file that has the same checksum by this algorithm, and then tricking the user into applying the patch to that. Such security measures are beyond the scope of XRIF.
Adler-32 was chosen over more comprehensive, security-minded algorithms (such as MD5) because of the above, and because it is much easier for mere mortals to comprehend and implement.
VARY: variation markerXRIF is not limited to a single patch for a single target file: variations allow for separate sets of patch data to be conveyed in a single XRIF file, while allowing data common to all variations to be marked as such. This can be used for:
It is not required for patchers to support variations; for this reason, a flag in the XRIF header indicates whether they're used. If they are supported and used, some other flags will give parameters for the handling of the variations.
The VARY chunk marks the beginning of a variation, which encompasses everything until the next VARY chunk (or the end of the file). If there is payload data (TREP, ...) before the first VARY, then that data is considered part of the common segment and forms a common part of all the variations in the patch.
Whether the common segment can be selected as a variation in its own right is determined by a flag in the XRIF header. By default, it can be, in order to allow patchers to handle variationless patches in the same manner as variable patches.
Another flag determines whether the common segment is to be applied before or after any selected regular variation. Most of the time it'll be applied first. However, using the full/upgrade example above, the common segment would contain the part that upgrades the original work, and so would be applied after the part (kept in a variation) that defines the original work.
The VARY chunk itself consists of an identifier (for the purpose of selecting a variation by name, using a command-line tool for example) and one or more descriptions (each with a locale declaration) that describe it.
There are two ways to encode this data in the VARY chunk, selected by a flag in the XRIF header. The default format is straightforward: the identifier is limited to 15 characters in length (not including the nul-terminator), and each description consists of a locale declaration (as for iLOC) followed by a 0A byte, followed by the description itself.
The alternate format has the identifier field fixed to exactly 16 bytes long with all unused bytes set to 00, of which there must be at least one, which serves as a nul-terminator, and each description string is prefixed by a 16-bit integer giving its length (including the locale and nul-terminator). The idea behind this is to make it easier for a computer program to parse, particularly one already well geared to the length of something being announced ahead of time, as it is throughout the rest of XRIF. This format became available in the 2006-05-01 edition of XRIF.
The description field is meant only as a short title for the purposes of distinguishing one variation from another. More-detailed “readme” text can be done using the usual annotation features.
It is not required for patchers to support both formats. It is allowable for a patcher to support just the alternate format without supporting the original too.
There is an arbitrary limit of 255 variations (not including the common segment) in a single XRIF file. As of the 2006-05-01 edition, each variation can be independently selected, and multiple variations selected at once, with the appropriate flag set in the header.
TCRC and annotationEach variation can have its own set of TCRC and annotation chunks.
A patcher could use each variation's TCRC to auto-select a variation based on which is the first to pass the TCRC check. However, the patcher should not actually apply the patch automatically, unless requested to; but it can certainly highlight it or make it the default variation if none other is selected by the user.
Annotation present in the common segment applies to the patch as a whole; annotation in an individual variation applies only to that variation.
ixxx chunks: Simplified AnnotationThe 2006-03-31 edition of XRIF added a simplified way to add annotation. Although the INFO chunk is very expressive and powerful, it is a bit much to process, and is quite different in structure than the rest of the XRIF format.
The simplified annotation chunks contain a basic set of annotation abilities, with each entry given in a separate chunk, as listed in the table below. The chunk contents consist of a nul-terminated Unicode text stream, encoded using UTF-8. It may consist of any valid Unicode text except U+0000 (which would be interpreted as the nul-terminator).
Unicode is a character repertoire, a set of characters encompassing all major writing systems in the world, and not a few minor ones. These characters are identified by codepoints, given in the form “U+NNNN”, where NNNN is the codepoint, expressed in base-16 (hexadecimal). Codepoints can range from U+0000 to U+10FFFF.
A “codepoint” is a rather abstract concept. There are several encoding forms which are used to represent Unicode text as an actual sequence of bytes for computer use. Foremost among these is UTF-8, which encodes each codepoint as a sequence of 1 to 4 bytes, with codepoints in the U+0000 to U+007F range (that is, US ASCII characters) encoded as a single byte, in the same way as US ASCII itself, which makes it much more palatable to existing computer systems.
Many people, when they think “Unicode”, they're actually thinking of UTF-16, which is another encoding form of Unicode, encoding each codepoint as one or two 16-bit values (codepoints U+10000 and above require two; the rest only one).
UTF-8 and UTF-16 are both different ways to represent the same thing: Unicode codepoints.
XRIF uses UTF-8 rather than UTF-16 because:
00) will not occur in a valid UTF-8 bytestream except to represent the U+0000 codepoint, which, for that reason, is not allowed to be used.XRIF uses Unicode rather than, say, ISO-8859-1 because it can represent scripts other than Latin, and avoids the problems that arise when one encoding is interpreted as another. Rather than having the text specify the encoding used (which would require patchers to detect and support all of them), I stuck with UTF-8 so there is only a single encoding that has to be dealt with.
As of the 2008-01-10 edition of XRIF, these chunks are changed so that they start with a lower-case “i” instead of upper-case. Also, for future compatibility, all chunks starting with a lower-case “i” are considered annotation, so that new annotation chunks can be defined in the future without having to give each of them their own flags.
The lower-case versions are marked with an advisory flag in the XRIF header (so that old patchers will accept them). The upper-case versions are deprecated.
iNAM | The complete name of the work, including subtitles (if any), for example: “Foo II: Revenge of Foo”. |
|---|---|
iCRE | The person who created the work. If there is more than one creator, each is given in a separate iCRE chunk. If desired, this can be followed by a parenthetical comment briefly explaining their role in the creation process, for example, “Whoever (Translator)”. |
iGRP | The group that created, directed and/or published the work; for example, “Yoyodyne Translations”. There can be more than one, in which case each is given in a separate iGRP chunk. |
iTHX | A person or group or such that the creators wish to thank. This implies that they weren't directly involved in the creation of the work (otherwise they would be named in iCRE or iGRP). It's suggested that, if desired, the name of the person/whatever being thanked be followed by a brief parenthetical comment explaining why; for example, “Whoever (advice and moral support)”. |
iTAR | Specifies the name of the target data (for example, the name of the game being modified). This can be a filename (if the file is expected to have a specific filename), or an arbitrary label for human consideration. Extensions might define other uses for this; XRIF+NES for example expects a GoodNES name here. |
iDAT | The date on which the work or patch was finished or published, or some other date as the patch creator sees fit. Unlike the other annotation chunks, this is in a binary format: a 16-bit value giving the year, a byte giving the month (01=January, ...) and a byte giving the day (starting at 01=1). For example:
D7 07 0B 14 \___/ \/ \/ | | | Year Mon DayThis represents the date November 20, 2007. |
iVER | The version number of the work, in an arbitrary text format such as “1.0”, “Beta 3”, whatever is appropriate. |
iTXT | General-purpose “readme” text, such as an overview of what the work is for, how to use it, and so on, whatever is deemed fit. This is plain text, but should be formatted for a fixed-width display 80 columns/characters wide. Also, U+000A is the only acceptable end-of-line sequence; it is up to the patcher to translate this to the local line-ending conventions (such as 0D 0A for Microsoft platforms) if necessary. |
iURL | A website address (or some other thing that can be expressed as a URL) for the last mentioned iNAM (that is, the whole work), iCRE, iGRP or iTAR. If mentioned before any of these, then it applies to iNAM. This must be a valid absolute URL in accordance with RFC 2396. |
iADR | As for iURL, but specifies an e-mail address. |
iIRC | As for iURL, but specifies an IRC channel: the name of the server, then a space, then a list of one or more channel names separated by commas. For example:
|
iGEN | The name of the patch generator (the program that created the patch). This is provided to give the patch generator authors a place to insert the program's name into a patch in a non-intrusive way. |
iAUT | The person who created the patch itself. (This is basically the human version of iGEN.) |
iLOC | Contains a short string specifying the locale for all subsequent annotation chunks; for example, “en-US” to represent American English. (This also affects INFO and VARY chunks that have empty locale fields.) Locale support is not required to be supported; if it isn't, the implementation shall at least recognize— that is, not complain about seeing— the iLOC chunk and ignore it. If other annotation chunks are given before iLOC, they are assumed to be in the “en” locale (that is, non-region-specific English). |
A patch can include more than one set of these chunks, separated by an iLOC chunk to introduce a new locale, but unless the implementation supports this, it can ignore any annotation chunks occuring after iLOC (unless the iLOC came before any other annotation chunks).
For example, suppose there is an iNAM chunk, in English, followed by an iLOC declaring the “de” (German) locale, followed by another iNAM containing the title in German. In this case, unless the implementation supports multiple locales in one patch, only the first one (the English one, in this case) is dealt with.
The use of these chunks is indicated with a flag in the XRIF header. New annotation chunks will use a lower-case “i”, so that only one flag needs to be set. It is not necessary to set both the original and lower-case annotation flags. Older patches, however, might use the upper-case versions (with the attendant flag in the header), so patchers should, but are not required to, be prepared to deal with them.
INFO chunk: Consolidated AnnotationThis was the original XRIF annotation method: one chunk containing all the annotation, in “Name=Value” format. This is semi-deprecated in favor of the simplified Annotation chunks (iNAM, iCRE, ...), since those are adequate and much less complex. However, there's nothing wrong with a patch using both methods; if a patch uses both and a patcher supports both, INFO takes precedence over the simplified annotation chunks.
As with VARY, there are two ways to encode this data.
In the default format, the chunk begins with a nul-terminated locale declaration (in the manner of iLOC), which applies to the entire chunk. This is followed by the entries that form the actual content, each being in a “Name=Value” format. The Name is in US ASCII, consisting only of upper-case letters, digits, underscores (“_”) and dashes (“-”), at a maximum of 64 characters in length, including the “=” which separates the name and value. Also, it may not start with a dash. The value part is in UTF-8 as for other annotation, and is nul-terminated.
The alternate format works the same way, except the locale declaration and each entry are prefixed by a 16-bit integer giving its length (including the nul terminator), to make it easier for programs to parse. This format became available in the 2006-05-01 edition of XRIF, and is invoked by a flag in the header.
Either way, an entire entry, including name, “=”, value, and nul terminator, is limited to 65536 bytes in length.
A given name may occur more than once, in which case they are taken to be members of a list. For example, if there is more than one creator, each gets their own CREATOR entry (plus CREATOR:LINK and so forth).
Some names can be attached to some other name, to convey some subset of information about the subject. For example, a CREATOR entry gives the name of the creator of the work; one could then associate, for example, a website address with them by using a CREATOR:LINK entry. As suggested, “:” is used for this purpose.
Names which are not attached to anything are assumed to be “attached” to the work conveyed in the patch.
The name field can be any arbitrary string fitting the above constraints, but for the purposes of interoperability, a few names are defined here with standard semantic properties; these should be used in preference to other labels, to the extent that they are adequate.
TITLEiNAM chunk.LOCALEINFO chunk. If the work is a translation, this entry refers to the destination language. (The source language is given with TARGET:LOCALE.)LINKiURL.EMAILiADR.IRCiIRC.VERSIONiVER.CREATORCREATOR entry. Equivalent to iCRE.CREATOR:ROLECREATOR played in the creation of the work. This can be arbitrary, but it is recommended that one of the following labels be used (case as written) if one of them is appropriate:
HackerTranslatorDesigner (for example, storyline designer)Writer (as of dialogue and such)Programmer (as for purpose-built utilities and such)Composer (as of music)GROUPiGRP.THANKSCREATOR), but whom the creators nevertheless wish to give credit. Equivalent to iTHX.THANKS:ROLECREATOR:ROLE, but applies to the last THANKS. Beta-testers, if listed, should be listed using THANKS instead of CREATOR, and for the purposes of THANKS:ROLEshould be credited as “Beta-tester”.TYPEHackTranslationUpdateBugfixTARGETITAR.TARGET:VERSIONTARGET:LOCALELOCALE.)DATEiDAT, except it's in text form, in the form “YYYY-MM-DD”. For example, January 10, 2008 becomes “2008-01-10”.READMEiTXT.README:FONT-FAMILYSpecifies the general type of font recommended for use with README text. This must be exactly one of “serif”, “sans-serif” or “monospace”. This does not specify a specific font, only the font type: if it's specified as anything other than those listed, it is ignored.
It is up to the patcher to associate specific fonts with these general font types; it is recommended the patcher allow the user to select them. For example, on MS Windows systems, it will probably work well to start out with Times New Roman for the “serif” type, Arial for the “sans-serif” type, and Courier New for the “monospace” type.
There is no default: if this isn't specified, then the text of README should not make any assumptions about the font type. For example, if the README text contains ASCII diagrams or something, then it should specify a monospace font.
It's not required for patchers to distinguish between serif and sans-serif.
README:TABSPACINGU+0009 character is encountered, the location at which the next character will be written, is advanced to the next multiple of this many characters. The default is 8.README:COLORSThis consists of two RGB color values in the form “RRGGBB”, where RR is red, GG is green, BB is blue, each expressed in capitalized hexadecimal. The two colors are separated by a comma.
The first color specifies the text color, the second specifies the background color. These apply to the README text, and perhaps to other text at the option of the patcher.
Patchers are free to disregard this, including being configured to do so by the end user.
PATCH:DATEPATCH:CREATORiAUT.PATCH:GENERATORiGEN.This list is not exhaustive, but you can mix and match attachments; for example, GROUP:IRC or PATCH:GENERATOR:VERSION (the latter demonstrating multiple attachments).
These chunks represent out-of-band data, relating to the patch itself. The XRIF and XAPP chunks logically fall under this group, but the labels of other chunks in this group all start with “.” (a period / full stop) as their first character.
Extensions are not allowed to redefine or remove these, or to add their own.
.END chunk: end-of-patch markerThis chunk marks the end of the patch, and is required to be used for this purpose in every XRIF file. Patchers shall ignore everything in the file past the end of this chunk.
The chunk itself has no content.
.SYS chunk: Implementation-specific dataThis chunk conveys program-specific information in a program-specific format. The chunk starts out with a nul-terminated identifier, 32 characters maximum (including the nul terminator) that identifies the program in question; other programs use this to ignore .SYS chunks other than their own. After this identifier is data in any format the program may specify, and the interpretation of it is left to that program.
An example of how this chunk might be used: a patch generator might record the local filenames of the two files from which the patch was created, as well as any relevant program options used in the process. Then, if the patch creator updated the hack, they might then open the patch in that program and tell it to re-create the patch, which it could then do without the user having to re-configure everything. It might even automatically create an “upgrade” version using variations.
It is not required for any program to use this chunk, and it's allowed for programs to look in other programs' .SYS chunks if it wants.
.PAD chunk: Padding or commentsThe purpose of this chunk is to take up space, such as to align the following chunk so its contents start on a 0x10-byte boundary, or to delete a chunk from a file without rearranging subsequent chunks, or whatever.
The contents are ignored. However, it's suggested that if the chunk doesn't start with 00 or 20, then it might be interpreted as an inline comment; but it's certainly not required to do so. Basically, there will not be any harm in ignoring any .PAD chunks you encounter: they have no effect on how the patch operates.
.AKA chunk: Chunk name aliasingThis chunk was created (in the 2006-01-22 edition of XRIF) to address a potential problem with Extensions. Suppose, hypothetically speaking:
FROB.FROB with a different meaning.Precedence, in this case, is not an issue: if the Extension is used, then FROB chunks would be interpreted as the Extension's and not XRIF's. But in such a case, the patch could not make use of XRIF's FROB.
To address this problem, an .AKA chunk can be used to assign an alias to the Core XRIF FROB chunk, so that it is known by another name. Supposing for example that it is aliased as “HOSE”, then instances of “FROB” chunks refer to the Extension's FROB, and instances of “HOSE” chunks refer to the Core XRIF FROB.
The .AKA chunk itself consists of nine bytes: two chunk names (each four bytes), and a byte identifying the Extension layer.
The first chunk name is the name of the chunk being aliased. The second chunk name is the alias itself. (In the above example, FROB is being aliased as HOSE, so they would be listed in that order in the .AKA chunk.) The byte at the end indicates which Extension layer the first chunk name refers to: layer 00 refers to the XRIF Core, layer 01 refers to the Extension invoked by the first XAPP, and so forth. In the example above, you want to create an alias for the Core XRIF FROB, so the Extension layer would be 00.
It is an error if any of the following are true:
.END and so forth) or is XRIF or XAPP.If the chosen alias refers to a chunk that is already defined somewhere, but requires a flag to be set in its XRIF/XAPP header in order to be used, and that flag isn't set, then it's a legal alias name. For example: if the patch does not use TADD (according to the header), then “TADD” is a legal alias name.
This is to allow future updates to XRIF and/or the Extension in question to add new chunks: otherwise you might define an alias name for your patch, then a later edition of XRIF or the Extension is published which specifies a new chunk that happens to use the same name as the alias you chose, and then a program is written against the new edition, and suddenly the once-legal alias name in the already-created patch will have been clobbered: which would be ironic, given the purpose of the .AKA chunk.
So, if XRIF or the Extension adds a new chunk, then it would have to be invoked by a flag in its header, so that old patches will continue working with new programs.
Once a valid alias is created, then the alias becomes a legal chunk name for the remainder of the patch, and to use that alias is equivalent to using the chunk that was aliased.
Being itself an addition to XRIF, the .AKA chunk has a flag in the XRIF header indicating whether it's used, and it is not requird to be supported (in fact, when dealing with Core XRIF files, it's required to not be supported).
It is recommended that XRIF files have the filename extension “.xrif”. For filesystems (such as that of MS DOS) that do not allow filename extensions longer than three characters, then “.xri” shall be used, because this is what would be produced by automatically truncating “.xrif” to three letters, as would be done by MS Windows (which supports longer filename extensions) when copying to an MS DOS environment, for example.
For Macintosh filesystems, which use a Type Code in lieu of (or in addition to) a filename extension, “XRIF” (if anything) is recommended for this purpose. The resource fork of the file shall be empty; the data fork shall contain an XRIF file as described in this document.
XRIF itself does not address the issue of data compression. This is expected to be handled externally, for example by gzip-compressing an XRIF file. If the target file is compressed, it is up to the patcher and/or user to decompress it first so that it can be patched, and if necessary to recompress it later.
For example, this could be done by using a decompressor to pipe the uncompressed data to the patcher, which in turn can pipe the patched data to a recompressor, assuming of course that the patcher supports piping:
zcat foo.gz | xrif - foohack.xrif | gzip -c9 - > foohack.gz
Such an approach deals with compression external to XRIF itself, which means the patcher itself is not required to support it. This, in turn, means you can use any compression format you have suitable decompressors and compressors for.
The same considerations exist for the patch itself:
zcat foo.xrif.gz | xrif foo.bin -
There's nothing wrong with a patcher being equipped to do this itself; but in any case, compression issues are expected to be dealt with externally to XRIF itself.
I usually pronounce the name of the format as “eks-riff”, and the standards of basic human decency compel you to do likewise. To do otherwise— especially on a Tuesday— will inexorably lead to Burnination.
Consider yourself warned.
XRIF has a number of features that could be difficult for patch generators, without human intervention, to take full advantage of (such as block-copy operations). Strictly speaking, such matters are beyond the scope of XRIF itself: this document only defines the language, so to speak, and it's up to you to speak it well.
Having said that, I might offer some advice on certain details.
The primary use of TCFR and TCFS is to make it practical to insert, remove, or otherwise move data around, by copying the moved data to its new location before performing other modifications.
For some uses, such as the NES, they could be used on a smaller scale: for dealing with pointers. In many games, there might be an array of data, such a dialogue text, where each entry has an arbitrary length. There will be a list of addresses for the start of each item in the array, so that the game can find them quickly without needing to scan the data from within the game itself.
Suppose you change one of the array entries, and in the process, you increase its length. All subsequent array entries would have to be moved forward to make room, but otherwise they would not be changed.
For a patcher to go in and detect which ones were changed and which were simply moved, without having any context (such as knowing that this is in fact a variable-sized array with a corresponding address list), could be a difficult task, but it would need to be done in order to use TCFR/TCFS operations to move the unchanged strings, which is more difficult than hard-wiring the byte-for-byte changes with simple-minded TREP operations, and conveys less of the original data in the patch itself.
Now, one approach might be to tell the patcher just where such data structures exist: then at least it knows where to look and how to interpret that data. It can analyze the data (when it might not have analyzed it in that way before), and see which array entries were merely moved but otherwise unchanged, and insert the appropriate copy operations, followed by TREP to implement actual changes in the contents— and not merely positions— of the other entries.
As an aside, it might be easier to use TCFS, rather than TCFR, to do that: with TCFR, the patch generator would be tasked with trying to juggle array entries around without clobbering data that needs to be copied later. Even if it could be done, it would be difficult and error-prone to write a program to do it. Instead, the program can just use TCFS and not have to worry about such things. It is for precisely this reason that TCFS support is required, and not merely optional, for the XRIF+NES Extension.
For much the same reasons: if you insert data into the middle of a file, it would be more effective if the patch creator told the patch generator where these things had occured, and it can then insert the appropriate copy rules beforehand, rather than trying to compare every body sequence to every other byte sequence and apply heuristics to determine where such deliberate data shifts had occured; or taking the crude, IPS-style approach of comparing the shifted data with the unrelated data that used to be at that file location.
For graphical programs, it is suggested that annotation and VARY descriptions be handed off to the window system, which is probably already well-equipped to handle Unicode text. If you have a means to, then pass the locale information along as well (for example by writing <html lang="whatever">, if using an HTML engine for the purpose).
If you must perform text processing yourself (such as if writing a terminal-based program), you might do any of the following:
U+0000 to U+007F), displaying the rest with placeholder text. Just be sure to indicate the number of characters correctly: codepoints beyond US ASCII range take up multiple bytes apiece. This isn't a very attractive approach, but is about as simple as it gets, and will be sufficient for English text.U+0000 to U+00FF), displaying the rest with placeholder text. This might be done under the assumption that the display environment expects text encoded in ISO-8859-1. This isn't attractive either, but will suffice for most western European languages.If you are using the C programming language— or some other language that can use function pointers or otherwise “assign” a function or method call to a variable— you might deal with loading block values (addresses and lengths) by using function pointers.
Just as an example, you could define the functions/methods readint8(), readint16(), readint31() and so forth. These would all return the largest datatype your program supports (64-bit if your program supports that, 32-bit otherwise).
Then, have two function pointers, say read_address and read_length, and assign one of the above functions to them depending on the contents of the XRIF header, such as by using a switch statement. Then, you can handle block values simply by invoking the relevant function pointers, without having to check which one to load every time.
The following things are required to be supported: that is, patch creators are free to use these and can expect that they will be widely supported by XRIF implementations; they are also not marked by flags in the XRIF header.
0x7FFFFFFF).XRIF (file header chunk)TREP (replacement)TRLE (repeated replacement)TCFR (copy from result).END (end-of-patch marker)The following chunks can be ignored, but must at least be recognized without an error being reported or something:
TCRC (checksums)INFO (consolidated annotation).SYS (implementation-specific data).PAD (padding/comments)The following are not required to be supported at all. For all of these, the XRIF header will report, ahead of time, which ones are used:
XAPP (extension declaration)Extensions have never been required to be supported. Patchers that do support a given Extension will necessarily need to support XAPP in order to detect that the Extension has been used. Supported Extensions come with their own support requirements, on top of XRIF itself.
TCFS (copy from source)If this is used, the patcher will need to make arrangements to keep the original, unmodified target data on hand throughout the patching process. This can be impractical, such as with large files, and some patchers might be designed or configured to operate directly on the target file rather than on an on-disk or in-memory copy.
TFSM (data size change)This is optional as of the 2006-01-22 edition of XRIF. Prior to then, it was required to be supported, except in situations where it's impossible— for whatever reason— to resize the file.
TAND, TIOR, TXOR (bit twiddlers)These are all optional as of the 2006-01-22 edition of XRIF. Prior to then, they were required to be supported. Originally, their format was closer to that of TRLE; this was simplified to having the patch-side data be only a single byte. This simplification was done because, at the time, these chunks were required to be supported.
The 2008-01-10 edition added an alternate, TREP-style format in order to allow for XOR-type patches (that use TXOR rather than TREP as their primary patching mechanism). This is not required to be supported either. In fact, it's permissible to support just one of the two to the exclusion of the other.
TADD (addition), TROT (shifting/rotation)These have always been optional on account of relative complexity.
VARY)This has always been optional, though encouraged to be supported.
.AKA (chunk name alias)This is the first new chunk (not counting simplified Annotation) since the original 2006-01-03 edition of XRIF, which necessarily makes it non-required. However, even if it had been a part of XRIF from the very beginning, I don't think I would have made it required to be supported anyway.
In fact, it's not allowed to be used at all for Core XRIF patches (ones which don't use Extensions). It would only be needed if there were a chunk name collision between XRIF and an Extension (or between two Extensions if they can be used together in the same patch).
0x7FFFFFFFBecause not all filesystem libraries support addressing beyond this range, and some development environment make it difficult, if not impossible (or non-portable) to obtain datatypes that can represent such integers anyway.
Optional as of the 2006-04-26 edition of XRIF.
Initial publication of XRIF.
TFSM flag to XRIF header.TCFR to allow the source and target ranges to overlap.TCFR and TCFS so their Data Length fields are of the Address type rather than the Length type.TFSM operation may not set the file size to be anything larger than what a block address can specify.TAND, TIOR and TXOR so the patch-side data is just one byte, instead of being in a TRLE-style format..AKA chunk.TFSM, TAND, TIOR, TXOR are no longer required to be supported.XRIF flag bytes now alternate between Advisory and Mandatory.README:COLORS to INFO.INFO and VARY, where each string is prepended with a 16-bit integer giving its length.ICRE, IGRP, IURL, IADR now contain only a single item per chunk, instead of possibly multiple values in a single chunk.IURL and IADR no longer have a label; just an address.TAND, TIOR and TXOR to give them a structure like that of TREP, chiefly to allow for XOR-style patches. Added Mandatory flag to that effect.TFSM's relative-sizing methods are deprecated.TFSM, if used, be used before any Payload chunks.TCRC to allow multiple checksums in a single chunk, instead of each being in a separate chunk.INFO is semi-deprecated; nonetheless, THANKS and IRC entry names were added to the official list.i” instead of an upper-case one, and added an Advisory flag to that effect, and rendered the upper-case versions deprecated.iTHX, iGEN, iAUT and iIRC annotation chunks.TRLE block's range of effect (length × iterations) can't exceed 64 KB unless using block lengths greater than 16-bit.This document— as well as the XRIF format itself— was created by Vystrix Nexoth. However, this document, and the format it describes, are both committed to the public domain. This means you're free to distribute copies of this document, and to implement the format described, with no restrictions or requirements imposed on you by me.