Target file

What it is and why it’s needed?

Different machines decompress the same file with different decompression times. So, in order to properly compute the decompression time of a compressed file for a given target machine, bc-zip needs a decompression time model tuned for that target machine (which may be different than the machine used in compression!).

bc-zip does not ship with any decompression time model by default. Instead, bc-zip needs a target file when it is invoked with the “compress” command.

A target file is a small, human-readable file which encodes the decompression time model for a particular machine.

How do I get it?

A target file can be obtained in an automatic or semi-automatic fashion through the use of the provided calibrator tool.

A calibrator obtains the target file of the machine in which it is executed.

The calibrator tool is composed of two objects: a shell script calibrator.sh and an executable calibrator, both located in the tool/ directory.

The user must invoke the shell script, taking care of having both the shell script and the executable on the same directory.

The tool is invoked as follows:

./calibrator.sh 	<target name> <memory hierarchy descriptor>

Where:

• target name: the name of that particular machine. If X is passed, the outputted target file will be named X.tgt.
• memory hierarchy descriptor: A file with a line for each memory level (e.g. L1 and L2 caches, main memory), sorted by capacity, with just two fields for each line separated by some space: level capacity, in kilobytes, and latency, in nanoseconds.

A valid memory hierarchy descriptor could be the following:

32           0.4
256          1.2
6144         3.3
4294967296   80

Which means that that particular machine has a L1 cache of 32KB with access latency of 0.4ns, a 256kb L2 cache with latency 1.2ns, a L3 cache with latency 3.3 and a main memory of 4GB with latency of 80ns.

This file can also be generated automatically via the get_latencies tool, which can be invoked in the following way:

./get_latencies lmbench_file

Where lmbench_file is the output of tool lat_mem_rd in the LMBench3 suite, with the first line (starting with ") and last line (empty), stripped (look at the examples if that’s unclear).

Examples

./lat_mem_rd 64 256 2>&1 | tail -n +2 | head -n -1 > latencies.txt
./get_latencies latencies.txt > memory_hierarchy.txt

First get latencies.txt, a “probe” of memory latencies of the first 64 megabytes of main memory, by invoking lat_mem_rd, and stripping the first and last line with head and tail. Then, pass this file to the get_latencies tool, which finally obtains a valid memory hierarchy descriptor.

./calibrator client memory_hierarchy.txt

Obtains the target file client.tgt related to the machine in which it is executed, provided that memory_hierarchy.txt is a valid memory hierarchy descriptor of the same machine (such as the one obtained in the previous example).