Terminally confused (part five)

In part three, I gave an overview of terminal ioctls and terminal capabilities, and promised I would discuss both in part four. In part four, I discussed terminal ioctls in detail, and ran out of space to talk about terminal capabilities!

Warning: This post is a bit rambly, and it may not be rewarding to read the whole thing. Scroll to the bottom—below the final horizontal rule—for the summary. Or, if you care more about job control, skip to part six.

In part three, I mentioned three features of terminals that are implemented as terminal capabilities: colour output, the terminal bell, and moving the cursor. I pointed out that terminal capabilities, unlike terminal ioctls, are features that can be activated simply by writing to terminals, and also that they primarily affect what is displayed rather than how data are processed.

The upshot is that as long as you are reading from or writing to a terminal device, you are entitled to the assumption that certain terminal ioctls are supported, because the kernel device driver for the terminal device will take care of them. On the other hand, the terminal capabilities depend on what device is actually displaying output. On a local terminal like tty1, you can’t really see the distinction. However, the distinction is important for a serial line, as an example. Suppose you are at a thin client connected by a serial line to some mainframe in another room. When you type in a command, the program actually gets launched at the mainframe. The program on the other side knows that it is outputting to a device called /dev/ttyS1 or something similar, but that doesn’t give it any useful information about the capabilities of the thin client. For example, suppose you are running ls --color=auto. It would be nice if it had some way to tell whether your thin client supports colour output, so that it doesn’t try to write colour escape codes to a monochrome terminal, which would likely result in stray escape codes cluttering the output. It cannot, however, discover this using an ioctl. So what does it do?

The answer is quite ugly, perhaps surprisingly so. Every Unix-like system has a database called terminfo(5). terminfo consists of a collection of files, each of which describes the capabilities of a single kind of terminal. For example, on my system, the file /lib/terminfo/l/linux describes the capabilities of the Linux console (system console or numbered virtual terminal), and the file /lib/terminfo/v/vt100 describes the capabilities of the VT100.

A VT100 terminal. Although you are not likely to see one of these around these days, its legacy lives on as the de facto standard for terminal capabilities, a bare minimum that terminals strive to conform to.

When a program that produces terminal output runs, it expects to find an environment variable called TERM that tells it what kind of terminal it is outputting to. When you start a shell on tty1, for example, either init(8) or getty(8) should at some point set TERM=linux, which is inherited by the shell, and in turn inherited by any programs you start from the shell, such as ls(1), so that it will know that it’s running on a virtual terminal whose capabilities are determined by the Linux terminal driver; and whenever it wants to know whether it can do something, it consults the corresponding terminfo entry. When you use a terminal emulator, on the other hand, the terminal emulator has full control over what the output looks like, whether it produces an audible bell or visible bell, and so on, so sets TERM to some value to reflect its own capabilities. Most terminal emulators, for example, set TERM=xterm because they have the same capabilities as the program xterm, the original terminal emulator for the X window system. If you are using a serial terminal, you’ll have to set TERM manually on the remote machine (by typing export TERM=name into the shell, for example, once you’ve started it on the remote machine) in order to tell the program on the other side what kind of display and keyboard you have. ssh(1) will automatically forward the value of the TERM variable from the local machine to the remote machine.

xterm running in twm. Believe it or not, modern graphical terminal emulators have the same set of capabilities as this ugly thing.

The terminfo files are not human-readable, unlike the earlier termcap format that it replaced. However, most systems come with a utility called infocmp(1) that can be used to print out a terminfo entry in human-readable form. Invoking it as simply infocmp will print out the capabilities of the terminal it is running on, as determined by the TERM variable. You can also specify which type you want to know about, for example, by invoking infocmp linux or infocmp xterm. It prints out the capabilities in the same format as the source files used to generate terminfo entries in the first place, as specified by POSIX. This is what the output looks like:

#       Reconstructed via infocmp from file: /lib/terminfo/l/linux
linux|linux console,
        am, bce, ccc, eo, mir, msgr, xenl, xon,
        colors#8, it#8, ncv#18, pairs#64,
        acsc=+20\,21-30.^Y0\333`04a\261f\370g\361h\260i\316j\331k\277l\332m\300n\305o~p\304q\304r\304s_t\303u\264v\301w\302x\263y\363z\362{\343|\330}\234~\376,
        bel=^G, blink=\E[5m, bold=\E[1m, civis=\E[?25l\E[?1c,
        clear=\E[H\E[J, cnorm=\E[?25h\E[?0c, cr=^M,
        csr=\E[%i%p1%d;%p2%dr, cub=\E[%p1%dD, cub1=^H,
        cud=\E[%p1%dB, cud1=^J, cuf=\E[%p1%dC, cuf1=\E[C,
        cup=\E[%i%p1%d;%p2%dH, cuu=\E[%p1%dA, cuu1=\E[A,
        cvvis=\E[?25h\E[?8c, dch=\E[%p1%dP, dch1=\E[P, dim=\E[2m,
        dl=\E[%p1%dM, dl1=\E[M, ech=\E[%p1%dX, ed=\E[J, el=\E[K,
        el1=\E[1K, flash=\E[?5h\E[?5l$<200/>, home=\E[H,
        hpa=\E[%i%p1%dG, ht=^I, hts=\EH, ich=\E[%p1%d@, ich1=\E[@,
        il=\E[%p1%dL, il1=\E[L, ind=^J,
        initc=\E]P%p1%x%p2%{255}%*%{1000}%/%02x%p3%{255}%*%{1000}%/%02x%p4%{255}%*%{1000}%/%02x,
        kb2=\E[G, kbs=\177, kcbt=\E[Z, kcub1=\E[D, kcud1=\E[B,
        kcuf1=\E[C, kcuu1=\E[A, kdch1=\E[3~, kend=\E[4~, kf1=\E[[A,
        kf10=\E[21~, kf11=\E[23~, kf12=\E[24~, kf13=\E[25~,
        kf14=\E[26~, kf15=\E[28~, kf16=\E[29~, kf17=\E[31~,
        kf18=\E[32~, kf19=\E[33~, kf2=\E[[B, kf20=\E[34~,
        kf3=\E[[C, kf4=\E[[D, kf5=\E[[E, kf6=\E[17~, kf7=\E[18~,
        kf8=\E[19~, kf9=\E[20~, khome=\E[1~, kich1=\E[2~,
        kmous=\E[M, knp=\E[6~, kpp=\E[5~, kspd=^Z, nel=^M^J, oc=\E]R,
        op=\E[39;49m, rc=\E8, rev=\E[7m, ri=\EM, rmacs=\E[10m,
        rmam=\E[?7l, rmir=\E[4l, rmpch=\E[10m, rmso=\E[27m,
        rmul=\E[24m, rs1=\Ec\E]R, sc=\E7, setab=\E[4%p1%dm,
        setaf=\E[3%p1%dm,
        sgr=\E[0;10%?%p1%t;7%;%?%p2%t;4%;%?%p3%t;7%;%?%p4%t;5%;%?%p5%t;2%;%?%p6%t;1%;%?%p7%t;8%;%?%p9%t;11%;m,
        sgr0=\E[0;10m, smacs=\E[11m, smam=\E[?7h, smir=\E[4h,
        smpch=\E[11m, smso=\E[7m, smul=\E[4m, tbc=\E[3g,
        u6=\E[%i%d;%dR, u7=\E[6n, u8=\E[?6c, u9=\E[c,
        vpa=\E[%i%p1%dd,

Each capability has a name in the terminfo specification, and there are three kinds of capabilities: boolean, numeric, and string. Boolean capabilities are given first, as simple names without values: here, the strings am, bce, ccc, eo, mir, msgr, xenl, xon are boolean capabilities. A boolean capability means the terminal has a feature, which might not necessarily be good. For example, the ccc capability means that the terminal’s colours are not hard-coded but can be redefined (typically by giving RGB values). On the other hand, hz is actually a glitch: if the terminal has this capability, then it’s incapable of printing the tilde character. (The reason why this capability is included is that there was once a terminal with this very glitch; hz stands for Hazeltine, the manufacturer of this terminal.) Following the boolean capabilities are numeric capabilities, given as the capability name followed by “#” and then the value. The entry colors#8, for example, says that the terminal supports 8 colours, and it#8 means that the tab is (initially) eight spaces wide. After that we see the string capabilities, which are of the form cap=str. There are a lot of these. Although all the capabilities are documented in the terminfo(5) man page, I will give some examples of string capabilities here, with a typical value taken from the linux terminal as above. The string \E represents the escape character (ASCII 0x1B, echoed as “^[“).

• bel=^G: Terminal bell. Note that you can’t change this, the way you could if it were a terminal attribute.
• blink=\E[5m: Put the terminal into blink mode; all subsequent characters written will blink until blink mode is left.
• bold=\E[1m: Put the terminal into bold mode; all subsequent characters written will be bold until bold mode is left.
  

  Bold mode on a Linux virtual terminal, obtained by echoing the bold terminal capability string. Note the appearance of the normal cursor.

• civis=\E[?25l\E[?1c: Make the cursor invisible.
• clear=\E[H\E[J: Clear the screen and “home” the cursor (move it to the upper left corner of the screen).
• cnorm=\E[?25h\E[?0c: Make the cursor normal. Undoes civis and cvvis. On a linux-type terminal, the normal cursor is a horizontal underline.
• cr=^M: Carriage return; returns the cursor to the leftmost column.
• cub=\E[%p1%dD: Move the cursor left (backward) a given number of columns. The string %p1%d means that the first (and only) parameter should be printed in %d format (as in printf(3)) at that position, so, for example, ^[[3D moves the cursor 3 columns left.
• cub1=^H: Move the cursor a single column left. This is, of course, equivalent to using cub and specifying 1 as the parameter.)
• cud=\E[%p1%dB: Move the cursor down a given number of columns.
• cud1=^J: Move the cursor down one column. You can observe this if you turn off ONLCR (stty -onlcr), which will force Ctrl+J to produce a literal ^J rather than ^M^J, and type stuff into cat(1); pressing Ctrl+J, generating a newline character, will move the cursor down one line without moving it back to the leftmost column as it usually does.
• cuf=\E[%p1%dC, cuf1=\E[C: Move the cursor forward a given number of columns, or one column, respectively.
• cup=\E[%i%p1%d;%p2%dH: Move the cursor to the given row (parameter 1) and column (parameter 2). Notice that %p1 means the first parameter and %p2 the second. So ^[[4;2H will move the cursor to the fourth row and second column. The code %i is actually an instruction to increment the first two parameters, translating from zero-based coordinates (row 3 and column 1) to one-based coordinates.
• cuu=\E[%p1%dA, cuu1=\E[A: Move the cursor up a given number of columns, or one column, respectively.
• cvvis=\E[?25h\E[?8c: Make the cursor very visible. In a linux-type terminal, the very visible cursor is a rectangle whose width is that of the entire column and whose height is that of the entire row.
  
  

  Very visible cursor, shown after echoing the cvvis string capability.

• dch=\E[%p1%dP, dch1=\E[P: Delete a given number of characters, or one character, respectively, at the current cursor position. This behaves like the Delete key, in that it causes the characters to the right of the deleted characters to be shifted over, but it does not delete characters from the input buffer.
  

  First, the string “foo” is echoed, moving the cursor to the fourth column of the second row. Then, the sequence \E[1A moves the cursor up one row, placing it on top of the “t” or “root”. Then the escape sequence \E[1P is used to delete that character, shifting everything to the right of it over one character. The following character, “@”, is then overwritten by a space, and finally a ‘\n’ is written to go to the next line, followed by the extra ‘\n’ that echo(1) normally appends.

• dim=\E[2m: Turn on “dim mode”. On the linux-type terminal, this simply results in the foreground (text) colour being ignored, so that all text appears dark grey.
• dl=\E[%p1%dM, dl1=\E[M: Delete a given number of lines, or one line, respectively, starting from the current line; analogous to dch and dch1. These commands do not move the cursor.
• ech=\E[%p1%dX: Erase a given number of characters. This is analogous to Backspace, but it does not erase characters from the input buffer. Thus, unlike Delete, it moves the cursor backward p1 times.
• ed=\E[J: Erase the screen starting from the current cursor position, that is, all characters on the current line on or to the right of the cursor, and all lines below the current line. Does not move the cursor.
• el=\E[K: Delete all characters on or to the right of the cursor on the current line, without moving the cursor.
• el1=\E[1K: Delete all characters strictly to the left of the cursor on the current line, without moving the cursor.
• flash=\E[?5h\E[?5l$<200/>: Visible bell (flash the screen). The $<200/> is not actually written to the terminal; instead, this string instructs the program to delay by 200 milliseconds when it is writing this capability. So a program that wants to produce a visible bell should write \033[?5h\033?5l and then sleep for 200 ms before doing anything else.
• home=\E[H: Move the cursor to home position (upper left corner of screen).
• ht=^I: Horizontal tab. Normally, pressing Tab produces this character.
• ich=\E[%p1%d@, ich1=\E[@: Insert a given number of spaces, or one space, respectively, shifting all characters on or to the right of the cursor to the right, without moving the cursor.
• il=\E[%p1%dL, il1=\E[L: Insert a given number of blank lines, or one blank line, respectively, shifting down all lines from the current line down, without moving the cursor.
• initc=\E]P%p1%x%p2%{255}%*%{1000}%/%02x%p3%{255}%*%{1000}%/%02x%p4%{255}%*%{1000}%/%02x: This command, which sets colours, takes four parameters; the first indicates which colour to set, and the other three give the red, green, and blue channels, in that order. This complicated string is actually a sequence of instructions for how to generate the actual string, which involves pushing operands onto a stack and performing arithmetic and so on. This programming language is described in the manual page. (In case you’re wondering: it always halts, so it’s not Turing complete.)
• kb2=\E[G, kbs=\177, kcbt=\E[Z, kcub1=\E[D, kcud1=\E[B, ...: These give the strings produced by various keys; for example, kbs=\177 indicates that the Backspace key produces the character '\177'.
• nel=^M^J: Move the cursor to the first column of the next line. Typically, this is produced by the Enter key.
• rev=\E[7m: Enter reverse video mode. This means that the current foreground colour is used as the background, and vice versa.
  

  Reverse video mode. The terminal is placed in reverse video mode with \E[7m, causing “foo” to be echoed with black foreground and grey background. Note that rev and setaf commute. Thus, the sequence \E[31m is used to change the foreground colour to red. However, since we are still in reverse video mode, subsequent characters have background in red.

• rmir=\E[4l: Exit insert mode. See smir
• rmso=\E[27m: Exit standout mode. See smso
• rmul=\E[24m: Exit underline mode. See smul
• rs1=\Ec\E]R: Reset the terminal. This clears all text styles, resets all colours to their defaults, clears the screen, and places the cursor in the upper-left corner of the screen, among other things.
• setab=\E[4%p1%dm, setaf=\E[3%p1%dm,: Set the background or foreground colour, respectively. For example, colour 1 is usually red, so writing \033[31mHello, world! to the terminal prints Hello, world! in red.
• sgr=\E[0;10%?%p1%t;7%;%?%p2%t;4%;%?%p3%t;7%;%?%p4%t;5%;%?%p5%t;2%;%?%p6%t;1%;%?%p7%t;8%;%?%p9%t;11%;m: Set multiple character attributes simultaneously. This string is a complicated sequence of instructions like that of initc.
• sgr0=\E[0;10m: Resets all attributes (text colour, background colour, bold, underline, and so on).
• smir=\E[4h: Enter insert mode. In this mode, whenever a character is written to the terminal, all characters on or below the cursor are first shifted right one column.
• smso=\E[7m: Enter standout mode. On my linux terminal, standout mode appears to be identical to reverse video.
• smul=\E[4m: Enter underline mode. On my linux terminal, underlining is not supported, and instead underline mode results in all characters being printed with a cyan foreground. On the other hand, Konsole (which has xterm capabilities) will underline just fine.
  

  Underlining on Konsole. Notice that the first part of the next prompt is affected, but not the colon and directory. We can infer that this means that the shell’s PS1 variable sets the colour to green before printing the user and host, but does not reset first, and then resets before printing the colon. If you’re curious, examine the output of echo PS1.

---

Now, the terminfo files are not guaranteed to be in any fixed location, nor are they guaranteed to be in any fixed format. The proper way to access them (in POSIX) is through a set of functions defined in curses.h and term.h (which must be included in that order). I’m not going to describe them in detail, because I’ve already explained the low-level details of terminal capabilities, so these are included just for interest:

• int setupterm(char *term, int fd, int *errret);: Initializes the interface by setting up appropriate data structures for terminfo data. This must be called before any of the other functions. Here term is the name of the terminal. If this is null, the TERM environment variable is used. This function returns either OK or ERR, and if the latter, then it places a return code into *errret (as long as it is not null).
• int tigetflag(char *capname);
int tigetnum(char *capname);
char *tigetstr(char *capname);
  These functions retrieve the value of a boolean, numeric, or string capability with name given by capname, respectively, or a negative value on error.
• char *tparm(char *str, ...);: Interpreter for the strange programming language visible in complex string capabilities such as initc and sgr. The value of the string capability returned from tigetstr is passed in as str, followed by the parameters in order. For example, tparm("\033[3%p1%dm", 1) will return a pointer to the string "\033[31m".
• int tputs(const char *str, int affcnt, int (*putc)(int));: This function should be used to actually write string capabilities (after formatting with tparm if necessary) to the terminal device. See the manual page for details.

Additionally, the utility functions mvcur(3), vidputs(3), vidattr(3), vid_puts(3), and vid_attr(3) are provided to facilitate cursor movement and outputting text with colour and attributes (bold, underline, and so on). Consult the manual pages for details.

Furthermore, most of the time, even those functions are too low-level, and you will want to use the curses library. (On Linux, you will almost always come across an implementation of curses called ncurses. This is not a different library from curses, just as CPython is not a different programming language from Python.) The curses library provides an abstraction that frees applications from having to deal with the irritating details of dealing with terminfo. It is extensively documented online and there are many tutorials for it, so I will not discuss it here.

---

As I mentioned previously, you cannot, in general, edit terminal capabilities. For example, there is no way to make it so that some character other than ^G produces the bell on a given terminal, or to force the terminal to stop producing the bell. (Well, actually, you can turn off the speakers, but that doesn’t count, because the terminal will still try to ring the bell.)

However, what you can do is trick applications into thinking that terminal capabilities don’t exist, by feeding them misleading terminfo files. You’ve already seen that infocmp(1) will decompile terminfo. You can then easily edit this and recompile it using tic(1). For example, we can decompile the linux terminfo file, and then change the entry bel=^G to bel@, where the @ symbol is an explicit statement that the bel capability does not exist. We could then recompile this source to some file called linux-nb, say, for “no bell”. You could either put this in with the system terminfo files, if you have administrative privileges, or you could put it in the .terminfo subdirectory of your home directory, since ncurses always searches here first. You could then add a line to your .profile to tell bash that if TERM=linux then it should change the value to TERM=linux-nb. Conforming applications will then be tricked into thinking that there is no bell, and may produce a visible bell instead. (Nonconforming applications will just output ^G anyway, which is unfortunate.) You could trick all conforming applications into producing visible bells by making a file called linux-vb (this suffix is by convention) where the bel capability is set to the string that actually produces a visible bell.

By the way, the program setterm(1) can be used to configure some terminal capabilities. For example, you can use it to turn off the cursor with setterm -cursor off. Behind the scenes, of course, it consults terminfo to find the value of the civis capability. Note that setterm(1) writes a string to standard output indicating how to achieve the changes you requested, so you could, for example, redirect its output or pipe it through xxd(1) or something like that; but normally, standard output will be the terminal itself, which is why those changes are applied immediately. Curiously, setterm(1) knows how to change the length and frequency of the audible bell on a Linux virtual terminal, even though there is no terminfo capability specifying how to do this. Also, the manual page appears to be in error for failing to specify that those two features are not known to terminfo. (It appears that the escape sequences that do this are undocumented; indeed, setterm(1) is part of the util-linux package, so it can make whatever assumptions it wants about the Linux virtual terminals.)

---

So there you have it—more about terminal capabilities than you probably ever wanted to know. The most important things to take away from this post are:

1. Most terminal capabilities, such as coloured text and cursor movement, are activated by writing escape sequences to the terminal, starting with the escape character (ASCII 0x1B);
2. To find out which escape sequence performs a given action, you have to consult a terminfo file corresponding to the terminal you’re outputting to, whose name is usually given by the TERM environment variable, assuming your parent processes behaved as they were supposed to;
3. You can experiment with the capabilities of your terminal by decompiling the terminfo with infocmp(1) and then echoing strings to the terminal; remember that Ctrl+V Esc produces the escape character (unless you’ve messed around with the terminal attributes from part four);
4. A small number of terminal capabilities can be configured using setterm(1);
5. You can hide and falsify terminal capabilities if necessary by editing the decompiled terminfo and recompiling with tic(1);
6. This low-level overview of terminal capabilities is not so useful when you actually write programs; there, you will want to use the curses library instead of messing around with this stuff.

In part six, I will finally start talking about job control, which is one of the topics I most wanted to write about.

Advertisement