Good questions. The job ID is basically a shell. The kernel has support in the form of signals that are involved in job management, and how the kernel knows exactly which processes send job control signals.
Strictly speaking, the answer to your first question is that the job id is just a wrapper. It exists because the pipeline (or, more rarely, another grouped shell) can consist of several processes that must be managed as a whole.
To answer your last question, the shell starts all the processes, first by fork(2) and then by execve(2) . The only difference from & is that the shell does not execute wait(2) (or a related option), and therefore the program can continue "in the background." On Unix, there is virtually no difference between the foreground and background.
A process group is an association defined by shells, so the kernel is aware of a single “foreground” process that processes many different “background” processes. It is mainly important that background processes generate a signal if they decide to suddenly read from the terminal. (Such a terminal is probably connected to standard inputs.) This will cause the “task” to generate a signal, and the shell will prompt the user to do something.
Try (sleep 5; read x)& and after 6 seconds enter return or something else so that the shell wakes up. This is when you see something like ...
[1] + Stopped (sleep 5, read x)
... and then type fg to bring it to the front.
Unix originally had pipelines, and it had & , but there was no way to move a command or pipeline between the front and back backgrounds and not help the background process, which suddenly decided to read standard input.
Job management and kernel support for it were added by Bill Joy and others in earlier versions of BSD and csh (1). They were taken as a commercial Unix line and cloned for a working Linux kernel.
Regarding questions about process groups and ps(1) ... To support job management in shells, the state of the kernel process includes a process group identifier and a session identifier. The process group and the job are the same thing, but the job number is just a shell handle. A process is a session leader if the session identifier matches the pid identifier, and a process is the leader of a process group if pgid matches the pid identifier. I believe something a little more subtle happens to + , which prints ps(1) . Each terminal knows what a group of foreground processes is, so I think the process gets + if pid == pgid && (pgid is the foreground pg for the control terminal).
Thus, the kernel stores several status elements: pid, pgid, sid, and the process can have a control terminal, and the terminal can have a pgid front panel. These credentials are primarily intended to support job management, but are also used to deny access to the terminal when a user logs off.