seegongsik
Operating systems

What is a system call

At the end of the last lesson we spotted a curious gap. The operating system sets permissions on every file, yet my program cannot pull even a file it may read straight off the disk. In fact a program cannot freely touch the disk, the screen, or the network at all; it is shut inside a sturdy fence. So how does it ever open a file or put letters on the screen?

01

A Program Lives Inside a Fence (User Mode)

Earlier the operating system put up walls so programs could not trespass on each other's memory. The same kind of wall stands in front of the hardware. An ordinary program cannot reach into the disk, cannot light up a dot on the screen, and cannot handle the network device directly.
This fenced-in state is called user mode.
It sounds cramped, but there is a reason. If any program could handle the hardware directly, one buggy or ill-intentioned program could bring down the whole computer.

program
disk

Press the button and the program reaches straight for the disk. The wall in the middle turns red and stops it: a program in user mode cannot touch the hardware directly.

Every direct reach gets stopped, which is exactly what safety calls for.
But that seems odd. The programs we use every day open files, show photos, and go online without any trouble. If they are fenced in, how do they manage all that?
The secret is that they do not do it themselves. They ask.

02

The Way Out: Ask at an Official Counter (a System Call)

At a bank we do not walk into the vault ourselves. We hand a request slip across the counter, and a clerk with the keys takes care of it for us.
A program works the same way. Since it cannot open the disk itself, it goes to the operating system's counter and asks: "Open this file." "Put these letters on the screen." "Send this over the internet." Each request handed to the operating system like this is called a system call.
The key point is that the program does not do the work; it has the operating system do it, because only the operating system has the right to touch the hardware.

program
→
OS counter
→
disk

Press the button and the program (gold) asks the operating system's counter (blue) to open a file; the operating system does the job at the disk and hands back the result (green). The program never touches the disk itself.

Asking at the counter turned an impossible job into a possible one. The program is still inside the fence, yet it got outside work done through an agent, the operating system.
One thing is puzzling, though. It is the same disk, so why is the program stopped when it touches it while the operating system is not?
Something must change in that brief moment of crossing the counter.

03

Crossing the Counter Switches the Privilege (User Mode ↔ Kernel Mode)

The secret is that the CPU has two modes. One is user mode, the limited world our programs live in. The other is kernel mode, the operating system's world, where the hardware and everything else can be touched.
A program normally runs in user mode. The instant a system call crosses the counter, the CPU switches briefly into kernel mode. With that strong privilege the operating system does the requested job, then switches straight back to user mode and hands the result to the program.
The strong power is given only for the moment of handling the request, and only through this counter, and then it is taken away again.

user mode

Press the button once and the system call crosses the counter; in kernel mode (blue) the operating system does the work. Press it again and you are back in user mode (gold) with the result in hand.

This short round trip, out of one mode and back again, is the heart of the operating system's safety. The strong power is not left on; it flicks on only when a request comes in and then goes off.
One question remains. If crossing the counter meant the operating system did whatever it was told, how would that be any different from the program touching the hardware itself?
So the counter has one more job: checking.

04

One Counter Means Everything Can Be Checked (the Permission Check)

If the operating system did everything the moment it was asked, no questions, the counter would be nothing more than a passage. But the operating system does not work that way.
Because every request has to go through this one counter, the operating system can check before doing any work: "Does this program have permission to read this file?" The permission check from the last lesson happens right here.
If the permission is there, the job gets done; if not, the request is refused and an error comes back. One narrow counter becomes the checkpoint that every risky action must pass.

Pick one of the three requests and send it to the counter. The operating system checks permission: reading my file and printing to the screen go through (green), while writing a friend's file is refused with an error (red).

The allowed requests went through and the forbidden one was stopped at the door. The narrow counter turned out to be a safeguard rather than a nuisance.
Since every risky action has to pass one spot, the rules only have to be enforced at that spot.
Now we can string it all together: the fence of user mode, the request that is a system call, the brief switch to kernel mode, and the check at the counter.

05

In Summary: the Round Trip on One Line, and the Kernel Behind the Counter

On one line, it goes like this. A program runs inside the fence of user mode, unable to touch the hardware. When it needs something, it asks at the official counter with a system call.
The moment it crosses, the CPU switches briefly into kernel mode, and the operating system checks permission first and then does the work. When it is finished, it switches back to user mode and hands over the result.
So a system call is the one well-guarded door between a program and the operating system's power. Opening a file, putting a single dot on the screen, sending something over the internet: all of it passes quietly through this door.

Press the five steps in order and the round trip lights up: program, system call, kernel mode with the permission check, doing the work, returning the result. When all five are lit, the loop is complete.

So who, behind that counter, actually does the work with the power of kernel mode? The deepest center of the operating system, the part that holds every privilege and talks to the hardware directly, has a name: the kernel. A system call was a request sent to this kernel all along.
In the next lesson we finally meet the kernel face to face, and see in one place whose hands have been running the scheduling, memory, files, permissions, and counters we have looked at so far.

In one lineA program is kept inside a fence (user mode) so that it cannot touch the hardware directly, and to get work done it must ask the operating system. That request is a system call: handing something like "open this file" or "show this on screen" to an official counter. The moment the request crosses the counter, the CPU briefly switches to strong privilege (kernel mode), the operating system does the work, and then the CPU switches straight back. Because there is only one counter, the operating system can check every request first (this is where the permission check happens), and so the fence stays firm while programs still get done what they need.
← Operating systems
Was this helpful? Support seegongsik