ds.cs.ut.eeds.cs.ut.ee/courses/course-files/osworkshop.pdf · title: how to start writing a hobby...

How to start writing a hobby operating system

Manuel Hohmann

Laboratory of Theoretical Physics - Institute of Physics - University of TartuCenter of Excellence “The Dark Side of the Universe”

Univers

e European UnionEuropean Regional Development Fund

Investingin your future

Hardware programming and operating systems workshop22. April 2019

Manuel Hohmann (University of Tartu) OSDev for beginners 22. April 2019 1 / 26

Outline

1 Preliminary questions and general advice

2 Resources and required knowledge

3 Necessary and helpful tools

4 Some advice on getting started

Outline

Reasons not to write an OS

I want to write an operating system because. . .

. . . it will make me rich and / or famous like Bill Gates, Steve Jobsand Linus Torvalds, with millions of people using my OS.⇒ That is very, very unlikely. . .. . . I don’t like the way things look.⇒ The “look and feel” is one of the most marginal things thatcome with an OS. Try writing a UI for another OS instead.. . . I want to unify Windows, Linux, Mac OS and run all theirprograms on one computer at the same time.⇒ Getting an OS to run even programs for one other OS is amajor task, let alone several of them.. . . I want it to be more secure than any other OS written before.⇒ Writing anything secure is highly non-trivial and any subtle bugmay become a security risk. Consider joining e.g. Linux kerneldevelopment to fix security holes instead.

I want to write an operating system because. . .. . . it will make me rich and / or famous like Bill Gates, Steve Jobsand Linus Torvalds, with millions of people using my OS.⇒ That is very, very unlikely. . .

. . . I don’t like the way things look.⇒ The “look and feel” is one of the most marginal things thatcome with an OS. Try writing a UI for another OS instead.. . . I want to unify Windows, Linux, Mac OS and run all theirprograms on one computer at the same time.⇒ Getting an OS to run even programs for one other OS is amajor task, let alone several of them.. . . I want it to be more secure than any other OS written before.⇒ Writing anything secure is highly non-trivial and any subtle bugmay become a security risk. Consider joining e.g. Linux kerneldevelopment to fix security holes instead.

I want to write an operating system because. . .. . . it will make me rich and / or famous like Bill Gates, Steve Jobsand Linus Torvalds, with millions of people using my OS.⇒ That is very, very unlikely. . .. . . I don’t like the way things look.⇒ The “look and feel” is one of the most marginal things thatcome with an OS. Try writing a UI for another OS instead.

. . . I want to unify Windows, Linux, Mac OS and run all theirprograms on one computer at the same time.⇒ Getting an OS to run even programs for one other OS is amajor task, let alone several of them.. . . I want it to be more secure than any other OS written before.⇒ Writing anything secure is highly non-trivial and any subtle bugmay become a security risk. Consider joining e.g. Linux kerneldevelopment to fix security holes instead.

I want to write an operating system because. . .. . . it will make me rich and / or famous like Bill Gates, Steve Jobsand Linus Torvalds, with millions of people using my OS.⇒ That is very, very unlikely. . .. . . I don’t like the way things look.⇒ The “look and feel” is one of the most marginal things thatcome with an OS. Try writing a UI for another OS instead.. . . I want to unify Windows, Linux, Mac OS and run all theirprograms on one computer at the same time.⇒ Getting an OS to run even programs for one other OS is amajor task, let alone several of them.

. . . I want it to be more secure than any other OS written before.⇒ Writing anything secure is highly non-trivial and any subtle bugmay become a security risk. Consider joining e.g. Linux kerneldevelopment to fix security holes instead.

I want to write an operating system because. . .. . . it will make me rich and / or famous like Bill Gates, Steve Jobsand Linus Torvalds, with millions of people using my OS.⇒ That is very, very unlikely. . .. . . I don’t like the way things look.⇒ The “look and feel” is one of the most marginal things thatcome with an OS. Try writing a UI for another OS instead.. . . I want to unify Windows, Linux, Mac OS and run all theirprograms on one computer at the same time.⇒ Getting an OS to run even programs for one other OS is amajor task, let alone several of them.. . . I want it to be more secure than any other OS written before.⇒ Writing anything secure is highly non-trivial and any subtle bugmay become a security risk. Consider joining e.g. Linux kerneldevelopment to fix security holes instead.

Reasons to write an OS

I want to write an operating system because. . .

. . . my teacher / professor wants me to.

. . . I want to learn and understand how computers and operatingsystems work.. . . it is fun.

⇒ Decide what you want to achieve, and try to set a realistic goal whatyour OS should be able to do.

I want to write an operating system because. . .. . . my teacher / professor wants me to.

. . . I want to learn and understand how computers and operatingsystems work.. . . it is fun.

I want to write an operating system because. . .. . . my teacher / professor wants me to.. . . I want to learn and understand how computers and operatingsystems work.

. . . it is fun.⇒ Decide what you want to achieve, and try to set a realistic goal whatyour OS should be able to do.

I want to write an operating system because. . .. . . my teacher / professor wants me to.. . . I want to learn and understand how computers and operatingsystems work.. . . it is fun.

Design considerations

General design considerations:Target a single architecture / platforms or several of them?Which kernel model - monolithic, microkernel?Which task model - monotasking, multitasking, real time?Single processing or multiprocessing?

Choosing how to implement things:Memory management and protectionScheduling of processes / threads (if multitasking).Synchronization and inter-process communication.Supported executable formats, file systems. . .

Hardware specific design choices:Interface between kernel and user programs (system calls).Possible workarounds for CPU bugs.Security measures.Supported hardware.

Which programming language to use

C+ Easy to write low level code.+ Almost no support code / runtime environment required.+ Good optimizing C compilers available (fast code).- Not much language support for structured programming.

C+++ Most of the advantages of C.+ Language supports structured programming (object oriented etc.).- Requires slightly more support code.- May obscure things and accidentally create bloat if used wrongly.- Some features must be disabled or implemented by hand.- Steep learning curve (especially newer standards).

Other languages (Rust, Haskell, D, C#, Objective C, Pascal. . . )+ May allow for better structure in high level parts.- May require more runtime support for low level code.

Assembler+ Full control over CPU, optimization. . .- Tedious to write everything by hand.- Code is not portable to other architectures.

Outline

Andrew S. Tanenbaum, “Modern Operating Systems”.

Andrew S. Tanenbaum, “Operating Systems: Design andImplementation”.Mark Russinovich, David A. Solomon, and Alex Ionescu,“Windows Internals”.Scott Andrew Maxwell, “Linux Core Kernel Commentary”.. . . and many other books.

Andrew S. Tanenbaum, “Modern Operating Systems”.Andrew S. Tanenbaum, “Operating Systems: Design andImplementation”.

Mark Russinovich, David A. Solomon, and Alex Ionescu,“Windows Internals”.Scott Andrew Maxwell, “Linux Core Kernel Commentary”.. . . and many other books.

Andrew S. Tanenbaum, “Modern Operating Systems”.Andrew S. Tanenbaum, “Operating Systems: Design andImplementation”.Mark Russinovich, David A. Solomon, and Alex Ionescu,“Windows Internals”.

Scott Andrew Maxwell, “Linux Core Kernel Commentary”.. . . and many other books.

Andrew S. Tanenbaum, “Modern Operating Systems”.Andrew S. Tanenbaum, “Operating Systems: Design andImplementation”.Mark Russinovich, David A. Solomon, and Alex Ionescu,“Windows Internals”.Scott Andrew Maxwell, “Linux Core Kernel Commentary”.

. . . and many other books.

Andrew S. Tanenbaum, “Modern Operating Systems”.Andrew S. Tanenbaum, “Operating Systems: Design andImplementation”.Mark Russinovich, David A. Solomon, and Alex Ionescu,“Windows Internals”.Scott Andrew Maxwell, “Linux Core Kernel Commentary”.. . . and many other books.

Hardware manuals and standards

CPU architecture manuals:x86:

Intel 64 and IA-32 Architectures Software Developer ManualsAMD64 Architecture Programmer’s Manual

ARM (AArch32, AArch64): ARM Information CenterMIPS32 & MIPS64 Instruction Set Architecture manuals

Platform documentation:Buses: ISA, PCI, USB. . .Storage devices.Human interface devices.Video graphics adapters.Network cards.

Common standards:Low level hardware interfaces: ACPI, UEFI. . .File systems.Executable formats.

Tutorials

Bare BonesHow to write a simple “Hello world” on bare metal, without any OSor runtime support.

Meaty SkeletonHow to organize and structure your project, create a sourcetree. . .Common kernel tutorials for x86 architecture (be aware of bugs!):

James A. Molloy’s tutorial (bugs)BrokenThorn tutorial (bugs)Bran’s kernel development tutorial (bugs)

Many, many other tutorials. . .

Tutorials

Bare BonesHow to write a simple “Hello world” on bare metal, without any OSor runtime support.Meaty SkeletonHow to organize and structure your project, create a sourcetree. . .

Common kernel tutorials for x86 architecture (be aware of bugs!):James A. Molloy’s tutorial (bugs)BrokenThorn tutorial (bugs)Bran’s kernel development tutorial (bugs)

Tutorials

Bare BonesHow to write a simple “Hello world” on bare metal, without any OSor runtime support.Meaty SkeletonHow to organize and structure your project, create a sourcetree. . .Common kernel tutorials for x86 architecture (be aware of bugs!):

Tutorials

Bare BonesHow to write a simple “Hello world” on bare metal, without any OSor runtime support.Meaty SkeletonHow to organize and structure your project, create a sourcetree. . .Common kernel tutorials for x86 architecture (be aware of bugs!):

Operating system internals

Take a look at books on operating system internals, developerdocumentation and open source operating systems, such as. . .

Teaching operating systems:xv6Minix

Major operating systems:LinuxFreeBSD

Interesting alternative operating systems / kernels:ReactOSseL4HaikuFuchsia / Zircon

Successful hobby operating systems:ToaruOSSortix

Community resources

OSDev.org operating system developer community:

ForumWikiIRC channel #osdev on freenode

Outline

Compilers

GCC+ Comes with compilers for C, C++, Ada, Fortran.+ Can target many different architectures.+ Actively developed, supports up to date standards.+ Widely supported standard among OSDev community.- Must be compiled for every architecture separately.

Clang+ Targets many different architectures by command line parameter.+ Actively developed, supports up to date standard.- Less widely used.

Microsoft Visual C+++ Comes with development environment.- Designed to produce Windows executables and libraries.- Requires some tweaking to be used for OSDev.

Compilers

Cross compilation

Why do I need a cross compiler?You might want to compile code for a different architecture (e.g.,compile ARM code on a x86 computer).Compilers distributed with your operating system (e.g., Linux) aredesigned / configured to produce programs running under that OS,and getting them to target bare metal is at least tricky.Setting up a cross compiler gives you a standard developmentenvironment that is used by other people, who can give advice.

How do I get a cross compiler?Clang: Already works as a cross compiler.GCC: Build from source and configure for bare metal target.

Cross compilation

Why do I need a cross compiler?You might want to compile code for a different architecture (e.g.,compile ARM code on a x86 computer).Compilers distributed with your operating system (e.g., Linux) aredesigned / configured to produce programs running under that OS,and getting them to target bare metal is at least tricky.Setting up a cross compiler gives you a standard developmentenvironment that is used by other people, who can give advice.

How do I get a cross compiler?Clang: Already works as a cross compiler.GCC: Build from source and configure for bare metal target.

Assemblers

GNU Assembler (GAS):+ Targets many different architectures.+ Widely used standard tool.+ Well integrated with GCC and GNU toolchain.- Claimed to be harder to use because of AT&T syntax on x86.

Netwide Assembler (NASM):+ Claimed to be easier to use because of Intel syntax on x86.+ Well developed macro capabilities.- Targets only x86 architectures.- Less well integrated into common toolchains.

Flat Assembler (FASM):+ New version supports different target architectures.+ Rather easy to use.- Less well integrated into common toolchains.

Assemblers

Linkers

GNU Linker (LD):+ Well integrated with GCC and GNU toolchain.+ Supports many input and output formats.+ Highly configurable output via scripting language.

gold:+ Faster than LD.- Supports only ELF format.

LLD:+ Well integrated with LLVM toolchain.+ Faster than GNU linkers.

Linkers

Build systems

GNU Make+ Well supported standard tool.+ Known to work “out of the box” for OS development.- Can become messy for large projects, multiple architectures. . .

Autotools (Automake & Autoconf)+ Creates “configure” script + makefiles for configurable building.+ Avoids code duplication.- Sometimes considered problematic for bare metal targets.

CMake:+ Popular alternative to Autotools.- Designed for applications, not for kernel development.

Build systems

Emulators and virtual machines

Bochs:+ Provides rather accurate emulation.+ Highly configurable virtual hardware.+ Highly configurable and verbose log output.+ Integrated debugger (both CLI and GUI).- Supports only x86 platforms.

QEMU:+ Emulates many CPU architectures and machines.+ Highly configurable virtual hardware.+ Integrated monitor allows inspecting machine state.- Some targets are not yet fully implemented.

Virtual machines (VirtualBox, VMware, Virtual PC. . . ):+ Provide additional virtual hardware to test your kernel.- Less features to debug or inspect virtual machine state.- Support only x86 platforms.

Debuggers

Bochs integrated debugger:+ Very powerful debugger with many features.+ Allows inspecting and manipulating full machine state.+ Scriptable for automated debugging / generating reports.- Supports only x86 architectures.

GNU Debugger (GDB):+ Very powerful debugger with many features.+ Supports many different target architectures.+ Can connect to a “GDB stub”, which is integrated in. . .

BochsQEMUyour kernel (if you implement it, for remote debugging).

- Steep learning curve.- Cannot access full machine state on all architectures.

VirtualBox debugger:+ Comes integrated with VirtualBox.- Very limited features.

Debuggers

Other tools you might need / want

Tools for working with disk images:Emulators use disk images as virtual storage media.Images can be copied on physical storage media.Images provide a convenient way to distribute your OS binary.

Boot loader:Loads the kernel (and other files) from disk to memory.Sets up a basic execution environment.Allows installing / booting more than one OS.

Version control systems:You might accidentally delete / overwrite some code.It might turn out that a change you did made things worse.You might want to fork off branches to work on specific features.You might want to have a look at earlier versions and compare.You might want to maintain a changelog and version history.

Continuous integration:You might want to test your code in several configurations.Use automatized builds and tests on push to repository.

Outline

How to actually get started

Make sure you have gained the required knowledge.

Choose your design - kernel model, target platform(s). . .Choose a programming language for your implementation.Get the tools you need for your platform(s) and language.Make sure your tools work, compile some “Hello world” example.Keep the hardware manuals ready at hand.Make a structured plan of your implementation.Create a structured directory tree for your source code.Start coding!

Make sure you have gained the required knowledge.Choose your design - kernel model, target platform(s). . .

Choose a programming language for your implementation.Get the tools you need for your platform(s) and language.Make sure your tools work, compile some “Hello world” example.Keep the hardware manuals ready at hand.Make a structured plan of your implementation.Create a structured directory tree for your source code.Start coding!

Make sure you have gained the required knowledge.Choose your design - kernel model, target platform(s). . .Choose a programming language for your implementation.

Get the tools you need for your platform(s) and language.Make sure your tools work, compile some “Hello world” example.Keep the hardware manuals ready at hand.Make a structured plan of your implementation.Create a structured directory tree for your source code.Start coding!

Make sure you have gained the required knowledge.Choose your design - kernel model, target platform(s). . .Choose a programming language for your implementation.Get the tools you need for your platform(s) and language.

Make sure your tools work, compile some “Hello world” example.Keep the hardware manuals ready at hand.Make a structured plan of your implementation.Create a structured directory tree for your source code.Start coding!

Make sure you have gained the required knowledge.Choose your design - kernel model, target platform(s). . .Choose a programming language for your implementation.Get the tools you need for your platform(s) and language.Make sure your tools work, compile some “Hello world” example.

Keep the hardware manuals ready at hand.Make a structured plan of your implementation.Create a structured directory tree for your source code.Start coding!

Make sure you have gained the required knowledge.Choose your design - kernel model, target platform(s). . .Choose a programming language for your implementation.Get the tools you need for your platform(s) and language.Make sure your tools work, compile some “Hello world” example.Keep the hardware manuals ready at hand.

Make a structured plan of your implementation.Create a structured directory tree for your source code.Start coding!

Make sure you have gained the required knowledge.Choose your design - kernel model, target platform(s). . .Choose a programming language for your implementation.Get the tools you need for your platform(s) and language.Make sure your tools work, compile some “Hello world” example.Keep the hardware manuals ready at hand.Make a structured plan of your implementation.

Create a structured directory tree for your source code.Start coding!

Make sure you have gained the required knowledge.Choose your design - kernel model, target platform(s). . .Choose a programming language for your implementation.Get the tools you need for your platform(s) and language.Make sure your tools work, compile some “Hello world” example.Keep the hardware manuals ready at hand.Make a structured plan of your implementation.Create a structured directory tree for your source code.

Start coding!

Make sure you have gained the required knowledge.Choose your design - kernel model, target platform(s). . .Choose a programming language for your implementation.Get the tools you need for your platform(s) and language.Make sure your tools work, compile some “Hello world” example.Keep the hardware manuals ready at hand.Make a structured plan of your implementation.Create a structured directory tree for your source code.Start coding!

Typical ingredients for an OS kernel

Linker script:Determines where your kernel is in memory.Defines sections and their properties (read/write, executable. . . ).

Assembler stubs:Need to set up environment for high level language (stack. . . ).Entry points for interrupts, syscalls, CPU mode switches.CPU specific system instructions1.

High level language (C, C++, . . . ) sources:Probably the largest part of your code.Possible to port parts of it to other architectures.Reduces code duplication.Easier to maintain than assembly.

Makefiles / build script:Compilation instructions for your OS.Determine in which order to build, dependencies. . .

1One may also use inline assembly code here.

1One may also use inline assembly code here.Manuel Hohmann (University of Tartu) OSDev for beginners 22. April 2019 25 / 26

Which things should I implement next?

Printing strings and integers (decimal and hexadecimal):Essential for any debug output. You know a lot more about yourkernel’s state and what it does if it can tell you that.

Interrupt / exception handling:Your kernel will have bugs that trigger CPU faults / exceptions.Dumping the machine state and possible even cause of theexception is crucial in debugging.Memory management:Almost anything you do will require memory management.Memory management tasks (in the kernel) can usually be split inthree categories:

Physical memory management - which pages are free, which areused? Finding free pages, allocating & freeing pages. . .Virtual memory management - how to map physical pages intovirtual memory, managing paging structures, address spaces. . .Kernel heap - memory used by the kernel for dynamic allocation /deallocation (malloc(), free() or new, delete).

Printing strings and integers (decimal and hexadecimal):Essential for any debug output. You know a lot more about yourkernel’s state and what it does if it can tell you that.Interrupt / exception handling:Your kernel will have bugs that trigger CPU faults / exceptions.Dumping the machine state and possible even cause of theexception is crucial in debugging.

Memory management:Almost anything you do will require memory management.Memory management tasks (in the kernel) can usually be split inthree categories:

Printing strings and integers (decimal and hexadecimal):Essential for any debug output. You know a lot more about yourkernel’s state and what it does if it can tell you that.Interrupt / exception handling:Your kernel will have bugs that trigger CPU faults / exceptions.Dumping the machine state and possible even cause of theexception is crucial in debugging.Memory management:Almost anything you do will require memory management.Memory management tasks (in the kernel) can usually be split inthree categories:

ds.cs.ut.eeds.cs.ut.ee/courses/course-files/osworkshop.pdf · title: how to start writing a hobby...

Documents