Out with the ugly hacks

Considering the present time (0:14, at UTC+1) I'm going to keep this one short.

I've been working on getting scheduling working, and I found the GDT trick to be a little bit of a nuisance. As it turns out, I had a design in mind as a replacement for this trick.

What does it do?

Basically what we want is to have the kernel start at somewhere very high in memory, both tricks allow us to migrate the kernel code without endangering our more volatile code.

The GDT trick requires us to exploit some feature in the CPU which isn't guaranteed (as far as I could find). Now this doesn't sound very nice, and isn't really portable too.

Now what I did was remove this trick and enabled paging very early on. This means that paging will actually replace the function of the GDT trick. Now this was kinda tricky since booting with the GDT trick might require one to do some very nasty stuff later on.

This code still exists, so I had to enable that in the paging trick. Basically what we have now works.

More info on this can be expected later.

Still waiting

So we didn't quite reach our deadline for November 4th.

I think it has been a little too ambitious, considering the situations all developers are in now.

So we're definitely working towards the 0.1.1 release, and among the things we already mentioned would be in there, we'll also feature file buffer support.

Now that doesn't say a lot though. Basically all it says is that we have something that resembles a file, which you can write to, and read from. Seeks are also supported, but unfortunately, we can't sync with disk, or even clean up read items in a one way stream/pipe ...

Michel still is working on his PCI and APIC support and is thinking of including C++ code into the project (I wish him luck implementing the special operators, I'll continue doing C).

Doubted is whether or not we will be able to put PCI and APIC into the next release. There'll be a small discussion on that and we'll get back to you when we've got more news about the next release.

Until that time, the developers will be working on both school and the kernel.

Also, I've managed to set up gitweb on my own server, so if you like you can subscribe to the rss/atom feeds on there: http://orion-os.eu/gitweb/

I hope to have more news soon.

A new release already?

A new release is coming up on the horizon. We now have the date of Friday 4th of November 2011 planned for the release of Andromeda-0.1.1.

This will feature a brand new paging system, a new memory mapping system, minimal ACPI support and basic support for the Advanced Programmable Interrupt Controllers (APIC).

The paging system has been made in such a fashion that the memory requirement scales to the pages requested, instead of permanently occupying 4 Megs of RAM.

The same goes for the new memory map, and did I already mention it is based on the size of the actual physical memory!

This means that we always occupy 0.2% of physical memory available for the memory map, and depending of the amount of virtual memory usage a little bit more for the page tables.

Also we've conjured up a way to put the kernel at 3 GiB. It's called the overflow trick. The principle behind it is pretty simple. All you do is set up a segment which maps memory address 0  to the 3GiB location and make it 4 GiB long or so. Then you jump into the higher half code.

Simple as that. All you need to do is page the code to both low and high memory, disable the overflow trick and unmap the low memory code, leaving only high memory in place.

Whenever some low address is required, all you need to do is add 1 GiB to the address and the 32-bits integer flows over when the segment is applied, and the correct pointer is used.

Simple, stable and effective (do watch out with virtual machines that don't properly emulate the 32-bits overflow, they can be a real pain sometimes).

The ACPI support is created by Michel Megens, so not much for me to mention on that part. The same pretty much goes for APIC support.

And speaking of dates! My birthday was last Sunday (October 30th) at which I turned 19!

Continuing to move

So I found this bug I was telling you about last blog post. It basically boils down to me having forgotten to map the entire lowest 15 MiB.

That means page faults, which cause a double fault (because the page fault isn't mapped), which causes a triple fault (reboot, because the double fault isn't mapped).

That's nice, I think, but now I have that settled, I need to remove it again after I disabled segmentation, otherwise we've got a security leak on our hands, greater than the great wall of china.

In the mean time I am trying to build a way to keep track of the memory that is mapped, and the memory that isn't. At the moment I've got a system in place that produces 0.29296875% overhead on the entire memory system, at MOST versus a huge 6.6666....% using the old system (and yes I used a calculator for that! I'm not that smart).

This is without counting the heap overhead, which go through the roof on very small allocations, but because that's too unpredictable I've chosen to only count the paging system in.

What needs to be done now?

First of all, the page fault system is to be written. This will in the future include swapping, but since file systems aren't supported yet, a swap file would be pretty useless. So if we run out of memory now, we just panic!

Second the lowest 15 MiB must be released again. Once that's done, we can start doing work on getting all the features we broke by this, to get to work again.

I see a mammoth task in the future, which means I'm happy it's nearly autumn brake.

In the mean time Michel is working on the ACPI tables and stuff like that. I hope he'll get to explain it to me in the near future. I sure wish to know this.

Also I'm getting an Arduino board. Just for playing around. Maybe I'll attempt to port Andromeda, if the chip is even capable of the tricks we require.

And there is a new server coming up. It's made with absolutely NO moving parts so that I can keep it running longer.

Up, up and, well, you know the story

Basically what I've been working on is getting back to the higher half mode.

The trick I've chosen this time is the GDT trick, designed by Tim Robinson.

What this means is that the image is linked to be at 3GiB and then we use paging to trick the CPU into thinking the code is where it should be.

This is nice and all, but it does make things with physical memory addresses a little harder, and since that's the case I still haven't been able to get paging to work.

The weirdest thing is that the debugging values seem to be correct, while the virtual machine still triple-faults (for non OS-developers, that's basically the PC throwing it's hands up in the air, saying: "I can't fix this, do it your self!").

In the mean time Michel has been continuing work on the ACPI tables.

Besides that there is lack of a clearly defined website to go to for the kernel.

We're working on that, but do keep in mind that we have higher priorities at the moment (such as school).

A new release!

The 0.1.0 release is here! Time to celebrate!

But exactly what are we celebrating?

Well, in the 0.1.0 release we've successfully merged Andromeda and OpenLoader under the name of Andromeda.

If you were to look at the code, there still is a lot with of code with ol_ prefixes. This is legacy, and although re-factoring is a possibility, we're not actively going to search for this.

Also I want to give credit for at least half of the functions implemented in the 0.1.0 release to our new co-project-leader: Michel Megens.

Below is a list of what has changed since the 0.0.4 release in what we call Andromeda.

• Better heap allocator
• Heaps of bugs have been fixed
• New paging system
• A complete rewrite
• A new page frame allocation scheme
• A page mapping scheme
• Work is to be done on integrating the frame allocation and mapping
• A start on a virtual file system
• Files are implemented minimally
• Directories have been implemented
• File references need to be created
• Mount points aren't done yet
• A possibility for very very low resolution graphics
• VGA at 320*240 like resolutions
• Work is being done on 800*600
• Better keyboard support
• Capitals and backspace working
• PS2 controller now supported!
• A start on disk drivers
• ATA support now has a stub
• Error code centralisation
• Error codes can now be found in a single file.
• The removal of floppy.img
• Make test works again.
• floppy.img no longer required to test
• Repository is cleaner
• A start on the ACPI
• Added ACPI data structures
• Implemented ACPI table search schemes
• A start on the APIC
• Detection implemented but we're still using the legacy chip.
• A better CPU interface
• The CPUID instruction now has a better implementation.
• PCI support added
• Multi processor support now has a stub.

On the move

The project is going forward steadily and while we're still merging, some things are already moving forward again.

One of them is the paging subsystem. Well, actually it's gone backwards but that's mainly because it's a rewrite (can't clean up without making a mess first).

One thing I'm working on is getting the pages to be put in the page tables to be put in a page directory (yes, Intel couldn't make it simpler).

That's not really my issue here, but what is my issue is that I have to find out a way to get the pages to be marked free and used, without requiring 4MiB as I used to.

I have got some designs ready, one of which is the old design, which was underwhelming to say the least. Another is a simple bitmap, with the processes maintaining their own page locations. The last one is to have a list of regions, which is a doubly linked list so the processes can also mark the pages as used for them selves. This does however put me at a 8MiB max. The sunny side, however is that there also is an 8 byte minimum, and that with a bit of work, the fragmentation could be resolved, to make the entire requirements just a bit smaller.

Now this 8 MiB max is only on 32-bits systems, as on 64-bits this can be doubled. The bitmap approach has got several advantages, one of which is that it's small and easy to maintain. That is until we're going to think about swapping pages out to disk. At that moment, all hell breaks loose, and code is flying around randomly.

So my choice is this: "Am I going for low memory usage, easy swapping or easy maintenance?" You'll hear the answer soon (hopefully).