Continuing to move

So I found this bug I was telling you about last blog post. It basically boils down to me having forgotten to map the entire lowest 15 MiB.

That means page faults, which cause a double fault (because the page fault isn't mapped), which causes a triple fault (reboot, because the double fault isn't mapped).

That's nice, I think, but now I have that settled, I need to remove it again after I disabled segmentation, otherwise we've got a security leak on our hands, greater than the great wall of china.

In the mean time I am trying to build a way to keep track of the memory that is mapped, and the memory that isn't. At the moment I've got a system in place that produces 0.29296875% overhead on the entire memory system, at MOST versus a huge 6.6666....% using the old system (and yes I used a calculator for that! I'm not that smart).

This is without counting the heap overhead, which go through the roof on very small allocations, but because that's too unpredictable I've chosen to only count the paging system in.

What needs to be done now?

First of all, the page fault system is to be written. This will in the future include swapping, but since file systems aren't supported yet, a swap file would be pretty useless. So if we run out of memory now, we just panic!

Second the lowest 15 MiB must be released again. Once that's done, we can start doing work on getting all the features we broke by this, to get to work again.

I see a mammoth task in the future, which means I'm happy it's nearly autumn brake.

In the mean time Michel is working on the ACPI tables and stuff like that. I hope he'll get to explain it to me in the near future. I sure wish to know this.

Also I'm getting an Arduino board. Just for playing around. Maybe I'll attempt to port Andromeda, if the chip is even capable of the tricks we require.

And there is a new server coming up. It's made with absolutely NO moving parts so that I can keep it running longer.

Up, up and, well, you know the story

Basically what I've been working on is getting back to the higher half mode.

The trick I've chosen this time is the GDT trick, designed by Tim Robinson.

What this means is that the image is linked to be at 3GiB and then we use paging to trick the CPU into thinking the code is where it should be.

This is nice and all, but it does make things with physical memory addresses a little harder, and since that's the case I still haven't been able to get paging to work.

The weirdest thing is that the debugging values seem to be correct, while the virtual machine still triple-faults (for non OS-developers, that's basically the PC throwing it's hands up in the air, saying: "I can't fix this, do it your self!").

In the mean time Michel has been continuing work on the ACPI tables.

Besides that there is lack of a clearly defined website to go to for the kernel.

We're working on that, but do keep in mind that we have higher priorities at the moment (such as school).

A new release!

The 0.1.0 release is here! Time to celebrate!

But exactly what are we celebrating?

Well, in the 0.1.0 release we've successfully merged Andromeda and OpenLoader under the name of Andromeda.

If you were to look at the code, there still is a lot with of code with ol_ prefixes. This is legacy, and although re-factoring is a possibility, we're not actively going to search for this.

Also I want to give credit for at least half of the functions implemented in the 0.1.0 release to our new co-project-leader: Michel Megens.

Below is a list of what has changed since the 0.0.4 release in what we call Andromeda.

• Better heap allocator
• Heaps of bugs have been fixed
• New paging system
• A complete rewrite
• A new page frame allocation scheme
• A page mapping scheme
• Work is to be done on integrating the frame allocation and mapping
• A start on a virtual file system
• Files are implemented minimally
• Directories have been implemented
• File references need to be created
• Mount points aren't done yet
• A possibility for very very low resolution graphics
• VGA at 320*240 like resolutions
• Work is being done on 800*600
• Better keyboard support
• Capitals and backspace working
• PS2 controller now supported!
• A start on disk drivers
• ATA support now has a stub
• Error code centralisation
• Error codes can now be found in a single file.
• The removal of floppy.img
• Make test works again.
• floppy.img no longer required to test
• Repository is cleaner
• A start on the ACPI
• Added ACPI data structures
• Implemented ACPI table search schemes
• A start on the APIC
• Detection implemented but we're still using the legacy chip.
• A better CPU interface
• The CPUID instruction now has a better implementation.
• PCI support added
• Multi processor support now has a stub.

On the move

The project is going forward steadily and while we're still merging, some things are already moving forward again.

One of them is the paging subsystem. Well, actually it's gone backwards but that's mainly because it's a rewrite (can't clean up without making a mess first).

One thing I'm working on is getting the pages to be put in the page tables to be put in a page directory (yes, Intel couldn't make it simpler).

That's not really my issue here, but what is my issue is that I have to find out a way to get the pages to be marked free and used, without requiring 4MiB as I used to.

I have got some designs ready, one of which is the old design, which was underwhelming to say the least. Another is a simple bitmap, with the processes maintaining their own page locations. The last one is to have a list of regions, which is a doubly linked list so the processes can also mark the pages as used for them selves. This does however put me at a 8MiB max. The sunny side, however is that there also is an 8 byte minimum, and that with a bit of work, the fragmentation could be resolved, to make the entire requirements just a bit smaller.

Now this 8 MiB max is only on 32-bits systems, as on 64-bits this can be doubled. The bitmap approach has got several advantages, one of which is that it's small and easy to maintain. That is until we're going to think about swapping pages out to disk. At that moment, all hell breaks loose, and code is flying around randomly.

So my choice is this: "Am I going for low memory usage, easy swapping or easy maintenance?" You'll hear the answer soon (hopefully).

The dust is settling

While the dust of the cosmic collision starts to settle we're still a long way away from the total merge of the projects.

At the moment it's more the drivers that are being implemented in the Andromeda kernel. Michel is the one responsible for that.

In the mean time Steven feels like he can't do a thing since the change is going so rapidly.

And me, what am I doing?

Well, I'm primarily focussing on the same stuff as before the merge and that is getting the paging system up to par. At the moment work is being done on the basics of the paging system and while I feel it isn't sufficient in the long run (when this might be ported to 64 bits systems) for now it'll do and will probably even be good (according to the standards we're working with now).

So what is going to change?

To start with work is being done on drivers such as Intel's Advanced Programmable Interrupt Controller or APIC. In the mean time work also is being done on ACPI or Advanced Configuration and Power Interface (which is a disaster to work with). That is what is imported from Openloader.

In the mean time we're also taking over the Openloader boot procedure and this is also a work in progress.

Also we're doing work on getting the kernel into higher half, which is where the new paging system comes in. It has to support identity paging which is that the same page can be referenced by multiple virtual addresses. This, in combination with relocatable code and a whole lot of rather simple arithmetic on relocating the heap, will allow us to relocate the entire kernel into the 3GiB region or the -2 GiB region on AMD 64 systems.

Work also is being done on the graphics systems, but that is basically what has been going on all the time since Steven joined, and we're looking to merge the makefiles.

The heap code had some significant bugs, which are now resolved, but no guarantees can be given that it is now bug free.

So while the dust is settling, the merging is far from complete and the legacy of the two projects will probably remain visible for months, if not years to come.

Cosmic collision

This blog post marks quite an important moment in what now used to be Andromeda as we know it, since we're doing something bonkers.

Basically what we're doing is calling the boot loader by Michel Megens the Milky way and sped up time.

If you're not aware of what's going to happen in a couple of million years, well, here's a quick explanation: "Boom!"

If that doesn't cut it for you, well, Andromeda and the Milky way are both galaxies (not sweets) and Andromeda is the only galaxy moving towards the Milky way (aka us).

This means the two will collide in a couple of million years and the two galaxies will likely merge.

That's exactly what Openloader and Andromeda (the kernel) are doing. We're fusing.

Since the end result will still be a kernel the name of the kernel will persist, but the project structure will likely change.

In galactic terms this means the two black holes at the centre of the galaxies will revolve around each other but not merge. In project terms, this will mean we have two project leaders.

Michel being involved with the direct hardware interfacing, while Bart will be focussing on the higher level algorithms such as scheduling, paging and other kinds of memory management.

The merging of two galaxies is a lengthy process and often messy. I don't see why the merging of the two projects will be different, but all together we will likely gain more mass and momentum (at least that's the general idea).

You can still use https://github.com/bemk/andromeda as main repository, but the official one now resides at https://github.com/Andromeda-Kernel/andromeda.

We hope the dust of this collision will soon settle down so we can continue what we were doing before this.

Bugs, bugs, bugs and more bugs

Yeah, the title says enough, I'm afraid.

I think it would be an understatement to say that we found SOME bugs last summer break. We found a lot.

First up was a series of bugs related to the fact that the ELF loader didn't fully comply with the standards. Once that was found it was an easy fix.

Further, what we've unearthed is quite an impressive bug, this bug is related to the paging system, which by the way, is by no means finished yet.

It's this last bug that I've been working on since a while and the last several days I've been getting some small clues of where the bug could be.

At first we thought it could be a memset issue, but by replacing the current memset with a simpler one, we eliminated that as the possible and most obvious cause.

We then continued with some other obvious culprit which was the memory allocation system, which we started testing. During these tests we figured out that the allocator wasn't the issue.

This led me to thinking about the issue in a new light. From the paging perspective, and so I did a little test, to figure out where in memory this bug starts.

As it turns out, once we enter a level of memory above a certain number, we cause page-faults, which in turn cause more page-faults, and so forth.

This issue is probably due to the way I approach paging, but it leads to the more interesting question. What is precisely going wrong, since without knowing that, we can't fix the issue.

That's what we're doing now, so we probably won't meet this sprints target, but that doesn't really matter, as the bugs resolved will have a way higher priority.

That's it for now,

Hope you had a nice summer break, and see you next post.