Bugs, bugs, bugs and more bugs

Yeah, the title says enough, I'm afraid.

I think it would be an understatement to say that we found SOME bugs last summer break. We found a lot.

First up was a series of bugs related to the fact that the ELF loader didn't fully comply with the standards. Once that was found it was an easy fix.

Further, what we've unearthed is quite an impressive bug, this bug is related to the paging system, which by the way, is by no means finished yet.

It's this last bug that I've been working on since a while and the last several days I've been getting some small clues of where the bug could be.

At first we thought it could be a memset issue, but by replacing the current memset with a simpler one, we eliminated that as the possible and most obvious cause.

We then continued with some other obvious culprit which was the memory allocation system, which we started testing. During these tests we figured out that the allocator wasn't the issue.

This led me to thinking about the issue in a new light. From the paging perspective, and so I did a little test, to figure out where in memory this bug starts.

As it turns out, once we enter a level of memory above a certain number, we cause page-faults, which in turn cause more page-faults, and so forth.

This issue is probably due to the way I approach paging, but it leads to the more interesting question. What is precisely going wrong, since without knowing that, we can't fix the issue.

That's what we're doing now, so we probably won't meet this sprints target, but that doesn't really matter, as the bugs resolved will have a way higher priority.

That's it for now,

Hope you had a nice summer break, and see you next post.

Chaos in the project?

Strange title, maybe?

Well, yes. There is a little bit of chaos in the project, mainly because it's vacation season, so that means that everybody will spend some time away from the project and most things which would normally go well, don't now.

Another source of chaos might be the fact that, up until the last release I had a pretty good idea of where I would like to go with the project. I need it to be in high memory and I want that to be done in a simple fashion, which I think I have completed by now.

Next up I thought I'd look at streams and everything, but it turns out now that we need at the very least a simple virtual file system. Now that very simply put means that I don't have a clue what I'm doing.

I have been reading this book written by Andrew S. Tanenbaum and it does give a nice overview, but it's all theory, and like we all know, there is a difference between theory and practice. Oh-yeah, there are some differences all right.

Now I'm not going to sum it all up for you, but it boils down to the fact that I have a lot of questions and pretty much nothing at hand to answer it all, besides the book, which I'm still reading.

I'm at the same time also looking at the OS dev wiki and have found some interesting bits of info there as well.

So, what can one expect to happen until September the first, when this sprint should be done.

Basically just us lagging severely behind. I think I might have overlooked the gravity of some of the functions and that in planning terms will cost us dearly.

Now this doesn't matter anyway, as I still don't have a clue of when the 1.0.0 release should be.

One thing is for sure though, and that is that the 0.0.5 release might have a little less than one'd expect it to have, more because of the gravity of the features, than because of the lack of planning in the vacation (which is why the sprint ends first of September instead of August the first).

Speaking of summer break, what am I going to do?

Well, I'm going to France for a couple of weeks, and the neighbours will be watching over the plants and everything.

My sister is not going with us for the very first time, because she gets to go to language training camp in Cambridge (lucky arse).

In the mean time I'm pretty sure Steven will keep going with the graphics drivers and stuff like that.

So I think that'll be it for now and I plan on posting again when I'm back.

And here comes the fourth release

Yep, its time for the fourth development release also known as 0.0.4.

This release features the ability to:

• Respond to interrupts (again),
• Load in the core image and
• Compile the core image.

The previous release (the overhaul of the project layout) should actually have been part of this release, but it was so much of a change that I thought it was worth its own release, this making this the fourth release.

I certainly did a short silly dance once I had the elf loading done for sure (probably not a soul that's seen it, lucky me), because this was a point I was working toward from the beginning of the project.

Up next (at least for me) is the revising of the printf functions and the way text is displayed on screen. This actually might get in the way of Steven a little, but I don't care as he is on vacation anyway.

The next release is scheduled in September, and that might seem an awful long time from now, but keep in mind that the summer breaks are in the middle of this, and that I'll be gone for 21 days here.

Steven has been delivering code, but unfortunately this is not testable, and thus is shifted to the next sprint, which actually starts today (Monday, June the 11th of 2011).

We wish Steven a nice vacation and hope he will be able to make some good progress in my absence.

We also wish to thank him for the server space he's provided.

This is a link to the tag formed during the 0.0.4 release.

Major break through

I know this post is a little early, but I don't care, since we have had this major breakthrough last nigt.

It doesn't seem like much, but if you're a little familiar with the project you'll realise that the fact that we have Elf loading working, is a HUGE thing.

The reason why it such an impressive feat is because everything has been made from scratch.

"So?" You might ask.

Well, We've had to initialise a heap, use that heap (did that with a first fit algorithm), print some text to screen, and set up paging.

The first big breakthrough was when I had a heap working, the second was paging, even though that one isn't really finished yet. The last in this sequence is Elf loading, since it puts every thing together.

It basically pulls the interesting stuff out of the image and pasts it somewhere in memory (defined by the image). The weird thing is that the batch of memory requested might not even exist. That's difficult since we actually need it.

Not to worry, we have a way of doing things to make this all possible. It's called virtual memory. Basically what it does is split the address space up into 4KB pages, and a page is then connected to a physical batch of memory, allowing for pages to exist in memory addresses which don't exist in the physical address space.

This does however come with a problem. Since the virtual address space is 4 GB in size and the actual memory size is something random, that might spawn the problem of a lack of physical memory. Now that's solved by swapping the pages out to disk.

This however hasn't been implemented, which has a very simple reason. We don't have file system support, and thus we aren't able to write to or read from disk.

Guess what's up next in the planning?

Well, yes, graphics, but besides that?

Indeed, file systems support. This will likely be a three tier approach.

Tier 1: The virtual filesystem (vfs), this is what you interface with as user/programmer.

Tier 2: The filesystem driver(fs driver), which is what looks up the physical position on a physical hard drive, regardless of what type of drive you use.

Tier 3: The disk driver, which takes commands from the filesystem driver to access specific blocks on the drive, without the need to know about filesystems.

How this will be implemented needs some thinking though. Are we going to implement the inodes in the vfs, or will that be supported in the fs driver, and if so where will we keep track of mount points.

If you have ideas, they're always welcome of course.

Another thing that desperately needs to be done is the design of a logo. Now for more info go here.

The fourth release already?

So it's been a while since last blog post. I know. Last time we were still in 0.0.2, and now I'm already talking about 0.0.4!

What's happened you might ask. Well, if we open the new site, we will see several things there. There is a sprint backlog over there, which explains what we're currently working on before the next release. There's a product backlog which represents all the future features, and there's some more general info.

The sprint page shows you what we're doing and what's done. I'll make a quick summary for you, since this page will likely be updated in the near future.

We've:

• Set up a site,
• Finished the header analysis,
• Done a new project layout,
• and verified the elf header.

What's more is, I've actually just finished work on making the core image compile.

Now the work on the core image isn't finished yet, as it still needs testing (it's now in to verify) en might thus go back into the in progress, or even the todo if the issues appear to severe.

In the mean time Steven has been doing some work on getting the graphics going, and even though we don't always agree, he does his best, and quite often produces good work. Hence he's also got several topics.

He's also working on a feature which isn't in the sprint. The feature needs to be done for sure, but will probably be introduced in the master branch at around 0.0.5 (Oh my god, I'm already talking about an 0.0.5 release!).

In the mean time I have had official verification that I may continue at school into the second year!

Now for more info check out the site at http://orionos.tk/.

A quick side note though. Because of energy saving and the target audience primarily being in Europe, the server will be down between 23:00 UTC until 7:00 UTC.

What's going on here?

If you have been following the code lately you will have seen several things changing, quite dramatically.

One of which is that there is a new contributor, named Steven van der Schoot. His name already was in the  CREDITS file, but his role actually has changed dramatically.

Previously, all he was known for was the printhex function, which couldn't do much more that print out numbers in hex. I thought that was a bit messy, as I also wanted something that could do binary, decimal and octal, and just hacked them together into a single function. (If you know the maths it's not that dificult, I admit).

Now he has more time because he has passed his final secondary school exams, for which I wish to congratulate him.

But he has decided to do several things with his spare time. One of which is Mincecraft. The other? The VGA driver for the platform.

Yes, we already had text output, and that probably won't change in the near future, but what he is enabling here is much more than basic text printing. Basically the bootloader configures you to use VGA text mode, which enables you to write characters to a location somewhere in memory. Now that a driver is coming up, we still be writing to this buffer, be it in another location, only god (and my paging code) knows where in physical memory. But once this is done, what he can do, is start focussing on drawing shapes to screen. Such as rectangles, triangles and circles.

Now that sounds fantastic and all, we've got graphics working, or ... do we?

Technically, yes, but don't get too impressed. I still don't know anything about resolution, but for all I know VGA is basically crappy, not being able to go over 640x480, at, what was it? 256 or 16 colours.

Now this is nice and all, but it doesn't really have my priority. I'm more worried about getting the source code in a new layout, which enables me to make a difference between the compressed image (or nano kernel. Yes I'm changing the terminology here) and the core image on basis of path, in contrast to a complex, hard to notice, easy to forget while writing ifdef structure.

This is actually the reason why you see these huge spikes in the impact graphs on github. It basically boils down to the entire project being assigned a new location, which generates a huge impact, while in reality, nothing changes in the content. It's a mere process of copying and removing.

I am going to have to do some work on the makefiles though, and I probably am not going to like doing so. The make files need to get more scalable, as I know that they aren't exactly doing so as of now.

But is your elf code done?

Well, no.

I needed to get my loaf away from it for a second. Making the project more scalable had a higher priority than elf code. Despite the elf code being half done. This is due to the fact that I'm no longer the sole developer of this project and I need to make it possible to give the other developer(s) all the space they need.

I still don't know where I should put Stevens drivers for example. The drivers could be necessary for the nano kernel, since the drivers need to be installed from unreal mode, and the core image can't go there any more (unless you have 3,5 gigs or more). I could also do v8086 mode, but I feel like that could pose security issues.

I still haven't figured it out yet, but I'm working on it. Fortunately with a new developer.

Is it me, or has it been a while?

It's not you. It indeed has been a while and it is all due to the fact that this week is exam week at school. Those things require my attention a little more than the kernel.

So you've done nothing to the kernel since last update?

Well, that's not entirely true. I've been reading the Elf32 specifications and have managed to pull some more info from the Elf headers. Among which is a segment with a type number that doesn't appear to make sense according to the specifications.

However, after doing some googling I found out that Linux uses its own types. One of those is a stack segment, in which the kernel is specified to reserve room for the stack. In normal circumstances that's nice, but since my kernel sets up its own stack I'm just going to ignore this segment.

As for the rest?

Well I've been redoing my printhex function so that it now prints numbers in every base in the range from 2 to 36. All printf has to do now is call this printnum function with the number (named index), base, a boolean called signed and a boolean for capital characters.

So I guess that's all.

Well, not entirely. I have also changed the layout of the Orion project itself. Andromeda now no longer is merged into the Orion source tree. It rather is imported as sub-module. This means that when you import the Orion source tree, all you get is a .gitmodules file and a bunch of empty directories.

You then have to do git submodule init; git submodule update; to get the sources from the sub-projects.

Now that's all.

And what can we expect for the future?

Well, school is about to break up for the summer, which renders me a lot of spare time. I'll be spending some of that spare time in the woods with my dog, and some of that time in France. I will also be spending some of that time on the project, so don't worry about that.

I think the summer break up will give me more time to do work on the project so you might see the impact graphs show a lot more impact.