hckrnws

Author here. I’m happy to see one of my projects on Hacker News. This has been a fun one. One evening you just try to disassemble it and wonder where the code is. The following months were a truly satisfying experience, reverse-engineering this diamond.

There is still a functioning Forth interpreter implemented in the game. If they hadn’t removed all the word names, it would have been possible to debug at any time and analyze the program state and call functions live with real Forth code. Some crazy feature at that time.

Some early source code snippets of the game have even survived and can still be found on archive.org [0]. By today’s standards, they’re almost unreadable.

[0] https://web.archive.org/web/20030906124225/http://www.sonic....

"By today's standards"? no that's just FORTH code

A fantastic read, and really interesting to see the use of Forth. I remember Forth having a bit of popularity in the 80s. This was such an amazing game, especially in that you felt like the world was huge with the encouragement to just explore.

The other game this reminds me of is a game for the TI99/4a called Tunnels of Doom. It was a cartridge game that also had a floppy or cassette data load. It had a dynamic dungeon creation so every time you played the game you got a new unique experience. That would be an equally challenging one to reverse engineer due to the oddity of the GROM/GPL architecture in the TI99/4a.

There is a VM project now using the reversing work as a basis: https://git.information-superhighway.net/SpindleyQ/starfligh...

Microblog: https://gamemaking.social/@SpindleyQ/115058149348319018

Hard to convey how effective Starflight’s game design was within the limits of the day.

The embedding of the story within what was almost entirely free-form exploration & adventure across a huge galaxy was masterful.

You could feel how close the creators were to the edge of what was possible with the save game system: Basically, the disk was a memory image. As you played the game would rewrite itself, so if you got stuck there was no “reset”. The documentation was emphatic: Only play with a copy of the original disk!!

Everyone goes on about Elite, but Elite was just a sandbox… (an amazing sandbox)

This was crazy! World gen is hard. Proc world gen is NP hard. Story driven proc world gen with persistence in 1986 was Kaku level brilliance.

I loved Starlight but I'm not sure it was procedural world generation. I mean there was a map of stars printed with the game so they weren't changing. There was a small bit of variation in terms of what one found on planets and so on...the key was it felt like an open world because it was big enough and there was nothing stopping you from doing what you liked and when (except resources).

It was procedural at least in the sense that you couldn't store the data for all the planets in memory (or even store it on disk) on the 1980s systems it ran on. So you needed a way to generate the data on the fly.

yeah but you're dismissing the fact that this was just a pregen table of data back then. They made a map based on that, sure, but from that table came... everything else and you can't store all that data on floppy.

Similar techniques apply today. Pregen like 100,000 stars. Give them names and locations in the galaxy, treat them as your "locations of interest" with a seed. The rest can just be another cloud of particles with no interest and if the player visits, you can RNG whatever based on the seed. No two systems can share a seed. They can, however, share a branch.

They bragged about it being procedural in interviews.

What I was never clear on was the degree of cherry picking they did. There were 800 worlds on something like nine disks, each unique and peppered with minerals and artifacts.

Star Control 2 (which takes many Starflight influences and also hired Greg Johnson) also used a lot of procedural generation. It has around 500 stars and 4000 worlds with minerals and lifeforms. I recall the SC2 lead programmer Fred Ford saying they used a fixed seed, and they went through many seeds until they found one that looked good. I presume Starflight was of the same mindset.

I wonder if QA was given the job of vetting the seeds.

Procedural generation can use a fixed seed, it's not too uncommon. For instance, Elder Scrolls 2: Daggerfall's map was procedurally generated, but is the same for every player.

O cool! This approach was used in WASTELAND too, on four floppies (the progenitor of the FALLOUT series).

If anybody is interested in just playing this game, the Sega Genesis version of this game is arguably the definitive one. It was released later on and took everything in the PC version and amped it up a step. It's amazing and easy to find emulators + the ROM online. I replayed it fairly recently and it definitely held up very well.

It's open world, clue based, and with some amazing plot. Like the article mentions, the ending twist alone makes it worth suffering through your notepad of a million little hints thrown at you throughout the game. It's also the sort of game you can beat in [literally] 5 minutes if you know everything ahead of time. So I would strongly recommend avoiding walkthroughs/spoilers. The whole game is about piecing together the puzzle of what's happening and it's amazingly immersive.

Generally a game that was way ahead of its time.

Wow, so cool! I did not know Starflight was written in Forth.

Starflight was my first experience with game patching. When I got my copy it would not run on my PC. I believe it was because I had an EGA graphics adapter. I wrote a letter explaining my problem and mailed it to Electronic Arts. They mailed a letter back. It said there should be a program on my computer named debug.com, and gave instructions about how to load the Starflight executable in it, replace a couple of bytes, and save it out. I followed the patch instructions and it got the game working! After which I spent many happy hours with my friends exploring the Starflight universe.

Early in the 2000's one of the original authors of Sfarflight posted a massive dump of information online: design documents and source code. That is the "Technical Articles Saved from Oblivion" link in this repository. Sadly no one seems to have managed to save a full copy! This is one I kick myself about: I browsed this back when it was up but didnt think to download. These days I make a point to request Internet Archives to save things like this if I ever come across it.

About halfway through my reverse-engineering process, I came across those documents and realized that roughly 50% were missing. I searched pretty much the entire web to see if anyone had stored these files elsewhere, but so far I haven’t had any luck.

I'm not sure what you're referring to since the files T.C. Lee posted on that geocities site are the only design documents he (or anyone else) ever released, and those are preserved in the SFFiles.zip found at the oocities mirror linked from the github there. That zip contained only a partial dump of some source code, while that sonic.net page contains a more complete copy (but lacks any kind of design documents whatsoever). Was there something else you were referring to?

I seem to recall browsing what appeared to be the complete source back in the day. I put in a bit of an effort to get it compiling, but it was only released as poorly scanned printouts of the source code and OCR wasn't so good then so the project was bigger than I hoped.

That would be the source code posted to that sonic.net site. It was a mix of raw dumps of FORTH blocks and printouts converted to PDFs. It wasn't the full source code, and there weren't any design documents in there.

You can see on archive.org where it captured directory info for files which were not stored:

https://web.archive.org/web/20031218202130/http://www.sonic....

When you navigate those directory archives many of the files are missing. E.g.

https://web.archive.org/web/20030529230704/http://www.sonic....

Some design documents ARE there, e.g. "Specification of 3d display for starquest" : https://web.archive.org/web/20030719111039if_/http://www.son...

Well....... shit. I'd been in there previously and never noticed that. Disregard.

> for this game you can throw the usual tools away...

> The reason is that Starflight was written in Forth

I recently reverse-engineered another game from the 80's, and had a similar issue because it was written with Turbo Pascal.

The different tools didn't like quirks such as data stored inline between the instructions or every library function having a different calling convention.

Turns out the simplest way to handle this was to write my own decompiler, since Turbo Pascal 3 didn't do any optimisations.

> There is no set path, allowing players to switch freely between mining, ship-to-ship combat, and alien diplomacy. The broader plot of the game emerges slowly

This reminds me of Star Control 2, aka The Ur-Quan Masters

> The game influenced the design of numerous other games for decades after its release.

And in SC2 wiki page I see this:

> Once Reiche and Ford conceived Star Control 2, they would draw large inspiration from Starflight.

https://en.wikipedia.org/wiki/Star_Control_II

Never played Starflight before but seems right up my alley as SC2 is one of my favorite games

Edit: Just finished reading the rest of the readme. Very cool! I honestly knew nothing about Forth (just that it was a programming language) and now I want to play around with it.

It goes beyond drawing inspiration. Greg Johnson, designer of Starflight, worked directly on Star Control 2.

You can read more about this gem of a game upon the digital antiquarian:

https://www.filfre.net/2014/10/starflight/

> Back in the 80ths, an unknown company called Binary Systems published the game Starflight.

80s - 80ths is a fraction (1/80).

Also - the publisher was Electronic Arts, already well-established at that point. Binary Systems were the developers.

One of my all-time favourite games. A family member had Starflight on Genesis. I still pull it up in an emulator from time to time.

Random edit: It's also how I learned the word obsequious.

The lore in the manual was entertaining as well; whoever wrote it was clearly having fun. All these years later I still clearly remember a quip from the parody religious prophecy in Appendix C: "And then, within the bowels of Arth¹ shall there once again be movement." Never fails to crack me up.

¹ The name of the planet that the player starts on.

lol! Me too! I remember the alien interaction dialog and this word.

Starflight is the great grandfather of No Man’s Sky, and the first game I availed myself of using Internet databases to find obscure facts about the game. There was no website. It was some group of people curating a text document that was reposted to Usenet periodically with additions and corrections.

I don’t know when I started the game, but I finished it the summer after my freshman year, which is where I first got Internet access.

Awesome! I got this game for Christmas in 3rd grade and was completely hooked! It really captured my imagination and immersed me into this seemingly limitless world. It took me a couple of years to complete, and I kept of journal of exploration experiences.

Thank you for posting this!

I find it curious that the game was written in Forth. Certainly a very unusual choice for a commercial game.

Probably worth mentioning that writing a big project in Forth is more like creating an OOP framework.(if you are disciplined)

The end result of that is one doesn't write the program in "Forth" per se but in the domain specific language you create for the job. This is how Forth gets more productive than pure assembly language if the team documents things well and follows the prescribed system.

This was the era before optimizing compilers.[1] The overwhelming majority of commercial games were shipping hand-coded assembly still. Forth had the advantage of low overhead, no-worse-than-a-compiler speed, and better-than-assembly productivity. It was a small time window, but a good fit in the moment.

[1] Non-trivial optimizations were just starting to show up on big systems, but Microsoft C in 1985 was still a direct translator.

Forth generated code is basically a long series of "assembler macros", always doing the same maximally-generic thing for each primitive operation. Even a very simple-minded compiler of the 1980s could already beat that.

    VAR1 @ VAR2 @ + VAR3 !

    ; push address of VAR1
    inc    bx
    inc    bx
    push   bx
    ; fetch and jump to next primitive
    lodsw
    mov    bx,ax
    jmp    [bx]
    ; push contents of variable
    pop    bx
    push   [bx]
    ; next primitive...
    ; push address of VAR2, next...
    ; push contents of variable, next...
    ; add top two stack elements, push sum, next...
    ; push address of VAR3, next...
    ; store to address, next...

Still, an incredibly stupid compiler could do better (for execution speed, definitely not size) by outputting similar code fragments - including all the pushes and pops, who needs a register allocator? - just without the interpreter overhead (lodsw etc.) in between them.

A compiler producing worse code likely didn't exist before today's glorious world of vibe coding ;)

A slightly better compiler would directly load the variables instead of first pushing their addresses, etc. You don't need to be an expert in compiler theory to come up with enough ideas for boiling it down to the same three instructions that a competent assembly programmer would write for this operation. And at least for this case, Forth doesn't even have the size advantage anymore, the code would only take 10 bytes instead of 14.

The compiler space in 1985 was really thin. You were basically looking at Microsoft/Lattice C and Turbo Pascal. And while I don't have any of them handy for a test, that's pretty much exactly the character of the code they'd generate. In particular the 8086 calling conventions were a kludgey dance on the stack for every function, something forth could actually improve on.

I know Turbo Pascal produces really bad code (even in later versions), but it's not on the same level as a non-optimized Forth. Function prologue on x86 doesn't have much overhead either.

It's somewhat closer for 8-bit processors, the most popular ones had either no convenient addressing mode for stack frames at all, or an extremely slow one, like IX/IY on the Z80. For even more primitive CPUs, you might already need a virtual machine to make it "general-purpose programmable" at all -- for example if there is only one memory address register, and no way to save its contents without clobbering another one. I think that some of Chuck Moore's earliest Forth implementations were for architectures like that.

Also memory constraints could have been more important than efficiency of course. I'm not saying Forth is worthless, but it's not competing with any compiled language in terms of speed, and IMHO it also does away with too many "niceties" like local variables or actually readable syntax. Your mileage may vary :)

Loved this game as a young kid. Great to see it. Somewhere I also found a web implementation of that black and white codewheel that came with it.

I think we will see more and more games and applications be fully reversed engineered in the coming month due to increase coding agent capabilities.

This is my all time favorite DOS game. It felt open world before people used that phrase. Amazing game.

I believe this is only the second game I ever finished. But also the first I used the internet to finish.

This is the way. Best thing since the reincarnation of Wall Street Raider.

...Forth? Wow. I wonder how much code change was necessary between the various systems. It's hard to imagine a Megadrive Forth compiler, but then again, the game was on several other M68k systems so maybe it wasn't as hard...

It is really, really easy to write a Forth interpreter (You can write a simple one in an afternoon). It's often the first software written for an architecture. The structure of Forth means that the hardware-dependent parts are contained in a small number of words (sort of like functions in other languages but not exactly). Forth can be implemented on tiny microcontrollers; a Megadrive would be luxury.

It's.. not a compiler (besides I had Forth on my C64). Maybe one can call it a translator to ad-hoc bytecode. I also had USCD Pascal on that C64 which translated to bytecode. This was more JVM-like. So nothing hard about it.

Forth is a compiler but what it "compiles" is not standard. The implementor decides what they need. Forth can compile pointers to native code that are the VM's instructions, called direct threading. Forth can compile pointers to pointers to native code VM instructions, called indirect threading. Forth can compile byte code like OpenFirmware/OpenBoot.

And modern systems compile optimized native code (VFX Forth, SwiftForth) but still remain fully interactive at the console.

Point of information: By and large FORTHs did not use bytecode. Execution tokens (XTs) were usually stored as a function of the default word size, which typically was 16 bits. There were some FORTHs that went out of their way to use token threading so they could store programs in byte codes, but those were pretty rare. Rarer still were programs that mixed byte code with word-sized code (although one such scheme is described in an issue of Forth Dimensions).

This game looked very familiar and after googling around it looks like it is recognized as the progenitor of Starcontrol game.