Posts Tagged retrocomputing

Some old-school UNIX shell hackery using “tr” for a UNIX v6 kernel build

I had to resort to some old-school UNIX shell hackery to get V6 UNIX running in SIMH as an automated process.

If you refer to the earlier posts, you’ll see that there were three main steps in getting a well-running UNIX kernel with new device drivers created:

  1. Boot from “tape” and copy an image of the root filesystem onto a simulated RK disk
  2. Boot from the RK disk, modify source code for several programs such as “df” and the kernel. Compile the new kernel, copy it into place in the root filesystem.
  3. Boot the new kernel from the new root filesystem.

This post covers step 2, which involves creating and editing several files to include information about new device drivers, then then compiling the new kernel. The entire process is very well documented at: https://gunkies.org/wiki/Installing_Unix_v6_(PDP-11)_on_SIMH so I won’t cover it in detail.

Recall that while we have a running V6 system, there’s no way to copy files into the virtual machine. We can enter shell commands, but not copy files into the emulated system. This means that we have to play games with “cat”, “ed” and other UNIX commands to create new files or edit existing ones. There’s a bit of a complication in that to enter commands into UNIX running within the SIMH emulator, we have to use the SIMH EXPECT/SEND commands. These are documented in the SIMH user’s guide for Version 4.

Also, remember that this is the 1975 “sh” shell. It’s very primitive. If we had the modern “here document” feature, this would have been much simpler. I’m not sure, but I believe that feature didn’t appear until 4BSD (1980). Instead we’ll have to rely on the commands that were there 44 years ago.

For example, this set of commands would normally just be entered on the command line.

chdir /usr/sys/conf
cc mkconf.c
mv a.out mkconf

But to use the EXPECT command, it looks like this, using “;” as the command separator to do this all on a single input line:

expect '#' send 'chdir /usr/sys/conf; cc mkconf.c; mv a.out mkconf\r'; continue

So far so good, but what about something more complex, where we’re even farther down the rabbit hole? Passing input to a program that’s running in the shell, in the emulated OS is one step further in and requires some “old school” shell hackery. For example, what about sending input into the “mkconf” program:

# ./mkconf
rk
tm
tc
8dc
lp
done
#

It turns out that we can’t just feed this in as multiple EXPECT/SEND combinations due to the need for embedded newlines. Any newlines in the EXPECT script would end the line, any newlines embedded in the SEND strings would also be lost.

This was a head scratcher for an hour, until I remembered some similar problems I’d had years ago doing stream editing (sed) to patch binary program files on the fly instead of re-compiling the source (don’t ask, ugly).

This led me to using the “tr” program to “send” newlines without ever using the newline character in my command.

expect '#' send 'echo \'rkXtmXtcX8dcXlpXdone\' | tr X \\\\012 | mkconf\r'; continue

You can see that the newline character never appears anywhere in the SEND string. The echo command will emit the needed lines, but with X instead of newline. The tr command will replace the X character with an escaped-and-escaped-again 012, the octal for newline. This feeds 6 newline separated strings (lines) into the mkconf program, without ever actually using a newline!

After that, it was back to vanilla scripting, until I hit another similar glitch. To add the RK disk to the list of supported devices in the “df” command, you need to edit the source to add 2 lines into an array that lists the supported devices. Interactively, this is pretty trivial (if you know “ed”).

# chdir /usr/source/s1
# ed df.c
/rp0/d
.-2a
  "/dev/rk0",
  "/dev/rk1",
.
w
q
# cc -s -O df.c
# cp a.out /bin/df
# rm a.out

And here we are again, needing to enter multiple lines to a program, without using the newline character. It’s “tr” to the rescue again:

expect '#' send 'chdir /usr/source/s1 ; echo \'/rp0/dX.-2aX  "/dev/rk0",X  "/dev/rk1",X.XwXqX\' | tr X \\\\012 | ed df.c\r' ; continue

And finally, one last time:

# ed /etc/ttys
1,8s/^0/1/p
w
q
# 

becomes

expect '#' send 'echo \'1,8s/^0/1/pXwXqX\' | tr X \\\\012 | ed /etc/ttys\r' ; continue

You can find all this hackery (and some other uglier things) in the “buildunix.ini file in the github repository.

For me, this was a fun trip down memory lane and the weird things we had to do, back when computers were more primitive and yet sometimes more fun.

, , , ,

Leave a comment

V6 UNIX – first boot and “driving” the SIMH emulator

In the previous post, I alluded to some “extreme expect hackery” needed to configure and install a new UNIX kernel.

Note: The repository name has changed to: https://github.com/tomperrine/unix-v6-pdp11-simh-gcp

To get the most out of this post, get the files from the GitHub repo to follow along.

During the era of PDP-11 and even VAX UNIX, adding device drivers to the kernel required changing the source code. Specifically, there are a set of data structures that define the mapping from UNIX device numbers to the names of the device driver entry points (C functions). Hence, adding a device driver to the kernel requires source code to be edited and a new kernel compiled from the new sources.

The entire process is pretty well documented here.

To get from “first boot” to “booting from rk disk with all patches applied” takes a few steps:

  1. Boot from “tape” and copy an image of the root filesystem onto a simulated RK disk
  2. Boot from the RK disk, modify source code for several programs such as “df” and the kernel. Compile the new kernel, copy it into place in the root filesystem.
  3. Boot the new kernel from the new root filesystem.

This post will cover step 1 and the SIMH scripting needed to automate this process.

If we could just copy new files from “outside” (Ubuntu OS) into the file system of the “inside” or guest OS (v6 UNIX), none of this would be necessary. However, since we can’t, the only way to create files is by executing commands.

We’ll get started by figuring out how to send commands into a program that’s running inside the OS that’s inside the emulator, then (in a future post) work up to some very old-school UNIX tricks to create the needed files.

The first challenge is to pass commands to the SIMH emulator. This is done simply by giving the emulator a script file on the command line. See this example:

$ more buildunix.ini 
set cpu 11/40
set tto 7b
set tm0 locked
attach tm0 dist.tap
attach rk0 rk0
attach rk1 rk1
attach rk2 rk2
dep system sr 173030
boot rk

When you start the emulator with that script file, you get this result:

$ ./simh-master/BIN/pdp11 buildunix.ini 
PDP-11 simulator V4.0-0 Current   git commit id: 0de9b628
sim> set cpu 11/40
Disabling XQ
sim> set tto 7b
sim> set tm0 locked
sim> attach tm0 dist.tap
sim> attach rk0 rk0
sim> attach rk1 rk1
sim> attach rk2 rk2
sim> dep system sr 173030
sim> boot rk0
@

As you can see, we just gave the emulator a list of commands and they were executed by the emulator, which loads and then runs the UNIX bootloader. It’s the bootloader, running in the emulator that presents the “@” prompt. At this point, the emulator stops passing its input file to the console, leaving us “stranded” at the boot prompt. The bootloader needs the name of the kernel to load, which would normally just be entered by the user. Since we want to automate the entire process, we need to find a way to enter data but not just a a line in the script.

The emulator provides a way to SEND input into the running programs, using its own internal implementation of “expect“. This means that we can use EXPECT/SEND combinations to enter information in the programs that are running inside the OS that’s running inside the emulator. Clear as mud, right?

The “hook” is to set up an EXPECT/SEND combination BEFORE we enter the boot command, so that when the boot command executes and presents the “@” prompt, the emulator knows what to send in response. Now the script looks like this:

$ more buildunix.ini 
set cpu 11/40
set tto 7b
set tm0 locked
attach tm0 dist.tap
attach rk0 rk0
attach rk1 rk1
attach rk2 rk2
dep system sr 173030
: this sets up the rkunix information to be sent later,
: after we enter the boot rk command
expect "@" send "rkunix\r"; continue
boot rk

Which results in something more like this:

$ ./simh-master/BIN/pdp11 buildunix.ini 
PDP-11 simulator V4.0-0 Current   git commit id: 0de9b628
sim> set cpu 11/40
Disabling XQ
sim> set tto 7b
sim> set tm0 locked
sim> attach tm0 dist.tap
sim> attach rk0 rk0
sim> attach rk1 rk1
sim> attach rk2 rk2
sim> dep system sr 173030
sim> boot rk0
@rkunix
mem = 1035
RESTRICTED RIGHTS

Use, duplication or disclosure is subject to
restrictions stated in Contract with Western
Electric Company, Inc.
#

And now we have UNIX running inside the emulator, and a command prompt. At this point, our terminal is attached to the UNIX shell, and we can start to manually enter commands. But that’s not enough. We have an entire set of commands that we need to enter to add the new device drivers to the source code so we can compile a new kernel. If it was a single shell script, it would be about 70-90 lines, and of course, we still can’t just copy files into the V6 file system.

Next time, entering all the commands needed to configure and build a new kernel.

, ,

Leave a comment

PDP-11 running UNIX v6 in Google Compute Platform (GCP) using SIMH

Wow!  This post is months overdue!  I blame work, more work, Destiny2, other work, Edinburgh Fringe Festivalother other work, and beer.

This post is a quick overview of my GitHub repo (pdp-11-in-gcp) and how it works to create a fully functional UNIX system from 1976 (UNIX V6) “in the cloud”. It has everything you need to run your own piece of UNIX history.

For the most part, this is an automation of the instructions from http://gunkies.org/wiki/Installing_Unix_v6_(PDP-11)_on_SIMH

This assumes that you have a functioning GCP account with billing enabled, and have at least skimmed earlier posts in this series.

This repo includes several scripts and configuration files:

* launch-pdp11.sh – The master script creates a place to run the SIMH emulator, and builds the emulator. Part of this process is loading another script on to the GCP instance.

* update-os-build-simh.sh – This script is copied on to the GCP Ubuntu instance and gets the SIMH PDP-11 emulator running in the instance. When this script completes, you have a running Ubuntu system with a PDP-11 emulator ready to install v6 UNIX.

The end of the launch-pdp11.sh script provides instructions on how to install V6 UNIX into the emulator. This requires manually running three commands while logged into the GCP Ubuntu instance. Due to limitations of EXPECT, there are a few places where you will need to manually halt the emulator (^E).

* simh-master/BIN/pdp11 tboot.ini – This starts the emulator and does a “tape boot” from an emulated tape image and copies the minimal root filesystem on to the emulated RK disk (which is a file on the Ubuntu host).

* simh-master/BIN/pdp11 buildunix.ini – This script uses extreme expect hackery to do LOTS of customization of the kernel to support an RK disk 

* simh-master/BIN/pdp11 normalboot.ini – boots the fully functional PDP-11 with all software. Use this for all subsequent boots of the UNIX guest

One of the most fun parts of this project was dealing with SIMH’s internal EXPECT function. In the “olden days” you had to change the kernel source code to configure tables for each device driver that you wanted included in a new kernel.  I’ll show some of that in the next post.

, , , ,

1 Comment

Scripting a fast Ubuntu install in Google Cloud Platform (GCP)

In this post I’ll show how to script GCP instance creation, Ubuntu installation and patching in order to support the customized SIMH installs that we’ll do later.

All of my GCP/SIMH installs are based on Ubuntu Linux, running on tiny or small GCP instances. Since one of my goals is quick iteration and making it fast and easy for other people to install the SIMH emulator and the guest OSes, I’ve scripted everything. I’ve been a fan of infrastructure-as-code for two decades, so how could I not apply that to my GCP estate?

For this we need four scripts:

  • create-instance – create an instance, install and patch Ubuntu
  • stop-instance – stop (pause) the instance, preserving the instance state (boot volume)
  • start-instance – (re)start the instance from the saved state
  • destroy-instance – destroy the instance (which deletes the associated boot volume)

All of the examples start with a common Linux base in GCP, so it made sense to script a fast Ubuntu install and update.  While I could use a common SIMH install for almost all the guest operating systems, it makes sense to keep them separate so that people can install just the single OS that they want to play with, instead of them all.

These examples all assume that you have created a Google Cloud account, created at least one project, and enabled billing for that project. You may want to start with these tutorials.

You also need to set a few environment variables as described in this earlier post.

Everything below should be self-explanatory. Essentially, the main steps are to create the instance, then wait for the instance to be up and running. After that, another loop waits until the SSH daemon is running, so that some commands (apt-get update and apt-get upgrade) can be run.

#!/bin/bash

# given a GCP, etc account and the SDK on the install-from host, build and install a new server

. ./set-cloud-configuration.sh

# If you don't use ssh-add to add your key to your active ssh-agent
# you're going to be typing your passphrase an awful lot

#
# create the instance
#
gcloud compute instances create ${INSTANCENAME} --machine-type=${MACHINETYPE} --image-family=${IMAGEFAMILY} --image-project=${IMAGEPROJECT}
gcloud compute instances get-serial-port-output ${INSTANCENAME}

# add the oslogin option so I don't need to manage SSH keys
gcloud compute instances add-metadata ${INSTANCENAME} --metadata enable-oslogin=TRUE

#
# it can take some time, and sometimes(?) the create returns much faster than expected, or the system
# takes a long time to boot and get to the SSH server, so wait for it to be READY
SSHRETURN="dummy"
while [[ "RUNNING" != ${SSHRETURN} ]]; do
    SSHRETURN=`gcloud compute instances describe ${INSTANCENAME} | grep status: | awk -F\  ' {print $2}' `
    sleep 5
done
echo "instance running..."

#
# now wait until the SSH server is running (we get a response without a timeout)
SSHRETURN=255
while [[ ${SSHRETURN} -ne 0 ]]; do
    gcloud compute ssh ${CLOUD_USERNAME}@${INSTANCENAME} --project ${PROJ} --zone ${CLOUDSDK_COMPUTE_ZONE} -- hostname
    SSHRETURN=$?
    sleep 3
done
echo "SSH up and listening..."

# All we have is a "naked" Ubuntu OK, its always a good idea to update+upgrade immediately after installation
 gcloud compute ssh ${CLOUD_USERNAME}@${INSTANCENAME} --project ${PROJ} --zone ${CLOUDSDK_COMPUTE_ZONE} -- sudo apt-get --yes update
 gcloud compute ssh ${CLOUD_USERNAME}@${INSTANCENAME} --project ${PROJ} --zone ${CLOUDSDK_COMPUTE_ZONE} -- sudo apt-get --yes upgrade

exit

The start, stop and destroy shell scripts are much simpler.

All the code is available in my github repo: https://github.com/tomperrine/create-simple-google-instance

, ,

Leave a comment

Setting configuration variables for the SIMH instance in Google Compute

In this short installment, we’ll create a BASH script that will be re-used as we script the creation of the Linux instance, SIMH installation and guest OS installation.

This assumes that you’ve followed the prior posts in the series, and have a functioning Google Cloud account, with a project created, and billing enabled. You need billing enabled even if you’re using the “free tier” or your initial account credit.

There are (for now) three things we need to have set up: account information for logging in, a project name, and a description of the instance we want to run. The description includes the physical location (region/zone) and the operating system we want.

This simple script will set the variables that we will want and can be included into all the other scripts we’ll write later.

Save this as set-cloud-configuration.sh

#!/bin/bash
#
# set user-specific configuration info
# we're going to use "oslogin" so set a username
# THIS MUST MATCH your GCP account configuration
# see https://cloud.google.com/compute/docs/instances/managing-instance-access for details
export CLOUD_USERNAME=YOU_NEED_TO_SET_THIS_FOR_YOUR ACCOUNT!!!!!!!

# Set project information - this project MUST already exist in GCP
# This project MUST have billing enabled, even if you plan to use the "free" tier
export PROJ=retro-simh
gcloud config set project ${PROJ}

# set configuration info for this instance
# pick a region
export CLOUDSDK_COMPUTE_ZONE="us-central1-f"
# set information for the instance we will create
export INSTANCENAME="simh-ubuntu-instance"
export MACHINETYPE="f1-micro"
export IMAGEFAMILY="ubuntu-1804-lts"
export IMAGEPROJECT="ubuntu-os-cloud"

In order to continue with the series, you’ll need to make sure you have enabled billing AND configured “oslogin”.

You should also make sure you have ssh-agent running, unless you want to type your password, a lot.

In the next installment, we’ll create, stop, start and destroy GCP instances in order to prepare for compiling and running SIMH.

, ,

1 Comment

Using SIMH in Google Compute to retrace my (UNIX) OS journey

After being introduced to SIMH and getting Multics running, I thought about using SIMH to retrace the steps (and operating systems) that I’ve used in my career. For now, I’ll focus on the UNIX and UNIX-derived systems.

Before coming to UNIX, I had already used Honeywell GECOS, Multics, CP-V and CP-6, and well as DEC’s VMS and TOPS-10. My first UNIX experience was Programmer’s Workbench (PWB) UNIX, which was an interim version between versions 6 and 7.

But after that I used 4BSD, SunOS, UNICOS, HPUX, DomainOS, SGI IRIX, and a host of other UNIX-flavored systems until finally coming to Linux. Along the way I help to extend or create two security kernels – KSOS-11 and KSOS-32.

So my plan is to bring up as many of these operating systems up as possible using SIMH, and focusing on the UNIX family.

Here’s the dependency graph of what I have in mind to begin, and it’s a roadmap for the rest of this series. I have no idea how long it will take, or how far I’ll get.

To date, I’ve got Multics and V6 UNIX, so I’ll show the tooling for those first. Using this information, you should eventually be able to run any OS for which a SIMH emulator exists for the CPU, and for which you can find a bootable or installable image.

, , , ,

Leave a comment

Retrocomputing – using SIMH to run Multics on Google Cloud Platform (GCP)

Last Fall (Oct 2018) I started playing with SIMH, and using it to run some rather ancient operating systems in the Google Cloud (GCP). So far I’ve been able to run Multics, UNIX V6 (PDP-11), and 4.0BSD (VaX).

I started down this path by using the dps8m fork of SIMH to run Multics on a Raspberry Pi 3. This worked very well, and produced performance that for a single user, matched the original mainframe hardware. Not bad for a US$35 pocket sized computer emulating a US$10+ MILLION mainframe (of the 1980s). Of course, Multics supported 100s of simultaneous users using timesharing, but at its heart, Multics (up to 8) CPUs were about 1-2 MIPS each and the system supported up to 8M 36-bit words (32 Mbytes) per memory controller,  up to 4 controllers per system for a grand total of 128 Mbytes per system. Yes, that’s Mbytes, not Gbytes.

For comparison, The $35 Pi 3 B+ runs at about 1000 MIPS, and has 1Gbyte of RAM. The Google Compute f1-micro uses 0.2 of a ~1 Ghz CPU and has 0.60 Gbytes (600 Mbytes) of RAM, making it a reasonable fit.

I’ve been building tools to allow anyone to install SIMH and any of these operating systems in the cloud, so that they can be experienced, studied and understood, without having to use dedicated hardware, or understand the details of using GCP, or SIMH.

In this series of posts, I’ll introduce how I’m using GCP (with scripting), a little about SIMH, a little bit about the hardware being emulated, and the historical operating systems and how to run them all in the GCP cloud, pretty much for free.

You should start by looking into Google Cloud Platform (GCP) and using some of their tutorials.

All of the SIMH examples I will show are running Ubuntu Linux on tiny or small GCP instances.

You can get started by reading about SIMH on Wikipedia, at the main SIMH web site, or at the Github repository for the software.

, , ,

Leave a comment

%d bloggers like this: