Jack Whitham - Virtual Lab - Board Server Hardware

		JACK WHITHAM PhD MEng Professional Activities - Publications - Software - Articles

		Home -> Software -> Virtual Lab -> Board Server Hardware


	On this page: Board Server Hardware Using EDK Baseboard LEDs MAC UARTLite Teaports JTAG GPIOs Console switch Flash Boot loader Programming the FX12 Platform Flash Wiring for the Prototypes


	VL2 documentation: Introduction Administrating VL2 Authorisation File Board Server Board Server Hardware Debug Device Design Goals Downloads EMS Client Extending VL2 Mutual Exclusion Daemon PPC Linux for FX12 FPGAs Relay Shell Using VL2 vlab Module vlab Protocol vlabif Module vlabif Protocol vlabifhw

Board Server Hardware

The board server hardware is hosted by a Xilinx FX12 FPGA, which contains a PowerPC 405 CPU. The FX12 mini module that we are using (left) is sold by Avnet: here is the product page. It is a tiny circuit board comprising a Xilinx Virtex-4 FPGA, SDRAM, Flash and an Ethernet header. The board is capable of running a full Linux 2.6 system on the PowerPC CPU within the FPGA, which is documented elsewhere.

The bit file for this FPGA is currently built using Xilinx ISE/EDK 10.1 service pack 3. Based on past experience, it is quite likely that future versions of ISE/EDK will not be able to generate the hardware correctly due to incompatibilities introduced by upgrades. For this reason, you should maintain a "frozen" ISE/EDK 10.1 installation that will never be upgraded.

The hardware consists mostly of IP cores provided by Xilinx and is generated entirely by EDK. It was originally based on an Avnet reference design for FX12. There are two custom IP cores. Here is the memory map:

Device type	Instance name	Base address	High address	Note
RAM	DDR_SDRAM_32Mx16	0x00000000	0x03ffffff
GPIO	BaseBoard_LEDs_3Bit	0x81400000	0x8140ffff	Note
Int Cntlr	xps_intc_0	0x81800000	0x8180ffff
MAC	TriMode_MAC_GMII	0x81c00000	0x81c0ffff	Note
UARTLite	RS232	0x84000000	0x8400ffff	Note
Custom	tp0 (UART 0)	0x8f000000	0x8f00ffff	Note
Custom	tp1 (UART 1)	0x8f010000	0x8f01ffff
Custom	tp2 (UART 2)	0x8f020000	0x8f02ffff
Custom	tp3 (UART 3)	0x8f030000	0x8f03ffff
GPIO	JTAG_Host_In	0x8f100000	0x8f10ffff	Note
GPIO	JTAG_Host_Out	0x8f110000	0x8f11ffff	Note
GPIO	Relay_Out	0x8f120000	0x8f12ffff
Custom	consoleswitch_0	0x8f200000	0x8f20ffff	Note
Flash	FLASH_2Mx16	0xff800000	0xffbfffff	Note
Block RAM	xps_bram_if_cntlr_1	0xfffff000	0xffffffff	Note

Using EDK

To rebuild the FX12 bit file within EDK, obtain the file fx12-linux-hardware.zip from the download page and unpack it in a suitable location. Open the system.xmp project file using EDK. Then follow this procedure:

Select: Project -> Clean All Generated Files.
Select: Software -> Software Platform Settings. Ensure that "OS" is standalone. Do not select any of the libraries (xilmfs, xilflash, etc.). Click "OS and Libraries" and ensure that stdout and stdin are both set to RS232. Click Ok.
This step seems counter intuitive, because the intention is to run Linux on the platform. However, the first program that runs will be the boot loader, which is "standalone" and built into FPGA block RAM.
Select: Hardware -> Clean Netlist.
This step has the side effect of generating the Makefile.
Launch an EDK Shell (using the Start Menu group for EDK on Windows, or by sourcing the EDK settings.sh file on Linux). Move to the project directory and run:
```
        make -f system.make init_bram
```
This step is also counter intuitive, because the same operation is supposed to be provided by Update Bitstream in the Device Configuration menu. However, ISE/EDK 10.1 has some bugs that make some menu options unreliable. These bugs tend to mask real errors with GUI exceptions. Therefore, it is best to build the project at the command line.
When the process completes, use project dir/implementation/download.bit as the bit file.

If you add new devices or change the memory map, you have to make a new version of the Linux kernel. For this, you need to generate a new "fdt.dts" file in EDK. The procedure is as follows:

Ensure that you have installed "gen-mhs-devtree" in EDK. This is included in fx12-linux-software.zip from the download page.
Select: Software -> Software Platform Settings. Change "OS" to fdt. Click Ok.
On the "OS and Libraries" page, change "bootargs" to read "console=ttyUL0 root=/dev/ram0 init=/sbin/init rw".
Select: Software -> Generate Libraries and BSPs.
When the process completes, search for fdt.dts in the project directory, and copy it to the Linux kernel directory linux-2.6-xlnx/arch/powerpc/boot/dts. You will need this in order to build the Linux kernel.
Consider restoring the standalone OS settings (see above) to avoid the risk of building a bit file without a working boot loader.

Baseboard LEDs

The three baseboard LEDs are used to report the status of the system. The codes vary depending on the baseboard being used:

LED State						Meaning
Avnet FX12 Prototyping Baseboard			In-House Virtex 4 Motherboard
DS1	DS2	DS3	J261	J263	J264
off	off	off	on	on	on	Undefined
off	off	on	on	off	on	Boot loader running (waiting 10 seconds before starting kernel) OR Embedded board server software starting
off	on	off	on	on	off	Boot loader expecting Intel Hex records OR Embedded board server software has exited abnormally
off	on	on	on	off	off	Linux kernel is booting
on	off	off	off	on	on	User space has been reached - init scripts are running
on	off	on	off	off	on	User space was reached, but either the network update failed, or the Flash memory was not initialised with configuration data
on	on	off	off	on	off	Embedded software is doing something
on	on	on	off	off	off	Embedded software is waiting for a command

MAC

The FX12 FPGA includes two Ethernet MACs which can operate at 10MB/s, 100MB/s and 1000MB/s speeds. (Note: the software driver has only been tested at 100MB/s.) One of these MACs is connected to the RJ45 socket on the FX12 board, and the custom Linux kernel has been built with the appropriate driver. The driver uses "local link FIFO" buffers to store received data and data waiting to be sent. Xilinx doesn't support this configuration any more; it has been deprecated in favour of DMA operation, but a memory corruption bug may exist in the current LLTEMAC DMA code, so FIFO mode is used instead.

Version 1.01.b of the xps_ll_temac IP core is used. Earlier versions have an packet loss bug that is triggered by unrecognisable packets, such as the 802.1d messages generated by some Ethernet switches. The bug discards the packet following the 802.1d packet, causing problems with virtually all network connections.

The IP and MAC address of the Ethernet connection can be set by editing /jffs2/network.

UARTLite

This component is the system console; it is connected to the serial port on the FX12 prototyping board (if available). The serial settings are fixed: 115200 baud, with 8 bits, no parity and 1 stop bit.

When the FX12 bit file is loaded, the boot loader uses this component to display a boot message and receive Intel Hex records if booting from Flash is prevented. As the Linux kernel boots, it writes debugging information to this component. Once Linux has started, a user can log in using a terminal connected to the serial port. The Linux device name for this component is /dev/ttyUL0 (major 207, minor 187).

UARTLite has to be used during initial setup of an FX12 board because an Intel Hex record download is the only way to boot the Linux kernel when the Flash is empty. Once Linux has booted, the Flash can be loaded with the kernel and root file system.

The physical port that is connected to UARTLite on bootup can be exchanged with the physical port of UART 1. This allows both of the physical RS232 ports on the Virtex 4 motherboard to be used as virtual lab UART ports. To activate this functionality, run fx12tool switch within a bootup script, e.g. /jffs2/custom.

Teaports

Four UARTs are used for communication between the FX12 board and virtual lab FPGAs. These UARTs support standard baud rates below 500000 baud, and some other non-standard baud rates. The Linux device names of these components are /dev/ttyS0 through /dev/ttyS3: the major and minor numbers match the Linux standard. The supported baud rates are:

Baud rate	Baud rate
300	115200
600	230400
1200	460800
1800	500000
2400	568181
4800	892857*
9600	1041666*
19200	1562500*
38400	2083333*
57600	3125000*'
	6250000*^

* = this baud rate may require a special serial cable.
' = not recommended if more than two FPGA boards are in use simultaneously.
^ = not recommended if more than one FPGA board is in use.

In early versions of the FX12 system, these UARTs were 16550A-compatible devices (instances of the xps_uart16550 component from Xilinx). However, the 16550A design has two significant disadvantages: (1) the FIFO size is a puny 16 bytes, and (2) every byte read from the UART requires two entire words to cross the data bus, creating a total overhead of at least 64 clock cycles per byte. This makes the 16550A unsuitable for operation above 115200 baud; and even at that relatively low speed, data was occasionally lost when more than one UART was active.

To support higher speeds, a custom UART named Teaport was implemented. It has its own page. The design uses a block RAM as a FIFO, giving 1kb send and receive buffers. It also permits data and status bits to be fetched for three bytes simultaneously, speeding up the interrupt service routine by a factor of six. Its disadvantage is a poorer feature set: it only supports "8 bit, no parity, 1 stop bit", cannot use RTS or CTS, and does not send breaks.

JTAG GPIOs

These components are used for programming FPGAs via JTAG. They are standard Xilinx GPIOs; the JTAG-specific features are all implemented in kernel and user software. JTAG_Host_In is a single bit GPIO that is used to sample TDO. JTAG_Host_Out is a three bit GPIO that is used to generate TCK, TMS and TDI. The pin assignments are set in jtag.h in the board server source code:

TCK - Least significant bit
TMS
TDI - Most significant bit

All GPIOs are controlled using Linux character devices. The major and minor device ids assigned by the kernel are:

major 10 minor 185 - Baseboard LEDs
major 10 minor 186 - Baseboard DIP switches
major 10 minor 187 - JTAG_Host_Out
major 10 minor 188 - JTAG_Host_In
major 10 minor 189 - Relay_Out

JTAG programming can be carried out by using ioctl commands to manipulate these GPIOs. But this is slow, since every operation requires a system call. Therefore, a custom JTAG programming driver exists within the custom Linux kernel.

Console switch

This component allows the console port (UARTLite) to be exchanged with the ports for UART 1. It provides a single register which is 1 on reset, meaning that exchange is disabled. The fx12tool program can change this register to activate the exchange, allowing two FPGA boards to be connected to a single FX12 on a Virtex 4 motherboard. fx12tool can also make the change semi-permanent so that it survives reboots (but not FPGA reprogramming).

Flash

This component must not be confused with the platform Flash which holds the FPGA bit stream. This Flash memory is part of the system memory map and holds two things:

A 64kbyte read/write filesystem for configuration data, using the JFFS2 filesystem driver. Configuration data includes the MAC address, the IP address and the board server configuration. The root user can change these files once Linux has booted. The Linux device name is /dev/mtdblock0. A new filesystem can be created by using the following sequence of commands on the embedded system:
```
    mkdir /tmp/newfs
    cd /tmp/newfs
    touch dummyfile
    mkfs.jffs2 -v -o /tmp/newfs.bin
    flashcp -v /tmp/newfs.bin /dev/mtd0
```
Having initialised the filesystem in this way, you should mount it (mount /jffs2) then install and edit the system-specific configuration files.
A 4032kbyte data area for the Linux kernel and compressed root filesystem. The Linux device name is /dev/mtd2. A zImage.bin file (produced by the build-kernel script) should be loaded into this space by a command like:
```
    flashcp -v zImage.bin /dev/mtd2
```
Danger! Don't turn off the power during the flashcp process as this will damage the Flash memory.
You can use the wget utility to download zImage.bin to an FX12 board from any web server. thttpd is useful here.

Initially, the Flash memory is empty and unformatted. The best way to prepare a new system is to run /install, which makes the the /jffs2 filesystem, downloads zImage.bin, and flashcps it into the Flash memory. The only additional step is editing of the files in /jffs2 to set up the network and configure the VL2 software.

Having loaded a kernel into Flash, you won't need to send the kernel via the serial line again, unless you flashcp a buggy kernel that won't boot.

Boot loader

The boot loader is a short C program. It is not standard Xilinx software; it was written in-house for this project. This is because Xilinx boot loaders don't include functionality to allow programs to be loaded. Xilinx expects you to load programs by putting the PowerPC or Microblaze CPU into debug mode, then use debugging commands to write to memory. This isn't suitable in this environment because it would force you to connect each FX12 board to a PC running EDK for the initial setup. Therefore, like the Linux/Microblaze company Petalogix, we have developed our own boot loader that allows new programs to be loaded. (Petalogix code has not been reused due to licencing restrictions.)

The boot loader uses the UARTLite serial port to communicate with the user. It begins by printing a message. Then it looks for a bootable Linux kernel at memory address 0xff810000. It expects kernels to begin with the machine code:

      48 00 00 05     bl      ff810004 
      7c 13 ea a6     mfccr0  r0

If it sees this code, then it waits for 10 seconds. During this waiting time, you can press Enter to abort the boot process. When the boot process is aborted, the boot loader waits for Intel Hex records, which are executed according to the Intel Hex record specification. If a bootable kernel is not found, then the boot loader begins waiting for Intel Hex records immediately.

The build-kernel script produces both a binary kernel image (zImage.bin) and an Intel Hex version (zImage.hex). Once the boot loader is waiting for Intel Hex records, you should send the latter file. If you are using a Linux system as your terminal, then you can simply:

    cat zImage.hex >/dev/ttyS0

to do this. Otherwise, look for a "send ASCII file" feature in your terminal. When the file is being sent, a rotating - symbol will be displayed. When transmission is complete, the Linux kernel will boot.

Programming the FX12 Platform Flash

The FX12 board includes an XCF08P platform Flash IC that can store up to 8MB of bitfile data, which is loaded onto the FX12 FPGA on power up. Each FX12 bitfile uses about 4.7MB. A compression feature is available; this allows more than one bitfile to be stored on the IC, although it is not clear why this is useful in this particular application. The main benefit of compression is faster updates.

You'll need to program the XCF08P in two situations:

You're updating the hardware on a virtual lab board, perhaps to add a feature or fix a bug. In this case, you will have used EDK to generate a new "download.bit" file.
You're adding a new virtual lab board. In this case, you'll be reusing an existing "download.bit" file.

You won't need to program the XCF08P if you're just testing a new bitfile. You can program the FPGA directly in that case.

To program the XCF08P, plug the FX12 board into the Avnet/Memec prototyping board, then connect the prototyping board to a PC via a JTAG connector. Then set up the Xilinx IMPACT software on the PC. Version 8.1 is usable with service packs; later versions presumably also work. (Because IMPACT is required for many aspects of virtual lab maintenance, it is worth keeping a PC specifically for IMPACT.)

In IMPACT, select "Create a new project", then "Prepare a PROM file". On the next page, you want to "target a Xilinx PROM". Give the PROM file a suitable name and location. Select "Serial mode" if you get the option. On the next page, select "xcf08p" and click Add. Then click Finish. Now you add the "download.bit" file (produced by EDK) to the PROM. This is the "device file". IMPACT should tell you that the startup clock has been changed to "CCLK" - this is normal. You only have one device file to add, so don't attempt to add any more if IMPACT asks.

You should now see that "xcf08p" is about 56% full. Click "Generate File" (on the left). Once "PROM File Generation Succeeded", you have created three PROM programming files ending in .mcs, .cfi, and .sig. Click "Boundary Scan" (on the left) and then select "File -> Initialize Chain". Assign your new .mcs file to the XCF08P device, and click Bypass for the FPGA. Now click Program and ensure that "Erase before programming" is selected before pressing Ok. When programming is complete, power cycle the board. If you did it right, the bootloader message will be written to the prototyping board's serial port (at 115200 bps, 8N1).

At this point, you might want to download the Linux kernel and root filesystem to the separate software Flash memory.

Wiring for the Prototypes

Avnet FX12 Prototyping Baseboard		In-House Virtex 4 Motherboard
The figure below shows the wiring arrangements for the FX12 board in the prototype VL2 system. This setup exposes four UARTs and the JTAG interface. This wiring arrangement does not necessarily match that of any finalised version of VL2.		The figure below shows the wiring arrangements for the FX12 board on the Virtex 4 motherboard made in-house. This setup exposes one UART, the JTAG interface, and two relay outputs. The JTAG header has the pins shown below.


		Copyright (C) Jack Whitham 1997-2010