APM X-C1 (Mustang)
The APM X-C1 (code name Mustang) is a server chip under their X-GENE family of products.
First Look
U-Boot
The board came with U-Boot loaded. When the system is first turned on, you will see the following in the console:
U-Boot 2013.04-mustang_sw_1.12.09-beta (Jun 12 2014 - 10:48:45)
CPU0: APM ARM 64-bit Potenza Rev A3 2400MHz PCP 2400MHz
32 KB ICACHE, 32 KB DCACHE
SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Boot from SPI-NOR
Board: Mustang - AppliedMicro APM883208-xNA24SPT Reference Board
I2C: ready
DRAM: ECC 16 GiB @ 1600MHz
SF: Detected N25Q256 with page size 256 Bytes, total 32 MiB
MMC: X-Gene SD/SDIO/eMMC: 0
PCIE0: (RC) link down
In: serial
Out: serial
Err: serial
CPUs: 11111111
Net: eth0
Here are the environment variables that are set in the U-Boot shell out of the box:
Mustang# printenv
addip=setenv bootargs ${bootargs} ip=${ipaddr}:${serverip}:${gatewayip}:${netmask}:${hostname}:${netdev}:off panic=1
addmisc=setenv bootargs ${bootargs} earlyprintk=uart8250-32bit,0x1c020000 debug maxcpus=${num_cores} swiotlb=65536
addtty=setenv bootargs ${bootargs} console=ttyS0,${baudrate}
baudrate=115200
bootcmd=run sata_self
bootdelay=5
bootfile=uImage
dnsip=10.66.18.10
enet=8
enet_mode=x:n
eth1addr=00:01:73:02:21:35
eth2addr=00:01:73:02:21:36
eth3addr=00:01:73:02:21:37
ethact=eth0
ethaddr=00:01:73:02:21:34
fdt_addr_f=0x1000000
fdt_addr_r=0x4003000000
fdt_file=apm-mustang.dtb
fdt_size=7285
flash_nfs=run nfsargs addip addtty addmisc; bootm ${kern_addr_f}
flash_self=run ramargs addip addtty addmisc; bootm ${kern_addr_f} ${ramdisk_addr_f} ${fdt_addr_f}
fw_addr_r=0x4003800000
gatewayip=10.66.19.1
hostname=dhcp-10-66-19-173
initrd_high=0x43e0000000
ipaddr=10.66.19.173
kern_addr_f=0x100000
kern_addr_r=0x4002000000
kern_size=a9a050
kernel_addr_r=0x4002000000
load=tftp ${uboot_addr_r} ${user_dir}/${u-boot}
load_fdt=tftp ${fdt_addr_r} ${user_dir}/${fdt_file}
load_kern=tftp ${kern_addr_r} ${user_dir}/${bootfile}
load_ramdisk=tftp ${ramdisk_addr_r} ${user_dir}/${ramdisk_file}
loadaddr=0x100000000
media_addr_r=0x4001000000
media_img=mustang_media.img
mgmt_enet=e
mslim_bootfile=uImage-mslim
mslim_fdt_addr_r=0x43f0600000
mslim_fdt_file=mslim-mustang.dtb
mslim_kern_addr_r=0x43f1000000
mslim_net_self=run net_load_mslim; startmslim
mslim_ramdisk_addr_r=0x43f2000000
mslim_ramdisk_file=uRamdisk-mslim
mslim_sd_self=run sd_load_mslim; startmslim
mslim_uboot_addr_r=0x43f0000000
mslim_uboot_file=u-boot-mslim.bin
mslim_user_dir=mslim
net_load_mslim=run nload_mslim_uboot nload_mslim_kern nload_mslim_ramdisk nload_mslim_fdt
net_nfs=run load_kern load_fdt; run nfsargs addip addtty addmisc; bootm ${kern_addr_r} - ${fdt_addr_r}
net_sata=run load_kern load_fdt; run sataargs addip addtty addmisc; bootm ${kern_addr_r} - ${fdt_addr_r}
net_self=run net_self_load; run ramargs addip addtty addmisc; bootm ${kern_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
net_self_load=run load_kern load_ramdisk load_fdt
netdev=eth0
netmask=255.255.255.0
nfsargs=setenv bootargs root=/dev/nfs rw nfsroot=${serverip}:${rootpath}
nload_mslim_fdt=tftp ${mslim_fdt_addr_r} ${mslim_user_dir}/${mslim_fdt_file}
nload_mslim_kern=tftp ${mslim_kern_addr_r} ${mslim_user_dir}/${mslim_bootfile}
nload_mslim_ramdisk=tftp ${mslim_ramdisk_addr_r} ${mslim_user_dir}/${mslim_ramdisk_file}
nload_mslim_uboot=tftp ${mslim_uboot_addr_r} ${mslim_user_dir}/${mslim_uboot_file}
num_cores=8
pcie_gen=3:3:3:3:3
pcie_mode=RP:NA:NA:NA:NA
pcie_reset=y:n:n:n:n
pxefile_addr_r=0x4001000000
ram_nfs=run nfsargs addip addtty addmisc; bootm ${kern_addr_r} - ${fdt_addr_r}
ram_self=run ramargs addip addtty addmisc; bootm ${kern_addr_r} ${ramdisk_addr_r} ${fdt_addr_r}
ramargs=setenv bootargs root=/dev/ram rw
ramdisk_addr_f=0x1100000
ramdisk_addr_r=0x4004000000
ramdisk_file=uRamdisk_basic
ramdisk_size=344abe
rgmii_phyid=3
rootpath=/opt/nfsrootfs/arm64/release
sata=0x3c
sata_enet=e:e:n:n
sata_load_fdt=ext4load scsi 0:1 0x4003000000 apm-mustang.dtb
sata_load_kern=ext4load scsi 0:1 0x4002000000 uImage
sata_self=scsi init; run sata_load_kern sata_load_fdt; run sataargs addip addtty addmisc; bootm 0x4002000000 - 0x4003000000
sataargs=setenv bootargs root=/dev/sda1 rw
script_addr_r=0x4004000000
sd_load_mslim=mmc dev 0; run sdload_mslim_uboot sdload_mslim_kern sdload_mslim_ramdisk sdload_mslim_fdt
sd_self=mmc dev 0; ext2load mmc 0 ${kern_addr_r} ${bootfile}; ext2load mmc 0 ${fdt_addr_r} ${fdt_file}; ext2load mmc 0 ${ramdisk_addr_r} ${ramdisk_file}; run ram_self
sdload_mslim_fdt=ext2load mmc 0 ${mslim_fdt_addr_r} ${mslim_fdt_file}
sdload_mslim_kern=ext2load mmc 0 ${mslim_kern_addr_r} ${mslim_bootfile}
sdload_mslim_ramdisk=ext2load mmc 0 ${mslim_ramdisk_addr_r} ${mslim_ramdisk_file}
sdload_mslim_uboot=ext2load mmc 0 ${mslim_uboot_addr_r} ${mslim_uboot_file}
sdphy=1
serverip=10.66.19.133
sgmii_phyid=4:5::
spi_fdt_upd=run load_fdt; setenv fdt_size ${filesize}; save; run update_fdt
spi_fw_img=mustang_slimpro_ext_spi.bin
spi_fw_load=tftp ${fw_addr_r} ${user_dir}/${spi_fw_img}
spi_fw_upd=run spi_fw_load;run spi_fw_update
spi_fw_update=bmc stop; i2c dev 1; eeprom write ${fw_addr_r} 0x0 ${filesize}
spi_kern_upd=run load_kern; setenv kern_size ${filesize}; save; run update_kern
spi_load=tftp ${media_addr_r} ${user_dir}/${media_img}
spi_ramdisk_upd=run load_ramdisk; setenv ramdisk_size ${filesize}; save; run update_ramdisk
spi_self=sf probe 0; sf read ${kern_addr_r} ${kern_addr_f} ${kern_size}; sf read ${fdt_addr_r} ${fdt_addr_f} ${fdt_size}; sf read ${ramdisk_addr_r} ${ramdisk_addr_f} ${ramdisk_size}; run ram_self
spi_upd=run spi_load;run spi_update
spi_update=sf probe 0; sf erase 0x0 ${filesize}; sf write ${media_addr_r} 0x0 ${filesize}
stderr=serial
stdin=serial
stdout=serial
u-boot=u-boot.bin
uboot_addr_f=0x8100000000
uboot_addr_r=0x4001000000
uboot_size=0x80000
upd=run load; setenv uboot_size ${filesize};run update
updall=run spi_kern_upd; run spi_ramdisk_upd; run spi_fdt_upd
update=protect off ${uboot_addr_f} +${uboot_size}; erase ${uboot_addr_f} +${uboot_size}; cp.b ${uboot_addr_r} ${uboot_addr_f} ${uboot_size}
update_fdt=sf probe 0; sf erase ${fdt_addr_f} ${filesize}; sf write ${fdt_addr_r} ${fdt_addr_f} ${filesize}
update_kern=sf probe 0; sf erase ${kern_addr_f} ${filesize}; sf write ${kern_addr_r} ${kern_addr_f} ${filesize}
update_ramdisk=sf probe 0; sf erase ${ramdisk_addr_f} ${filesize}; sf write ${ramdisk_addr_r} ${ramdisk_addr_f} ${filesize}
user_dir=/1.12.09-Beta/BUILDS/potenza
xgmii_phyid=:
Environment size: 5832/32764 bytes
The system comes with their version of Linux installed. It's using a custom version of the 3.12 Linux Kernel. Interestingly, the hostname does not reflect the actual IP address, so most likely it's the IP from a network install from the factory.
POWERED BY:
X X GGGG
X X G G
X X G EEEEE N NNNN EEEEE
X ####### G GGG E E NN N E E
X X G G EEEEEEE N N EEEEEEE
X X G G E N N E
X X GGGGG EEEEE N N EEEEE (TM)
Welcome to Applied Micro X-Gene 64-bit ARM Server Platform
Applied Micro Linux 3.12.0-mustang_sw_1.12.09-beta (aarch64) (ttyS0)
dhcp-10-66-19-173 login: root
Password:
[root@dhcp-10-66-19-173 ~]# cat /proc/cpuinfo
Processor : AArch64 Processor rev 0 (aarch64)
processor : 0
model name : APM X-Gene Mustang board @ 2.40GHz
cpu MHz : 2400.00
fpu : yes
processor : 1
model name : APM X-Gene Mustang board @ 2.40GHz
cpu MHz : 2400.00
fpu : yes
processor : 2
model name : APM X-Gene Mustang board @ 2.40GHz
cpu MHz : 2400.00
fpu : yes
processor : 3
model name : APM X-Gene Mustang board @ 2.40GHz
cpu MHz : 2400.00
fpu : yes
processor : 4
model name : APM X-Gene Mustang board @ 2.40GHz
cpu MHz : 2400.00
fpu : yes
processor : 5
model name : APM X-Gene Mustang board @ 2.40GHz
cpu MHz : 2400.00
fpu : yes
processor : 6
model name : APM X-Gene Mustang board @ 2.40GHz
cpu MHz : 2400.00
fpu : yes
processor : 7
model name : APM X-Gene Mustang board @ 2.40GHz
cpu MHz : 2400.00
fpu : yes
Features : fp asimd
CPU implementer : 0x50
CPU architecture: AArch64
CPU variant : 0x0
CPU part : 0x000
CPU revision : 0
Hardware : APM X-Gene Mustang board
Installing Fedora
Before you begin, read through the guide on the FedoraProject wiki: https://fedoraproject.org/wiki/Architectures/AArch64/Hardware/Mustang?rd=Architectures/ARM/AArch64/Mustang
To install Fedora on the system, we need to get the UEFI loader running first. You can either chainload this from U-Boot by having U-Boot run the UEFI loader from the SD card on start-up, or flash it directly to the system and replace the U-Boot loader altogether. Since I don't want to replace U-Boot, I will chainload it during startup.
Once the UEFI loader is installed and is made to start automatically, you will need to install Fedora from either USB or PXE. USB will be easier to do unless you already have a PXE server setup to handle aarch64 and UEFI.
Preparing the USB Stick
The FedoraProject wiki has instructions on setting up the USB stick in addition to a USB image that you dd to the drive. Unfortunately the instructions and image exist only for Fedora 21 and not the newest version, Fedora 22.
To install Fedora 22, you will need to make a few changes to the USB image. First, download the USB image Fedora-Server-netinst-aarch64-21.img.xz and extract the archive. The disk image file will contain two partitions:
Disk Fedora-Server-netinst-aarch64-21.img: 110 MiB, 115343360 bytes, 225280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 70AE0993-389E-4DB6-9455-DB9A75729515
Device Start End Size Type
Fedora-Server-netinst-aarch64-21.img1 2048 22527 10M EFI System
Fedora-Server-netinst-aarch64-21.img2 22528 225246 99M Linux filesystem
Mount the first partition:
mount -o loop,offset=$((22528*512)) /tmp/Fedora-Server-netinst-aarch64-21.img /mnt/usb
Then replace the kernel and initrd image under /images/pxeboot with the Fedora 22 versions.
Mount the second partition and update the grub.cfg file so that it points to the Fedora 22 HTTP repository. (You may also change this when grub runs later on, but you'll have to do it every single time you start grub)
Unmount the filesystem and dd the image to the USB drive.
Chain-Loading UEFI
Download mustang_uefi_upgrade.tar.gz and dump the contents onto a 2GB FAT16 formatted SD card partition. U-Boot doesn't appear to see the partition if it is any larger than 2GB. Insert the SD card into the X-C1 board and turn it on. Enter the U-Boot shell and run the following:
U-Boot 2013.04-mustang_sw_1.12.09-beta (Jun 12 2014 - 10:48:45)
CPU0: APM ARM 64-bit Potenza Rev A3 2400MHz PCP 2400MHz
32 KB ICACHE, 32 KB DCACHE
SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Boot from SPI-NOR
Board: Mustang - AppliedMicro APM883208-xNA24SPT Reference Board
I2C: ready
DRAM: ECC 16 GiB @ 1600MHz
SF: Detected N25Q256 with page size 256 Bytes, total 32 MiB
MMC: X-Gene SD/SDIO/eMMC: 0
PCIE0: (RC) link down
In: serial
Out: serial
Err: serial
CPUs: 11111111
Net: eth0
Hit any key to stop autoboot: 0
Mustang# setenv rh_sd_tiano_sec_ubt fatload mmc 0:1 0x1d000000 /mustang/mustang_tianocore_sec_ubt.fd
Mustang# setenv rh_sd_tiano_ubt fatload mmc 0:1 0x4002000000 /mustang/mustang_tianocore_ubt.fd
Mustang# setenv rh_run_tiano_ubt go 0x1d000000
Mustang# setenv bootcmd run rh_sd_tiano_ubt rh_sd_tiano_sec_ubt rh_run_tiano_ubt
Mustang# saveenv
Saving Environment to SPI Flash...
SF: Detected N25Q256 with page size 256 Bytes, total 32 MiB
.Erasing SPI flash.....SF: STMicro: Successfully erased 262144 bytes @ 0xc0000
Writing to SPI flash.......done
Mustang# reset
resetting ...
cold reset ...▒
U-Boot will now run run rh_sd_tiano_ubt rh_sd_tiano_sec_ubt rh_run_tiano_ubt everytime it boots by default, which loads the UEFI loader from the SD card.
Installing Fedora
Insert the Fedora USB stick to the first USB port (the one nearest the board under the Ethernet jack). When the UEFI loader starts, enter the shell.
## Starting application at 0x1D000000 ...
X-Gene Mustang Board
Boot firmware (version 1.1.0 built at 12:03:43 on Mar 23 2015)
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3020002 I0
PROGRESS CODE: V3020003 I0
PROGRESS CODE: V3021001 I0
TianoCore 1.1.0 UEFI 2.4.0 Mar 23 2015 12:03:18
CPU: APM ARM 64-bit Potenza Rev A3 2400MHz PCP 2400MHz
32 KB ICACHE, 32 KB DCACHE
SOC 2000MHz IOBAXI 400MHz AXI 250MHz AHB 200MHz GFC 125MHz
Board: X-Gene Mustang Board
Slimpro FW:
Ver: 2.1
The default boot selection will start in 4 seconds
[1] Shell
[2] Boot Manager
[3] Reboot
[4] Shutdown
Start: 1
In the UEFI shell, execute:
FS1:\EFI\BOOT\BOOTAA64.EFI
FS0 should be the SD card. If you aren't sure, enter the drive by running FS0: and list the directories using dir.
The BOOTAA64.EFI program will eventually start grub and from there the Fedora installer. Continue the Fedora installation via VNC. You should use auto partitioning since it will automatically create the necessary EFI partition for the UEFI loader to work properly.
Troubleshooting
For some reason, the Fedora 22 kernel (4.0.4-303.fc22.aarch64) is unable to bring the other CPUs online on startup:
[ 0.018839] ------------[ cut here ]------------
[ 0.018853] WARNING: CPU: 0 PID: 1 at drivers/firmware/dmi_scan.c:649 dmi_matches+0x120/0x128()
[ 0.018861] dmi check: not initialized yet.
[ 0.018866] Modules linked in:
[ 0.018874] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.0.4-303.fc22.aarch64 #1
[ 0.018882] Hardware name: APM X-Gene Mustang board (DT)
[ 0.018887] Call trace:
[ 0.018894] [<fffffe0000097738>] dump_backtrace+0x0/0x150
[ 0.018901] [<fffffe00000978a8>] show_stack+0x20/0x30
[ 0.018909] [<fffffe000076e9b4>] dump_stack+0x78/0x94
[ 0.018915] [<fffffe00000c0888>] warn_slowpath_common+0xa0/0xe0
[ 0.018921] [<fffffe00000c0928>] warn_slowpath_fmt+0x60/0x78
[ 0.018927] [<fffffe00005de86c>] dmi_matches+0x11c/0x128
[ 0.018933] [<fffffe00005de8a4>] dmi_check_system+0x2c/0x70
[ 0.018940] [<fffffe0000c4f5e0>] lockup_detector_init+0x10/0x54
[ 0.018946] [<fffffe0000c40abc>] kernel_init_freeable+0x108/0x260
[ 0.018953] [<fffffe000076b76c>] kernel_init+0x1c/0xe8
[ 0.018967] ---[ end trace 514f64f4445b269e ]---
[ 0.018973] ------------[ cut here ]------------
[ 0.018979] WARNING: CPU: 0 PID: 1 at drivers/firmware/dmi_scan.c:649 dmi_matches+0x120/0x128()
[ 0.018987] dmi check: not initialized yet.
[ 0.018991] Modules linked in:
[ 0.018998] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 4.0.4-303.fc22.aarch64 #1
[ 0.019005] Hardware name: APM X-Gene Mustang board (DT)
[ 0.019010] Call trace:
[ 0.019015] [<fffffe0000097738>] dump_backtrace+0x0/0x150
[ 0.019021] [<fffffe00000978a8>] show_stack+0x20/0x30
[ 0.019027] [<fffffe000076e9b4>] dump_stack+0x78/0x94
[ 0.019033] [<fffffe00000c0888>] warn_slowpath_common+0xa0/0xe0
[ 0.019038] [<fffffe00000c0928>] warn_slowpath_fmt+0x60/0x78
[ 0.019044] [<fffffe00005de86c>] dmi_matches+0x11c/0x128
[ 0.019051] [<fffffe00005de8a4>] dmi_check_system+0x2c/0x70
[ 0.019057] [<fffffe0000c4f5e0>] lockup_detector_init+0x10/0x54
[ 0.019063] [<fffffe0000c40abc>] kernel_init_freeable+0x108/0x260
[ 0.019068] [<fffffe000076b76c>] kernel_init+0x1c/0xe8
[ 0.019073] ---[ end trace 514f64f4445b269f ]---
[ 1.018060] CPU1: failed to come online
[ 2.018179] CPU2: failed to come online
[ 3.018300] CPU3: failed to come online
[ 4.018418] CPU4: failed to come online
[ 5.018544] CPU5: failed to come online
[ 6.018662] CPU6: failed to come online
[ 7.018782] CPU7: failed to come online
[ 7.018797] Brought up 1 CPUs
[ 7.018803] SMP: Total of 1 processors activated.
The CPUs will not come online post-startup either.
# echo 1 > /sys/devices/system/cpu/cpu1/online -bash: echo: write error: Input/output error # lscpu Architecture: aarch64 Byte Order: Little Endian CPU(s): 8 On-line CPU(s) list: 0 Off-line CPU(s) list: 1-7 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1