An Introduction to Fermilab
Accelerator Controls
December 1994 (revised 1/16/07)
This document will introduce the new user and new programmer to the
Fermilab Accelerator Control System. Part I describes its architecture
and key components. Part II will get a programmer started on writing
application programs for the end-user. This document presents simplified
descriptions with details omitted.
There is a group of Linux machines provided for code development of console
applications and for using several of the control system public facilities.
The discussion here assumes that you are a novice user on these machines and
that you have access to an accelerator control console.
Contents
Part I Control System Overview
Architecture a rough sketch
Network communication protocol
Front-ends connection to the real world
Centrals shared resources
Consoles man-machine interface
Devices logical atoms
Hardware Links some special data links
Part II Console Application Programming
Code Librarian application code development cycle
Application Shell the code environment
Programmer Tools console library and its routines
User Interface text manipulation and user dialog
Data Acquisition communicating with devices
Data Storage using files and databases
Graphics drawing pictures
Getting Started an on-line programming introduction
Appendices
Accelerators an overview of the Fermilab accelerators
Part I Control System Overview
Architecture
The control system consists of four main components: front-ends, centrals,
consoles and the network that ties them all together. Front-ends are
the computers that bridge between the hardware (e.g. power supply control
cards, analog read-backs...etc.) and the rest of the control system.
Centrals are the machines that provide centralized, shared tasks such as
databases, shared files, alarm reporting...etc. Consoles are the computers
that an end-user (e.g. an Accelerator Operator) uses to control the
accelerator. They provide a user interface to present accelerator data.
Consoles communicate with the front-ends and the centrals to acquire and
scale data. The network is the hardware and software that connect
consoles, centrals and front-ends; allowing them a means to communicate
with each other.
Network
There are hardware (and software protocol) links that connect up the
front-ends, centrals and consoles. Ethernet is the network protocol
used (see attached diagram). There are protocols layered on top of
these to provide the needed communication presentation. The most common
protocol is called ACNET (ACcelerator NETwork). ACNET is a protocol that
provides communications between any cooperating tasks. (Often the term ACNET
is used loosely to refer to all, or some part, of the control system.)
An ACNET task can send messages to any other ACNET tasks. The message can
be: a single request with an expected reply, a single request with many
expected replies or a single request with no reply expected. ACNET contains
many attributes that allow for reliable, connection-less communication
throughout the control system. ACNET tasks connect to ACNET itself and
can send and receive (and ignore) messages from any other ACNET-connected task.
The tasks do not have a point-to-point connection to each other.
Front-ends
A front-end is the machine and software used to bridge between hardware
and the rest of the control system. It contains software (called drivers)
that knows how to talk to the hardware control equipment. A front-end
typically contains a common set of services:
service purpose
---------------------------------------------------------------------
RETDAT reading data requests
SETDAT setting data requests
GETS32 reading data requests (new protocol)
SETS32 setting data requests (new protocol)
FTPMAN Fast Time Plot Manager (handle time plot requests)
ALARMS setting alarm limits
ALARMR scanning for alarms
ACNET ACNET itself
ACNAUX ACNET auxiliary
Not all front-ends have all these services, but a minimum set would contain
RETDAT, SETDAT and ACNET functionality. Front-ends receive data acquisition
requests from processes called Data Pool Managers on other machines. Requests
are clumped by a common requested frequency or global clock event (see T-clock
below). They also emit alarm messages to inform the central alarm task of
any devices in an alarm state.
Part I Control System Overview
Centrals
The centrals are machines that run tasks needed
by consoles and front-ends. These tasks include things like shared file
access, databases (including the database that contains scaling information
for devices), alarm broadcasting...etc. These tasks are maintained by the
Controls Department and are expected to always be running. In addition,
there are a set of special front-ends, called Open Access front-ends (OACs).
OACs are comprised of tasks that run on centrals. They may or may not address
hardware. OACs allow for calculated devices (devices derived from the value
of existing hardware devices) and for devices accessed through protocols
different from what traditional front-ends may use.
What runs on the central machines? Type "acnet show" and see.
Below is a typical output from "acnet show". The running processes are listed
and if any were missing, they would also be listed. This output has been
annotated with a short description of what each process does (see the comment
to the right of the '!' marker).
centra$ acnet show
PID NI %CPU VSZ RSS STAT START COMMAND
7705 0 0.0 4288 1056 Ss 2007 acnetd ! ACNET itself
7727 0 0.0 78072 13664 Sl 2007 dpm ! Data Pool Manager (data acquisition)
7744 0 0.0 34124 12764 Sl 2007 clockpeek ! T-clock receiver
8914 0 0.0 302816 21124 Sl Jan01 alarms ! broadcast alarms
7774 0 0.0 56304 16496 Sl 2007 almlog ! alarm logger
28681 0 0.0 64284 12576 Sl 2007 cpld ! Central Program Loader
7244 0 0.0 66816 10820 Ssl Jan07 dnldd ! front end setting downloader
7827 0 0.0 88472 15468 Sl 2007 fshare ! file sharing service
7852 0 0.0 94700 13264 Ssl 2007 nodesd ! ACNET node information server
7865 0 0.0 84672 13708 Sl 2007 sybset ! forwards settings to the database
16583 0 0.0 144556 22340 Sl 2007 tcport ! export ACNET data
Part I Control System Overview
Consoles
A console is the machine and system software that provide the human
interface to the accelerator. There are a fixed number of processes that
define a console including several system tasks and a fixed number of
application programs. The applications run in what is commonly called a SLOT.
Slot addressing provides a fixed environment for the user; and is maintained
for continuity with the past console architecture. A console is more exactly
defined as an INSTANCE. An INSTANCE is just a fixed set of processes. One
machine can support up to three INSTANCES. A console can have multiple
screens (used in Main Control Room) or a single screen (used in an office).
They take keyboard, mouse, and optionally knob, input.
There are two main kinds of applications that run on a console: Primary
Application (PA) and Secondary Application (SA). PAs run with character-based
windows and take keyboard input. SAs runs with graphics based windows and
usually do not use keyboard input. There are 3 PAs and 3 SAs allowed to
run on a given console (INSTANCE). PAs are started from a special console
application called the Index Page. The Index Page is just a menu of PAs with
their titles. The Index Page is the application running if no other PA is
running. SAs are started from an existing PA. PAs also have the ability to
use two dedicated screens which are pixel based, for graphics and text display.
There are several applications of general use worth noting:
page purpose
-----------------------------------------------------------------------------
Index Pages the index page has many useful features; see its 'CMDS' menu;
also try the 'Do' key to see its menu
D-index the D-index page, in particular, has many programmer tools
D15 display console information; program modification history...etc.
D10 allows edit of a static TV image associated with a PA
D7 allows testing of SAs
D22 allows debugging of applications by viewing CLIB routines
D33 display of T-clock events
D80 supports viewing/editing device database information
-----------------------------------------------------------------------------
What's running on a console? From any console terminal window, type "cnsshow"
and see. Below is a typical output from "cnsshow". This output has been
annotated with a short description of what each process does (see the
comment to the right of the '!' marker). Note that the output also
shows the node where the console is displayed and the user who is running
that console INSTANCE. This output shows a machine running three console
instances.
Part I Control System Overview
Consoles
PID NI %CPU VSZ RSS STAT START COMMAND
17686 0 0.0 3280 1064 Ss Jan09 acnetd ! ACNET itself
17741 0 0.0 33400 11372 Sl Jan09 clockpeek ! T-clock receiver
17706 0 0.0 76736 12312 Sl Jan09 dpm ! Data Pool Manager (data acquisition)
8964 0 0.0 55220 11888 Sl 01:05 pld ! Program Loader: loads images
17760 0 0.0 69480 13908 Sl Jan09 alarms_daemon ! console alarm client
3259 0 0.0 3332 940 S 2007 /bin/bash ./cnswatcher.sh ! console maintenance
-- ahn ---------- X8734 -- minotaur:0.0 --------- Up 2.38D Load nnnnn Idle 3s
23868 0 0.0 35256 12204 Sl Jan14 tvmgr_0 ! I/O for character-based windows
23905 0 0.0 75488 12672 Sl Jan14 lxmgr_0 ! I/O for graphics windows
27778 0 0.0 95196 12268 Sl Jan15 PA0001_01 ! Index page (slot 1)
23968 0 0.0 75304 11468 Sl Jan14 SA0001_02 ! NULL SA (slot 2)
24000 0 0.0 75416 11464 Sl Jan14 SB0001_03 ! NULL SA (slot 3)
25222 0 0.0 118668 12876 Sl 09:22 PE1661_04 ! alarm display
12046 0 0.0 96276 12256 Sl Jan14 PB0001_05 ! Index page (slot 5)
24099 0 0.0 96348 12692 Sl Jan14 UTILI0_06 ! Utility window (slot 6)
24131 0 0.0 76168 11464 Sl Jan14 SC0001_08 ! NULL SA (slot 8)
25330 0 0.0 95708 12256 Sl 09:24 PC0001_09 ! Index page (slot 9)
-- hendricks ---- X2448 -- minos:0 -------------- Up 9.34H Load nnnnn Idle 5s
8991 0 0.0 35304 12220 Sl 01:05 tvmgr_1 ! I/O for character-based windows
9028 0 0.0 75440 12676 Sl 01:05 lxmgr_1 ! I/O for graphics windows
25044 0 0.0 117624 12888 Sl 09:17 PF1008_11 ! PA1008 (slot 1)
9091 0 0.0 75504 11484 Sl 01:05 SF0001_12 ! NULL SA (slot 2)
9123 0 0.0 75280 11488 Sl 01:05 SG0001_13 ! NULL SA (slot 3)
25106 0 0.0 107128 12488 Sl 09:20 PJ1661_14 ! alarm display
25086 0 0.0 95164 12240 Sl 09:20 PG2074_15 ! PG2074 (slot 5)
9222 0 0.0 97236 12692 Sl 01:05 UTILI1_16 ! Utility window (slot 6)
9255 0 0.0 75592 11284 Sl 01:05 SI0001_17 ! null SA (slot 7)
9287 0 0.0 76120 11488 Sl 01:05 SH0001_18 ! NULL SA (slot 8)
9320 0 0.0 95184 12272 Sl 01:05 PH0001_19 ! Index page (slot 9)
Part I Control System Overview
Devices
Devices are an important logical construct within the control system. A
device is the common way to address a physical channel (hardware). More
specifically, 'device' refers to an ACNET device in the database. A device
is identified by a unique integer (called a device index, or DI) or a name.
Devices are defined to have some set of properties (PIs): reading, setting,
basic status, basic control, analog alarm, digital alarm...etc. By addressing
a DI/PI combination one can read or set that part of a device. Device
properties can also have scaling information stored in the database.
A common path for reading a device, in engineering units, goes as follows:
A console application uses a library routine to request the data, by DI/PI.
The data request is handled by the Data Pool Manager (DPM) inside the console.
DPM sends the request to the appropriate front-end and makes the reply (raw
data) available to the application. Once the application gets the raw data,
it scales it to engineering units using scaling information received from
the central database.
DABBEL is the name of the program that allows users to interact with the
device database. It is invoked simply by typing its name on the command
line. DABBEL allows defining and editing the definitions of devices in the
device database. DABBEL is not for novices. It requires knowledge about the
devices being edited and understanding of the control system's use of them.
Part I Control System Overview
Hardware Links
There are several hardware links that run throughout the accelerators.
T-clock (also called Tevatron clock) is a system of Manchester encoded events
on a 10 MHz link. Events are in the range between 0 and FF (hex). Each event
has a well-defined, global meaning. An occurrence of an event allows
synchronized activity. Events can be decoded in hardware and in consoles.
This allows both hardware and applications to react to (or in some cases
generate) events. Some examples of event use are: synchronized data
acquisition, starting event for power supply waveforms, notification of a
Tevatron quench...etc.
For more sensitive timing, a second timing link is used in each the
Main Injector and the Tevatron. This timing link, called Main Injector Beam
Synch (MIBS) for Main Injector and TeVatron Beam Synch (TVBS) is used for
timing of transfer kickers as well as longitudinal placement of beam in Main
Injector and the Tevatron. Beam injected into the Main Injector or the
Tevatron is injected into a specific RF bucket with respect to this marker.
This marker is then used by beam measuring equipment to determine where
specific RF buckets with are located.
Another important link is MDAT (Machine DATa). MDAT is a link that carries
global accelerator data (e.g. Tevatron energy). This data is packed in
type-coded frames that can be decoded by hardware. This allows certain devices
to scale their output by other accelerator parameters. For example, power
supply waveforms may scale their output by the field in the accelerator magnets.
MDAT updates at 720 Hz and allows up to 256 unique data frames.
There are a number of communication links used by the front-ends and
remote 'smart' modules data and command transfer. The majority of data and
control transfer is done using a CAMAC system. Two links are used for
PIOR/PIOX (Programmed I/O Receive and transfer) and BTR (Block TRansfer)
depending on the size of information to be transferred.
Another signal link that is used for Booster, Main Injector, Tevatron and
the Antiproton source is the Abort link. Each link has a CAMAC crate-based
transmitter card that transmits a signal around the accelerator to the various
abort cards as well as to abort kickers. Each abort card monitors critical
beam-related devices. If one of the inputs monitored by the abort card goes
into a bad state, the card will pull the abort link down, causing the abort
kickers to fire, kicking beam safely out of the accelerator.
Part II Console Application Programming
Code Librarian
You need a Linux account to develop console application programs. There are
common definitions on those machines to assist with program development.
Since all application programs end up within the code librarian, called
MECCA, it's useful to have some understanding of how it works. MECCA
addresses and stores applications by a unique name. Primary Applications (PAs)
are named PAxxxx and Secondary Applications (SAs) are named SAxxxx, where
xxxx is a sequentially assigned integer value. MECCA archives application
source code using CVS. It also maintains a copy of the current source in
a directory to support searching. PAs are stored under the directory
/usr/local/mecca_head/mecca/pas/paxxxx, where xxxx is the PA number.
SAs are stored under the directory /usr/local/mecca_head/mecca/pas/saxxxx.
MECCA will use all the source from your default directory (the one you
are in when you invoke MECCA) and compare it to what files it has. If any
files are changed or new ones are added they are copied, and a new version of
your program is built. MECCA supports FORTRAN and C/C++ source code. New
modules must be written in C++.
MECCA supports assigning an application to a particular console Index Page
by using the command: "mecca --ipe". Before assigning a program to the Index
Page, one should test the application. There is a special Index Page, called
the Z-index to allow temporary running of a PA. There's another page (D7) for
use with testing SAs. You can place your PA on the Z-index with the
MECCA command: "mecca --dev paxxxx". MECCA will build the application
in your area and place it on the Z-Index page for testing. For more details,
read the topics 'NOTES' and 'MECCA' under MECCA's on-line help (MECCA/HELP)
or on the Web at https://www-bd.fnal.gov/controls/mecca/mecca.html and
https://www-bd.fnal.gov/controls/mecca/mecca_manual.ps.
Part II Console Application Programming
Application Shell
All application programs have a very similar main routine. There is
an infinite loop starting with a special system routine called 'window_intype'.
window_intype should always be the first function called in the program. It
initializes the program context and returns environment parameters: interrupt
type, cursor row and column and other special information. There are four main
types of events (interrupts) that a console application receives:
initialization, termination, periodic and keyboard. An initialization
interrupt is delivered once, at program startup. Likewise the termination
interrupt is delivered once, when the user or the system indicates the desire
to terminate the program. Periodic interrupts are delivered at a 15 Hz rate.
This allows the program to schedule activities. Keyboard interrupts are
delivered any time the mouse-button (or return key) is pressed. This allows
the program to react to user interface point-and-click activity. Examples of
the application shell can be found in the directory CNS_INTRO: xskel.c or
xskel.for. There are several other useful files in this directory (see
'Getting Started' section below).
Programmer Tools
There are a large set of library routines provided for the console
application programmer. This library is called CLIB (Console LIBrary). CLIB
contains routines for user interface, data acquisition, program control...etc.
There is on-line help for this library (just type CLIBHELP). Start with:
CLIBHELP NEW_USER. CLIBHELP is broken into two levels of hierarchy: subjects
and within each subject is help on the actual routines. Calling sequences and
argument passing mechanisms are documented with a general syntax that is
described in CLIBHELP NEW_USER INTRO_HELP or on the Web at
https://www-bd.fnal.gov/www/controls/clib/intro_help.html. A very useful
technique for scanning CLIB routines is to search a one-line help file.
This one-line filename is defined in CNS_INTRO:help_def.com as the symbol
CLIB_ONELINE. In addition to CLIB there are several useful user libraries
within MECCA. CNS_INTRO:help_def.com describes these.
Help for CLIB routines is also available via the Web:
https://www-bd.fnal.gov/www/controls/clib/clib.html ! arranged by subject
https://www-bd.fnal.gov/www/controls/clib/clib_oneline.html ! one line help
https://www-bd.fnal.gov/www/controls/clib/new_user.html ! new user help
User Interface
PAs have the ability to create and manipulate 'window objects' within their
screen. These are character-based windows (see CLIBHELP TV_WINDOWS). These
objects include: moveable scrolling-text windows, dialog boxes, multiple
switches, menus, menu-bars...etc. Windows are created and then identified
by a 'window ID'. This window ID is then used to address the window for I/O.
Most of the routines that deal with these windows start with the prefix
'window_'. All the other routines in this topic of CLIB are things like
dialog boxes, switches, menus,...etc. Also see
CLIBHELP TV_WINDOWS INTRO_TV_WINDOWS or
https://www-bd.fnal.gov/www/controls/clib/intro_tv_windows.html.
Graphics
SAs and PAs have the ability to create and address pixel-mapped 'windows'.
These windows allow logical addressing of the SA screen or the PA graphics
screens. See CLIBHELP GRAPHICS_WINDOWS. One can initialize a physical
portion of the screen and assign scaled coordinates to it. Graphics windows
are created and then identified by a 'window ID'. This window ID is used to
address the window for I/O. Data can be plotted by specifying the window ID
and its real coordinates. Most of the routines that deal with these windows
start with the prefix 'wn_'. Graphics are different than TV windows. They are
used to logically address a pixel-mapped screen. There is no management of the
pixel space. For more details, see
CLIBHELP GRAPHICS_WINDOWS INTRO_GRAPHICS_WINDOWS or
https://www-bd.fnal.gov/www/controls/clib/intro_graphic_windows.html.
Part II Console Application Programming
Data Acquisition
There is a suite of routines to allow applications to read and set scaled
or raw device data. These routines are described in the topics
CLIBHELP DIO_xxx, where xxx takes on names of several types of data acquisition.
For example, DIO_SCALED topic includes routines for reading and setting
of scaled data. Also see CLIBHELP NEW_USER INTRO_DIO or
https://www-bd.fnal.gov/www/controls/clib/intro_dio.html. Reading or setting
of data is done either with the raw or scaled format of the data. The data
acquisition routines typically fall into two styles: manipulation of a
single device or manipulation of a list of devices. An important concept to
understand for reading of data is the FTD (Frequency Time Descriptor). When
reading a device (or device list) you specify the FTD. The FTD is either
a frequency (e.g. one-shot, 1 Hz...etc) or a global clock event (T-clock event).
This FTD tell the front-end how often, or when, to read and return the data.
Data Storage
There are three main methods available for storing application data on disk:
Filesharing, runtime files (called CBS_FILES) and a relational database
(called APPDB). Database tables should be your first consideration for
data storage.
APPDB is a Sybase maintained relational database. It is accessed using
the CLIB routines 'db_select' and 'db_modify'. One of the arguments of
this routine is a SQL string. The user is assumed to be familiar with
databases and SQL. Tables are created using SQL. For more details see
the file dwin_disk:[dwin.central.sybase.sql]aaareadme.txt. For
filesharing-like use, see dwin_disk:[dwin.central.sybase.info]file_share.txt.
Filesharing is a set of files that are accessed through a central process
called FSHARE. This facility allows shared access to files. A file can be
created using a facility called FILEMASTER. Once created, the file can be
opened for read or read-write access from multiple applications. Library
routines which access these files have names with a prefix of 'fm_'. For more
details, see CLIBHELP FILE_IO or
https://www-bd.fnal.gov/www/controls/clib/file_io.html.
CBS_FILES is just the name of an area which contains files created and
accessed through runtime functions (fopen, fread...etc.). The assignment
and use of this area is made on a as-needed basis. For more details
see /usr/local/cbs_files/aaareadme.txt.
Getting Started
There is on-line documentation for the programmer. Start by reading
several files in the CNS_INTRO directory: readme.txt, getting_started.txt
and help_def.com. These files are intended to lead the new programmer
through building a console application. You can also start with the command:
CLIBHELP NEW_USER or https://www-bd.fnal.gov/www/controls/clib/new_user.html.
Read the help under this topic.
Appendix: An Overview of Fermilab Accelerators
The accelerators presently operational at Fermilab are: Linac, Booster, Main
Injector, Tevatron, Antiproton source, Recycler, and Switchyard. A brief
description of the accelerators will be given below:
Linac: a 152m long Linear accelerator. This is actually three accelerators.
There are two Cockroft-Walton Pre-accelerators for producing 750 keV H- ions
and the 400 MeV Linac.
Booster: a 75m radius synchrotron. This accelerates protons from 400 MeV
to 8 GeV.
Main Injector: an alternating gradient synchrotron. It
accelerates protons and anti-protons from 8 GeV to 150 GeV.
Tevatron: a 1 km radius alternating gradient synchrotron. It can accelerate
protons and antiprotons from 150 GeV to 900 GeV.
Antiproton source: Consists of two storage rings of radius 90 and 75 m.
Antiprotons are stored at 8 GeV and injected into Main Injector.
Recycler: an antiproton storage ring where particles produced in the
antiproton source are kept until time for the nex Collider shot
Switchyard: This is a number of transfer lines which are used to distribute
150 GeV protons from the Main Injector to the various fixed target experiments.
Security, Privacy, Legal