Deterministic Boltzmann machines (DBMs) are typically fully interconnected networks that are trained using a two-phase settling process and a form of Hebbian weight updating. The input to a unit is the dot product of its incoming weights and the activations of the sending units. The units use a logistic transfer function (with an adjustable gain) and are output-integrating, meaning that the outputs change gradually over time. Groups in a DBM will use the IN_BOLTZ input function and the OUT_BOLTZ output function by default.
Training on each example event in a DBM occurs in two phases. At the start of the positive phase, both the input units and the output units are clamped to the inputs and targets, respectively, and the other units are reset to their initOutputs. Then the non-clamped units are allowed to settle for a given period of time or until the activations are no longer changing significantly. Usually the gain will be annealed with an exponential upward decay during the settling process. Initially there is high temperature (low gain), gradually settling to a lower temperature (high gain).
In the negative phase, the inputs remain clamped but the outputs are allowed to freely update. At the start of the negative phase, the activations of all of the units other than the inputs are shifted towards their initOutput. Early in training, the outputs may not be shifted at all, and would thus remain the same as they were at the end of the positive phase. Gradually, the normalization could be made stronger until the units are completely reset to the initOutput at the start of the negative phase. During testing, only the negative phase is run.
During training, each weight is updated in a Hebbian fashion according to the products of the outputs of the preceding and following units. The weight change will be proportional to the difference between the product at the end of the positive phase and the product at the end of the negative phase. Lens does not enforce the constraint that links in opposing directions between two units have the same weights, although they will receive the same changes so they will tend towards the same value if there is weight decay or will maintain a fixed offset otherwise.
Here is a sample file for creating a DBM, the full version of which is located in the boltz.in example:
addNet Boltz-big -i 4 -t 5 BOLTZMANN
addGroup input 20 INPUT
addGroup hidden 10
addGroup output 3 OUTPUT
connectGroups input hidden -bi
connectGroups hidden hidden
connectGroups hidden output -bi
connectGroups output output
loadExamples digits.ex
setObj digits.maxTime 3.0
setObj digits.graceTime 1.0
setObj digits.minTime 0.0
setObj trainGroupCrit 0.001
setObj testGroupCrit 0.001
setObj clampStrength 1.0
setObj initGain 0.1
setObj finalGain 1.0
setObj annealTime 1.0
The network is given the BOLTZMANN type on creation. When
adding groups, it is not necessary to given any types other than
INPUT and OUTPUT, as the group input and output
functions will default to the standard Boltzmann functions, which use
dot product inputs and a time-integrated logistic transfer function.
A group should be of type INPUT if it will ever be assigned inputs in an example. A group should be of type OUTPUT if it will ever be assigned targets. It is not necessary to provide inputs or targets to every input or output unit on every event. If a unit has no inputs or targets specified, it will freely update like any other unit.
It is possible to give a group both INPUT and
OUTPUT types. This may be useful if some units in the
group are to act as inputs on some events and as targets on others. If
a unit in such a group is given both an input and a target in an event,
the unit will be clamped to the input.
An OUTPUT group can be given an error function and the default error function is CROSS_ENTROPY. However, the error procedure doesn't affect the training. It is only used in assessing the difference between the actual and target outputs at the end of the negative phase.
It is customary to make most projections in a Boltzmann group
bidirectional. This can be done more easily with the -bi
flag to connectGroups. Note
that this will be equivalent to calling connectGroups twice with the two
groups names reversed. There will be no special relationship between
corresponding links in the forward and backward projections.
As mentioned above, when training the DBM, two settling phases are performed on each event. In the positive phase, any unit with an input specified is clamped to that value and any unit with a target specified is clamped to that value. All other units are reset to their group's initOutput, or the network's value if the group's is NaN. The gain, used in the logistic function, is initialized to the network's initGain.
On each tick, all the units in the network will be updated synchronously: first all calculating their inputs and then all calculating their outputs. After each tick, the gain will decay exponentially towards the finalGain (usually this involves increasing the gain). The half-life of the decay (in time intervals) is determined by the annealTime.
The positive phase ends at the end of the event's graceTime. The positive phase can end earlier, provided it is past the minTime and the network has settled. The network is considered to have settled when the change in the activation of every unit on the last update is less than the group's or network's trainGroupCrit (during training) or testGroupCrit (during testing). If all units are clamped during the positive phase, then it will only last for one tick, or the minTime is reached if that is longer.
At the start of the negative phase, the units with external inputs remain clamped but the units that only have targets are unclamped and the activations of all non-clamped units are reset towards the group's initOutput. The new value of one of these units will be weighted average of its current value and the initOutput. The weighting of the initOutput is determined by the network's clampStrength parameter. A value of 0.0 will cause no change. 1.0 will cause the activations to be set to the initOutput. 0.5 would move the activations halfway to the initOutput.
Essentially, the outputs are being WEAK_CLAMPed by the initOutput. One may wish to begin training with a very low clampStrength value and then gradually increase it to 1.0. There is no automatic mechanism for doing this increase so the user may want to use a preEpochProc function to adjust the clampStrength as it trains. However, just leaving the clampStrength at 1.0 might be optimal for many networks.
The gain annealing starts over with the negative phase and settling proceeds as in the positive phase, although the output units will not be clamped. The negative phase ends if the time since the end of the positive phase reaches maxTime - graceTime (that is, the difference between maxTime and graceTime). Thus, the event's maxTime still indicates the total time the event will ever last and the graceTime is the total time the positive phase can last. The difference is the total time the negative phase can last. The negative phase will also stop if the network has settled, but not before the minTime has elapsed since the start of the negative phase.
During testing, only the negative phase is run. At the start, the inputs will be clamped and the other units will be reset to their initOutput. In this case, the settling process lasts no more than maxTime - graceTime, the same as for the negative phase during training. Again, the gain will be annealed and settling will stop once the minTime is reached and activation changes are all less than the testGroupCrit.
Because DBMs typically use a large number of time intervals per example or ticks per interval, the history arrays allocated in each unit will tend to be very large and could use a significant amount of space. These arrays are not used in DBMs except for storing values to display in the Unit Viewer. Therefore, you may want to eliminate these arrays by setting the historyLength to 0 or some other small value. This can be done with the setTime command.
Finally, for most networks, the default behavior when selecting an example in the Unit Viewer or when calling doExample is to run the example testing procedures on that example. However, the default behavior for DBMs is to run the training procedures because this may be what the user prefers to see in the Unit Viewer. Note that this won't actually affect the weight updates unless training is going on when the example is run. The testing procedures can be run instead of the training ones by calling doExample with the appropriate parameters or by using the "Procedure" pull-down menu in the Unit Viewer.
When training a Boltzmann machine, it is important that the gain annealing isn't too rapid and that the finalGain isn't too high. Otherwise, the network can become unstable. On the other hand, if the gain remains low for too long, the network will have difficulty driving units to extreme values. You will need to experiment to find a good balance.