Manual training with COBYLA

[mspronesti]

Hello,
reading some recent papers that use lambeq for classification tasks, I noticed they often optimize the loss function with COBYLA . I tried giving it a shot. This is my manual training. It's not really working very well though as the loss doesn't really decrease (except some slight changes from the 3rd digit). I'm trying to figure out if I made a mistake in the way I implemented it or if it's just that this doesn't fit my use case.

import numpy as np
from scipy.optimize import minimize
from sklearn.metrics import accuracy_score

# parse data, create diagrams, parameterise 
# ...
# ...
# ...

all_circuits = train_circuits + val_circuits # + test_circuits
N_EPOCHS = 200

model = NumpyModel.from_diagrams(all_circuits)
model.initialise_weights()

def make_pred_fn(circuits):

    def predict(weights):
        model.weights = weights
        return model(circuits)

    return predict


def make_cost_fn(pred_fn, labels, log: bool = True):
    def cost_fn(params):
        predictions = pred_fn(params)

        cost = - np.sum(labels * np.log(predictions+1e-9))/len(labels)  # binary cross-entropy loss
        costs.append(cost)

        score = accuracy_score(labels, predictions)
        accuracies.append(score)
        
        if log:
           print("cost: %s |  acc: %f "  % (cost, score))
        return cost

    costs, accuracies = [], []
    return cost_fn, costs, accuracies

train_pred_fn = make_pred_fn(train_circuits)
dev_pred_fn = make_pred_fn(val_circuits)
# test_pred_fn = make_pred_fn(test_circuits)


train_cost_fn, train_costs, train_accs = make_cost_fn(train_pred_fn, train_labels)
dev_cost_fn, dev_costs, dev_accs = make_cost_fn(dev_pred_fn, val_labels)


result = minimize(
    fun=train_cost_fn,
    x0=model.weights,
    method='cobyla',
    callback=dev_cost_fn,
    options={
        'disp': True,
        'maxiter': N_EPOCHS,

    }
)

[Thommy257]

Hi,

unfortunately I'm not familiar with the COBYLA optimiser, but according to the docs there are a couple of hyperparameters that you can set. have you tried playing around with rhobeg? rhobeg=0.05 or something might seem sensible.

Hope that helps!